# WebToe — full documentation for language models > Every page of the WebToe docs in one file, generated by tools/site/build.mjs from site/docs/en/. > The short index is https://webtoe.openaudiovisual.com/llms.txt. Site: https://webtoe.openaudiovisual.com/ · 繁體中文: https://webtoe.openaudiovisual.com/zh/ > Repository: https://github.com/frank890417/WebToe (MIT) --- # Documentation > Use the editor, import TouchDesigner projects, write expressions, embed WebToe in your own page, and see how the engine is built. WebToe is a node-based dataflow engine for real-time visuals that runs in the browser. You patch operators into a network, drive parameters with expressions, and watch the result cook every frame on WebGL2 or WebGPU. It also opens TouchDesigner projects. The fastest way in is to [open the editor](/app/) and load an example from the toolbar. [Getting started](getting-started.md) covers the editor, running it locally, and working on the source. These pages describe the code in [the repository](https://github.com/frank890417/WebToe) as of this build. Numbers that come from measurements say where they were measured. --- # Getting started > Open the editor in a browser, run WebToe locally with npx webtoe, or set up the monorepo for development. ## Try it in the browser {#online} Open **[the editor](/app/)**. It needs a browser with WebGL2 (any current Chrome, Edge, Firefox or Safari). Nothing is installed and nothing is uploaded: projects load and run in the page. - **Examples.** The toolbar's *examples…* menu loads ten bundled projects, from a four-node noise patch to a 3D scene. See [Examples](examples.md). - **Your files.** *load* opens a `.webtoe.json`; *save* downloads the current network as one. You can also drop a file anywhere on the page: a `.webtoe.json`, a TouchDesigner `.toe` / `.tox`, or a `.toe.dir` folder produced by `toeexpand`. See [Importing TouchDesigner projects](importing.md). - **Links.** `/app/?project=` loads a project from a URL (the server has to allow it with CORS). Relative paths resolve against `/app/`, so `/app/?project=examples/01-hello-noise.webtoe.json` opens the first example. ### Working in the network {#keys} | Action | How | |---|---| | Add an operator | Tab or double-click the network: family tabs, one search across all families | | Wire | drag from an output dot to an input dot | | Select | click a node; the parameter panel and the viewer follow the selection | | Expression on a parameter | the **ƒ** button next to the parameter | | Enter a COMP / go up | i / u | | Display flag (what the viewer shows) | Shift + D | | Show the output behind the network | d | | Delete | Backspace or Delete | ### URL parameters {#url} | Parameter | Effect | |---|---| | `?project=` | load a `.webtoe.json` on start | | `?backend=webgpu` | use the WebGPU backend instead of WebGL2 (falls back to WebGL2 if WebGPU fails to start) | | `?bridge=` | use an import bridge at this address instead of `http://127.0.0.1:9881` | | `?bridgeToken=` | bearer token for a shared bridge; remembered in this browser after the first visit | ## Run it on your machine {#local} ```bash npx webtoe ``` This starts the WebToe bridge on `127.0.0.1:9881`, serves the editor at `http://127.0.0.1:9881/app/` and opens it. The bridge is what lets the page use your own TouchDesigner's `toeexpand` (see [Importing](importing.md#bridge)). It needs Node 20 or newer and has no dependencies. | Flag | Default | Meaning | |---|---|---| | `--port ` | `9881` | port to listen on | | `--no-open` | | do not open a browser (bridge only, for the hosted editor) | | `--host ` | `127.0.0.1` | bind address; anything beyond loopback should use `--token` | | `--token ` | | require `Authorization: Bearer ` on `/expand` | | `--toeexpand ` | auto | path to `toeexpand` if it is not in the standard install location | | `--app ` | built app | serve a different build of the site | ## Work on the source {#dev} WebToe is an npm-workspaces monorepo written in TypeScript. The shipped app has no runtime dependencies; development uses Vite, Vitest, TypeScript and playwright-core. ```bash git clone https://github.com/frank890417/WebToe.git cd WebToe npm install npm run dev # editor at http://localhost:8643/app/ npm run check # type check + test suite npm run build # apps/web/dist: the site at /, the editor at /app/ ``` | Script | What it does | |---|---| | `npm run dev` | Vite dev server for the editor (`apps/web`) | | `npm run check` | `tsc --noEmit`, then every Vitest suite under `tests/` | | `npm run build` | builds the editor into `apps/web/dist/app/`, then this site around it | | `npm run site:check` | builds the site in memory and fails on any broken link or missing translation | | `npm run bridge` | the local bridge from the checkout (serves `apps/web/dist` once built) | Start with [Architecture](architecture.md) before changing the engine: the package dependency rule and the GPU pass contract are the two things everything else rests on. --- # Examples > The ten projects bundled with the editor, what each one shows, and a link that opens it. Ten projects ship with the editor. Load them from the toolbar's *examples…* menu, or open one directly with the links below. They are ordinary `.webtoe.json` files in [`apps/web/public/examples`](https://github.com/frank890417/WebToe/tree/main/apps/web/public/examples): save one, change it, load it back. ## Authored patches {#authored} ### 01 · hello noise {#hello-noise} Animated fBm noise; an LFO drives the level's brightness through an expression. The smallest complete patch: four operators, one expression. [Open in the editor](/app/?project=examples/01-hello-noise.webtoe.json) ### 02 · feedback trails {#feedback-trails} Move the mouse over the viewer. A feedback loop with a fading level leaves trails behind a mouse-driven rectangle. [Open in the editor](/app/?project=examples/02-feedback-trails.webtoe.json) ### 03 · lfo garden {#lfo-garden} Three ramp chains, each rotated by its own LFO through an expression, composited additively with the hue slowly drifting. [Open in the editor](/app/?project=examples/03-lfo-garden.webtoe.json) ### 04 · webcam displace {#webcam-displace} The webcam displaced by animated noise, with an edge overlay. Allow camera access; without a camera a placeholder shows instead. [Open in the editor](/app/?project=examples/04-webcam-displace.webtoe.json) ### 05 · chop playground {#chop-playground} Select `merge1` to scope the channels: a raw sum against its lagged copy. The lagged sum rotates the ramp. [Open in the editor](/app/?project=examples/05-chop-playground.webtoe.json) ## Imported from TouchDesigner {#imported} Three 2022 daily sketches, converted from their original `.toe` files by the importer, lightly adapted for the web (a movie source swapped for noise, for example). Both engine bugs they exposed (composite layer order, quoted string constants) were fixed and have regression tests. ### 06 · sketch: pseudo voronoi (2022) {#voronoi} A pseudo-voronoi pattern from animated noise. [Open in the editor](/app/?project=examples/06-sketch-voronoi.webtoe.json) ### 07 · sketch: fractals (2022) {#fractals} Fractal-like feedback noise. [Open in the editor](/app/?project=examples/07-sketch-fractals.webtoe.json) ### 08 · sketch: chop study (2022) {#chop-study} Click and move in the viewer: lagged mouse channels scale the composite. [Open in the editor](/app/?project=examples/08-sketch-chop-study.webtoe.json) ## Showcases {#showcases} ### 09 · showcase {#showcase} Every family at once, in 27 nodes: camera through edge detection, a kaleidoscope COMP with in/out tunnels, colour noise, a mouse-driven source switch, noise displacement, hue-drifting feedback trails, and a CHOP rig (lag, speed integrator, parameter reader, full math pipeline) driving it through live expressions. Move the mouse over the viewer: x switches the source, y sets the displacement. [Open in the editor](/app/?project=examples/09-showcase.webtoe.json) ### 10 · 3d lines {#3d-lines} The 3D pipeline end to end: skinned, wobbling line ribbons (SOPs inside a Geometry COMP with a line MAT), noise-scattered instanced spheres, an orbiting look-at camera, point and ambient lights, a Render TOP, and a 2D glow chain after it. Uses the WebGL2 backend. [Open in the editor](/app/?project=examples/10-3d-lines.webtoe.json) ## Raw TouchDesigner files {#raw-toe} Two of the author's own 2022 daily sketches as the original `.toe` files, saved by TouchDesigner 2021.16410. Picking one decodes the container natively in the browser (a few milliseconds) and imports it; the import report says what ran and what was kept as a stub. Native .toe decoding is provided for research purposes only. ### 11 · raw .toe: pseudo voronoi {#raw-voronoi} Pseudo-voronoi cells from animated noise, as saved in TouchDesigner. [Open in the editor](/app/?project=examples/toe/2022-pseudo-voronoi.toe) ### 12 · raw .toe: fractals {#raw-fractals} A fractal feedback sketch, as saved in TouchDesigner. [Open in the editor](/app/?project=examples/toe/2022-fractals.toe) --- # Importing TouchDesigner projects > Drop a .toe or .tox on the editor. What comes across, what becomes a stub, how expressions are translated, and the bridge and toeexpand paths. Drag a `.toe` or `.tox` onto the editor (or pick it with *load*). WebToe rebuilds the network: supported operators run immediately, everything else is kept as a stub, and an import report says exactly how much of each. ## Native decoding {#native} WebToe decodes `.toe` / `.tox` files **natively in the browser**: no TouchDesigner install, no bridge, and the file never leaves your machine. A 20 MB show file decodes in 0.35 s (33,923 files) and the whole drop-to-running-graph takes about half a second in Chrome. The decoder is validated byte for byte against TouchDesigner's own `toeexpand` on 125 production project files. If native decoding fails, the bridge and `toeexpand` path below takes over automatically; add `?toe=bridge` to the editor URL to force it. Details: [The .toe format](toe-format.md). Try it: [open a raw 2022 project file](/app/?project=/examples/toe/2022-fractals.toe), saved by TouchDesigner 2021.16410. Native .toe decoding is provided for research purposes only. ## What comes across {#recovered} - **Operators and hierarchy.** Node types are mapped through a conservative type table (about 90 TouchDesigner types today); COMP networks nest as they do in TouchDesigner. - **Wires**, including wires across COMP boundaries and through `in` / `out` operators (TouchDesigner's tunnels; the input index comes from the name, `in2` → index 1). - **Parameters**, with token-accurate value maps for the operators WebToe implements (menu tokens, colour components gathered into one colour, TouchDesigner's own parameter names). - **Node flags.** Render, display and bypass flags come across; the Render TOP only draws Geometry COMPs whose render flag is on, as in TouchDesigner. - **Expressions.** Live Python expressions are translated when the translation is faithful, and kept inert otherwise (see [below](#expressions)). The parameter mode field is a bitfield (bit 0 = expression), so flagged expression modes import too. - **DAT text and tables.** TouchDesigner stores DAT bodies in a framed binary sidecar, not plain text; WebToe decodes the frame. - **Layout.** Node positions are kept, and the network view frames the imported content on entry. ## What does not {#stubs} - **Unmapped operators become stubs** of the right family. A stub keeps the node's name, wires, layout, parameters and Python code, and passes its input through so the chain downstream still cooks. The original type is shown on the node. - **Imported Python never runs.** Untranslatable expressions stay visible on the parameter's ƒ field, inactive. Python DATs (parameter execute, panel execute, script) are a permanent boundary. - **Media paths** become placeholders; image and movie files are not inside a `.toe`. - **Wires between different networks** that are not COMP tunnels are skipped and counted in the report (a listed follow-up). ## The import report {#report} Every import ends with a report: nodes imported, how many run, how many are stubs, expressions translated and kept inert, and a histogram of the stubbed types. That histogram is how WebToe decides what to implement next ([TouchDesigner parity](td-parity.md)). Measured on a real 20 MB production show file through the bridge: **14,710 nodes, 10,239 runnable (70%), 2,244 expressions translated, 2,251 DAT bodies recovered, 8.0 s** from drop to a running graph, 4.6 s of it in `toeexpand` (WORKLOG, 2026-08-01). ## Expression translation {#expressions} The translator rewrites a known subset of TouchDesigner Python into WebToe's expression language, then compiles the result and dry-runs it against an inert scope. Anything that fails either step stays inert. | TouchDesigner | WebToe | |---|---| | `absTime.seconds`, `me.time.seconds` | `time.seconds` | | `absTime.frame`, `me.time.frame` | `time.frame` | | `math.sin(x)`, `mod.math.sin(x)` | `sin(x)` | | `math.pi` | `PI` | | `a if cond else b` | `cond ? a : b` (one level) | | `and`, `or`, `not` | `&&`, `\|\|`, `!` | | `int(x)`, `float(x)` | `trunc(x)`, `(x)` | | `True`, `False`, `None` | `true`, `false`, `null` | | `op('lfo1')['chan1']`, `parent().par.speed`, `me.par.x` | unchanged: the same forms exist in WebToe | Kept inert on purpose: f-strings, `lambda`, loops, `mod(...)` module calls, `tdu.*`, `ext.*`, `project.*`, `.menuIndex`, `panel.*`, `me.digits`, floor division `//`, and nested conditionals. ## The bridge {#bridge} The bridge is a small local service that runs **your own** TouchDesigner's `toeexpand` (the official command-line tool that ships with every install) and hands the expansion to the page. It binds to `127.0.0.1`, has no dependencies, contains nothing of Derivative's, and deletes each upload when it is done. The editor looks for it automatically when a `.toe` is dropped. ```bash npx webtoe # serves the editor and the bridge, opens http://127.0.0.1:9881/app/ npx webtoe --no-open # bridge only, for the hosted editor at webtoe.openaudiovisual.com ``` No Node? Every TouchDesigner install ships Python, so the same bridge exists as one standard-library file, protocol-identical. Download [`bridge.py`](/bridge.py) and run it: ```bash # macOS: any Python 3.8+, including the one inside TouchDesigner python3 ~/Downloads/bridge.py ``` ```text REM Windows: TouchDesigner's own Python (adjust the install path if needed) "C:\Program Files\Derivative\TouchDesigner\bin\python.exe" %USERPROFILE%\Downloads\bridge.py ``` If no bridge is running, dropping a `.toe` opens a guide with these commands, and the guide keeps watching: start a bridge and it continues on its own. ### Sharing one bridge {#shared} A bridge can serve other machines (TouchDesigner on one computer, the browser on another): ```bash npx webtoe --host 0.0.0.0 --token # then open: https://webtoe.openaudiovisual.com/app/?bridge=http://:9881&bridgeToken= ``` Read this before doing it: `toeexpand` is a native program parsing files in an undocumented format, so a bridge that accepts uploads from others belongs on a machine you can wipe. Use HTTPS (a tunnel) for anything beyond your LAN, since browsers only exempt `127.0.0.1` from mixed-content rules. The bridge caps concurrent expansions at three. ## Without any bridge {#manual} Expand the file yourself and drop the resulting folder on the page (or use *import .toe.dir*): ```bash # macOS "/Applications/TouchDesigner.app/Contents/MacOS/toeexpand" myproject.toe ``` ```text REM Windows "C:\Program Files\Derivative\TouchDesigner\bin\toeexpand.exe" myproject.toe ``` For batch conversion there is a command-line converter that writes a `.webtoe.json`: ```bash node packages/cli/toe-convert.mjs myproject.toe # → myproject.webtoe.json ``` `toeexpand` cannot open file names that are not plain ASCII (it reads the path as Latin-1). The bridge and the converter stage a copy under an ASCII name, so projects named in Chinese, Japanese or Korean import normally. ## How it is tested {#tests} The import pipeline is covered by an original committed fixture: a real binary `.toe` authored for this repository plus its canonical `toeexpand` expansion ([provenance](https://github.com/frank890417/WebToe/blob/main/tests/fixtures/README.md)). A CI layer asserts the reconstructed graph (types, COMP-boundary and tunnel wires, parameter modes, translated expressions evaluated in the engine, stubs, report numbers); a second layer, skipped where TouchDesigner is not installed, expands the binary with the real `toeexpand` and runs the converter and the bridge end to end, including a project whose file name is not English. WebToe reads project files that you open with your own licensed TouchDesigner, for interoperability. It contains no Derivative code, binaries or assets. --- # The .toe format > How WebToe reads TouchDesigner's binary project container in the browser. Native .toe decoding is provided for research purposes only. It exists to study interoperability with project files you own, is not affiliated with or endorsed by Derivative Inc., and may change with any TouchDesigner release. Derivative has announced an official JSON project format; WebToe's `ProjectLoader` interface is ready to load it beside this decoder. Your own TouchDesigner's `toeexpand`, through the bridge, remains the reference and the automatic fallback. Source: [`packages/io/src/toeBinary.ts`](https://github.com/frank890417/WebToe/blob/main/packages/io/src/toeBinary.ts) · tests: [`tests/toe-native.test.ts`](https://github.com/frank890417/WebToe/blob/main/tests/toe-native.test.ts) ## What it does {#what} A dropped or linked `.toe` / `.tox` is decoded in the browser into exactly the file set `toeexpand` writes: node files (`.n`), parameters (`.parm`), DAT bodies and layout. The same importer the bridge path uses then builds the network. ## Measured {#measured} Validated against `toeexpand` (TouchDesigner 2025.33070) on 125 production project files, 3 KB to 151 MB, saved by builds from 2021.16410 to 2025: | | | |---|---| | identical file set, byte for byte | **125 / 125** (553,230 files) | | total decode time, native (Node) | 17.7 s | | total time, `toeexpand` | 266.5 s | | 20 MB show file, in Chrome | 0.35 s (33,923 files) | | 151 MB show file, in Chrome | 1.8 s (29,419 files) | | a 4 KB daily sketch, in Chrome | 1–6 ms | ## Format {#format} All integers are big-endian unless noted. ``` file = SEGMENT+ kind "10" "10" | u32 payloadLen | u32 rawLen | payload kind "12" "12" | u32 chunks | u32 8 | u32 payloadLen₀ | u32 rawLen₀ | payload₀ then for each further chunk: u32 payloadLenᵢ | u32 rawLenᵢ | payloadᵢ (large archives, ≤ 128 MiB of output per chunk; chunk headers are not encrypted) payload = TEA-ECB( zlib stream ), padded to a multiple of 8 bytes payloadLen counts the padding; rawLen is the inflated size ``` **TEA** is the standard Tiny Encryption Algorithm: 32 rounds, delta `0x9E3779B9`, little-endian 32-bit word pairs, ECB over 8-byte blocks, 128-bit key (little-endian words) `9241fa3a 59f32012 c612aa09 a98123d4`. The zlib stream is stock RFC 1950. Inflated payloads concatenate, in order across chunks and segments, into one archive of records. A record may straddle a chunk boundary: ``` 0x34 file [tag] [u8 nameLen] [u32 contentLen] [name ] [content] 0x36 push [tag] [0x34] [u8 nameLen] [name ] enter a subdirectory 0x35 pop [tag] leave it ``` A file record named `TRAILER!!!` closes the archive. ## Rules that matter {#rules} 1. **The directory stack persists across segments.** Resetting it per segment silently flattens large projects. 2. **Every chunk is checked.** Its inflated size must equal `rawLen`, and the next segment must start where the lengths say. A segment whose fields do not check out (an unseen variant) is recovered without them: zlib streams are walked on the TEA block grid, each one's end found by its Adler-32 trailer and confirmed by what follows. No file in the corpus needed it. 3. **Filenames.** TouchDesigner allows sibling operators whose names differ only in case (`geo_I` and `geo_i`). `toeexpand` writes to the file system, so on macOS and Windows those collide: it appends `.2`, `.3` and merges directories. The native decoder keeps the original names. Nine corpus files match `toeexpand` only once its case-insensitive collisions are emulated in the comparison. 4. **Browser inflate.** The 8-byte padding after each zlib stream makes `DecompressionStream` error, and in Chrome that error discards output not yet read. WebToe feeds the last 8 bytes one write at a time, so the error arrives only after every chunk of real output has been read. ## In the editor {#paths} | path | when | |---|---| | native decode | first, for every dropped or linked `.toe` / `.tox` | | bridge + `toeexpand` | if native decoding fails or the result does not look like a TouchDesigner expansion; `?toe=bridge` forces it | | guide | neither works (no bridge running) | The import report states which path was used and how long it took. ## Provenance {#provenance} The format description was established by studying project files and TouchDesigner's own command-line tools, and is validated byte for byte against `toeexpand` output. WebToe contains no Derivative code, binaries or assets. --- # Operators > Every operator WebToe ships, by family, generated from the engine's registry, plus how cooking, stubs and custom operators work. An operator is a node with a type such as `top:noise` or `chop:lfo`: a family, a name, typed parameters, a number of inputs, and a cook function. Families follow TouchDesigner's: what flows along a wire depends on the family of the operator that produced it. | Family | Carries | Cooks on | |---|---|---| | **TOP** | images (textures) | the GPU, as render passes | | **CHOP** | channels: named arrays of samples | the CPU, behind a WASM-ready kernel interface | | **SOP** | geometry: typed-array points, primitives, normals, colours | the CPU | | **MAT** | materials for the scene renderer | descriptions consumed by the Render TOP | | **COMP** | networks (containers) and 3D objects (geometry, camera, lights) | their children / the scene | | **DAT** | text and tables | the CPU | ## Every operator {#all} | Family | Ops | Operators | |---|--:|---| | TOP | 28 | blur, camera in, composite, constant, displace, edge, feedback, flip, hsv adjust, image in, in, level, lookup, math, monochrome, ndi in, ndi out, noise, null, out, ramp, rectangle, render, reorder, select, switch, transform, video in | | CHOP | 14 | constant, in, lag, lfo, math, merge, mouse in, noise, out, parameter, select, sop to, speed, switch | | SOP | 21 | add, box, circle, copy, facet, grid, in, line, merge, noise, null, out, point, rectangle, skin, sphere, switch, torus, transform, tube, twist | | MAT | 7 | constant, line, lit (phong/pbr), null, point sprite, switch, wireframe | | COMP | 5 | ambient light, camera, container, geometry, light | | DAT | 6 | in, null, out, select, table, text | | Total | 81 | | The names follow TouchDesigner's operators where WebToe implements the same idea, and the importer maps TouchDesigner's type tokens onto them (Composite TOP is `comp` in a project file, for example). Notable details: - **render** draws the Geometry COMPs that match its pattern (`geo* ^geo7`, TouchDesigner's syntax) with a camera and lights; the scene pass runs on WebGL2. - **geometry** COMPs instance their geometry over SOP points or over CHOP channels (one channel per transform component, a channel shorter than the instance count holds its last sample). - **lookup** maps an image through a ramp; a **ramp** can read its colour keys from a table DAT. - **feedback** returns its input's previous frame and is the one operator allowed to close a loop. - **ndi in** / **ndi out** talk to the local [NDI bridge](ndi.md). - **in** / **out** in TOP, CHOP, SOP and DAT tunnel data into and out of a COMP. ## How cooking works {#cooking} Cooking is pull-based. Each frame the engine starts at the displayed output and asks each input for its result; every operator cooks at most once per frame (memoized), and a cycle guard reports loops instead of hanging. Operators that depend on time, media or input devices are always re-cooked. TOPs describe their work as passes against a backend-agnostic contract, so the same operator runs on WebGL2 and WebGPU ([Architecture](architecture.md)). ## Stubs {#stubs} Every family has a stub operator. The importer uses it for any TouchDesigner type WebToe does not implement: the stub keeps the original type, name, wires, parameters and code, and passes its input through, so a partly supported network still cooks. Saved files carry each node's family, so an unknown type in a `.webtoe.json` also degrades to the right stub instead of failing. ## Writing your own {#custom} Operators register through the public plugin API in `@webtoe/core`. A minimal CHOP: ```ts import { registerOp } from '@webtoe/core'; registerOp({ type: 'chop:sine', // family:name, unique family: 'CHOP', label: 'sine', inputs: { min: 0, max: 0 }, alwaysCook: true, // depends on time params: [{ key: 'freq', type: 'float', default: 1, min: 0, max: 10 }], cook(ctx) { const v = Math.sin(ctx.time.seconds * ctx.paramNum('freq') * Math.PI * 2); return { kind: 'chop', rate: 60, channels: [{ name: 'chan1', data: new Float32Array([v]) }] }; }, }); ``` The palette, the parameter panel and the file format find registered operators through the registry; nothing else has to change. TOPs additionally ship a shader per backend (GLSL for WebGL2 and WGSL for WebGPU, written by hand), and a contract test checks that both exist. --- # Expressions > Drive any parameter with a JavaScript expression that reads time, channels, other parameters and values sent by a host page. Any parameter can switch from a constant to an expression with its **ƒ** button. An expression is a single JavaScript expression, compiled once per source string and evaluated every time the parameter is read, against a fixed scope. ```js op('lfo1')['chan1'] // channel chan1 of the CHOP lfo1 op('lfo1')[0] * 10 // first channel, by index parent().par.speed * 0.5 // parameter `speed` of the COMP this node is in op('noise1').par.period // another node's parameter time.seconds * 0.2 // engine time fract(time.seconds * 0.02) // a slow 0..1 ramp clamp(op('mouse1')['ty'], 0, 1) // mouse in, clamped ext('energy', 0.5) // a value from the host page, 0.5 until one arrives ``` ## The scope {#scope} | Name | Value | |---|---| | `time.seconds`, `time.frame`, `time.delta`, `time.fps` | engine time: seconds since start, frame count, last frame's duration, smoothed frame rate | | `me` | this node: `me.name`, `me.path`, `me.par.` | | `op(path)` | another node. `op('x')['chan']` or `op('x')[i]` reads a CHOP channel (its last sample); `op('x').par.` reads a parameter | | `parent(n = 1)` | the COMP `n` levels up, with `.par.` | | `ext(name, fallback = 0)` | a number set from outside the patch ([Embedding](embedding.md)) | | `PI`, `abs`, `sin`, `cos`, `tan`, `asin`, `acos`, `atan`, `atan2`, `floor`, `ceil`, `round`, `trunc`, `min`, `max`, `pow`, `sqrt`, `exp`, `log`, `sign` | as in JavaScript's `Math` | | `clamp(v, lo, hi)`, `fract(v)`, `lerp(a, b, t)` | the usual helpers | | `rand(seed)` | a deterministic 0..1 hash of `seed` (the same seed always gives the same value) | Paths work like TouchDesigner's: `op('noise1')` looks in the network the node lives in, `op('../lfo1')` one level up, `op('/geo1/out1')` from the root. ## Reading other nodes {#reading} `op()` cooks the node it points at when needed, so an expression always sees the current frame's value. Reading parameters through `.par` is live too, and a guard stops parameter cycles (two parameters that read each other): the node reports an expression cycle instead of recursing. A CHOP read by an expression needs no wire: the path in the expression is the connection. ## Errors {#errors} A syntax error marks the field red. When an expression throws, or returns something other than a number, string, boolean or array of numbers, the parameter falls back to its constant value and the node shows the error on its badge. Expressions are user-authored patch code, evaluated with `new Function` against the scope above: the trust model of a patching environment, not a security sandbox. Do not paste expressions you would not run. ## From TouchDesigner {#from-td} Imported Python expressions are translated when the translation is faithful (`absTime.seconds*0.2` → `time.seconds*0.2`, `math.sin(x)` → `sin(x)`, `a if c else b` → `c ? a : b`) and kept inert otherwise; imported Python is never executed. The full table is in [Importing](importing.md#expressions). --- # Embedding and external control > Put the editor in an iframe or mount it in your own page, load projects by URL, and drive patches from outside with postMessage and ext(). There are two ways to put WebToe in another page: embed the hosted editor in an iframe and talk to it with `postMessage`, or mount the editor package directly. ## Load a project by URL {#project} ```text https://webtoe.openaudiovisual.com/app/?project=https://example.com/patch.webtoe.json ``` `?project=` takes any URL the browser can fetch; a project on another origin must be served with CORS headers that allow it. Relative URLs resolve against `/app/`, which is how the bundled examples are linked: `/app/?project=examples/03-lfo-garden.webtoe.json`. ## Drive a patch from outside {#ext} Inside a patch, `ext('name', fallback)` reads a number set from outside. The host page sends values with `postMessage`; any parameter expression can read them: ```html ``` In the patch, set a parameter to an expression such as `ext('energy', 0.5) * 2`. The fallback is used until the first value arrives. | Message | Shape | Effect | |---|---|---| | `webtoe:ext` | `{ type: 'webtoe:ext', values: { [name]: number } }` | sets external values; non-numbers and non-finite numbers are dropped; names not sent keep their last value | | `webtoe:load` | `{ type: 'webtoe:load', url: string }` | loads the project at `url` and replaces the current one | Messages are accepted from any origin. That is deliberate and matches the trust model of the patch itself: a message can only set numbers that expressions read, or load a project the page could load from a link anyway. ## The open-audiovisual adapter {#oav} [open-audiovisual](https://openaudiovisual.com/) uses exactly this. Its `@openav/world-webtoe` package wraps an embedded WebToe editor as an openav *world*: the show's mapping layer resolves inputs (MIDI, audio, chords, hands, a timeline) into parameters, and the adapter posts them to the patch each frame as `webtoe:ext` values, sending only when something changed. In the patch they are plain `ext()` reads, so the same network runs standalone with its fallbacks or performed by a show. ## Mount the editor in your page {#mount} The editor package is framework-free and has no dependencies, so it can be mounted into any element: ```ts import { registerAllOps } from '@webtoe/ops'; import { mountEditor } from '@webtoe/editor'; import { setExternals } from '@webtoe/core'; registerAllOps(); const editor = await mountEditor(document.getElementById('app')!, { examples: [{ name: 'my patch', url: '/patches/my.webtoe.json' }], // toolbar menu backend: 'webgl2', // or 'webgpu'; ?backend= in the URL still wins starterPatch: false, // start empty instead of with the default patch repoUrl: 'https://github.com/you/your-fork', }); await editor.loadUrl('/patches/my.webtoe.json'); setExternals({ energy: 0.8 }); // what webtoe:ext does ``` The packages are workspace packages in the repository today, not yet published to npm; build from the monorepo (`apps/web/src/main.ts` is the complete reference host, including the `postMessage` listener). ## Compatibility with old links {#compat} Until the site moved the editor to `/app/`, it lived at the site root. Links and embeds that still point at `https://webtoe.openaudiovisual.com/` keep working: when the root page is opened inside an iframe, or with a query such as `?project=`, `?bridge=` or `?backend=`, it forwards to `/app/` with the same query and hash before rendering. Campaign-tracking parameters alone (`utm_*`, `fbclid`, `gclid` …) do not trigger the forward. --- # NDI bridge > Receive and send NDI video from the browser through a small local bridge, with the pixel conversion running in a 1 KB WebAssembly kernel. Browsers cannot join an NDI network: there is no mDNS discovery and no raw TCP or UDP in a web page, and the NDI SDK is closed source. WebToe therefore splits the job in two. A small local bridge owns the NDI side using **your own** NDI runtime, and the `ndi in` / `ndi out` TOPs do the pixel work in the browser, talking to the bridge over a WebSocket on localhost. ```text NDI network ⇄ packages/ndi-bridge (Node; ws, optional grandiose) ⇄ ws://127.0.0.1:9980 JSON control + binary frames ⇄ ndi in / ndi out TOPs ⇄ video kernels (WASM, JS fallback) ⇄ GPU textures ``` ## Try it without NDI {#mock} From a checkout of the repository (after `npm install`): ```bash node packages/ndi-bridge/index.mjs --mock ``` Mock mode needs nothing from NDI: the bridge generates an animated UYVY test pattern as a source called *WebToe Mock (Pattern)* and accepts frames sent from the browser. In the editor, add an **ndi in** TOP: it connects to `ws://127.0.0.1:9980` and shows the pattern. Add an **ndi out** TOP after any image and the bridge logs the frames it receives. ## Real NDI {#real} 1. Install the NDI runtime from Vizrt. 2. Install the optional binding in the bridge package: `npm i grandiose` inside `packages/ndi-bridge`. 3. Run `node packages/ndi-bridge/index.mjs` (no `--mock`). Without `grandiose` it falls back to mock behaviour and says so. | Operator | Parameters | |---|---| | **ndi in** | `bridge url` (default `ws://127.0.0.1:9980`), `ndi source` (blank = the first source found) | | **ndi out** | `bridge url`, `sender name` (default *WebToe Out*), `active`, `send rate` (default 30 fps), `send width` × `send height` (default 1280 × 720) | Bridge options: `--port ` (default 9980), `--mock`, `--mock-name `. ## The protocol {#protocol} Version 1, kept in sync between `packages/ndi-bridge/index.mjs` and `packages/ops/src/video/protocol.ts`: - **Control** messages are JSON text: `hello` (version and mode), `sources` (list of names), `subscribe` `{ source }`, `send-open` `{ name, w, h, fps }`. - **Frames** are binary: a 24-byte header (magic `WTNF`, `u32` width, `u32` height, a fourcc `RGBA` or `UYVY`, an `f64` timestamp) followed by the pixels. ## Why WebAssembly here {#wasm} Converting UYVY to RGBA (BT.601) and swizzling BGRA, every pixel of every frame, is the one place in the engine where a WASM kernel pays for itself. The kernel is written in AssemblyScript (`packages/wasm-kernels`), compiled to a 1 KB `.wasm` that the app loads at startup; the JavaScript implementation is the permanent fallback and the unit-tested reference. The console says which one is running (`video kernels: wasm`). ## Limits {#limits} `ndi out` reads pixels back synchronously on WebGL2 (throttled by its send rate); an asynchronous readback for WebGPU and compressed transport are listed follow-ups. Source discovery is by name in the `ndi source` parameter. NDI® is a trademark of Vizrt NDI AB. WebToe ships no part of the NDI SDK; real mode uses the runtime you install. --- # Architecture > The packages and their one dependency rule, the GPU pass contract behind both backends, the cook model, the file format and the decisions behind them. WebToe is an original engine for the web. It implements the TouchDesigner workflow (operator families, wired networks, expression-driven parameters, a live cook loop) on WebGL2 and WebGPU, in TypeScript, with no runtime dependencies. ## Packages and the dependency rule {#packages} ```text apps/web ──▶ @webtoe/editor ──▶ @webtoe/ops ──┐ │ │ ├──▶ @webtoe/core ├──▶ @webtoe/gpu ───────┤ └──▶ @webtoe/io ────────┘ @webtoe/cli (standalone node script; mirrors io's tables) ``` **Imports flow downward only, and `core` imports nothing.** No package touches the DOM except the editor, the app, and the media operators (image, video, camera). `core` and the CHOP side of `ops` run headless in the test suite. | Package | Role | |---|---| | `@webtoe/core` | graph model, pull-based cook engine, expressions, the backend-agnostic GPU pass contract, versioned serialization, `registerOp` | | `@webtoe/ops` | operator definitions; CHOP kernels behind a WASM-ready interface; TOP shaders written per backend (GLSL and WGSL, by hand) | | `@webtoe/gpu` | the WebGL2 and WebGPU backends: texture pools, ping-pong feedback, scene rendering, readback | | `@webtoe/io` | `.webtoe.json` load/save and the TouchDesigner importer, behind a `ProjectLoader` adapter | | `@webtoe/editor` | the framework-free editor, `mountEditor(el, options)` | | `webtoe` (packages/bridge) | the local import bridge behind `npx webtoe` | | `webtoe-ndi-bridge` | the local NDI bridge | ## The contracts everything rests on {#contracts} 1. **The GPU pass contract** (`core/passes.ts`). TOPs never touch WebGL or WebGPU. They describe work as passes (`TexturePassSpec`: shader id, uniforms, inputs, output; `ScenePassSpec` for 3D) against a `GpuFacade`, and each backend owns resources, pipeline caches, per-node texture pools, media upload, blits and readback. This is what makes two backends possible, and what a compute pass will extend. 2. **The registry** (`core/registry.ts`). `registerOp(spec)` is the public plugin surface. The palette, the parameter panel and the serializer discover operators through it; type keys are namespaced `family:name`. 3. **`ProjectLoader`** (`io`). Importers are adapters (`canLoad` / `load` → graph + import report). The TouchDesigner importer is one; an official JSON format from Derivative would slot in beside it without engine changes. 4. **The kernel seam** (`ops/chop/kernels.ts`). CPU kernels sit behind an interface with the TypeScript implementation as the permanent fallback; a WASM build is adopted only on a profiled win of at least 2×. 5. **Versioned files** (`serialize.ts`). `.webtoe.json` carries a version and a chain of migrations; every saved node carries its family, so unknown types degrade to the right stub. ## Cook model {#cook} - Pull-based and memoized per frame, with a cycle guard. Expressions can pull other nodes (`op('x')`) re-entrantly. - **Feedback** is the deliberate cycle-breaker: its input is not pre-cooked, and it returns the input's previous-frame texture from the backend's pool (transparent black on the first frame). - Operators driven by time, media or input are flagged to cook every frame. ## GPU backends {#gpu} | | WebGL2 | WebGPU | |---|---|---| | Status | complete: every TOP, plus the 3D scene renderer | 2D parity: every shader-driven TOP ships WGSL; the 3D scene pass is pending | | Passes | fullscreen triangle, per-shader program and uniform-location caches | one shared explicit bind-group layout: globals @0, operator uniforms @1 (256-byte aligned), sampler @2, textures @3+ | | Uniforms | by name | sorted by name, one `vec4f` each; enforced by a test | | Readback | synchronous `readPixels` | asynchronous buffer copy, one frame latent | WebGL2 is the default; `?backend=webgpu` opts in and falls back to WebGL2 if initialisation fails. Previews are painted by one transparent compositor canvas over the editor, which blits the viewer and every visible TOP thumbnail each frame; there are no CPU readbacks for previews. ## 3D {#3d} SOPs produce typed-array geometry, versioned so the renderer can cache vertex arrays. Geometry COMPs pick their display SOP, apply a material, and instance over SOP points or CHOP channels; the Render TOP gathers matching Geometry COMPs, a camera and lights, and draws them through four original shaders with depth, instancing and a translucent-last sort. Declared limits of this first version: the scene pass is WebGL2-only, lines are 1 px, and parent transforms are not inherited. ## Editor {#editor} DOM node boxes over an SVG wire layer, pan and zoom with CSS transforms, a palette, a parameter panel with per-parameter expressions and pages, and a viewer that shows a TOP, scopes a CHOP or prints a DAT. DOM was chosen over a canvas-drawn network for free hit-testing and text; it holds up to a few hundred visible nodes, and a canvas renderer could replace it behind the same interface. ## Decisions {#decisions} - **WebGL2 floor, WebGPU ceiling.** Compute shaders are the path for the particle (POP) family, hence the pass contract. - **Hand-written shaders for both backends, no transpiler.** Auditability and clean provenance over brevity; a contract test enforces that both exist. - **Expressions are JavaScript against a fixed scope.** The trust model of a patching tool, not a sandbox. Imported Python never executes. - **Zero runtime dependencies.** For longevity, for embedding, and so the whole system stays readable. - **No Derivative code, binaries, sample files or UI assets in the repository.** Shaders and UI are original; the importer reads files you open with your own TouchDesigner. --- # TouchDesigner parity > What "toward TouchDesigner" means for a browser engine, how it is measured on 60 real projects, where WebToe stands, and what comes next. WebToe aims at **creative-semantics parity** with TouchDesigner (networks, cooking, operators, expressions, rendering, interaction), not native-device parity. Progress is measured, not estimated: on a corpus of real projects, and against the official operator inventory. ## How it is measured {#method} The corpus is 60 real TouchDesigner projects from a daily generative-art practice, 2022–2026, sampled across eras from 199 available: **28,698 nodes**, expanded with the official `toeexpand` and counted by operator type, parameter mode and expression. The analyzer lives outside this repository because it reads private files; only aggregate numbers are published. The family mix of the corpus: TOP 7,536 · DAT 6,332 · COMP 5,417 · POP 3,148 · SOP 2,942 · CHOP 2,663 · MAT 660. 13,708 parameters carry live Python expressions, plus about 7,000 more in flagged expression modes. | Step | What shipped | Corpus nodes runnable | |---|---|--:| | v1 | first release: core TOPs and CHOPs, importer | **32.3%** | | Cycle 1 | expressions v2 (`parent()`, `.par`, Python conditionals), routing TOPs (switch, select, math, reorder, flip), CHOP switch/speed/parameter, DAT tables, parameter-mode bitfield | **47.1%** | | R3 | the 3D pipeline: SOPs, MATs, Geometry/Camera/Light COMPs, Render TOP, instancing | **62.3%** | On a 213-node reference project the same steps took runnable nodes from 56 to 71 to 88. After R3, a 2022 sketch of a DNA helix went from 55% to 71% runnable and another from 65% to 74%; mapping POPs onto the SOP implementations took a POP-heavy 2025 project from 43% to 73%. ## Operators against the official inventory {#inventory} The official counts come from the operator categories on docs.derivative.ca (crawled 2026-06-11). Not every official operator should exist in a browser; each one falls into one of four classes: - **Portable**: pure compute or render semantics (most TOPs, CHOPs, SOPs, POPs, MATs). Implementable as they are. - **Web-equivalent**: device or network I/O with a browser counterpart: Video Device In → `getUserMedia` (shipped as camera in), MIDI In → Web MIDI, Audio Device In/Out → Web Audio, Screen Grab → `getDisplayMedia`. - **Web-equivalent through a local bridge**: I/O with no browser primitive gets a small local process the user runs, the pattern of the import bridge. [NDI in/out](ndi.md) shipped this way; Art-Net, Syphon/Spout capture and serial would follow it. - **Native-only**, declared out of scope: DirectX/SDI/ST 2110, vendor SDKs (Kinect, ZED, Oculus), C++ operators, Notch and Substance hosts. ## Fidelity: the same numbers as TouchDesigner {#fidelity} Op counts say what exists; fidelity says whether an imported network renders the *same pixels* and moves at the *same pace*. The behaviour below was measured black-box against TouchDesigner 2025 (known inputs in, outputs read back) in the author's own production web port of two TouchDesigner shows, and ported here by the author. CPU references for every formula are tested, and the real shaders are compared with them in Chrome. | Operator | What matches TouchDesigner | Measured tolerance | |---|---|---| | Level | TouchDesigner's order of operations, black level, ranges, low/high, post page; opacity scales RGB and alpha | ≤ 1e-3 (14 × 2 cases) | | Edge | TouchDesigner's formula (√strength, black level, offset), Rec.709 luminance, edge over input | ≤ 1.2e-7 (65 cases) | | Monochrome, Lookup | Rec.709 luminance, TouchDesigner's channel menus | exact formula | | Ramp | wrapping keys (up to 32), phase/period rules, extend, interpolation, fit aspect | median ≤ 1e-5 (35 cases); antialias not reproduced | | Blur | full-width size, kernel integrated per texel, single-tap preshrink | ≤ 7.5e-7 (52 images) | | Noise | Gustavson Perlin/simplex 2D–4D, TouchDesigner's coordinates, transform page, seeds and octave count | ≤ 3.7e-3, 99.9% ≤ 6e-4 | | Composite | premultiplied, all 46 operations, transform page | 37 ops ≤ 1e-7, 9 fitted ≤ 3e-5 | | Cook clock | fixed-rate steps at the project cook rate; per-step constants behave as in TouchDesigner on any display | deterministic, tested | | Lag, Speed, Feedback | lag = time to 90% (a = 1 − exp(−dt·ln10/lag)); speed outputs the sum before this step; feedback returns its Target TOP's previous step | tested | Both GPU backends render these shaders identically within 1/255. The full table, with what is inferred or approximate, is in [docs/TD-PARITY.md](https://github.com/frank890417/WebToe/blob/main/docs/TD-PARITY.md#fidelity--tops-that-reproduce-touchdesigners-numbers). ## Engine concepts {#concepts} | Concept | Status in WebToe | |---|---| | 3D: geometry, materials, cameras, lights, Render TOP, instancing | shipped (R3) on WebGL2; WebGPU scene pass pending | | Multi-sample CHOPs and time slicing | not yet: channels are single-sample at control rate. The prerequisite for audio, trail, resample and wave | | Audio | planned on Web Audio (AudioWorklet ↔ multi-sample CHOPs, AnalyserNode for spectra) | | GLSL TOP | specified: TouchDesigner's injected contract (headerless GLSL, `sTD2DInputs`, `vUV`, `uTD2DInfos`, `TDOutputSwizzle`) shimmed onto GLSL ES 3.00. Used by 32 of the 60 corpus projects | | POPs | import and run their geometry through the SOP implementations; dynamics (forces, solvers) are stubs. Native POPs are planned on WebGPU compute | | Parameters | pages and expressions shipped; pulse parameters, binding and custom COMP parameters to come | | Panels | button, slider and container panels planned as a DOM overlay | | Replicator | planned with table-driven templates; `COMP:replicator` appears in all 60 corpus projects | | Python | **permanent boundary**: imported Python is kept and shown, never executed. The planned answer is an opt-in JavaScript callback DAT with a TouchDesigner-shaped API | ## The loop {#loop} 1. **Measure**: run the corpus analyzer; read the coverage and the histogram of stubbed types. 2. **Pick**: the highest *projects affected × low effort*, respecting prerequisites (multi-sample before audio, 3D before render-dependent TOPs). 3. **Implement** on the public contracts: `registerOp`, shaders for both backends, importer tables, honest stubs for the rest. 4. **Verify**: unit tests, fixture round-trips, and real sketches in the browser. Every bug a real project exposes gets a regression test. 5. **Record** the new number in docs/ROADMAP.md, then measure again. ## Next {#next} In order: **multi-sample CHOPs and time slicing → GLSL TOP → POPs on WebGPU compute → WebGPU 3D scene pass → replicator, tables and panels.** The full reasoning and measured gates are in [docs/ROADMAP.md](https://github.com/frank890417/WebToe/blob/main/docs/ROADMAP.md) and [docs/TD-PARITY.md](https://github.com/frank890417/WebToe/blob/main/docs/TD-PARITY.md).