Alpha (0.1.0). Motion vectors for any WebGPU pipeline, and a way to prove they are right.
Temporal techniques (upscaling, TAA, denoising, GI history) reproject last
frame through motion vectors. When those vectors are wrong, history smears or
resets. three.js's velocity node is exact for rigid and camera motion but
reports deformation as motion: a mesh displaced by positionNode or held on a
morph target reads 3–64+ px of motion every frame while standing still
(docs/MOTION.md §7). This package defines what a motion vector
must mean and measures any source against it.
- CONTRACT.md: the motion contract (value, jitter, frame semantics, unknowns) and the conformance bars.
@pmndrs/motion-vectors: contract constants and CPU reference math.@pmndrs/motion-vectors/webgpu: engine-free GPU passes on a bareGPUDevice: camera motion from depth, cut detection, warping error, a warp-residual audit, an exact oracle comparison, luma histograms.@pmndrs/motion-vectors/three: a drop-invelocityfor three.jsWebGPURendererthat is exact where three's is not:positionNodedeformation, morph targets,displacementMap, several renders per frame (docs/MOTION.md §8).
npm install @pmndrs/motion-vectorsthree (>=0.186.0), pixi.js and @pixi/3d are optional peers: install the
one your adapter needs. The root and /webgpu need none. Changes:
CHANGELOG.md.
import { mrt, output } from 'three/tsl';
import { MotionVectors } from '@pmndrs/motion-vectors/three';
const motion = MotionVectors.for(renderer); // one per renderer, shared page-wide
renderer.setMRT(mrt({ output, velocity: motion.velocity })); // the output must be named `velocity`
renderer.setAnimationLoop(() => {
motion.beginFrame(); // optional with one render per loop tick; required with more
renderer.render(scene, camera);
});-
Jitter: set
motion.projectionMatrixto the unjittered projection (anyMatrix4, read every frame); without it the camera's own projection is used. -
Output: the
velocityslot gets the full contract texel (CONTRACT.md):.xymotion,.zaMotionFlag(1 exact, 3 rigid fallback, 4 no previous position),.wthe previous linear depth. Give it anrgba16floattexture to keep.zw. -
Cuts:
motion.reset()makes the next frame report zero motion, flagged as no previous position. -
Deformation:
positionNodegraphs are re-evaluated with last frame's uniforms and previous position automatically, and so are morph targets anddisplacementMap(with an animated scale or bias). Skinning,InstancedMeshandBatchedMeshare exact too, skinning even with several renders per frame. -
What the node can't see: a
positionNodethat reads a texture, buffer or storage that changes per frame. The node can't know last frame's contents, so those pixels report flag 3 (rigid: camera and object motion only). If you know the previous position, say so:// positionPrevious = last frame's position before positionNode MotionVectors.setPreviousPosition(material, positionPrevious.add(normalLocal.mul(lastFrameHeight))); MotionVectors.setPreviousPosition(material, 'rigid'); // or: hold the deformation, flagged 3
toVar()variables made outside aFnaren't followed either (docs/MOTION.md §8). -
three releases: r186 and r187 pass every lab bar. r185 doesn't: its own instancing and batching history is wrong, and the node relies on it.
MotionFromDepthPass runs camera motion from depth and the GPU cut detector
(below) from a three depth texture and camera, and publishes both as three
textures. Use it for a reset signal, or as a velocity source where a scene has
no velocity output (static geometry only: moving objects read zero, flagged).
import { MotionFromDepthPass } from '@pmndrs/motion-vectors/three';
const depthMotion = new MotionFromDepthPass(renderer);
await depthMotion.init();
// Per frame, after rendering into `target` (which has a depthTexture), while the
// camera still holds the projection it rendered with:
depthMotion.update({
depth: target.depthTexture,
camera,
projectionMatrix: unjittered, // only when jittering
motions: [target.textures[1]], // the velocity output, optional
});
// texture(depthMotion.motion): camera motion in the contract format
// depthMotion.cutFlag: 1×1, .r = 1 resets history this frameencode(encoder, frame) does the same into your own command encoder.
Reversed depth works; a logarithmic depth buffer can't be unprojected and is
refused.
CutDetector tells a temporal pass when its history stopped being valid: a
camera cut, a teleport, a scene switch. It runs on the GPU with no readback and
writes a 1×1 rgba16float flag the consumer reads in the same frame (r = 1:
reset history).
import { CutDetector, MotionFromDepth } from '@pmndrs/motion-vectors/webgpu';
const fromDepth = new MotionFromDepth(device);
const cuts = new CutDetector(device);
await Promise.all([fromDepth.init(), cuts.init()]);
fromDepth.configure(width, height);
// Per frame, after the scene render:
fromDepth.encode(encoder, depthTexture, { viewProjection, inverseViewProjectionJittered, view });
cuts.encode(encoder, fromDepth, { motions: [velocityTexture] }); // velocity is optional
fromDepth.endFrame();
// Your temporal pass binds cuts.flag.gpu and resets where .r is 1.The signal is the share of surface pixels no motion (zero, camera, the
engine's velocity) maps onto a matching previous depth. Lighting changes,
fades and flashes don't touch depth, so they never fire; on the lab's 14-event
cut timeline it fires on every cut whose history is really gone and on nothing
else (docs/MOTION.md §9). It needs depth: inputs without it
fall back to the luma histogram (jensenShannon, earthMovers). A three app
uses MotionFromDepthPass (above). cuts.read() reads the verdict back if you
reset on the CPU instead.
WarpingError measures temporal stability without a reference image. Each
frame it warps the previous image onto the current one through a motion field
and averages the luma difference. A depth mask leaves out disoccluded pixels.
Flicker, smearing and ghosting under motion read higher; a stable output
approaches the scene's own shading change. It's a measurement for benches and
A/B runs, not a runtime feature.
import { WarpingError } from '@pmndrs/motion-vectors/webgpu';
const warping = new WarpingError(device);
await warping.init();
// Per frame, after the image you're judging (e.g. an upscaler's output) exists:
warping.encode(encoder, { current: outputTexture, motion: motionTexture, linearDepth });
device.queue.submit([encoder.finish()]);
const { meanError, scored, occluded } = await warping.read();motionis the judge: any field in the contract format (CONTRACT.md) whose.wis filled, since the mask compares against it. Use the most correct motion you have,MotionVectorsin three, and use the same judge for everything you compare. Camera motion (MotionFromDepthPass.motion) is a judge for static scenes only: it calls a moving object still, and when the object moves along the view axis the mask can't catch it, so it rewards outputs built from that same wrong motion (docs/MOTION.md §10).linearDepthis this frame's linear depth, e.g.MotionFromDepthPass.linearDepth. Without it nothing is masked.- The image can be at display resolution and the motion and depth at render resolution.
- The class keeps its own copy of the previous image and depth, so
currentneedsCOPY_SRC.reset()after a cut.heattakes anrgba8unormstorage texture for a per-pixel map. - Read
meanErrortogether withscored: a field that leaves more pixels to the mask covers less of the image.
@pixi/3d's own velocityOutput is camera motion only: a moving object reads
zero. MotionVectors from @pmndrs/motion-vectors/pixi adds object motion for
the meshes you track, in this package's contract.
import { MotionVectors, motionOutput } from '@pmndrs/motion-vectors/pixi';
const motion = new MotionVectors();
motion.track(car); // gives the mesh its own copy of its material
app.ticker.add(() => {
car.position.x += 0.1;
motion.update(); // once per frame, before the view renders
});
// A pass that needs it: requiredOutputs = [motionOutput]; graph.getSceneOutput(motionOutput)Rigid motion (translation, rotation, scale, parenting) is exact; skinning and
morph targets report the mesh's rigid motion. npm run check:pixi checks it
against analytic motion on the GPU. Why each tracked mesh needs its own material,
and what pixi-3d could add to remove that, is in
docs/PIXI.md.
No adapter yet. Babylon's own velocity is exact on camera and rigid motion
and never reports deformation as motion. It misses deformation motion and,
under its default bone texture, skinning. It also goes stale on a second
render per engine frame and on a mesh's first frame after it wasn't drawn.
npm run audit:babylon measures it (docs/MOTION.md §11).
npm run examples # http://localhost:5612- three-compare: the same scene with three's
velocity(left) andMotionVectors(right), as motion vectors, motion blur or colour. Pause it: the right half goes flat grey (zero motion), the left keeps reporting motion on every deformed object. - pixi-compare: one pixi-3d frame split at the middle, pixi-3d's
velocityOutputon the left andMotionVectorson the right. The sliding box, spinning box and orbiting ball read as still on the left.
lab/ renders scenes whose every object also writes an oracle: its exact
previous-frame position, from the scene's own math. Because the oracle and the
source under test describe the same rasterized point, the error is measured per
pixel, in render pixels, with no image warping in the way.
npm install
npm run lab # http://localhost:5611 (Chrome/Edge with WebGPU)
npm run measure # every experiment, headless over CDP, exact GPU readback (~2.5 min)
npm run check:gpu # the conformance bars; add -- --adapter swiftshader for no GPUResults land in results/<label>/ (gitignored).
npm run typecheck && npm test && npm run build && npm run verify:packThree.js is an optional peer (≥ 0.186); development pins 0.187.0. Incubated in pmndrs/upscaler; the history came with it. Status, phases and gates: docs/PLAN.md.
MIT licensed.