How I built a marble-sorting machine that paints in real time (C++, Box3D, Vulkan)

Nine hundred glass marbles start out shuffled in a hopper. About two and a half minutes later they're stacked in thirty narrow glass lanes, and the lanes read as a picture of a sunset. Nothing is faked: no marble changes color, nothing gets teleported into place, and the video plays at the speed the simulation ran. The machine reads each marble's color, decides where it goes, opens a gate at the right moment, and sends the ones it can't use yet back up to the hopper.

It's a simulation, not a workshop build. The physics is Box3D, Erin Catto's 3D physics engine (v0.1, pinned to one commit), driven from C++20. The rendering is a small Vulkan path tracer built on hardware ray queries. I made it for a YouTube Short, and this is the long version: how the mechanism works, the code that runs it, the long list of ways it jammed, and what hardware you need to render something like it yourself.

It's a departure from my usual infrastructure posts. But most of the work felt familiar: a system that worked in the demo, wedged under real load, and needed one more guard after every incident.

The one rule: honest sorting, no tricks

My first marble machine wasn't honest. It dropped 50,000 marbles through pegs and rocking plates, and the picture was painted on afterward: each marble got the color of wherever it happened to land. Plenty of "satisfying" videos work that way. Once you know, though, you can't un-know it, and I wanted a version where the machine does the work.

So the sorter has one hard rule. Before the simulation runs a single step, every marble's color is decided and written to colors.bin. The machine is six independent sorters side by side, and each one's hopper chamber holds exactly the colors its five lanes need, in the right counts, shuffled with a seeded generator:

// Every marble's group and color, decided now and written out before
// anything moves: each group's chamber holds exactly its lanes' colors,
// shuffled.
for ( int i = count - 1; i > 0; --i )
{
    state = state * 6364136223846793005ull + 1442695040888963407ull;
    std::swap( colors[base + i], colors[base + ( state >> 33 ) % ( i + 1 )] );
}
WriteColors( cfg.outDir() / "colors.bin", palette, colors );

When the run ends, the bake reads each lane bottom-up by the marbles' actual resting heights and compares each marble's pre-assigned color with the target pixel. It writes picture_result.png and picture_target.png side by side. On the final bake those two files are byte-for-byte identical: 900 of 900 marbles are on the right pixel.

The picture is 30 lanes by 30 rows: a procedural sunset, quantized to a 20-color marble palette by nearest match in Oklab. Each lane is one marble wide, so it fills bottom-up in arrival order. The sorter's whole job is getting the right marble to the right lane at the right time.

The mechanism, from hopper to pixel

The six groups share a cabinet and a glass hopper, and nothing else. Here's the path a marble takes, in order:

• Glass hopper with a vibrating funnel. Each chamber ends in a steep pyramid funnel that narrows to a slot. The funnel panels sit on a vibrator, moving 4 mm at 15 Hz like a real bowl feeder.
• Rotary vane feeder. A brass roller with four pockets sits under the slot and carries a few marbles at a time into the feed tube, turning three-quarters of a revolution per second. Two level sensors gate it: it turns only while the tube is running short (fewer than 10 marbles queued) and the trough below is empty. Each pocket holds a whole marble. One only partly in would be sheared between a vane and the housing.
• V trough, open run, elbow, drop. The trough funnels each pocket's marbles into a bore just over one marble wide. An open glass channel falls at 25° in plain view, turns through a smooth elbow, and drops vertically to the escapement in the sensor head.
• Escapement and color sensor. The escapement lets one marble through every 0.27 s onto a 12° entry ramp, and holds if more than three are already backed up. The sensor reads each marble where the ramp meets the lanes, and the controller assigns it a lane.
• Rail and gates. The rail over the lanes falls just 3.5°, so marbles cross at a steady pace. Each lane has a copper shutter in the rail floor and a brass stop pin, both of which slide out of the marbles' plane when idle. To drop a marble, the shutter pulls back and the pin slides in to stop it over the hole.
• Rejects and the return line. A marble no lane wants right now rolls off the end of the rail into a pipe. It rolls under gravity along a glass pipe across the front of the cabinet, falling 3°, to a riser that lifts it over the hopper and drops it back into its own chamber.
• LED display. Each sensor head shows one square per lane, lit in the color that lane needs next. It makes the controller's state visible.

Under the hood there are two Box3D worlds. The machine world is a glass slab one marble deep, with marbles held to the plane by motion locks. The hopper world is fully 3D, because a hopper locked to a plane doesn't behave like a hopper. Releasing a marble at the escapement moves its body from one world to the other. One honest caveat: the return pipe is animated, not simulated. A reject's position along it comes from a rolling-ball formula (5/7·g·sin 3°, then a 9 m/s lift), because nothing on that pipe interacts with anything else.

How the controller decides where a marble goes

The controller's core is one choice: when a red marble arrives, which of the lanes that want red gets it?

// Send each to the fullest lane whose next pixel is its color,
// or let it run on to the reject pipe.
int best = -1;
for ( int j = grp.firstLane; j < grp.firstLane + grp.laneCount; ++j )
{
    if ( assigned[j] >= S.rows || target[j * S.rows + assigned[j]] != colors[i] ) continue;
    if ( best < 0 || assigned[j] > assigned[best] ) best = j;
}

Fullest, not emptiest. My instinct was to balance the lanes, but that's backwards. A sunset's rows are bands of one color, so if every lane is at the same height, every lane wants the same color at once and everything else gets rejected. Lanes at different heights want different colors, so far more random draws find a home. Even so, about half the marbles read at the sensor go around again.

Where you read the color matters. I first assigned lanes as marbles left the sensor head. Now and then one overtook another in the drop zone and two pixels swapped. The sensor now sits where the ramp meets the lanes, where marbles are single file and arrive in the order they were assigned.

A few gate rules, each added after something went wrong:

• A gate opens only when its marble is close and nothing else on the rail is near the hole. It stays open until the marble is down, because closing a shutter under a marble wedges it.
• The next marble for a lane waits until the previous one is three diameters down. Otherwise the two meet in the lane's mouth and can wedge.
• If a marble overshoots its lane, it and everything queued behind it become rejects, and those rows are offered again. The machine recovers instead of painting a wrong pixel.

The gates are kinematic bodies snapped into place with b3Body_SetTransform, like solenoids. My first version gave them a velocity instead, and a kinematic body with a velocity keeps drifting after it arrives. Teleporting doesn't wake a marble resting against the body, though, so sleeping is off in both worlds. Otherwise a queue that dozed off against a stop pin would never notice the gate opening.

Everything that jammed, and what fixed it

The controller worked within a day. Making the marbles actually flow took most of the project, because granular material is famously good at jamming. Every item here stopped a bake at some point, and every fix is still in the code with a comment saying why:

• Arching over the hopper outlet. Marbles form a stable arch over a small opening and stop. That's a real, well-studied granular effect, and it's why the funnel vibrates.
• Column load. Even without an arch, the marbles under the slot were pinned by the weight of the column above. The vane feeder fixes it by carrying marbles out instead of waiting for them to fall.
• Funnel mouths arching. In a slim chamber, two marbles bridged across the funnel. The chambers got deeper front to back, and the funnel is steeper front to back than side to side.
• Thick funnel walls made a shelf. Where two chambers meet, a thick wall's corner poked through its neighbor and made a ledge for marbles to park on. The funnel walls are now thin.
• A marble riding another, wedged under a tube roof. A covered feed tube looks right, but a marble landing on top of another wedges under the roof. The run is now an open channel.
• Arching across a sharp corner into the drop. At a sharp corner, the lead marble falls against the drop's far wall and the next one, pushed by the queue, pins it there. The fix is a smooth elbow, 2.5 diameters in radius, with the same bore all the way round, so the queue's push turns down the drop. The "tube short" sensor also keeps the queue inside the vertical drop and off the corner.
• Fast marbles sinking into each other. Marbles hitting the queue at the bottom of the drop could interpenetrate deeply enough to jam across the bore. The hopper world now steps at 240 Hz with 16 substeps, and Box3D's contactSpeed is raised to 2.0 so overlap is pushed out briskly.
• Tunneling. A marble falling about 5 units down an empty lane hits roughly 10 units/s. At 60 Hz it moved more than half its diameter per step and tunneled through the marble at the bottom, swapping two pixels. At 240 Hz it moves 0.04 units per step against a marble 0.1 across.
• Glass panes pinning marbles. A pane 5 mm from a marble sits inside Box3D's speculative contact margin, so its friction pinned marbles that should roll. The panes left the physics entirely, and marbles are locked to the plane instead. The same margin wedged marbles in lanes that were geometrically wide enough, so lanes are 1.25 diameters wide and the steel fins between them have friction 0.05.
• The tube tapper. If an escapement has asked for a marble and nothing has arrived for 1.5 s, the sim "taps" the tube: marbles in the drop are re-centered and nudged down, and any marble riding the queue is popped back up the run. It's the simulated finger-flick, and the one fix I'd call a crutch.
• The catch tray. Box3D occasionally lets a fast marble squeeze through a thin wall. A marble that escapes the hopper goes back in over the top of its own chamber, like a real overflow tray, and a lid catches the odd marble a nearly empty feeder flicks upward.
• The stall detector. The most backend-engineer fix on the list. If nothing is sorted or rejected for 60 simulated seconds, the bake stops and dumps what every lane wants next and where every unplaced marble is. Most fixes above started from one of those dumps.

Anyone who has run production systems will recognize the pattern. None of these failures showed up in the first minute. They showed up at marble 400, or when a chamber was nearly empty, or when two marbles arrived 30 ms apart. A bake that has to place 900 of 900 is a long soak test, and the bugs live in the tail.

Why six sorters instead of one fast one

The video plays in real time, so the video clock is the simulation clock and there's no speeding up the slow parts.

My first working machine had one hopper, one sensor head and one rail over all 30 lanes. It managed about 3.3 marbles per second at the sensor and took about 16 minutes to finish the picture. The escapement can't just go faster either: the 0.27 s spacing is what lets the gate rules work.

So: parallel sensor heads, each with its own rail over a slice of the lanes. My first version had them all drawing from one shared hopper. A quick Monte Carlo of just the controller (random draws, lane assignment and rejects, no physics) showed that a shared hopper starves the endgame. Near the end, each group needs a few specific colors, and its head keeps drawing marbles only another group can use.

So each group got its own chamber holding exactly its lanes' colors: 5 lanes × 30 rows = 150 marbles. No head ever draws another group's marble, and all six finish in about two and a half minutes. Each group's rail sits a step above the previous one so the next sensor head fits over its end, and the lanes run unbroken across the cabinet, so the picture has no seams.

Rendering: a path tracer built on Vulkan ray queries

The renderer is one Vulkan compute shader, one invocation per pixel, using VK_KHR_ray_query rather than the full ray-tracing pipeline. Ray queries keep the whole path tracer an ordinary GLSL loop, with no shader binding table.

All the geometry is procedural. There's exactly one BLAS, containing a single unit AABB. Every marble, glass panel, gate, roller vane and tube segment is a TLAS instance of that box with its own transform, and a custom index says what it really is: sphere, box or cylinder. On a candidate hit the shader intersects the real primitive analytically in object space. Each frame only rewrites instance transforms and rebuilds the TLAS.

Marbles are a glass shell around a swirled core. The core is an opaque sphere at 75% of the radius with twisted noise bands, rotated per marble so no two match. Inside a marble, paths are traced analytically between core and shell with no BVH at all. The shell's reflection is evaluated exactly, because random reflections made every highlight sparkle with grain. Glass is transparent to shadow rays, the usual cheat that keeps everything behind the front glass from being lit only by caustics.

Power-weighted light sampling. At each diffuse hit, the shader picks one light in proportion to radiance times area instead of uniformly. Uniform picking wasted most samples on dim LED strips.

A deterministic live preview. The same shader has a second mode with every random choice removed: fixed points on each softbox, exact reflection and refraction, a small ambient term instead of bounced light, and a fixed 4×4 sub-pixel grid. Nothing shimmers in motion, so I could scrub the whole video at full speed to judge camera moves. T switches to the real path tracer.

Edge-aware à-trous denoising. For the live path-traced view (and optionally on export with --denoise) there's an à-trous filter after Dammertz et al. 2010. It divides out albedo so it smooths lighting and not marble colors, and it's guided by the first hit's normal, depth and albedo.

ACES tone mapping. Exposure in stops, the Narkowicz ACES filmic fit, then sRGB. The filmic shoulder keeps glass highlights from clipping harshly against the dark walnut cabinet.

Motion blur, the camera, and a 1080×1920 export

The bake records every marble's position at 30 fps, quantized to 16 bits per axis: about 24 MB for the whole run. Gates, rollers and LED states go in kinematics.bin in the same format. Each roller's angle is stored as (cos 4a, sin 4a), because four vanes repeat every quarter turn and that pair interpolates cleanly.

Motion blur comes from re-sampling that trajectory across the shutter. The exporter renders each frame in chunks of 16 samples, and each chunk sees the scene at a different instant across a 180° shutter (1/60 s at 30 fps):

// Each chunk of samples sees the scene at a different instant across the shutter.
float ts = t + shutter * ( ( done / chunk + 0.5f ) / chunks - 0.5f );
frame.Sample( *ep, ts );
scene.Update( cmd, ts, frame.marbles, frame.tankMarbles, frame.parts );

At 128 samples per pixel that's eight instants per frame, enough that a marble falling down a lane reads as a streak rather than eight ghosts.

Output is 1080×1920, the native Shorts canvas, and the live preview renders at that size too, so framing in the preview is exactly what ships. The camera opens with a 0.7-second flash of the finished picture as the hook. It then follows one feeder's marbles down the tube to the sensor head and along the rail, and from the 17-second mark eases out to the whole machine while the picture fills in.

The pipeline: bake, render, soundtrack, ffmpeg

Each episode is one small config file holding every knob the physics depends on, and Box3D is pinned to one commit in CMake, so a re-bake simulates identically:

# episodes/002-marble-sorter/episode.ini
machine = sorter
lanes = 30
groups = 6
rows = 30
marble_radius = 0.05
picture = procedural:sunset
seed = 1

Each step is its own binary:

cmake -S . -B build -G Ninja      # fetches Box3D, glm, stb, Vulkan headers, volk, VMA
cmake --build build

./build/bake episodes/002-marble-sorter --workers 8      # physics -> trajectory.bin, kinematics.bin, hits.bin
./build/render episodes/002-marble-sorter                 # live preview window
./build/render episodes/002-marble-sorter --frames frames --spp 128
./build/soundtrack episodes/002-marble-sorter             # hits.bin -> soundtrack.wav

ffmpeg -framerate 30 -i frames/%05d.png -i episodes/002-marble-sorter/out/soundtrack.wav \
       -c:v libx264 -pix_fmt yuv420p -crf 18 -c:a aac -shortest marble-sorter.mp4

The bake is the only step that touches physics: 240 Hz with 8 substeps, recording every 8th step. Every contact above 0.2 m/s is logged with the sound materials of the two shapes involved (copper shutters, brass pins, steel fins, glass, wood).

The soundtrack turns those hits into audio with modal synthesis. Each hit becomes a few decaying sine modes plus a click of noise, voiced by the two materials, louder for harder impacts and panned by position, then glued together with a small room reverb and a gentle compressor. Every click in the video is a real collision at the moment it happened.

What hardware you need to run and render this

I built and rendered this on an RTX 4090 with an Intel Core i9-14900KF. The final export is about 4,600 frames at 1080×1920 and 128 samples per pixel, and it takes about an hour: roughly 0.8 seconds a frame, including eight TLAS rebuilds per frame for the motion blur. Here's what matters if you want to do something similar.

GPU: you need hardware ray tracing. The renderer needs Vulkan 1.3 with VK_KHR_acceleration_structure and VK_KHR_ray_query. In practice that's an NVIDIA RTX card (20-series or newer), an AMD Radeon RX 6000-series or newer, or Intel Arc, on a current driver. A card without dedicated ray-tracing hardware is the wrong tool even if its driver exposes the extensions.

VRAM barely matters here. One BLAS holding a single box, a TLAS of a few thousand instances, and five full-resolution float images for accumulation and the denoiser add up to a few hundred megabytes. An 8 GB card has plenty of headroom. Pay for ray-tracing throughput, which is what sets your render time, not for memory.

Smaller cards just take longer. I've only rendered the final export on the 4090, so this is an estimate, not a benchmark. Render time scales roughly with ray-tracing throughput, so a current mid-range card (4070- or 4060 Ti-class) should land somewhere around two to four hours at the same settings. An RTX 3060 or RX 6000-series card is an overnight job. To go faster, drop to --spp 64 with --denoise, or render a short range with --from/--to to check a shot first.

CPU: cores for Box3D's workers, not a workstation. The bake is CPU-only and defaults to 8 worker threads, one per performance core on the 14900KF. With only about 900 marbles plus walls, I'd expect gains to taper past six to eight fast cores rather than keep scaling on a 32-core chip. Any modern 8-core desktop CPU is a sensible target. Per-core speed matters more than core count, because you feel bake time most when you're debugging a jam and re-baking over and over.

Memory and disk. 16 GB of RAM is plenty. Disk is what people forget: thousands of full-resolution PNG frames run to many gigabytes before ffmpeg shrinks them, so give the frames directory a fast SSD with room to spare.

What I'd buy. If you already own any RTX or RX 6000/7000-series card, use it and accept longer renders. If you're buying with this kind of work in mind, buy for ray-tracing performance per dollar. A current mid-range RTX card is the sensible spot, and a 4090 only buys shorter waits. Pair it with a current 8-core CPU, and don't spend extra on cores for a bake this size.

What I'd tell you if you're building one

• Set the honest constraint first. Fixing the colors before the run, and checking the result at the end, kept me from patching jams with quiet cheats. Every fix had to be something a real machine could do.
• Separate the bake from the render. A recorded trajectory in between lets you iterate on physics and visuals independently.
• Build the stall detector on day one. A jam at marble 400 is invisible until something notices nothing is moving.
• Prefer geometry fixes to engine tuning. The elbow, the open channel and the thin walls fixed more jams than any solver setting.
• Model the controller on its own. A few minutes of Monte Carlo with no physics answered a throughput question that hours of bakes couldn't.

Watch it run

The finished video is up as a Short. Every gate, every click and every marble is the real simulation at real speed, and the close-ups follow the middle sorter. Watch the feed tubes in the first ten seconds: that elbow took longer to get right than the whole path tracer.

Watch the marble-sorting machine on YouTube →

If you build something similar, or know a better way to unjam a granular hopper than "vibrate it and carry the marbles out," tell me in the comments.