Model from pictures: where we are, and the plan for Codex's reset

A one-page summary for choosing the direction before Codex resets (Wednesday, 4 am Ontario time). The decisions are in chat; the detail behind each result is on the earlier pages, linked below.

The goal

AI picturesclay drawings of the creature, straight pose, several views

A 3D shapecarved from the outlines; most of the loss happens here

Partswhere to split the shape

One tool per partlump, tube, branched tube, sheet, hollow

Blender starting objectplain, roughly sized and placed, for the Blender agent to refine

Where the AI pictures get us today

The last lab runs finished overnight: each of the seven AI animals carved from six straight views plus four special angles, and the five tools fitted to the carved shape. Each sheet shows, top to bottom, the drawing, the carved shape and the fitted parts. There is no true model behind a drawing, so these are judged by eye.

Flamingo: drawing, carved shape and fitted parts — recognisableFlamingo. A body with a few long branches (neck, legs): the parts come out in the right place and size.

Agama: drawing, carved shape and fitted parts — not yetAgama. Where its views disagree the carving leaves a slab (the tail), and the feet come out as crumpled pieces.

Scorpion: drawing, carved shape and fitted parts — not yetScorpion. The eight legs and the claws blur into blobs; the tail is one straight bar.

The other four AI animals

Octopus: drawing, carved shape and fitted parts — Octopus

Jellyfish: drawing, carved shape and fitted parts — Jellyfish

Seahorse: drawing, carved shape and fitted parts — Seahorse

Racer snake: drawing, carved shape and fitted parts — Racer snake

Where the parts get lost

Parts that pass, of 56 (six true models, parts painted perfectly)

Green: the five tools given each part's true 3D shape. Blue: from pictures of the same models. On a wider set (37 animals), the five tools given the true 3D parts pass 72% of parts, against 24% for one lump alone.

What each test taught us

VIEWS

More views help, but slowly

Six straight views carve the same shape as four (opposite views are mirror outlines). Four special angles, picked per part, take 7 parts to 14. Views page

DEPTH

Exact depth helps a lot; estimated depth doesn't

Carving with exact depth brings the shape closer to the true one in 23 of 24 cases. The open depth estimator we tried is typically 4–13 cells off, so carving with it cuts away true shape. Depth page

CUTS

Cut-in-half pictures open the mouth, but the tools close it again

The anglerfish cut in half: true model, carved shapes and fitted parts

With five cuts only 11% of the anglerfish's open mouth stays filled in the carved shape, but the fitted parts fill 46–64% of it, even when fitted to the true model (63%). Features page

FIT TO PICTURES

Fitting straight to the pictures drifts from the true part

Refitting each part to match the pictures matches them better, but nothing in a picture says what lies behind another part, so fits spread into their neighbours: 14 parts pass before, 12 after. Features page

TRUTH

Our "true parts" are ragged (your observation)

The agama's three torso rows: one real torso and two slivers

The AI painted each part on pictures of the true model and the paint was carried onto the 3D shape. Paint edges come out ragged and stray paint becomes "parts" of its own: two of the agama's three torso rows are slivers no fit can match. This affects both the scores and the fitting, which uses the same paint to split the carved shape.

MOUTH

Teeth may hide the open mouth

Your point: the teeth are in the way. Where they fill the gap in the side outline, carving cannot open the mouth; with the teeth gone, the side view alone could show the gape. Not tested yet (track T2).

The new direction: simplify first, then a part tree

A creature as a tree: thick body, thinner branches, thinner sub-branches

Simplify first. Small details are removed before any fitting: in the pictures, by asking the image model for a plain clay version, and in our truth, so we score against clean parts.

A part tree. The thickest core is the body; thinner pieces branching off it are the second level; pieces branching off those are the third. It fits nearly all our animals: the octopus and the jellyfish are a body with many arms, the snake is a trunk alone. The tree is found from the shape's thickness, not from AI paint.

Hands and feet as their own step: a palm with finger branches.

How artists do it: creature artists often start from exactly this, a tree of points each with a thickness (ZBrush's ZSpheres, Blender's Skin modifier), and Blender turns it into a base mesh. That could become what the fitter hands to the Blender agent; to be checked by reading before we build on it.

The plan for Codex's reset

What Codex costs. One picture is about 0.02% of the week (47 pictures cost 1% on Oct 31). A build task is roughly 0.1–0.3%, a code review under 0.1%. Wave 1 below is about 10% of the week. The real limit on how fast we can use Codex is how fast the lab runs and results get checked, so wave 2 leans on pictures, the big lever. The Voyage scene work draws on the same Codex pool.

Running in parallel: three Codex workers at a time, the lab running alongside.

NOWBefore the reset (no Codex)

Read up on part trees: how artists block out creatures (ZSpheres, Skin modifier), finding a tree in a 3D shape, and the recent animal-from-photos methods that use a spine with leg chains.
A quick first tree on the agama and the anglerfish, to sharpen the briefs.
Write the five briefs and stage the lab data, so three workers start at the reset.

T1A clean truth: the part tree of each true model

Codex: Two workers build two ways of finding the tree, in parallel: by thickness, and by a centre-line skeleton. Both are tested on made-up shapes with known trees: a hand, a star, a four-legged animal of tubes, a fish with open jaws.
Lab: Run on all 37 true models.
You see: The tree's parts beside today's ragged ones.
Cost: about 1%

T2Simplified pictures

Codex: The simplify prompt on pictures of four true models (anglerfish, agama, flamingo, scorpion) and on all seven AI animals.
Check: The outline stays put except where a detail was removed; the anglerfish's mouth opens when carved from the simplified pictures.
Cost: about 130 pictures, 3%

T3Fitting along the tree

Codex: The body first, then each branch from where it joins, then fingers and toes; feet as a palm with fingers. Optionally the whole tree as a Skin-modifier skeleton.
Lab: The seven true models and the seven AI animals, scored against T1's clean truth.
Cost: about 2%

T4A turnable 3D viewer

Codex: True parts, the tree and the fitted parts, turnable, side by side with the pictures, so you can judge parts that are unrecognisable in still pictures.
Cost: about 0.5%

T5More animals in AI pictures

Codex: Ten new kinds, one per group in the 37-animal set (so each has a true model to compare with): straight pose, simplified, plus a cut-in-half picture down the middle.
Cost: about 150 pictures, 3%

WAVE 2If wave 1 pays off

All 37 kinds as simplified AI pictures with cuts: about 450 pictures, 10%.
Competing versions of the tree fitter built side by side, the best kept.
Cut-in-half pictures per part (the jaws, the jellyfish bell), where the tree leaves a hollow.

The decisions are in chat.