Constrained JSON does not constrain the tool choice

TL;DR: Needle 3 is a small model built specifically for tool calls, structured extraction and embeddings. That narrowness is useful. But its grammar guarantees that the JSON parses, not that the model chose the right tool. The creator's own test makes the distinction visible.

This is worth watching if your application has a small, enumerable tool surface and cloud latency is wasteful. It is also worth watching if your team still treats valid structured output as evidence that an action is safe to execute.

Four moments that change the design review

• 0:57: a smart-home demo turns on a fan, changes a thermostat and switches on a light. The creator reports the two calls in 66 milliseconds.
• 3:28: the explanation says schema validation means the model “cannot hallucinate”. That is too broad.
• 7:54: without resetting conversation state, “order me a pizza” produces an inaccurate thermostat call. The JSON can be valid while the decision is wrong.
• 10:01: typed invoice extraction works on clean text, while a more conversational input with an implied date breaks the extraction.

The video is strongest where it stops selling speed and shows the failure modes.

A parser guarantee is not a decision guarantee

Cactus Compute's current release page says a byte-level grammar is compiled from the schema, so the output parses and enum values stay inside their allowed set. The repository also describes a calibrated confidence head, a default confidence floor of 0.1 and a routing pattern that executes high-confidence calls, confirms middling ones and refuses empty results.

Those are separate controls because they answer separate questions. The grammar asks whether the call is well formed. Tool selection asks whether it is the right call. An authorisation layer asks whether the system should execute it now.

My rule would be to keep all three gates visible in an acceptance test. A syntactically perfect call that changes the wrong thermostat is not a partial success. It is the more dangerous failure because ordinary validation happily lets it through.

For a hypothetical building-control workflow, I would clamp temperature ranges in code, reset state between independent users, require confirmation for consequential changes and keep off-topic requests in the test set. That is labelled design judgment; I did not run Needle 3 or reproduce the notebook.

The numbers are version-sensitive

The 18 September video describes a 35 MB, 53-million-parameter model. The primary page I checked on 22 September describes one 8–29 MB CQ2 binary with 29–121 million parameters across its laddered subnetworks. It reports 400–4,000 decode tokens per second on a Raspberry Pi 5, while the video reports its own Colab and smart-home timings.

I would not collapse those figures into one benchmark. The project is moving quickly, and the published measurements come from Cactus or the video's author. Use them to decide whether a local trial is worthwhile, then run your tools, prompts, state boundaries and device in the acceptance test.

I inspected the complete automatic transcript and representative playback frames around the four moments above, plus the current release page and Apache-2.0 repository. I did not execute the model, notebook or fine-tuning service.

What's in it for you

• Use a narrow local model where general language generation buys nothing.
• Test wrong-tool selection separately from JSON validity.
• Put confirmation and hard policy checks after the model, not inside its confidence score.

Structured output can remove the parser from your failure path; it cannot remove judgment from your control path.