Claude connector
How the gate check works
A gate says what must be true for a step's work to pass. Its checker reads the step's inputs and answers GATE: PASS or GATE: FAIL. If the gate asks for a judgement nobody could decide from those inputs, such as "the user will be happy", the checker guesses, and a bad result can pass.
When Claude checks a graph through the connector, a small classifier on our server reads every gate against what its step receives. If a checker could not decide it, the answer carries a warning saying so, with how to fix it. It is a warning, never an error: it does not stop a graph from compiling or running.
What Claude sees
These are the model's real scores on the Ship a change starter, with its own gate swapped for each line. The score is the model's confidence that a checker could not decide the gate; the connector warns at 0.89 and above.
| Gate | Read as | Score | Warning |
|---|---|---|---|
| No blocking finding is left open, and every finding cites a line of the diff. | checkable | 0.02 | no |
| All tests pass. | checkable | 0.01 | no |
| Every finding names a file and line that exist. | checkable | 0.01 | no |
| The user will be happy with the result and find it useful. | not checkable | 0.92 | yes |
| The work is thorough, accurate and professional. | not checkable | 0.92 | yes |
| The fixes are elegant. | not checkable | 0.91 | yes |
The warning, as it appears in what validate_graph returns:
- [semantic-gate-uncheckable] step gate: The gate of "gate" ("The fixes are elegant.") is a judgement its checker cannot decide from the step's inputs (confidence 0.91). Say what must be true for it to pass, e.g. "every total matches the sum of its line items".Note "All tests pass.": the plain rule this replaces flags any gate under four words as too short, so it warned on that gate and passed every longer vague one. The model reads what a gate asks for instead of counting its words.
What kind of model it is
It is not a chatbot or a large language model, and it writes no text. It is a cross-encoder classifier: it reads two texts together, the gate and the list of what the step receives, and returns three probabilities: checkable, partly checkable, not checkable. This is exactly what it reads for the gate above:
[H4] [GATE] The fixes are elegant. [SEP] [INPUT] diff · Command · The output of git diff, unchanged. [INPUT] review_correctness · Worker · A numbered list; each finding cites a file and line of the diff and says blocking or minor. [INPUT] review_security · Worker · A numbered list; each finding cites a file and line of the diff and says blocking or minor.
| Network | MiniLM-L12-H384 (Microsoft): a Transformer encoder of 12 layers and 33 million parameters, distilled from a BERT-Base-sized model and about 2.7 times faster than BERT-Base. The starting checkpoint is cross-encoder/ms-marco-MiniLM-L12-v2, the same network already trained to judge whether one passage answers another. |
| Our training | Fine-tuned by us on gates labelled checkable, partly or not, set on generated graphs and on our starters, plus 16 facts about each step taken from the graph itself (its type, how many inputs it has, whether they are JSON). An open-weights model (Qwen2.5) in our training notebook wrote the gates; no data from users. |
| What ships | One head of the model, compressed to 8-bit integers: a 34 MB file (from 133 MB) that scores the same on our hand-written test set. It runs on ONNX Runtime inside the connector's server function. |
| Speed | About 70 milliseconds per gate. The first call after the server starts loads the model, about a second, once. |
How well it does
- On a test set written by hand, which the model never saw in training: it flagged 20 of 21 gates that cannot be checked, and none of 42 gates that can be, fully or in part.
- On the 14 starter graphs: it warns on none of their own gates. With each gate swapped for a vague one, it caught 68 of 84; with each swapped for a short checkable one ("Every claim has a link.", "The total is under $500."), it warned on none of 140.
- What it misses most: vague gates worded like a rule. "Everything makes sense and nothing important is missing from the answer" reads to it as partly checkable, so it does not warn.
Limits
- It reads the gate and the one-line description of each input, not the work itself. It cannot tell whether an answer meets the gate; the checker step does that.
- It was trained on English gates. A gate in another language may be read wrongly.
- It only checks gates. Other warnings, such as a missing input or a vague prompt, come from the plain rules, as before.
- It runs in the connector, not in the studio. The studio checks in your browser and still uses the four-word rule. If the model ever fails to load or takes longer than a second and a half, the connector answers exactly as it would without it.
Privacy
The model runs inside the connector on our server, on the graph Claude sent with the tool call, in memory. No other service sees it, and nothing is stored or logged: as for every connector call, we log only the name of the tool. Read the privacy policy.