jev-fit

Paste an idea. Jev decides what should run it.

Method

How to write questions for Jev

Ruben Roques ·

Jev answers the question you wrote. It does not answer the question you meant. It does not read the other questions in the same call, and it does not ask you anything back. Every piece of meaning has to be in the state and in the question.

I learned this while I built jev-fit, a tool that asks Jev if a software idea fits Jev. jev-fit is closed source and is not affiliated with TypeSafe AI. I keep its questions private, because they are the product. The method is not private. Almost every example here comes from two public sources, TypeSafe's docs and the jev-spam-eval study by bitnovus, so you can check each claim.

1. Ask atomic questions

TypeSafe's build guide says: "Break down complex or ill-defined questions into separate questions that each evaluate one property". It calls this the most important idea in the guide. The reason it gives is that broad questions hide several judgments behind one answer.

"Is this ticket important?" is a broad question. It mixes urgency, customer value and severity. When the answer is wrong, you cannot tell which part failed. Three questions are better: does the message convey urgency, is the sender a paying customer, and which severity level fits.

Splitting costs almost nothing, because all questions in one call run in parallel.

One honest limit. The spam study split its task into nine narrower questions. The authors report that this made the errors easier to inspect, and that it did not bring a comparable gain in accuracy. Split for debuggability, not for accuracy.

A test I use: can I write the false case in one sentence? The first question in jev-fit asks if the pasted text describes a task that software would perform. The false case is easy to state: a shopping list is not a task. If I cannot state the false case, the question is not atomic yet.

2. Write criteria as what, includes, not_for, examples

A Choice question takes a map of options. Each value can be a plain description or a structured object. The docs note that no field name is reserved, so what, includes, not_for and examples are a convention. The convention works because it makes you write the boundary down.

The docs give this example for two options that people confuse:

"return_policy": {
  "what": "Whether and how an item can be returned",
  "not_for": "Progress of a return already sent"
}

not_for does most of the work. It tells the model where this option ends and the next one starts. The docs put it this way: "write descriptions that separate the options from each other". They also tell you to add an other option when the list may not cover every input.

The spam study measured how much definitions matter. In its first three-way experiment with legitimate mail, spam and phishing, the category names alone scored 73.1%. The same options with written definitions scored 93.5%. The authors explain that much of the difference was semantic. With names only, Jev called many scams phishing, and the dataset counted them as spam. The model was not wrong about the emails. It did not know the dataset's definition of "phishing". Your team has definitions like that too. Write them into the options.

A Noul accepts criteria with a true and a false description. Use both when the boundary matters.

3. Write score levels as situations, never degrees

The Score page has the best single line in the docs: "Describe situations, not degrees. 'Broken or degraded feature, but workaround exists' gives the model something to match the state against. 'Moderately severe' doesn't".

The docs also explain why relative wording fails. Jev evaluates each level on its own, without knowledge of the levels next to it. "Worse than the previous level" therefore says nothing.

In jev-fit, one score asks what happens when the component gives a wrong answer. The levels are not "low, medium, high". They are situations. Nobody notices. A person corrects it later at small cost. A customer loses money or access. Each level is something you can recognise in the text of an idea.

The docs allow up to 10 levels and say that three is fine. The returned score is a probability-weighted mean, so it can fall between two levels. Use it to rank or to cross a threshold. Do not read 3.4 as a measurement.

4. Evidence beats wording

When a question performs badly, my first reaction was to rewrite it. The spam study says to look at the state first.

The first version of the study sent the subject, the sender and the body text. It removed HTML, so a link kept its visible words and lost its real destination. Then the authors added the Reply-To header, the link destinations with parsed hostnames, and the attachment names and types. They did not change the question or the definitions. Accuracy on the main set went from 93.62% to 97.98%. A fake notification often copies the real message text exactly, so only the link destination shows the fraud.

Then they improved the wording. Accuracy on the main set rose again, to 98.64%. On the most recent phishing messages the new wording caught fewer, 94.49% against 95.31% for the original question. The README states it directly: more instruction was not uniformly better.

So, before you edit a question, ask what a careful human would need to see to answer it. Put that in the state, and only that. The build guide says to include only the context relevant to the current questions. Let code do the parsing. Let Jev do the judging.

The study is exploratory, and its authors say so. Treat the numbers as directions, not as constants.

5. Compose and veto in code

Questions in one call are independent. The build guide says: "One primitive's result does not become hidden context that changes another primitive's result". Two of your questions can contradict each other, and Jev will not notice.

This happened to me. jev-fit asks if a task needs the system to look at an image, audio or video. Jev cannot take images, and it answered that question correctly for a screenshot task. The verdict question still picked Jev, because the verdict question never sees that answer. I tried to fix it with wording. The reliable fix was one line of code: if the task needs images, Jev is not a valid verdict.

The docs call the general pattern composite scoring: "Combine independent answers with deterministic rules or weighted sums". Their example multiplies three Noul answers by fixed weights. The weights and the thresholds are policy. They belong in your repository, where you can review and test them.

Keep deterministic work in code as well: counting, dates, and a deny-list that runs before Jev when the input can be hostile.

6. Low confidence is an answer

A Choice returns a confidence value. The docs describe it by shape: a flat distribution means low confidence, and a single peak means high confidence. A Noul returns no separate confidence. A value near 0.5 is its way to say "I cannot tell".

The build guide tells you to act on this: "Escalate uncertain cases to a person or a more expensive reasoning model". Its example sends answers with confidence below 0.8 to human review. Choose your own threshold, and fit it on your own labelled data.

Do not force a verdict when the distribution is flat. In jev-fit, an idea that is too vague gets a "not sure, add detail" state and no verdict, and that is the correct output.

The short version

One property per question. Boundaries written down, with not_for. Situations for levels. Better state before better wording. Policy and vetoes in code. A plan for the unclear cases. Keep a labelled set, and run it after every change.

You can see the result at jev-fit.com. Paste an idea and try to break it.

Sources

jev-fit is an independent project and is not affiliated with TypeSafe AI.