Testing Jev on Slovene

Over the past few days, I've been playing around with the model that caught everyone's attention recently. Specifically, I wanted to see how good it is with Slovene, so the experiment was built around a few tasks I had worked on before in one form or another, each with its own limitations. The next logical step was to see whether this new thing, apparently cheap and very fast compared with LLMs, could handle those same Slovene challenges. That meant putting Jev, TypeSafe AI's new model announced on 15 September 2026, through a few tests. The short answer is: it depends a lot on the task, and in one case, on something that shouldn't matter at all.

What Jev is (and isn't)

Jev isn't a chatbot. TypeSafe calls it a System One model, built "to make fast, structured decisions that software can use directly". It doesn't write text; you give it some input and a question, and it gives you back a number. Here, every question was a yes/no question (TypeSafe calls these Noul questions), and the answer was a score between 0 and 1: close to 1 means yes, close to 0 means no.

TypeSafe's launch post says Jev is "two orders of magnitude faster and more efficient" than existing LLMs on what it calls System One tasks, and that its answers are calibrated. It also says Jev "can't hallucinate", meaning that it gives up free-text generation and only returns structured answers; those answers can still be wrong. At the time of writing it's also cheap: you only pay for input, $0.042 per million tokens. Speed and cost weren't measured here; this post is only about the quality of the answers.

TypeSafe's documentation says English is where Jev's accuracy is "currently best" and that other languages are "handled but not equally well", which makes Slovene a useful test case. All experiments here used jev-1.13.0.

Three decisions instead of a benchmark

Slovene already has good, openly available datasets with manually annotated examples, so the experiment used three existing kinds of linguistic decisions rather than made-up test questions:

Roughly speaking, these cover lexical meaning, semantic roles and syntax.

Each task had two possible answers and a balanced test sample, so 50% is the baseline. The datasets and setups differ, though, so the percentages below aren't directly comparable. The figures also give ROC AUC, which shows how well Jev's scores rank the right answers above the wrong ones: 0.5 means a random ranking, 1 a perfect one.

Figure 1: Final test accuracy for each task, with 95% confidence intervals. The tasks use different datasets and setups, so this is an overview, not a ranking. Balanced samples, so 50% is chance. The SloIE result comes from an exploratory pilot, and some of the parser test cases had been seen in earlier work.

1. Idiomatic or literal?

An idiom is an expression whose meaning isn't fully predictable from the words themselves. Pasti v vodo, for example, can literally mean falling into water, but it can also mean that a plan failed. The task here is simply to look at one occurrence in a sentence and decide which reading it has: idiomatic or literal.

For annotation, the important thing is the particular occurrence. It isn't enough to recognise that the expression can be idiomatic in general.

The data comes from SloIE (Škvorc, Gantar and Robnik-Šikonja, 2020), where the occurrences are manually annotated. The test set had 380 examples from 19 idiom families (an expression and its variants) not used during development, half idiomatic and half literal.

The tricky part was how to point Jev at "this occurrence of this expression". Three versions were tested on the same examples. Version A named the expression but didn't mark where it was in the sentence. B marked the words in the sentence but didn't say which expression they were supposed to be. C did both. Only C really asks the question the annotators answered, so C counts as the result. Its accuracy was lower than both A and B.

Figure 2: SloIE, the same 380 test items asked three ways. A and B are diagnostics; C matches the annotated decision and is the final formulation, although it scored lowest (95% CI: accuracy 63.9–72.4%, AUC 0.711–0.809). Exploratory pilot, not a confirmatory result.

A and B got around 81% and 80%, C got 68.2%. Same examples, same model, a slightly different way of asking, and almost 13 percentage points of difference. So a higher score doesn't automatically mean Jev did the task better: A scored best while answering a looser question than the one in the annotation.

Most of C's mistakes went one way: half of the examples were literal, but Jev said "idiomatic" for 259 of the 380 and got only half of the literal ones right.

Two illustrative development examples, selected for readability rather than as a representative sample. Marked words are in bold, the score is Jev's number for "idiomatic", and the translations are mine.

Right

V ljubezenski zvezi Steffania in Anje očitno hlače nosi ona. 'In their relationship, she's obviously the one who wears the trousers.'

Gold: idiomatic · Jev: idiomatic · score 0.73

Wrong

Nekatere žuželke po ves dan letajo s cveta na cvet ter nabirajo pelod in medičino. 'Some insects fly from flower to flower all day, collecting pollen and nectar.'

Gold: literal · Jev: idiomatic · score 0.69

Figuratively, letati s cveta na cvet means hopping from one partner to the next. Here the insects really are just flying between flowers.

One caveat: this was the first experiment, and I ran it rather loosely. I had already seen the test results of A and B when I decided that C was the right version. I didn't pick C for its score (it was the lowest), but the SloIE numbers still come from an exploratory pilot, not a clean test.

The other two tasks were therefore run more strictly: the method was chosen on a separate development part of the data, everything was frozen, and only then was the test set run, once. The scripts wouldn't even allow a test run before that. The only catch: I had already looked at some of the parser test cases in earlier parser-evaluation work, so that test set isn't completely fresh.

2. Actor or patient?

The second task is about who is doing something and who is affected by it. Linguistic annotation uses labels for these roles; in the Slovene ssj500k scheme they are ACT and PAT. ACT is broader than "the one doing something": it covers the actor, the causer, the bearer of a situation and the logical subject. PAT is roughly the participant affected or described.

Jev does not have to find the verb or argument itself. Both are given, and its only job is to decide which of the two roles the argument has.

Two illustrative development examples, also selected for readability. Predicate and argument are in bold; the score is Jev's number for ACT, so below 0.5 means PAT.

Right

Trenirali smo zelo dobro in sposobni smo premagati Ukrajince. 'We trained very well and we're able to beat the Ukrainians.'

Predicate premagati ('beat') · argument Ukrajince ('the Ukrainians') · Gold: PAT · Jev: PAT · score 0.05

The Ukrainians are the ones who would be beaten, so they're the affected participant.

Wrong

Hladna rosa osveži in poživi krvni obtok, zato se noge segrejejo. 'The cold dew refreshes and stimulates the circulation, so the feet warm up.'

Predicate segrejejo ('warm up') · argument noge ('the feet') · Gold: ACT · Jev: PAT · score 0.19

In the ssj500k scheme the subject of a verb like 'warm up' counts as ACT, even though the feet can intuitively also look like the affected participant. Jev chose PAT.

On the 400 test items, Jev got 329 right (82.25%). Its errors were lopsided, though: it recognised 93.5% of the patients but only 71.0% of the actors.

It's worth being careful about what that 82% means. Who acts and who is affected is closely tied to sentence structure. A very simple rule, "if the argument comes before the verb, it's the actor", already gets 65.5%, and Jev itself was more accurate when the argument came before the verb (86.8%) than when it didn't (78.4%). A rule based on the grammatical annotation, which Jev never saw, gets 93.25%. So Jev can make this particular decision well, but that doesn't show it understands Slovene meaning independently of grammar.

3. Which parser is right?

The third task is syntactic. A dependency parser describes sentence structure by linking words to the words they depend on. The word that another word attaches to is called its head.

Two parsers can disagree about one of those links. In the cases used here, human annotation tells us that exactly one parser is right. Jev's job is simply to choose between them.

The two parser outputs came from CLASSLA-Stanza (classla 2.2.1, standard Slovene model 2.2) and SPOT-Trankit 1.2 (Trankit 1.1.2), with UD Slovenian-SSJ r2.18 as the human-annotated reference. Jev saw the sentence, the word in question and the two candidate heads, labelled option A and option B. It never saw which parser suggested which. Half the time the right answer was A, half the time B.

During development, something looked off: Jev picked A far less often than it should have. When the same questions were asked again with only A and B swapped, 127 of 400 answers changed; in 124 of those Jev picked whatever was in position B both times. So the final method asked every question in both orders and averaged the two scores. Then came the test set.

Figure 3: Each of the 380 test items was asked twice, with the two candidate heads in opposite order. 139 answers changed, and in 138 of them Jev picked whatever was in position B both times.

The effect was still there. On the test set, 139 of 380 answers changed when only the order changed, and 138 of those 139 went to whichever candidate was in position B.

Asking in both orders fixed the overall imbalance: in the end Jev chose A in about half of the cases, as it should. But it didn't make Jev any less sensitive to order. The final accuracy was 62.4% (237 of 380). That's better than chance, but more than a third of the answers were still wrong.

Two illustrative development examples, selected the same way: which word does the bold word attach to? The score is the averaged score for A, so above 0.5 means A.

Right

Člani skupine Monty Python so ga zelo občudovali. 'Members of the group Monty Python admired [him] greatly.'

A: skupine ('group') · B: Člani ('members') · Gold: A · Jev: A · score 0.84

Monty Python is the name of the group, so it attaches to 'group', not to 'members'.

Wrong

Tonove veke so se zagrinjale, mož je šepetal: 'Tone's eyelids were closing, the man was whispering:'

A: zagrinjale ('were closing') · B: Tonove ('Tone's') · Gold: A · Jev: B · score 0.205

The eyelids are what was closing, so veke depends on the verb. Jev chose Tonove, the word that only says whose eyelids they are. It made the same choice in both orders, so this error isn't the order effect.

One more odd thing: Jev preferred Trankit's answer (57.6% of the time, where 50% would mean no preference) and was more accurate when Trankit was the one that was right (70.0% vs 54.7%). The data don't explain why.

Can confidence help?

Jev gives a score, not just a yes or no, so the practical question is: if you only trust it when it's confident, does it get better? Here, confidence means how far Jev's score is from 0.5 in either direction, so 0.9 and 0.1 count as equally confident.

Figure 4: Accuracy on the test items where Jev's confidence was at least 0.6, 0.7, 0.8 and so on. For ACT/PAT accuracy rises quickly while most items are kept; for parser adjudication it rises slowly while coverage drops. The parser point at ≥0.90 has only 5 items. Observed rates, not guaranteed error rates.

For actor or patient, it clearly worked. Keeping only the answers where Jev's confidence was at least 0.70 still keeps about three quarters of the items, and 95% of those are right.

For the parser task, it helped much less. Accuracy went up slowly, and the number of items dropped fast. This analysis wasn't run for the idiom task.

And for Slovene?

Slovene isn't starting from zero when it comes to evaluating language models: there's SloBENCH, Slovenian-LLM-Eval and newer benchmarks for things like pragmatics and grammatical knowledge. What I couldn't find was a previous public evaluation of Jev or another System One model specifically on Slovene.

Jev had been public for just over a week when I ran these tests, so being early isn't the point. What matters more is that Slovene has enough good, manually annotated data to test new models on real linguistic decisions instead of assuming that results from English carry over. These three small tests show why: the same model looked good on one task, much weaker on another, and failed in ways that a general benchmark might not reveal.

What this doesn't show

This was one version of Jev, three narrow yes/no questions and a few hundred balanced examples per task. All three datasets are public, so some of this material may have been in Jev's training data. And none of it shows that Jev "understands Slovene": its best result came on the task where a simple grammatical rule already does very well, and its most striking habit was preferring the second option.

How this was built

Claude Code (mostly Claude Opus 5.5, with Opus 5 in the initial SloIE work) and OpenAI Codex (GPT-5.6 Sol) were used for implementation, checks, reproducibility and documentation. The tasks and methods as well as experimental decisions were chosen by me. Jev is the only model being evaluated here.

Conclusion?

It is useful, but with constraints. On actor or patient, Jev did surprisingly well. On the parser task it struggled much more, and the order of the answers affected the result far more than it should have.

I wouldn't use it to replace an annotator. It could help sort out the easy cases and leave the uncertain ones to a human, at least for decisions like actor or patient. Before trusting it on a new task, though, it would be worth testing the wording, swapping the answer order and checking whether confidence actually means anything.

The code, prompts, protocols, Jev outputs and full results are in the Jev-SL repository. Datasets: SloIE (Škvorc, Gantar and Robnik-Šikonja, 2020; CLARIN.SI 11356/1335; CC BY-NC-SA 4.0), ssj500k semantic-role data (via clarinsi/bilateral-srl; treated as CC BY-NC-SA 4.0, the licence of ssj500k) and UD Slovenian-SSJ r2.18 (CC BY-SA 4.0). The example sentences are quoted from these datasets, and they and my translations of them remain under their source licences. Full licences and citations: DATA_LICENSES.md.