Does Jev read Hebrew? 66 of my own posts, Hebrew vs English

v2 / 66 posts, blind labels, controls

The first version used 9 tweets and my own labels. @boardyai pointed out the obvious: my own labels, so of course it agrees. v2 fixes that. 66 Hebrew posts. English translations written blind. Labels from three models that never saw Jev's answers (Claude Opus, Claude Sonnet, Gemini 3.1 Pro). Questions rebuilt the way TypeSafe's docs recommend. And two controls: the same Hebrew asked twice, and Hebrew paired with the wrong English translation.

Dot: share of posts that agree. Line: 95% Wilson interval. κ is Cohen's kappa (agreement beyond chance; 0 = chance).

The original nine, relabelled by the blind panel

The panel disagreed with my own v1 labels on 2 of 9. Boardy was right to ask.

What changed in the questions

• No "none" option inside the Choice. A separate yes/no question first: does the post express any attitude at all? The stance Choice is read only when it does. In v1, "no clear stance" was the easy class that inflated agreement.
• Structured options. Each stance says what it covers and what belongs to its neighbour (mocking vs lamenting, endorsing vs mocking praise).
• Three-way sarcasm. At or above 0.7 is yes, at or below 0.3 is no, and the middle is "unsure", not forced to a side.

The panel is three AI models, not native-speaker humans; Fleiss' κ between them is on stance and on sarcasm. "Hard" posts were flagged by one judge (Gemini). Raw answers, translations and panel labels: data-v2-raw.json, scores: data-v2.json. Total cost of 264 Jev calls: about one cent.

Below: v1, the first 9 tweets with my own labels, kept as published.

01 / The question

The test, end to end

flowchart TD
  X["9 latest Hebrew posts
(@yonyoniz, public)"] --> T["My English translation
of each comment"]
  X --> S1["state: comment HE
+ quoted post"]
  T --> S2["state: comment EN
+ quoted post"]
  S1 --> J{{"Jev
jev-1.13.0"}}
  S2 --> J
  J --> C["Choice: stance
6 options + probabilities"]
  J --> N["Noul: sarcastic
probability of yes"]
  C --> P["Parity: HE pick = EN pick?"]
  N --> P
  C --> L["vs my own label
(one rater)"]
  N --> L
  classDef he fill:#fdeee6,stroke:#c2410c,color:#16181d
  classDef en fill:#e8eefc,stroke:#2f5bd3,color:#16181d
  classDef jev fill:#16181d,stroke:#16181d,color:#fff
  class S1 he
  class S2,T en
  class J jev

18 calls: 9 tweets, each asked once in Hebrew and once in English. Both questions ride in one request, so they cannot see each other.

02 / Verdict

Where each Hebrew answer went in English

Left: Jev's stance pick on the Hebrew comment. Right: its pick on my English translation. One band per tweet; only one band crosses.

03 / Per tweet

04 / What it told me

Sarcasm, Hebrew vs English, per tweet

Orange dot: Hebrew. Blue dot: English. A long bar means the language changed the sarcasm read. The dashed line is 0.5.

n = 9, one rater (me), my own labels, my own translations. This is a probe, not a benchmark. Only public posts were sent. Questions and raw answers: data.json.