multimodal on tomrochette.com

H2O-Lightning-4B

tom@tomrochette.com (Tom Rochette) — Sun, 11 Oct 2026 00:00:00 +0000

H2O-Lightning-4B is H2O.ai’s Apache-2.0 decision model, a Qwen3.5-4B fine-tune that answers choice, yes/no, and score questions about a record and its attached images, one forward pass and one output token per decision, served on unmodified vLLM 0.30.0 behind a small standard-library shim.

H2O-Lightning-4B v1.1 is the first open-weights system to outscore hosted Jev on an independent board, 72.5 against Jev’s 71.5 unranked reference on JevBench’s open-weights composite, and it did it as a 4B, ahead of every open 12B, 26B, and 31B on that board.

What it is
#

A multimodal decision fine-tune of Qwen3.5-4B (pipeline tag image-text-to-text) that returns a probability for every option: choice over named options, the probability a statement is true, and an ordinal score level with its distribution. The serving path is deliberately boring: stock vLLM 0.30.0 plus h2o_lightning_shim.py, a standard-library shim that exposes the decision surface, so nothing custom has to be installed or trusted beyond the weights. The model card documents the company’s own headline (v1.1 number one on the JevBench open composite as of 2026-10-07), and the measured board row (2026-10-04, composite 72.52, calibration axis 90.0, about $0.021 estimated per 1,000 decisions) comes from the benchmark’s own runs, not the vendor’s.

Status
#

Active and ten days old: the weights were created on Hugging Face 2026-10-02 and the card was last modified 2026-10-12, with v1.1 the current revision, 1,254 downloads and 54 likes as of 2026-10-12 (Hugging Face API). It entered the board the ordinary way, a 2026-10-02 add-request on the benchmark’s repository requesting measurement on one H100 with pinned weights, and the maintainer ran it: the row that makes it the open leader is the benchmark’s own measurement on its own protocol (add-request 181, JevBench aggregate, both fetched 2026-10-12). The company’s card carries the claim and the screenshot, and the 2026-10-12 card revision added a comparison section against Microsoft’s Microsoft-Decision-1 announcement (Microsoft’s accuracy numbers kept, latency corrected onto one H100 basis, both JevBench scores shown), which also explains the board’s new Capability Score ordering as the Intelligence-plus-Calibration average that excludes speed and cost. The Decision Index still does not carry the model at all, so its second-board reading does not exist yet.

Strengths
#

The open lead is measured, not claimed: the composite that puts it above Jev’s reference was produced by the benchmark’s own runner on pinned weights, which is more than most of the wave’s launch posts can say.
Calibration is a board-visible strength (axis 90.0), the axis the contract actually sells, and it is the highest among the open top five.
The serving story is the least custom in the category: unmodified vLLM plus a shim any engineer can read in an afternoon, with the cost being input tokens only.
A vendor with an existing ML platform standing behind a 4B open checkpoint changes the wave’s economics argument, which until now has been run by individuals and labs.

Cautions
#

The board evidence is one profile on one runner’s protocol: the same benchmark’s protocol changes have moved established families by tens of points (Decider’s v2 fell 64.13 to 41.24), and no second board has measured H2O-Lightning-4B at all.
The measured configuration wants an H100-class GPU with a pinned vLLM build, so the “4B” size does not translate into laptop-class serving the way the smaller encoders’ does.
The vendor has not published training data, recipe, or a calibration methodology, so the board’s calibration axis is a measurement of outputs, not a documented discipline.
Downloads are in the hundreds against tens of thousands for the community leaders, so the deployment record is essentially the board run plus the card.

Pricing
#

Free and open: Apache-2.0 weights on Hugging Face, no hosted service and no paid tier. The cost is the GPU the measured path wants and input tokens only at serving time, one output token per decision.

Compared to
#

Jev: the closed reference it outscores on one board; Jev keeps the hosted SLA, the longer track record, and the second-board lead, and its own note carries the seven development waves this model belongs to.
JevBench: the scoreboard that produced the claim; read the two together, because the model’s headline is that board’s composite and carries its methodology caveats.
Decider: the cautionary family on the same board, whose first-place run ended in a protocol change; the reason to wait for a second-board reading before shipping on this one.

Bottom line
#

Recommended for engineers who want the strongest measured open checkpoint today and can serve vLLM on data-centre hardware, reading the lead as one protocol’s verdict. Not for laptop-class serving, and not for anyone who needs a second independent reading before trusting a board row. The disagreeable claim I will defend: a company checkpoint taking the open lead on a hobbyist benchmark is the moment the wave stopped being a community curiosity, and the interesting question is no longer whether open weights can match Jev but whether vendor-built open checkpoints will crowd the individuals out of their own category.

Changes
#

2026-10-11 - Created from the entrant scan (awesome-jev cross-check), accepted on the measured open-weights lead of the primary independent scoreboard with the single-protocol caveat recorded.
2026-10-12 - Recorded the card’s 2026-10-12 revision (a comparison section against Microsoft’s Microsoft-Decision-1 announcement with both JevBench scores, plus the Capability Score explanation) and refreshed the checkpoint to 1,254 downloads and 54 likes.

References
#

https://huggingface.co/h2oai/h2o-lightning-4b - the model card: Apache-2.0, Qwen3.5-4B fine-tune, the typed primitives, the vLLM-plus-shim serving path, and the vendor’s own board claim (fetched 2026-10-11)
https://huggingface.co/api/models/h2oai/h2o-lightning-4b - created 2026-10-02, last modified 2026-10-12, 1,254 downloads, 54 likes, Apache-2.0 (fetched 2026-10-12)
https://github.com/fstandhartinger/jevbench/issues/181 - the add-request that entered the model: pinned v1.1 weights, requested H100 hardware, the serving commands (fetched 2026-10-11)
https://benchmarkheaven.com/jev-models - the live open-weights board ordered by the Capability Score with H2O-Lightning-4B v1.1 the composite leader (official #1, 72.5; capability 75.0, eighth), board v1.7.46 (fetched 2026-10-12)
https://benchmarkheaven.com/api/jevbench/v1.6.1 - the board aggregate behind the row: composite 72.52, calibration 90.0, measured 2026-10-04, estimated $0.021 per 1,000 decisions (fetched 2026-10-11)

Winnow

tom@tomrochette.com (Tom Rochette) — Sun, 11 Oct 2026 00:00:00 +0000

Winnow is EldanRing’s Apache-2.0 family of Gemma 4 fine-tunes (E2B, E4B, and 12B) for local typed decisions, chat, and image input, served through a MIT inference server that scores options through /v1/systemone without generating an explanation, distributed as merged GGUF weights with checksums.

Winnow is the community family the independent benches agree on: Ollaya’s human-labeled comparison calls winnow:12b its most accurate model (0.773, ahead of Ollama’s best), local-jev-bench puts it in the top group on every set it runs, and on its own frozen panel Winnow-12B Q8 matches Jev answer for answer on JevBench’s public subset while Jev keeps the transfer lead.

What it is
#

A Gemma 4 LoRA fine-tune with the adapter merged into the weights, published in three sizes (Winnow-E2B, Winnow-E4B, and Winnow-12B, the 12B created 2026-09-20 and last modified 2026-10-06), each with Q8_0, BF16, and NVFP4 GGUF targets plus an F16 vision projector for image input. The winnow-inference server (MIT, created 2026-09-21) loads one model for both /v1/systemone decisions and ordinary /v1/chat/completions, supports noul, choice, and score with a shared state prefill and cached prefixes, and documents an RTX 5070 Ti 16 GB deployment profile for the 12B Q8 target. The self-run panel (measured 2026-09-21, one RTX PRO 5000) scores Winnow-12B Q8 at 85.71 percent on JevBench’s 231-item public subset against hosted Jev’s 85.71, and 81.55 against Jev’s 87.00 on Kev-v9 clean, with the training data private and not distributed.

Status
#

Active and community-scale: the 12B checkpoint shows 57,761 downloads and 66 likes, the E4B 43,204 downloads, and the inference server 26 stars as of 2026-10-12 (Hugging Face and GitHub APIs).

On JevBench’s official open-weights composite Winnow-12B Q8 ranks seventh at its unchanged 68.89 (measured 2026-10-02; fifth before the mid-October additions crowded the board), and on the Decision Index 0.3.1 board winnow-12b-q8 scores 49.27 and winnow-e4b-q8 38.73 against Jev’s 60.11. The third-party local-jev-bench suite (12B measured 2026-10-10) reads it as top-group everywhere: transfer-v4 83 percent, level with Kev-9B and Clef, typed-decisions 71 percent level with its E4B sibling, and the suite’s regression note that Winnow now beats Kev-9B on typed-decisions with non-overlapping Wilson intervals. Ollaya’s Ollama comparison calls winnow:12b its most accurate model on Bespoke Labs’ 3,880 human-labeled questions, 0.773 against Ollama 0.35’s best 0.749.

Strengths
#

The third-party record is the strongest of any community family: two independent suites plus Ollaya’s human-labeled table all place it at or near the top, which is the verification most of this wave still lacks.
The GGUF distribution with checksums, documented 16 GB deployment profiles, and vision via a projector make it the wave’s most practical local download for mixed text-and-image decisions.
The self-run panel prints the losses in the same table as the ties (Jev’s 5.45-point transfer lead on Kev-v9 clean), the frankness discipline Jeff introduced, kept.
Chat and decisions share one loaded model, so a local stack can keep one weights file for both its assistant and its decision layer.

Cautions
#

The training data is private and the recipe undocumented, so the close-to-Jev headline cannot be audited for contamination, and the card’s own “clean” caveat concedes the question.
Transfer is where Jev still leads (87.00 versus 81.55 on the panel’s out-of-distribution additions, 69.23 against 85.64 on v9 additions), so near-parity holds on public subsets, not everywhere.
The official composite (68.89) sits well below the public-subset accuracy the card leads with, because the composite’s cost, speed, and sealed axes cut the other way.
The inference server is a 26-star companion repo, so the weights’ ecosystem rests on one small server plus third-party runtimes (Ollaya, llama.cpp-class).

Pricing
#

Free and open: Apache-2.0 weights and an MIT server, no hosted service and no paid tier. The cost is the GGUF download (12.67 GB for the 12B Q8 target) and a GPU in the 16 GB class for the documented profile.

Compared to
#

Jev: the closed original it ties on the public subset and trails on transfer; choose Jev when out-of-distribution accuracy leads, Winnow when local, private, image-bearing decisions do.
Kev: the other family the benches compare it to; Kev ships the eval discipline and the 27B flagship, Winnow the better typed-decisions accuracy per local-jev-bench and the easier GGUF deployment.
Decider: the other community generalist; Decider leads on transfer in the same third-party bench, Winnow on typed-decisions and multilingual sets.

Bottom line
#

Recommended as the default local decision model for engineers who want the most-third-party-verified open checkpoint with text and image states, budgeting a 16 GB-class GPU for the 12B. Not for out-of-distribution-critical routing where Jev’s transfer lead holds, and not for anyone who needs the training data behind a near-frontier score to be public. The disagreeable claim I will defend: Winnow matching Jev answer-for-answer on a public subset is the weaker half of its case, and the stronger half is that three independent readers agree on it, which is exactly the verification Jev still does not have.

Changes
#

2026-10-11 - Created from the entrant scan (awesome-jev cross-check), accepted on the strongest third-party verification record among the community families, the private-training-data caution recorded.
2026-10-11 - Added the star history chart.
2026-10-12 - The JevBench composite rank moved from fifth to seventh on the unchanged 68.89 as the open board grew to 152 ranked systems; refreshed the checkpoint to 57,761 downloads and 66 likes.

References
#

https://huggingface.co/EldanRing/Winnow-12B - the model card: Apache-2.0, Gemma 4 12B fine-tune, GGUF targets, the self-run panel against Jev, Kev, and Laya, and the private-training-data statement (fetched 2026-10-11)
https://huggingface.co/EldanRing/Winnow-12B/raw/main/docs/BENCHMARKS.md - the benchmark report: the 85.71 versus 85.71 public-subset tie, the 81.55 versus 87.00 transfer gap, and the measurement conditions (fetched 2026-10-11)
https://huggingface.co/api/models/EldanRing/Winnow-12B - created 2026-09-20, last modified 2026-10-06, 57,761 downloads, 66 likes (fetched 2026-10-12)
https://github.com/EldanRing/winnow-inference - the MIT server: created 2026-09-21, 26 stars, pushed 2026-10-06, the /v1/systemone plus chat surface and the 16 GB deployment profile (GitHub API, as of 2026-10-11)
https://benchmarkheaven.com/api/jevbench/v1.6.1 - the JevBench aggregate: Winnow-12B Q8 seventh on the open composite at 68.89, measured 2026-10-02 (fetched 2026-10-12)
https://huggingface.co/spaces/multimodalart/jev-decision-index/resolve/main/data/v03.json - the Decision Index 0.3.1 rows (winnow-12b-q8 49.27, winnow-e4b-q8 38.73), regenerated 2026-10-12, fetched 2026-10-12
https://raw.githubusercontent.com/tak-bro/local-jev-bench/HEAD/README.md - the third-party Apple Silicon readings placing Winnow in the top group, including the Kev-9B typed-decisions regression note (fetched 2026-10-11)
https://ollaya.dev/ - the human-labeled Ollama comparison where winnow:12b is the most accurate model at 0.773 (fetched 2026-10-11)

multimodal on tomrochette.com

H2O-Lightning-4B

What it is
#

Status
#

Strengths
#

Cautions
#

Pricing
#

Compared to
#

Bottom line
#

Changes
#

See also
#

References
#

Winnow

What it is
#

Status
#

Strengths
#

Cautions
#

Pricing
#

Compared to
#

Bottom line
#

Changes
#

See also
#

References
#

multimodal on tomrochette.com

H2O-Lightning-4B

What it is #

Status #

Strengths #

Cautions #

Pricing #

Compared to #

Bottom line #

Changes #

See also #

References #

Winnow

What it is #

Status #

Strengths #

Cautions #

Pricing #

Compared to #

Bottom line #

Changes #

See also #

References #

What it is
#

Status
#

Strengths
#

Cautions
#

Pricing
#

Compared to
#

Bottom line
#

Changes
#

See also
#

References
#

What it is
#

Status
#

Strengths
#

Cautions
#

Pricing
#

Compared to
#

Bottom line
#

Changes
#

See also
#

References
#