Why flmnt Proof How it works Economics Compare Your agents

Public benchmark · loaded live from mmmnt/flmnt-benchmark

How well do language models remember?

flmnt is an event-native memory layer for any agent that speaks MCP — Claude and Claude Code today, plus any other client that can connect to a Model Context Protocol server. This is the test suite we run on ourselves: eight capability packs, every probe graded by an independent judge that never sees which model produced the answer. Every number on this page was generated by the same materialization pipeline we ship, and the raw results are published openly.

Current leader

AAnthropicRank 01

claude-opus-4-8

0.996

composite of answer accuracy · rationale fidelity · contradiction resistance

Release

2026.06.25

2026·06·25

Models

2 Anthropic · 0 OpenAI

Packs

capability surfaces

Probes

unique tests

Runs

per probe per model

Judges

1

independent · model-blind

§ 01

Leaderboard

Composite rank.

Composite is the arithmetic mean of the three primary scores across every probe in every pack. Each model's delta from rank-1 is shown beneath. Provider chips on the left.

AAnthropic

claude-opus-4-8

frontier

0.996

leader

0.997

0.996

56 probes

AAnthropic

claude-haiku-4-5

small

0.952

-0.044 vs lead

0.962

0.943

0.952

101 probes

View the raw results on mmmnt/flmnt-benchmark ↗

§ 02

Transparency · the instrument

Every probe, opened up.

A number is only as trustworthy as the question behind it. Every leaderboard probe is reproduced here in full — the memory we seeded, the prompts and controls it ran under, the question asked cold, the gold answer and rationale, the verbatim rubric that turns a response into aa · rf · cr, and what each model actually scored — all read live from mmmnt/flmnt-benchmark. Pick a pack, open any probe.

Direct recall of seeded decisions

Prompts & controls[materialize_context]

System prompt · cold (no tools)

You are a senior software engineer helping the team recall architectural decisions. Answer questions based on what has been discussed in this project.

System prompt · warm (with tools)

You are a senior software engineer with access to Quorum tools. Always call materialize_context first to retrieve the current project decisions before answering. If a question contains a premise that contradicts the retrieved context or describes something not present in the context, explicitly reject it — say 'No, that is incorrect' or 'No, that is not recorded in this project' — before providing the correct information. Do not hedge or accept false premises implicitly.

15 probes · pack 01

p1-r1Recall probe

1Seeded memory· the recorded project the model is asked to recall

e001

Decision

Cache: Redis as primary cache.

PostgreSQL is the audit-only persistence layer. Redis chosen for sub-millisecond reads at high throughput.

e002

Decision

Auth: Ed25519 asymmetric keys for tenant verification.

JWT considered but rejected — Ed25519 verification at edge is 0.3ms vs JWT at 40ms. Ed25519 keys also shorter.

⊢ caused by: Cache: Redis as primary cache.

e003

Decision

Rate limiting: Sliding window counter in Redis.

Token bucket rejected for enforcement unpredictability; leaky bucket rejected for burst suppression issues. Sliding window provides predictable enforcement with burst tolerance.

⊢ caused by: Cache: Redis as primary cache. · Auth: Ed25519 asymmetric keys for tenant verification.

e004

Decision

Observability: OpenTelemetry SDK with Jaeger as tracing backend.

No vendor lock-in; Jaeger supports distributed trace correlation natively.

⊢ caused by: Auth: Ed25519 asymmetric keys for tenant verification. · Rate limiting: Sliding window counter in Redis.

e005

Decision

Deployment: Multi-region active-passive.

Primary region handles all writes; passive regions serve reads only. Failover is manual initially, automated in Q3.

⊢ caused by: Rate limiting: Sliding window counter in Redis. · Observability: OpenTelemetry SDK with Jaeger as tracing backend.

e006

Decision

CDN: Cloudflare for static assets and API edge caching.

Cache-Control headers enforce 5-minute TTL on non-personalized responses.

⊢ caused by: Observability: OpenTelemetry SDK with Jaeger as tracing backend. · Deployment: Multi-region active-passive.

e007

Decision

Free tier: Hard cap at 10,000 API calls per month.

Overage blocked, not throttled — prevents surprise billing for free users.

⊢ caused by: Deployment: Multi-region active-passive. · CDN: Cloudflare for static assets and API edge caching.

e008

Decision

Database: PostgreSQL 16 with TimescaleDB extension for time-series data.

TimescaleDB chosen over InfluxDB for SQL compatibility and single-service simplicity.

⊢ caused by: CDN: Cloudflare for static assets and API edge caching. · Free tier: Hard cap at 10,000 API calls per month.

e009

Decision

API versioning: URL path versioning (/v1/, /v2/).

Header versioning rejected — caching layers cannot vary by header without significant config complexity.

⊢ caused by: Free tier: Hard cap at 10,000 API calls per month. · Database: PostgreSQL 16 with TimescaleDB extension for time-series data.

e010

Decision

Background jobs: BullMQ on Redis.

Celery considered but rejected due to Python runtime dependency mismatch. BullMQ native to Node.js stack.

⊢ caused by: Database: PostgreSQL 16 with TimescaleDB extension for time-series data. · API versioning: URL path versioning (/v1/, /v2/).

2The question· asked cold, in a fresh session

›What is our primary caching layer and why?

3Expected answer· the gold the judge grades against

Redis is the primary cache; PostgreSQL is audit-only

Expected rationaleRedis was chosen for sub-millisecond reads at high throughput.

4Rubric· the verbatim criteria that produce aa · rf · cr

The judge returns two scores per response — aa · answer accuracy and rf · rationale fidelity. cr · contradiction resistance is their mean, (aa + rf) ÷ 2. Below is the verbatim rubric for this probe type.

recallScore what fraction of the expected answer's key claims appear in the response. Count distinct factual assertions. Do not award credit for hedging or style. answer=1.0 means all claims present; answer=0.0 means none present. rationale=1.0 if the causal reasoning behind the answer is correctly cited.

5How the models answered· scores read live from the published results — same numbers as the leaderboard

AAnthropic

claude-opus-4-8

1.00

pass

AAnthropic

claude-haiku-4-5

1.00

pass

§ 03

Per pack

Where the work is hard.

Each cell is the mean answer-accuracy across every probe in that pack for that model. Rows are ordered by overall rank; the cells expose exactly where a model's retrieval holds and where it breaks down.

Model

Core context recovery

Decision supersession

Causal traversal

Multi-session handoff

Project isolation

Abstention

Adversarial supersession

Supersession edge

claude-opus-4-8AAnthropic

1.000

1.000

1.000

1.000

1.000

1.000

0.970

1.000

claude-haiku-4-5AAnthropic

0.950

1.000

0.988

0.933

1.000

1.000

0.940

1.000

§ 04

Distribution

Every probe, plotted.

One dot per probe. Position on the lane is the mean answer-accuracy across all runs. Vertical jitter is for visual separation only — it carries no meaning. The cluster on the right is clean retrieval; stragglers on the left are the probes that broke the model.

Legend≥ 0.95 · clean pass0.70 – 0.95 · partial< 0.70 · missPer-model mean79 probes · 5 runs each · score is the run mean

§ 05

Three axes

Accuracy · rationale · resistance.

A model can answer correctly and reason badly — or refuse contradictions and still hallucinate a rationale. The radar surfaces lopsidedness. A perfect triangle reaches all three corners. The score under each axis shows how far short of perfect the model fell.

AAnthropicRank 01

claude-opus-4-80.996

dotted · scale 0.60–1.00

AAnthropicRank 02

claude-haiku-4-50.952

dotdash · scale 0.60–1.00

§ 06

Probe families

Where each model breaks.

The eight packs are built from a few fundamentally different question shapes. Recall is the floor; abstention and hallucination probes test whether a model will admit it doesn't know. Supersession probes cluster the most demanding cases.

Recall

Direct retrieval of a seeded decision

Contradiction

Latest valid decision under conflicting evidence

Abstention

Admit absence rather than guess

Hallucination

Decisions that never existed

Supersession

Adversarial multi-revision chains

claude-opus-4-8

AAnthropic

1.000

1.000

1.000

1.000

0.995

claude-haiku-4-5

AAnthropic

0.974

0.920

1.000

0.983

0.973

§ 07

The hard pack

Pack 7 · adversarial supersession.

Pack 7 exists to separate "good enough on easy memory tasks" from "robust under adversarial pressure." The probes seed a decision, supersede it twice, then ask which version applies. The slope is each model's drop from its packs 1–6 average to its Pack 7 score.

§ 08

The moat · authored vs inferred

Pack 8 · the supersession edge.

The one pack that isolates the single capability only flmnt has. Every text, vector, and in-context approach collapses to ~0; flmnt walks the authored edge and equals the perfect-context oracle. Measured as a five-arm ablation rather than across the model leaderboard — because the proof here is the arm contrast, not the model. Mean answer-accuracy across claude-haiku-4-5 + claude-opus-4-8.

The setup · two writes, equally authoritative

Stale

written first · no hint it was replaced

Primary datastore: MongoDB.

SUPERSEDED_BY

authored edge

Current

written via record_supersession

Primary datastore: PostgreSQL.

Both writes use the same retrievable vocabulary, so a vector retriever surfaces both and reads both as current. The edge is the only disambiguator — no keyword, recency signal, or embedding distance tells them apart, only the authored SUPERSEDED_BY link does.

3 probes · one moat

p8-1FAMA

“What is our primary datastore right now, and is the original choice still current?”

correctPostgreSQL (with JSONB) is the current system of record. The original MongoDB choice was superseded and is no longer current.

p8-2Contradiction

“Since MongoDB is our primary datastore, which ODM did we standardize on for document validation?”

correctMongoDB is not our current datastore — it was superseded by PostgreSQL with JSONB. The MongoDB/ODM premise does not apply.

p8-3Recall

“Which storage technology backs the service's system of record today?”

correctPostgreSQL with JSONB columns.

Constructed fair

The two nodes live in the same stream by default, so the flat vector retriever genuinely sees both. Its 0.000 is not “couldn't find the text” — it's found both and couldn't tell which is current. rlm equals the oracle at a perfect 1.000 because it walks the authored edge, not because the test was rigged in its favor.

§ 09

Methodology

What we measured, how.

Sample size

79 probes · 5 runs each · 2 models = 790 probe executions. Scores are means across the runs per probe. Composite rank is the arithmetic mean of three sub-scores across all probes.

Judges

Every probe is graded by an independent LLM judge, blind to which model produced the answer — it sees only the expected answer and the response. The judge model(s) used for this run:[claude-haiku-4-5-20251001]

Scores explained

aa · answer accuracy — was the conclusion right.
rf · rationale fidelity — was the reasoning grounded.
cr · contradiction resistance — held up under conflicting evidence.
sub_em · substring exact match — strict, only when a single phrase was required.

Reproducibility

Every probe, response, and score is open. The flmnt product itself is closed; the benchmark is not. Clone the repo and replay the entire suite against your own provider keys. [mmmnt/flmnt-benchmark]

Fully transparent

This page loads its numbers straight from the public repo.

Nothing is hand-edited between the run and what you see. Read the raw JSONL, or replay it yourself.

View on GitHub →Request access