Why measure conformance?

There's no official conformance suite for DynamoDB. AWS doesn't publish one, so every emulator author ends up guessing at how the real thing behaves and testing against their own assumptions. The closest the community had was Dynalite's test suite, and by the start of 2026 over half of its tests had drifted out of step with current DynamoDB. DynamoDB Local ships with no test suite at all.

That's the gap the conformance suite fills. It runs every test against real DynamoDB on AWS first, records what passes, and treats that as the baseline. An emulator only passes a test if it gives the same answer DynamoDB does. Real DynamoDB is the ground truth, which is why it sits at the top of every table diverging from itself nowhere - not because it scored well, but because it's the thing everything else is measured against. DynamoDB doesn't always behave the same way in every region, so the suite records the answer in each region and scores a target against all of them; the methodology has the detail.

"Works with the SDK" isn't the same as "behaves like DynamoDB"

Plenty of emulators will happily take your requests and hand something back. The question that matters is whether they hand back the same something. A subtly different error, a validation check that fires in a different order, a filter that returns one extra item - none of it breaks the happy path, all of it bites you in production when your code was quietly relying on the real behaviour.

So the suite tests observable behaviour and nothing else. It drives the standard AWS SDK against an HTTP endpoint and asserts on what comes back: the response shape, the error it returns (its type, the field it objects to, and the constraint it failed), the order things are validated in. No internal hooks, no privileged access. If your application talks to the target through the SDK, that's precisely what gets checked.

Three tiers, because one number hides too much

A target that gets 8% of the suite wrong tells you almost nothing on its own. Eight percent of what? Get 8% of the core operations wrong and you'll feel it constantly; get 8% of the strictest edge cases wrong and it's likely still fine for local development. Those are very different situations behind the same number, so the suite splits its tests into three tiers and reports divergence within each.

Tier 1 - Core. The operations roughly 90% of DynamoDB users rely on: CRUD, queries, scans, batch operations, GSIs, UpdateTable. The more an emulator diverges here, the more often everyday code will hit a difference.

Tier 2 - Complete. Documented but less common features: transactions, PartiQL, LSIs, TTL, streams, tags, vector search. A gap here only matters if you actually use that feature.

Tier 3 - Strict. Validation ordering, error behaviour at a range of strictness - exact where DynamoDB's wording is stable, structural (the type, field and constraint) where its rendering is non-deterministic - limits, and legacy API shapes. Missing some of this is usually fine when you're developing locally, where the exact error string rarely matters. It matters far more in CI: if your own test suite runs against an emulator and asserts on error messages or validation behaviour, a Tier 3 gap is exactly the kind of thing that lets a bug through a green build and only shows up against real DynamoDB in production.

So a target diverging 8% over the whole suite might be right about every core operation and wrong about a fifth of Tier 3, or the other way round. The tier columns say which, and only one of those two is a problem for most people.

Skips are scope, fails are bugs

Skips here are deliberate. Each test file probes for feature support up front and skips itself if the target doesn't implement that operation at all. A skip says "I don't do this"; a fail says "I do this, but I get it wrong". Those are different problems for whoever is relying on the thing: an operation a target declines is something you find in minutes and plan around, and one it quietly gets wrong is something you find in production.

So the board reports them apart. Divergence counts the fails against the whole suite. Coverage says how much of the suite the target implements. A target with a narrow surface that gets it right shows a low divergence and a low coverage, and both are true at once. Because the denominator is the whole suite either way, a fail that turns into a skip leaves both figures together: divergence and coverage fall by exactly the same amount, so a target that stops attempting something it got wrong cannot improve one without paying for it in the other. The letter grade on each row is a reading of the pair: divergence sets the letter and coverage can only lower it, never raise it, under criteria the methodology publishes in full. The letter is weaker than the figures under a withdrawal, and the methodology says by how much. The methodology works the rest through.

Why a whole site for it

The suite already publishes its latest numbers as a table in its README. That's fine for a single snapshot, but a markdown table can't show whether an emulator is improving or regressing, it can't carry a per-tier breakdown without overflowing the page, and it throws away every run before the current one.

The data to tell the fuller story was there all along. Every run stamps its results into the repo, and the git history holds every past snapshot. So this site reads that history and rebuilds the whole timeline: the latest results with run-over-run movement, a page for each target, and every recorded run you can step back through.

Every figure here is derived from the suite's own results at build time. None of it is typed in by hand, which is the point - the moment you copy a number into a second place, the two start drifting apart, and conformance figures that drift are worse than none at all.

On independence

The suite and this site are built and maintained by Martin Hicks, who also maintains Dynoxide - one of the targets scored here. That's why the no-figure-by-hand rule matters for more than drift: it's what keeps the scoring honest. Every number is derived from the suite's own results at build time, and the scoring logic is shared with the suite rather than restated here. A target's score can't be tuned without changing the suite's published results first, in the open, and the code that turns those results into the numbers on this page is in the same public repository as the tests that produced them. Real DynamoDB is the baseline, every figure carries the region and date it was measured, and anyone can suggest a target. There's more for programmatic consumers, and the raw data, on the agent guide.

What is reproducible is the scoring, not the site. Clone the repository and you can rerun the scorer over the committed results and get the figures published here, test by test. A local build of the site itself renders a smaller thing: the pages assemble a timeline by fetching the history of results/ from the API, and without a token that fetch falls back to a committed snapshot. The deploy sets FAIL_ON_FALLBACK precisely so a scheduled build refuses the snapshot rather than quietly publishing a thinner board.

The letter grades need a second answer, because that argument only covers the figures. Where the bands sit was a choice, and moving one regrades targets whose results never changed - no run required, nothing to notice. So the criteria are versioned and dated: they are grading criteria version 1, and any change to a band, the coverage weight or the A+ gate bumps the version and is dated on that page. Both figures print beside the letter on every surface the board controls, so you can recompute a grade yourself and check it, and a retune leaves a record rather than a quietly different board.

The bands are not the only hand-picked input, and the other one deserves naming. The split registry is written by hand by design: it records the behaviours where real DynamoDB's own regions disagree, with the evidence each region returned, and a target matching any recorded answer is scored as conformant rather than wrong. Admitting a row there can turn a fail into a pass with no re-run and no results file changing, which is the one thing "a score can't be tuned without changing the published results first" does not cover. So the registry is public, every row carries the captured evidence and the date it was refreshed, and a behaviour enters only once confirmed across regions. It currently holds five rows, and they are the entire set of behaviours on which matching one region rather than another can turn a fail into a pass.