Skip to content

Run

8 emulators 1054 tests

Results as of this run. The arrow shows each target's movement since the previous run it was tested in. The suite grew this run, so an upward arrow can be the new tests biting rather than a target getting worse.

Suite grew from 998 to 1054 tests this run.

That's 56 new tests measured against every target. Movement below compares to the previous run, so a rise here is as likely to be the stricter suite as a real regression.

What changed in the suite this run

Suite on

Read this first if you consume the JSON. The data endpoints go from schema 2 to schema 4 in one step. Schema 3 was never published on its own, so everything on 2 crosses both steps at once. Schema 3 breaks in four ways:

  • movement.state values changed from up/down to improved/regressed. This is the one that fails silently: the old names still parse, and now mean the opposite direction.
  • movement.delta keeps its shape but is computed from divergence rather than correctness, so the sign of a delta means the opposite of what it did.
  • A tier no longer carries pct and value. It carries divergence, coverage and correctness, each with a pct and a value of its own.
  • The whole-suite correctness percentage is correctness, not total. total also names the raw test count inside counts, so the same word meant a count in one place and a percentage in another.

Schema 4 is additive on top: each target carries its grade and the full criteria in metrics.grade, every envelope gains baseline.observation (how much of the suite the live-AWS row stands on, which passes reported, and what is carried), latest.json and runs.json gain divergence, project, configuration and isVariant, results/summary.json gains regionFailures, and /data/index.json lists the split registry.

A score is two figures, never one

Divergence is the share of the whole suite a target answers differently from real DynamoDB. Coverage is the share it implements at all. They are reported apart and never summed, because a declined operation is discoverable in minutes and a wrong one in production. No target was re-run for the change and no pass, fail or skip moved: what changed is how the same counts are expressed.

Everything else that was a percentage followed the headline down - tier figures, the per-region drilldown, the per-operation table, and the colour bands, which inverted with them. A target's history is two plots rather than one, because divergence falls when a target stops attempting something it used to get wrong, so a divergence line alone can render a withdrawal as an improvement.

Every target wears a letter

Divergence sets it - A under 5%, B under 15%, C under 25%, D under 35%, F beyond

  • and coverage can only lower it, never raise it: a third of whatever a target leaves unimplemented is added to its divergence before the bands are read. A+ is exactly zero divergence at full coverage. A row says when coverage is holding its letter down, so a capped row is not mistaken for one with room above it. Real DynamoDB reads baseline rather than a letter, because grading the yardstick against itself would seat it in a band an engine had to earn its way into.

These are grading criteria version 1, dated in the methodology, which carries the derivation. Where a threshold sits is a hand-picked input to a published letter, and moving one regrades targets whose results never changed, so any change to a band, the coverage weight or the A+ gate bumps the version.

The suite grows from 998 tests to 1054

Vector search (#125). 42 tests over the deterministic surface DynamoDB shipped in August 2026: the index lifecycle on both creation paths, request validation on each plane, rejection wording, write-path validation, search on a fixture where the nearest neighbour is unambiguous, the two new capacity shapes, and PartiQL's inability to reach a vector index. Every pinned value was characterised against real DynamoDB in eu-west-2 before it was asserted. Two findings worth naming: searching during a backfill is an error, which settles which side of a contradiction in AWS's own documentation is right - three developer-guide pages say the call fails, the tutorial page says results can be incomplete - and an overwrite leaving the stored vector unchanged reports no vector write capacity at all, because index replication is delta-based. Both sides are captured in captures/2026-08-12-vector-backfill-docs.json. No emulator implements the family yet, so every target skips it and every coverage figure drops with this release while divergence is untouched. Sending the new operations needs @aws-sdk/client-dynamodb 3.1103.0 or later.

Index write costs (#124). 14 tests. The suite's only per-index capacity assertion was on a Query, so the write side - the half you get billed extra for - went unmeasured. A sub-1KB write costs one unit for the table and one for each index it lands in, and LSI units fold into the total exactly as GSI units do. Moving an item to a new GSI key costs two on that index, a delete and an insert; touching a projected attribute costs one; touching a non-projected attribute costs nothing, and the response carries no arm for that index rather than a zero. An overwrite that leaves the item unchanged reports no index cost whatsoever.

The index exclusions create no indexed table (#116). !gsi and !lsi used to select the right tests and then build the tables anyway, so an engine with no secondary-index support died in setup whatever it had asked for. Shared tables are created on demand from what the running file declared, the composite table split into indexed and plain variants, and three guards keep the declarations and the tags honest.

Corrections

  • The suite counts its own tests. "The whole suite" had meant whichever target ran the most tests, so one of the measured things was setting the denominator every figure divided by. registry/suite-manifest.json lists every test by file and full name, generated from the suite rather than inferred from a run, and CI fails if it drifts. Publishing now refuses a row whose test population disagrees with it, which catches a results file carried across a rename that keeps its old total while naming tests that no longer exist.
  • Every figure on a row comes from one region. A headline came from the target's best-matching region while its tier split and raw counts stayed on the baseline region's basis. Correctness never had to reconcile, because each tier had its own denominator; divergence is additive, so it does.
  • The per-region overlay is matched to the run it describes, rather than keyed on the date real AWS was last swept, which had put a later run's figures on an earlier run's page.
  • A methodology claim was wrong. The page said that measuring divergence over the whole suite stops an engine implementing a sliver from posting a perfect score. It doesn't: zero fails is 0.0% divergence at any coverage. What does hold is the identity the page now states - a test going from failing to skipped leaves both numerators together over the same denominator, so withdrawal costs exactly as much coverage as it gains divergence, and is disclosed rather than silent.
  • The baseline row is measured rather than pinned once real AWS has been observed across the whole suite. It runs in three passes and only the main one reached the published artefact, so the row had claimed a full suite on a run that recorded less. The passes are merged before scoring, each with its own capture date, and the row stays pinned and says so until all three report.
  • The Atom feed carries no letter on any run measured before criteria version 1 took effect. It had been rewriting each entry's summary with a grade the run never had, while leaving <updated> alone so no subscriber re-notified.
  • Badges publish the grade under a parity label, having still been publishing the correctness percentage the board retired under a conformance label. The endpoint URL is unchanged; a badge whose target has no results this run reads no data rather than disappearing, because the URL sits in other people's READMEs.
  • Two more regional splits are admitted, both BatchGetItem with an empty RequestItems map. The nesting-depth row was rewritten from a full 32-region capture, having been written from four.
  • A build of an engine nests under it rather than taking a row beside it, and maintainedByAuthor is keyed on the project, so it now reads true for the WebAssembly build. That build runs in CI like every other target, where its row had been refreshed by hand.
  • The board leads with the highest-graded engine, since the baseline moved into a panel above the standings. Where that is the board author's own engine, the conflict-of-interest disclosure sits on the card.
  • The Region column is gone. It named the cohort a target matched at its best rate, which read as breadth. The count sits beside the figure instead, and the cohort listing stays on the target page.
  • Each target lists how it is actually distributed, with the project's own page for each. These are the only claims on the board the suite does not measure, so each carries the link that backs it.
Scored against 33 real regions. 1 dropped, out of scoring until it returns: me-south-1
  1. DynamoDB
    live (AWS)

    Runs via: AWS service

    Grade baseline

    no divergence in all 33 regions · covers 100.0% of the suite

    ground truth
    Tier breakdown
    Tier 1 · Core 0.0% diverges
    100.0% covered
    Tier 2 · Complete 0.0% diverges
    100.0% covered
    Tier 3 · Strict 0.0% diverges
    100.0% covered
  2. Dynoxide
    0.13.0 14 unsupported ; covers 98.6% of the suite

    Runs via: npx, Docker, Homebrew, binary, npm, cargo, embedded, GitHub Action, source

    Disclosure: maintained by this board's author

    Grade A

    no divergence in 6 regions · up to 0.3% in the other 26 · covers 98.6% of the suite · coverage lowers this row to A

    unchanged
    Tier breakdown
    Tier 1 · Core 0.0% diverges
    100.0% covered
    Tier 2 · Complete 0.0% diverges
    93.0% covered 14 unsupported
    Tier 3 · Strict 0.0% diverges
    100.0% covered
    Also built for WebAssembly / OPFS
    • WebAssembly / OPFS
      Grade B

      low divergence (0.9%) · covers 83.4% · coverage lowers this row to B

  3. ExtendDB
    v0.1.3 87 unsupported ; covers 91.3% of the suite

    Runs via: source

    Grade A

    low divergence (1.9%) in all 32 regions · covers 91.3% of the suite

    diverged 0.1 percentage points less
    Tier breakdown
    Tier 1 · Core 0.4% diverges
    100.0% covered
    Tier 2 · Complete 3.0% diverges
    58.3% covered 83 unsupported
    Tier 3 · Strict 3.4% diverges
    98.8% covered 4 unsupported
  4. Ministack
    63621de32116

    Runs via: pip, Docker, source

    Grade B

    moderate divergence (11.2%) in all 32 regions · covers 100.0% of the suite

    unchanged
    Tier breakdown
    Tier 1 · Core 3.2% diverges
    100.0% covered
    Tier 2 · Complete 17.1% diverges
    100.0% covered
    Tier 3 · Strict 19.4% diverges
    100.0% covered
  5. Dynalite
    4.0.0 200 unsupported ; covers 80.0% of the suite

    Runs via: npm, npx, embedded

    Grade C

    moderate divergence (12.3%) in 26 regions · up to 12.5% in the other 6 · covers 80.0% of the suite · coverage lowers this row to C

    diverged 0.2 percentage points less
    Tier breakdown
    Tier 1 · Core 8.6% diverges
    100.0% covered
    Tier 2 · Complete 14.1% diverges
    20.1% covered 159 unsupported
    Tier 3 · Strict 16.7% diverges
    87.3% covered 41 unsupported
  6. LocalStack
    2026.7.3 8 unsupported ; covers 99.2% of the suite

    Runs via: Docker, pip, Homebrew, binary

    Grade C

    high divergence (15.6%) in 26 regions · up to 15.7% in the other 6 · covers 99.2% of the suite

    diverged 0.1 percentage points less
    Tier breakdown
    Tier 1 · Core 6.5% diverges
    100.0% covered
    Tier 2 · Complete 18.1% diverges
    96.0% covered 8 unsupported
    Tier 3 · Strict 27.5% diverges
    100.0% covered
  7. DynamoDB Local
    ff89bd48ff32 21 unsupported ; covers 97.9% of the suite

    Runs via: Docker, JAR, Maven

    Grade C

    high divergence (15.9%) in 26 regions · up to 16.0% in the other 6 · covers 97.9% of the suite

    diverged 0.1 percentage points less
    Tier breakdown
    Tier 1 · Core 7.8% diverges
    100.0% covered
    Tier 2 · Complete 16.1% diverges
    89.4% covered 21 unsupported
    Tier 3 · Strict 27.8% diverges
    100.0% covered
  8. Floci
    eab36252ea43 9 unsupported ; covers 99.1% of the suite

    Runs via: Docker, Homebrew, install script, Scoop, binary, JAR

    Grade C

    high divergence (20.9%) in all 32 regions · covers 99.1% of the suite

    diverged 0.1 percentage points more
    Tier breakdown
    Tier 1 · Core 8.6% diverges
    100.0% covered
    Tier 2 · Complete 36.7% diverges
    95.5% covered 9 unsupported
    Tier 3 · Strict 29.3% diverges
    100.0% covered