Skip to content

Run

7 emulators 873 tests

Results as of this run. The arrow shows each target's movement since the previous run it was tested in.

When this run was published the board led with a single correctness percentage. The figures here are that run's own results restated as divergence and coverage, so they read differently from what this page showed at the time. The changelog has the change.

What changed in the suite this run

Suite on

The scores barely move in this release, but what they mean has changed.

Until now the suite pinned one region, eu-west-2, as ground truth. That was quietly unfair: real DynamoDB disagrees with itself in a handful of places, and a one-region baseline takes a side without saying so. The clearest case is the { NULL: false } attribute value - accepted and normalised to { NULL: true } in eu-west-2 and eu-central-1, rejected with a ValidationException in us-east-1 and ap-southeast-2. An engine matching us-east-1 on that behaviour was marked non-conformant for doing exactly what real DynamoDB does in Virginia. From 2.0.0, ground truth is per region.

What changed:

  • A weekly sweep runs the full suite against real DynamoDB in every commercial region and publishes per-region ground truth (.github/workflows/sweep.yml, ground-truth/). It gates nothing: PR scoring stays offline and deterministic.
  • Confirmed regional splits live in a checked-in registry (registry/splits.json), each row carrying what every named region actually returned, when, and who admitted it. Detection is automatic; admission is not. The sweep files an issue with the evidence, and only a human commits a row. The registry ships seeded with the { NULL: false } split.
  • Each target is scored against every observed region's expectations, and its published number is its best-matching region - named in the results table's new Region column, with the full per-region view in results/summary.json (a versioned, additive artefact; the per-target results/<slug>.json files are unchanged in shape).
  • A third result state, indeterminate, for a failed observation: a timeout, an exhausted throttle, a transport fault. It is excluded from both sides of the score and cannot become a split, a registry row, or a fail. An absent answer is not a different answer.
  • Region health is tracked in registry/regions.json. A region that cannot complete a sweep is published as unresolved rather than silently omitted; two consecutive misses drop it from the scored set and page a maintainer in the same act.

One deliberate departure from the RFC that proposed this (#75): the RFC suggested a behaviour conforms if it matches any real region. 2.0.0 scores each target against one region at a time and headlines the best match, so a target only passes a behaviour when at least one real region does what it did, and its headline reflects one coherent region rather than a mix. Match-any scoring would have accepted an engine that combines eu-west-2's answer on one behaviour with us-east-1's on another - a deployment that exists nowhere. That is stricter than the RFC asked for, and it is deliberate.

No score moves at release: the one admitted split pins eu-west-2, which is the only region in the health record until the first sweep runs. Per-target deltas will be published once the sweep admits more regions; the expected movement is roughly a tenth of a percent for the six engines that match us-east-1 on the { NULL: false } split.

The suite also grew to 954 tests, up 81, all characterised against real DynamoDB - the control-plane pins in eu-west-2, everything else across four regions (eu-west-2, eu-central-1, us-east-1, ap-southeast-2):

  • UpdateTable AttributeDefinitions reconciliation (#77, #78, #79): delta-fed GSI adds merge into the stored union rather than replacing it, a conflicting redeclaration of an existing key keeps the stored type, and deleting a GSI prunes only its orphaned key attributes. An unused definition on an add is silently dropped where CreateTable rejects the same shape.
  • ProjectionExpression validation (#81): duplicate paths, alias collisions and parent/child overlaps rejected identically on GetItem, Query, Scan and BatchGetItem, pinned with exact messages; legal shared-prefix projections guarded as accepted; GetItem's misfiled validation tests rehomed into Tier 3.
  • Expression-size limit (#80): every expression parameter caps at 4096 bytes, measured on the raw string before ExpressionAttributeNames substitution - 4096 accepted, 4097 rejected, on all five expression surfaces. No tracked target enforces this limit today, so the Tier 3 movement it causes is new coverage, not a regression.
  • Empty set members (#82): empty strings in an SS and zero-length members in a BS are accepted and round-trip intact through every write path, and contains() can find them; an empty NS member and duplicate empty members are rejected, with messages pinned. One existing assertion got stricter: the empty-binary round-trip now asserts byte length zero.
Scored against 1 real region.
  1. DynamoDB
    live (AWS)

    Runs via: AWS service

    Grade baseline

    no divergence in all 1 regions · covers 100.0% of the suite

    ground truth
    Tier breakdown
    Tier 1 · Core 0.0% diverges
    100.0% covered
    Tier 2 · Complete 0.0% diverges
    100.0% covered
    Tier 3 · Strict 0.0% diverges
    100.0% covered
  2. Dynoxide
    0.11.3 14 unsupported ; covers 98.4% of the suite

    Runs via: npx, Docker, Homebrew, binary, npm, cargo, embedded, GitHub Action, source

    Disclosure: maintained by this board's author

    Grade A

    no divergence in all 1 regions · covers 98.4% of the suite · coverage lowers this row to A

    unchanged
    Tier breakdown
    Tier 1 · Core 0.0% diverges
    100.0% covered
    Tier 2 · Complete 0.0% diverges
    91.0% covered 14 unsupported
    Tier 3 · Strict 0.0% diverges
    100.0% covered
  3. ExtendDB
    v0.1.1 43 unsupported ; covers 95.1% of the suite

    Runs via: source

    Grade B

    moderate divergence (13.3%) in all 1 regions · covers 95.1% of the suite

    unchanged
    Tier breakdown
    Tier 1 · Core 15.0% diverges
    100.0% covered
    Tier 2 · Complete 7.7% diverges
    72.3% covered 43 unsupported
    Tier 3 · Strict 13.6% diverges
    100.0% covered
  4. Ministack
    27304b7b6d09

    Runs via: pip, Docker, source

    Grade B

    moderate divergence (13.4%) in all 1 regions · covers 100.0% of the suite

    diverged 0.1 percentage points less
    Tier breakdown
    Tier 1 · Core 9.0% diverges
    100.0% covered
    Tier 2 · Complete 18.1% diverges
    100.0% covered
    Tier 3 · Strict 18.0% diverges
    100.0% covered
  5. Floci
    4f69631e5601 9 unsupported ; covers 99.0% of the suite

    Runs via: Docker, Homebrew, install script, Scoop, binary, JAR

    Grade C

    high divergence (15.1%) in all 1 regions · covers 99.0% of the suite

    unchanged
    Tier breakdown
    Tier 1 · Core 8.3% diverges
    100.0% covered
    Tier 2 · Complete 21.3% diverges
    94.2% covered 9 unsupported
    Tier 3 · Strict 22.8% diverges
    100.0% covered
  6. LocalStack
    2026.6.2 8 unsupported ; covers 99.1% of the suite

    Runs via: Docker, pip, Homebrew, binary

    Grade C

    high divergence (15.8%) in all 1 regions · covers 99.1% of the suite

    unchanged
    Tier breakdown
    Tier 1 · Core 7.0% diverges
    100.0% covered
    Tier 2 · Complete 11.6% diverges
    94.8% covered 8 unsupported
    Tier 3 · Strict 32.7% diverges
    100.0% covered
  7. DynamoDB Local
    d89f8fcc6b1a 13 unsupported ; covers 98.5% of the suite

    Runs via: Docker, JAR, Maven

    Grade C

    high divergence (17.1%) in all 1 regions · covers 98.5% of the suite

    unchanged
    Tier breakdown
    Tier 1 · Core 8.3% diverges
    100.0% covered
    Tier 2 · Complete 14.2% diverges
    91.6% covered 13 unsupported
    Tier 3 · Strict 33.1% diverges
    100.0% covered
  8. Dynalite
    4.0.0 68 unsupported ; covers 92.2% of the suite

    Runs via: npm, npx, embedded

    Grade D

    high divergence (22.9%) in all 1 regions · covers 92.2% of the suite · coverage lowers this row to D

    unchanged
    Tier breakdown
    Tier 1 · Core 8.7% diverges
    100.0% covered
    Tier 2 · Complete 49.0% diverges
    56.1% covered 68 unsupported
    Tier 3 · Strict 31.3% diverges
    100.0% covered