Skip to content

Run

14 emulators 1,920 tests

Results as of this run. The arrow shows each target's movement since the previous run it was tested in. The suite grew this run, so an upward arrow can be the new tests biting rather than a target getting worse.

Suite grew from 1,678 to 1,920 tests this run.

That's 242 new tests measured against every target. Movement below compares to the previous run, so a rise here is as likely to be the stricter suite as a real regression.

What changed in the suite this run

Suite on

ca-central-1 joins eu-west-2 on the five Scan rows. It gave the new wording on the 2026-11-07 and 2026-11-14 sweeps and on the three runs on 2026-11-13, which leaves me-central-1 the only region recorded on the old wording. On the 2026-11-14 sweep eu-north-1 stored UpdateItem writes that the statement-sized rule refuses, which it had not done before. Three runs later that day had it answering both ways, so the item-size rows stay as they are.

Ten regions join eu-west-2 on the five Scan rows: ap-northeast-1, ap-south-2, ap-southeast-6, ap-southeast-7, eu-central-2, eu-south-1, eu-south-2, eu-west-1, us-east-2 and us-west-2. Each gave the new wording on the 2026-11-07 sweep and at least one sweep before it, and on three runs against every region on 2026-11-13, which leaves ca-central-1 and me-central-1 on the old wording. ca-central-1 gave the new wording on the 2026-11-07 sweep and on all three runs, but the old wording on the two sweeps before, so it keeps its recorded answer until a second sweep agrees. eu-north-1 gave the old wording on one of the three runs, so it answers Scan both ways for now and keeps its recorded answer, the new wording. The BatchGetItem row doesn't change: me-central-1 gave both answers to an empty RequestItems across the same runs.

eu-west-2 now accepts a 32-level ExpressionAttributeValue in an UpdateItem condition, so the test that pinned its refusal now pins acceptance, and is renamed to say so. It asserts the ConditionalCheckFailedException a false condition gives, the same proof the 31-level test uses. The 2026-11-14 sweep found 19 regions accepting a value they had refused the week before, and by the last of three runs that day 23 did. ap-east-1, ap-southeast-2, ca-west-1, eu-south-2, eu-west-3, mx-central-1, us-east-1, us-west-1 and us-west-2 still refuse it, and me-central-1, which had accepted it until then, now refuses it too. The registry records those ten on that side. The same regions moved together on UpdateItem's ReturnValues error, which now lists UPDATED_NEW where it listed UPDATE_NEW. That test accepts both, and its row now pins the new wording. A stored item is still capped at 32 levels everywhere, and a 63-level value is still refused.

The wait for an index added by UpdateTable to become ACTIVE is now 2 minutes on any emulator, down from 40. Real DynamoDB keeps 40 minutes. The slowest passing index test on any emulator in the published results takes about 31 seconds, against about 25 minutes on real DynamoDB, so the shorter wait only shortens runs where an index never completes. An expired wait is recorded as no answer either way, never as a failure, so no score can change.

PartiQL reads whose OR branches stay in one partition now have paging and capacity tests. Under Limit, a set of sort keys ends on its last key, a missing last key keeps the token for one empty half-unit page, and a branch the read hasn't reached holds the token. Each shape pinned page by page gave the same pages on every walk in eu-west-2 and us-east-1 between 2026-11-03 and 2026-11-05. Page endings that DynamoDB answers at random, such as a page ending inside an open range, are checked only for the keys they return. Each branch costs a unit under ConsistentRead even when it finds nothing, a sort-key IN costs and pages exactly as the equivalent OR does, and a partition key that can't be a key reads the whole table, so beside sort-key ranges it costs what that read does: 2.0, or 4.0 consistent, on the small on-demand table the test uses.

Items tied on an index key come back in a fixed order that isn't the table's sort-key order, and descending is its exact reverse. The order was the same in nine fresh tables across two regions and two AWS accounts, whatever order the items were written in, so Query on an LSI and a GSI, and PartiQL on an LSI, now pin it.

The PartiQL refusals these reads run into are pinned word for word: a negated sort key inside an OR filter, a partition key given the wrong type or a boolean, ORDER BY over branches that each name the partition key, a KEYS_ONLY index asked to filter on an attribute it doesn't project, and ExecuteStatement Limit values of 0, -1 and a string. What DynamoDB accepts is pinned beside them: ORDER BY on an OR in one partition, the same key written twice, and Limit values past 2^31, past 2^64 and written with an exponent or a fraction.

An attribute name starting with an underscore is a syntax error in every expression type unless it goes through ExpressionAttributeNames, including key conditions. An UpdateItem that creates its item returns no Attributes under ALL_OLD or UPDATED_OLD.

Deep PutItem, UpdateItem and DeleteItem requests, read by the older validation layer, now have their joined errors pinned: every bad member, in each operation's own order, under the N validation errors detected envelope even when only one member is wrong.

A table name's length is counted in UTF-16 units on deep writes and on CreateTable, DescribeTable, DeleteTable and UpdateTable, and in characters on shallow item operations, ListTables' start name and index names. So two emoji are long enough for CreateTable and too short for PutItem. Deep writes also check the name in two stages: a missing or empty name is listed among the member errors, while length and pattern errors only show when nothing else is wrong.

Fourteen new rows in registry/splits.json record where me-central-1, and eu-north-1 on Scan, gave the older validation layer's answer to the shallow table-name and leading-underscore cases on 2026-11-05. The other regions agree with eu-west-2. The answers from all 33 regions on 2026-11-05 are in captures/2026-11-05-shallow-validation-rollout.json. On three later runs on 2026-11-13, me-central-1 gave eu-west-2's answer to the twelve cases that aren't Scan (once cut short, on UpdateItem) and eu-north-1 gave it on Scan, so the rows keep the 2026-11-05 answers until a sweep has run the tests. Those runs are in captures/2026-11-13-shallow-validation-follow-up.json.

Four more PartiQL index-qualifier answers are pinned. An unknown index read with ConsistentRead is reported as an unknown index, not as a consistent read on a GSI. Selecting an attribute a GSI doesn't project is refused on an unkeyed scan as well as on a keyed read. With two such attributes the refusal lists them in the same order whichever way round the statement names them, and that order isn't alphabetical, so the test pins the list for one pair. A BatchExecuteStatement SELECT on a composite table that gives the partition key but not the sort key is refused like any other member without the full primary key. All four gave the same answers on three runs each in eu-west-2 and us-east-1 on 2026-11-14.

Renaming or removing a released test, or changing a region's recorded answer, no longer fails the checks on its own pull request. The committed results are now checked against the suite and registry as they stood at the release they were measured at, as the rest of the published board already was, rather than against main. The change reaches the board at the next release, when every target is re-run. A board is now refused if real DynamoDB's own run, in its own region, fails a test or passes one on a split row with an answer recorded for a different region.

A project's own row on the board now names its configuration beside the project, as the README table already did, so ExtendDB's row reads "ExtendDB · PostgreSQL" before its other builds are opened. The ways to run a target gain one more, OS package, for an operating system package: a .deb or .rpm, or an apt or dnf repository.

AWS Emulator joins the board, suggested by @exoego in #304. It is AWS's own multi-service emulator, written in Rust, with DynamoDB and DynamoDB Streams among the services it serves. It installs from a Linux package, or on other systems runs in Docker from an image built with the Dockerfile in its user guide. It serves HTTPS and checks SigV4 signatures by default, with a certificate and key it generates on each start, and it is measured that way.

Alternator joins the board. It is ScyllaDB's DynamoDB-compatible API, built into the database node itself rather than run as a separate emulator, with global secondary indexes on materialised views and Streams on change data capture. It runs from the official ScyllaDB container image, and its version column shows the ScyllaDB release the node reports.

ExtendDB's MongoDB backend joins the run as a third build of the same release, beside PostgreSQL and SQLite. It runs against a single-node MongoDB 7.0 replica set and is held to the same TLS, SigV4 and IAM posture as the other two, so the storage engine is again the only thing that differs between them. ExtendDB's PostgreSQL row now lists its official Docker image beside the source.

A target whose run is not scored now publishes as one everywhere. Its badge reads not scored in grey rather than going missing, and its row in the results table reads not scored where the letter would be. Neither the table nor the board gives it a region count, because a cohort is only ever read off a scored headline, and its page says no region has a figure rather than naming a headline region. It still sits below every lettered row. The rule itself doesn't change: a run that gets no answer to any one test is not scored.

Phoenix-Adapters joins the board. It is an Apache project serving the DynamoDB API as a REST layer over Apache Phoenix on HBase, so the row measures an adapter over an external store rather than a self-contained emulator. There is no release yet, so it is built from main with the project's own Docker setup and the row records the commit it ran.

npm run suite:manifest no longer writes into results/. It lists the suite by running it against an endpoint nothing answers, and that run wrote its own "provisioning never completed" file beside the results of whichever target the shell named. With CONFORMANCE_TARGET exported, that target then showed as not scored until the file was deleted, and with nothing set it replaced the one a real local run had left. The listing run now keeps everything in a temporary directory and ignores the caller's target, results directory, endpoint and AWS settings (#342).

Five index tests now have a timeout longer than the waits inside them, which corrects an earlier note in this section: in these five an expired wait could still end as a failure. Two vector index tests could wait up to 20 seconds longer than their timeouts allowed, and three GSI validation tests waited up to 2 minutes for a new table inside the 30-second default. The test runner ended them first, so a slow target got a failure where it should have got no answer. No published result came near the old limits, so no verdict changes.

Floci's row no longer lists Scoop. Its link pointed at a repository that doesn't exist, and Floci's own install instructions don't mention Scoop.

Every target now names its licence. Each target's page shows it in How to run it, next to what the target needs, and each project's entry in the emulator directory shows it under What you need. In both places it is the licence's name, linked to its text or to the vendor's terms. Where a licence has an SPDX identifier, that identifier is the name shown. Builds of one project show the project's licence. Where the thing the suite runs is licensed differently from its source repository, the licence shown is the one for what is run. Each target in /data/latest.json carries the same fact as licence, with name, url and, where one exists, spdx. The one licence remark that sat in a target's run note has moved into the field, so no target carries a line about its licence that the others do not.

LocalStack's name in the results table, and the project links on its pages, now point at localstack.cloud rather than its GitHub repository, which is archived.

A test expecting an error now says so plainly when the request was accepted: expected ValidationException, but the request was accepted, where it used to read expected AssertionError: expected "should have thrown" ... to be an instance of DynamoDBServiceException. The verdict is the same, a failure. What does change is a request that got no answer while a test was waiting for an error. A throttle that outlasted the SDK's retries, a transport fault or a 5xx other than 501 (not implemented, which stays a real failure) used to be judged as the wrong error and recorded as a failure. It is now recorded as no answer, the same as anywhere else in the suite, so a run that meets one is not scored rather than carrying a failure for it. A test that expects one of those errors on purpose still passes when it gets it. This covers tests that use expectDynamoError. Tests that catch the error by hand are unchanged for now.

The sweep's confirmation re-runs now run the test they are meant to repeat (#332). On the 2026-10-31, 2026-11-07 and 2026-11-14 sweeps every confirmation re-run selected no test, so all 16 split candidates were discarded as not reproduced. A candidate that cannot be re-run is now reported on a confirmation-error issue rather than discarded.

The six TransactWriteItems and TransactGetItems tests on malformed numbers and duplicate set members now pin eu-west-2's answer, from the current validation layer, where they accepted either layer's. us-east-1, eu-west-1 and us-west-2, which gave the older layer's answer in October, have moved. On 2026-11-13 every region that answered gave the current layer's answer to all of them except ap-south-1 and me-central-1, and a run of the six tests on 2026-11-14 found ap-south-1 on the current layer as well. Six new rows in registry/splits.json record me-central-1 on the older layer, which reads 1e2147483647 as an underflow and prints set members as sent. An engine that answers these six in the older layer now fails them, and is credited only against me-central-1. me-central-1 has missed the sweeps since 2026-10-17 and is not among the regions the board scores against, so for now such an engine fails them everywhere the board shows. The answers are in captures/2026-11-14-transaction-value-layers.json.

A not-scored row now says why it is not scored, where it used to print its sentences with a dash in place of each figure. Its row on the board and its directory entry say how many of its tests got no answer, or that the whole run did. Its page and the text corpus say the same, and add the reason the suite recorded for each kind (a transport failure or 5xx, a table that never became active, and so on) and what the answered tests did. A row that implements none of the suite says that instead. The page lists the tests that got no answer by operation, each with its reason, and no longer lists them among the failures as divergences they never were. The counts above the lists, and the row's counts in the data endpoints, are read from the same results as the lists, so the failed count is the number of failures listed. The methodology gains a section stating the rule.

The rest of the row reads the same way. A run that got no answer at all lists every operation and feature as not scored, with no pass or fail counts beneath them and no findings, where it used to show figures tallied from tests the run never answered. A feature holding some tests that got no answer reads as not scored too, as an operation holding one already did, rather than wearing a support state read from the rest. The data endpoints add indeterminate to each row's counts, tiers, operation areas and features, so the counts add up to the total on a row that is not scored as on any other; give an area or feature where nothing was observed the state unanswered rather than unsupported; and withhold correctness, and each region's pass rate, wherever the other two figures are withheld. Movement says when a target loses its score or gets one, as not scored and scored, instead of reading unchanged on both runs.

Scored against 32 real regions. 2 dropped, out of scoring until it returns: me-central-1, me-south-1
  1. live (AWS)

    Runs via: AWS service

    Grade baseline
    0.0%Divergence
    100.0%Coverage

    no divergence in all 32 regions

    ground truth
    Tier breakdown
    Tier 1 · Core 0.0% diverges
    100.0% covered
    Tier 2 · Complete 0.0% diverges
    100.0% covered
    Tier 3 · Strict 0.0% diverges
    100.0% covered
  2. Runs via: npx, Docker, Homebrew, binary, npm, cargo, embedded, GitHub Action, source

    Maintained by the board’s author

    Grade A
    3.1%Divergence
    99.3%Coverage

    low divergence (3.1%) in 9 regions · up to 3.3% in the other 23

    diverged 1.5 percentage points more
    Tier breakdown
    Tier 1 · Core 0.8% diverges
    100.0% covered
    Tier 2 · Complete 1.9% diverges
    97.4% covered 14 unsupported
    Tier 3 · Strict 5.1% diverges
    100.0% covered
    Also built for WebAssembly / OPFS
    • Grade B

      low divergence (2.1%) in 9 regions · up to 2.3% in the other 23 · covers 89.4% · coverage lowers this row to B

  3. a0fbf944d7f3

    Runs via: Docker, Homebrew, install script, binary, JAR

    Grade A
    4.4%Divergence
    100.0%Coverage

    low divergence (4.4%) in 31 regions · up to 4.5% in the other 1

    diverged 4.9 percentage points less, grade B to A
    Tier breakdown
    Tier 1 · Core 0.8% diverges
    100.0% covered
    Tier 2 · Complete 11.1% diverges
    100.0% covered
    Tier 3 · Strict 2.5% diverges
    100.0% covered
  4. aef90fec1a45

    Runs via: pip, Docker, source

    Grade B
    8.4%Divergence
    100.0%Coverage

    moderate divergence (8.4%) in all 32 regions

    diverged 8.3 percentage points more, grade A to B
    Tier breakdown
    Tier 1 · Core 0.8% diverges
    100.0% covered
    Tier 2 · Complete 9.9% diverges
    100.0% covered
    Tier 3 · Strict 11.8% diverges
    100.0% covered
  5. 1.0.385

    Runs via: OS package, Docker, embedded, source

    Grade D
    26.0%Divergence
    100.0%Coverage

    very high divergence (26.0%) in 31 regions · up to 26.1% in the other 1

    new
    Tier breakdown
    Tier 1 · Core 3.9% diverges
    100.0% covered
    Tier 2 · Complete 24.8% diverges
    100.0% covered
    Tier 3 · Strict 39.4% diverges
    100.0% covered
  6. Runs via: Docker, pip, Homebrew, binary

    Grade D
    26.8%Divergence
    96.2%Coverage

    very high divergence (26.8%) in 1 region · up to 27.0% in the other 31

    diverged 7.3 percentage points more, grade C to D
    Tier breakdown
    Tier 1 · Core 1.0% diverges
    100.0% covered
    Tier 2 · Complete 23.5% diverges
    91.2% covered 47 unsupported
    Tier 3 · Strict 43.8% diverges
    97.0% covered 26 unsupported
  7. Grade D
    15.9%Divergence
    57.2%Coverage

    high divergence (15.9%) in 1 region · up to 16.0% in the other 31

    coverage lowers this row to D

    diverged 3.3 percentage points more
    Tier breakdown
    Tier 1 · Core 11.2% diverges
    100.0% covered
    Tier 2 · Complete 5.3% diverges
    7.5% covered 493 unsupported
    Tier 3 · Strict 25.0% diverges
    62.6% covered 329 unsupported
  8. Runs via: Docker, OS package, install script, binary, source

    Grade F
    28.5%Divergence
    58.3%Coverage

    very high divergence (28.5%) in 23 regions · up to 28.5% in the other 9

    coverage lowers this row to F

    new
    Tier breakdown
    Tier 1 · Core 20.1% diverges
    100.0% covered
    Tier 2 · Complete 4.5% diverges
    11.4% covered 472 unsupported
    Tier 3 · Strict 47.8% diverges
    62.6% covered 329 unsupported
  9. Runs via: Docker, binary, source

    Grade F
    44.7%Divergence
    67.6%Coverage

    severe divergence (44.7%) in all 32 regions

    diverged 0.3 percentage points more
    Tier breakdown
    Tier 1 · Core 46.9% diverges
    100.0% covered
    Tier 2 · Complete 11.1% diverges
    23.6% covered 407 unsupported
    Tier 3 · Strict 63.7% diverges
    75.5% covered 216 unsupported
  10. 6ada631ec7aa 79 unsupported

    Runs via: Docker, JAR, Maven

    Grade not scored
    3 of 1,920got no answer
    not scored this run
    Tier breakdown
    Tier 1 · Core 8.3% diverges
    100.0% covered
    Tier 2 · Complete not scored
    3 of 533 got no answer 61 unsupported
    Tier 3 · Strict 50.0% diverges
    98.0% covered 18 unsupported
  11. Runs via: Docker, source

    Grade not scored
    1 of 1,920got no answer
    not scored this run
    Tier breakdown
    Tier 1 · Core 0.8% diverges
    100.0% covered
    Tier 2 · Complete 2.3% diverges
    24.4% covered 403 unsupported
    Tier 3 · Strict not scored
    1 of 880 got no answer 216 unsupported
    Also built for SQLite and MongoDB
    • SQLite
      Grade D

      high divergence (15.4%) in 9 regions · up to 15.4% in the other 23 · covers 70.8% · coverage lowers this row to D

    • MongoDB
      Grade not scored

      not scored · 1 of 1,920 tests got no answer

  12. 04d5cbd9d541 119 unsupported

    Runs via: source

    Grade not scored
    603 of 1,920got no answer
    new
    Tier breakdown
    Tier 1 · Core not scored
    24 of 507 got no answer
    Tier 2 · Complete not scored
    351 of 533 got no answer 101 unsupported
    Tier 3 · Strict not scored
    228 of 880 got no answer 18 unsupported