Run
14 emulators 1,920 tests
Results as of this run. The arrow shows each target's movement since the previous run it was tested in. The suite grew this run, so an upward arrow can be the new tests biting rather than a target getting worse.
Suite grew from 1,678 to 1,920 tests this run.
That's 242 new tests measured against every target. Movement below compares to the previous run, so a rise here is as likely to be the stricter suite as a real regression.
What changed in the suite this run
Suite on
ca-central-1 joins eu-west-2 on the five Scan rows. It gave the new wording on the 2026-11-07 and 2026-11-14 sweeps and on the three runs on 2026-11-13, which leaves me-central-1 the only region recorded on the old wording. On the 2026-11-14 sweep eu-north-1 stored UpdateItem writes that the statement-sized rule refuses, which it had not done before. Three runs later that day had it answering both ways, so the item-size rows stay as they are.
Ten regions join eu-west-2 on the five Scan rows: ap-northeast-1, ap-south-2, ap-southeast-6, ap-southeast-7, eu-central-2, eu-south-1, eu-south-2, eu-west-1, us-east-2 and us-west-2. Each gave the new wording on the 2026-11-07 sweep and at least one sweep before it, and on three runs against every region on 2026-11-13, which leaves ca-central-1 and me-central-1 on the old wording. ca-central-1 gave the new wording on the 2026-11-07 sweep and on all three runs, but the old wording on the two sweeps before, so it keeps its recorded answer until a second sweep agrees. eu-north-1 gave the old wording on one of the three runs, so it answers Scan both ways for now and keeps its recorded answer, the new wording. The BatchGetItem row doesn't change: me-central-1 gave both answers to an empty RequestItems across the same runs.
eu-west-2 now accepts a 32-level ExpressionAttributeValue in an UpdateItem
condition, so the test that pinned its refusal now pins acceptance, and is
renamed to say so. It asserts the ConditionalCheckFailedException a false
condition gives, the same proof the 31-level test uses. The 2026-11-14 sweep
found 19 regions accepting a value they had refused the week before, and by the
last of three runs that day 23 did. ap-east-1, ap-southeast-2, ca-west-1,
eu-south-2, eu-west-3, mx-central-1, us-east-1, us-west-1 and us-west-2 still
refuse it, and me-central-1, which had accepted it until then, now refuses it
too. The registry records those ten on that side. The same regions moved
together on UpdateItem's ReturnValues error, which now lists
UPDATED_NEW where it listed UPDATE_NEW. That test accepts both, and its row
now pins the new wording. A stored item is still capped at 32 levels
everywhere, and a 63-level value is still refused.
The wait for an index added by UpdateTable to become ACTIVE is now 2 minutes on any emulator, down from 40. Real DynamoDB keeps 40 minutes. The slowest passing index test on any emulator in the published results takes about 31 seconds, against about 25 minutes on real DynamoDB, so the shorter wait only shortens runs where an index never completes. An expired wait is recorded as no answer either way, never as a failure, so no score can change.
PartiQL reads whose OR branches stay in one partition now have paging and
capacity tests. Under Limit, a set of sort keys ends on its last key, a
missing last key keeps the token for one empty half-unit page, and a branch the
read hasn't reached holds the token. Each shape pinned page by page gave the
same pages on every walk in eu-west-2 and us-east-1 between 2026-11-03 and
2026-11-05. Page endings that DynamoDB answers at random, such as a page ending
inside an open range, are checked only for the keys they return. Each branch
costs a unit under ConsistentRead even when it finds nothing, a sort-key IN
costs and pages exactly as the equivalent OR does, and a partition key that
can't be a key reads the whole table, so beside sort-key ranges it costs what
that read does: 2.0, or 4.0 consistent, on the small on-demand table the test
uses.
Items tied on an index key come back in a fixed order that isn't the table's sort-key order, and descending is its exact reverse. The order was the same in nine fresh tables across two regions and two AWS accounts, whatever order the items were written in, so Query on an LSI and a GSI, and PartiQL on an LSI, now pin it.
The PartiQL refusals these reads run into are pinned word for word: a negated
sort key inside an OR filter, a partition key given the wrong type or a
boolean, ORDER BY over branches that each name the partition key, a KEYS_ONLY
index asked to filter on an attribute it doesn't project, and ExecuteStatement
Limit values of 0, -1 and a string. What DynamoDB accepts is pinned beside
them: ORDER BY on an OR in one partition, the same key written twice, and
Limit values past 2^31, past 2^64 and written with an exponent or a fraction.
An attribute name starting with an underscore is a syntax error in every
expression type unless it goes through ExpressionAttributeNames, including
key conditions. An UpdateItem that creates its item returns no Attributes
under ALL_OLD or UPDATED_OLD.
Deep PutItem, UpdateItem and DeleteItem requests, read by the older validation
layer, now have their joined errors pinned: every bad member, in each
operation's own order, under the N validation errors detected envelope even
when only one member is wrong.
A table name's length is counted in UTF-16 units on deep writes and on CreateTable, DescribeTable, DeleteTable and UpdateTable, and in characters on shallow item operations, ListTables' start name and index names. So two emoji are long enough for CreateTable and too short for PutItem. Deep writes also check the name in two stages: a missing or empty name is listed among the member errors, while length and pattern errors only show when nothing else is wrong.
Fourteen new rows in registry/splits.json record where me-central-1, and
eu-north-1 on Scan, gave the older validation layer's answer to the shallow
table-name and leading-underscore cases on 2026-11-05. The other regions agree
with eu-west-2. The answers from all 33 regions on 2026-11-05 are in
captures/2026-11-05-shallow-validation-rollout.json. On three later runs on
2026-11-13, me-central-1 gave eu-west-2's answer to the twelve cases that aren't
Scan (once cut short, on UpdateItem) and eu-north-1 gave it on Scan, so the rows
keep the 2026-11-05 answers until a sweep has run the tests. Those runs are in
captures/2026-11-13-shallow-validation-follow-up.json.
Four more PartiQL index-qualifier answers are pinned. An unknown index read with
ConsistentRead is reported as an unknown index, not as a consistent read on a
GSI. Selecting an attribute a GSI doesn't project is refused on an unkeyed scan
as well as on a keyed read. With two such attributes the refusal lists them in
the same order whichever way round the statement names them, and that order
isn't alphabetical, so the test pins the list for one pair. A
BatchExecuteStatement SELECT on a composite table that gives the partition key
but not the sort key is refused like any other member without the full primary
key. All four gave the same answers on three runs each in eu-west-2 and
us-east-1 on 2026-11-14.
Renaming or removing a released test, or changing a region's recorded answer, no longer fails the checks on its own pull request. The committed results are now checked against the suite and registry as they stood at the release they were measured at, as the rest of the published board already was, rather than against main. The change reaches the board at the next release, when every target is re-run. A board is now refused if real DynamoDB's own run, in its own region, fails a test or passes one on a split row with an answer recorded for a different region.
A project's own row on the board now names its configuration beside the
project, as the README table already did, so ExtendDB's row reads "ExtendDB ·
PostgreSQL" before its other builds are opened. The ways to run a target gain
one more, OS package, for an operating system package: a .deb or .rpm, or an
apt or dnf repository.
AWS Emulator joins the board, suggested by @exoego in #304. It is AWS's own multi-service emulator, written in Rust, with DynamoDB and DynamoDB Streams among the services it serves. It installs from a Linux package, or on other systems runs in Docker from an image built with the Dockerfile in its user guide. It serves HTTPS and checks SigV4 signatures by default, with a certificate and key it generates on each start, and it is measured that way.
Alternator joins the board. It is ScyllaDB's DynamoDB-compatible API, built into the database node itself rather than run as a separate emulator, with global secondary indexes on materialised views and Streams on change data capture. It runs from the official ScyllaDB container image, and its version column shows the ScyllaDB release the node reports.
ExtendDB's MongoDB backend joins the run as a third build of the same release, beside PostgreSQL and SQLite. It runs against a single-node MongoDB 7.0 replica set and is held to the same TLS, SigV4 and IAM posture as the other two, so the storage engine is again the only thing that differs between them. ExtendDB's PostgreSQL row now lists its official Docker image beside the source.
A target whose run is not scored now publishes as one everywhere. Its badge
reads not scored in grey rather than going missing, and its row in the results
table reads not scored where the letter would be. Neither the table nor the
board gives it a region count, because a cohort is only ever read off a scored
headline, and its page says no region has a figure rather than naming a
headline region. It still sits below every lettered row. The rule itself
doesn't change: a run that gets no answer to any one test is not scored.
Phoenix-Adapters joins the board. It is an Apache project serving the DynamoDB
API as a REST layer over Apache Phoenix on HBase, so the row measures an adapter
over an external store rather than a self-contained emulator. There is no
release yet, so it is built from main with the project's own Docker setup and
the row records the commit it ran.
npm run suite:manifest no longer writes into results/. It lists the suite by
running it against an endpoint nothing answers, and that run wrote its own
"provisioning never completed" file beside the results of whichever target the
shell named. With CONFORMANCE_TARGET exported, that target then showed as not
scored until the file was deleted, and with nothing set it replaced the one a
real local run had left. The listing run now keeps everything in a temporary
directory and ignores the caller's target, results directory, endpoint and AWS
settings
(#342).
Five index tests now have a timeout longer than the waits inside them, which corrects an earlier note in this section: in these five an expired wait could still end as a failure. Two vector index tests could wait up to 20 seconds longer than their timeouts allowed, and three GSI validation tests waited up to 2 minutes for a new table inside the 30-second default. The test runner ended them first, so a slow target got a failure where it should have got no answer. No published result came near the old limits, so no verdict changes.
Floci's row no longer lists Scoop. Its link pointed at a repository that doesn't exist, and Floci's own install instructions don't mention Scoop.
Every target now names its licence. Each target's page shows it in How to run
it, next to what the target needs, and each project's entry in the emulator
directory shows it under What you need. In both places it is the licence's name,
linked to its text or to the vendor's terms. Where a licence has an SPDX
identifier, that identifier is the name shown. Builds of one project show the
project's licence. Where the thing the suite runs is licensed differently from
its source repository, the licence shown is the one for what is run. Each target
in /data/latest.json carries the same fact as licence, with name, url
and, where one exists, spdx. The one licence remark that sat in a target's
run note has moved into the field, so no target carries a line about its
licence that the others do not.
LocalStack's name in the results table, and the project links on its pages, now point at localstack.cloud rather than its GitHub repository, which is archived.
A test expecting an error now says so plainly when the request was accepted:
expected ValidationException, but the request was accepted, where it used to
read expected AssertionError: expected "should have thrown" ... to be an instance of DynamoDBServiceException. The verdict is the same, a failure. What
does change is a request that got no answer while a test was waiting for an
error. A throttle that outlasted the SDK's retries, a transport fault or a 5xx
other than 501 (not implemented, which stays a real failure) used to be judged
as the wrong error and recorded as a failure. It is now recorded as no answer,
the same as anywhere else in the suite, so a run that meets one is not scored
rather than carrying a failure for it. A test that expects one of those errors
on purpose still passes when it gets it. This covers tests that use
expectDynamoError. Tests that catch the error by hand are unchanged for now.
The sweep's confirmation re-runs now run the test they are meant to repeat
(#332). On
the 2026-10-31, 2026-11-07 and 2026-11-14 sweeps every confirmation re-run
selected no test, so all 16 split candidates were discarded as not reproduced.
A candidate that cannot be re-run is now reported on a confirmation-error
issue rather than discarded.
The six TransactWriteItems and TransactGetItems tests on malformed numbers and
duplicate set members now pin eu-west-2's answer, from the current validation
layer, where they accepted either layer's. us-east-1, eu-west-1 and us-west-2,
which gave the older layer's answer in October, have moved. On 2026-11-13 every
region that answered gave the current layer's answer to all of them except
ap-south-1 and me-central-1, and a run of the six tests on 2026-11-14 found
ap-south-1 on the current layer as well. Six new rows in registry/splits.json
record me-central-1 on the older layer, which reads 1e2147483647 as an underflow
and prints set members as sent. An engine that answers these six in the older
layer now fails them, and is credited only against me-central-1. me-central-1
has missed the sweeps since 2026-10-17 and is not among the regions the board
scores against, so for now such an engine fails them everywhere the board shows.
The answers are in captures/2026-11-14-transaction-value-layers.json.
A not-scored row now says why it is not scored, where it used to print its sentences with a dash in place of each figure. Its row on the board and its directory entry say how many of its tests got no answer, or that the whole run did. Its page and the text corpus say the same, and add the reason the suite recorded for each kind (a transport failure or 5xx, a table that never became active, and so on) and what the answered tests did. A row that implements none of the suite says that instead. The page lists the tests that got no answer by operation, each with its reason, and no longer lists them among the failures as divergences they never were. The counts above the lists, and the row's counts in the data endpoints, are read from the same results as the lists, so the failed count is the number of failures listed. The methodology gains a section stating the rule.
The rest of the row reads the same way. A run that got no answer at all lists
every operation and feature as not scored, with no pass or fail counts beneath
them and no findings, where it used to show figures tallied from tests the run
never answered. A feature holding some tests that got no answer reads as not
scored too, as an operation holding one already did, rather than wearing a
support state read from the rest. The data endpoints add indeterminate to each
row's counts, tiers, operation areas and features, so the counts add up to the
total on a row that is not scored as on any other; give an area or feature where
nothing was observed the state unanswered rather than unsupported; and
withhold correctness, and each region's pass rate, wherever the other two
figures are withheld. Movement says when a target loses its score or gets one,
as not scored and scored, instead of reading unchanged on both runs.
-
Grade baselineground truth0.0%Divergence100.0%Coverage
no divergence in all 32 regions
Tier breakdown
Tier 1 · Core 0.0% diverges100.0% coveredTier 2 · Complete 0.0% diverges100.0% coveredTier 3 · Strict 0.0% diverges100.0% covered -
Grade Adiverged 1.5 percentage points more3.1%Divergence99.3%Coverage
low divergence (3.1%) in 9 regions · up to 3.3% in the other 23
Tier breakdown
Tier 1 · Core 0.8% diverges100.0% coveredTier 2 · Complete 1.9% diverges97.4% covered 14 unsupportedTier 3 · Strict 5.1% diverges100.0% coveredAlso built for WebAssembly / OPFS
-
Grade B
low divergence (2.1%) in 9 regions · up to 2.3% in the other 23 · covers 89.4% · coverage lowers this row to B
-
-
Grade Adiverged 4.9 percentage points less, grade B to A4.4%Divergence100.0%Coverage
low divergence (4.4%) in 31 regions · up to 4.5% in the other 1
Tier breakdown
Tier 1 · Core 0.8% diverges100.0% coveredTier 2 · Complete 11.1% diverges100.0% coveredTier 3 · Strict 2.5% diverges100.0% covered -
Grade Bdiverged 8.3 percentage points more, grade A to B8.4%Divergence100.0%Coverage
moderate divergence (8.4%) in all 32 regions
Tier breakdown
Tier 1 · Core 0.8% diverges100.0% coveredTier 2 · Complete 9.9% diverges100.0% coveredTier 3 · Strict 11.8% diverges100.0% covered -
Grade Dnew26.0%Divergence100.0%Coverage
very high divergence (26.0%) in 31 regions · up to 26.1% in the other 1
Tier breakdown
Tier 1 · Core 3.9% diverges100.0% coveredTier 2 · Complete 24.8% diverges100.0% coveredTier 3 · Strict 39.4% diverges100.0% covered -
Grade Ddiverged 7.3 percentage points more, grade C to D26.8%Divergence96.2%Coverage
very high divergence (26.8%) in 1 region · up to 27.0% in the other 31
Tier breakdown
Tier 1 · Core 1.0% diverges100.0% coveredTier 2 · Complete 23.5% diverges91.2% covered 47 unsupportedTier 3 · Strict 43.8% diverges97.0% covered 26 unsupported -
Grade Ddiverged 3.3 percentage points more15.9%Divergence57.2%Coverage
high divergence (15.9%) in 1 region · up to 16.0% in the other 31
coverage lowers this row to D
Tier breakdown
Tier 1 · Core 11.2% diverges100.0% coveredTier 2 · Complete 5.3% diverges7.5% covered 493 unsupportedTier 3 · Strict 25.0% diverges62.6% covered 329 unsupported -
Grade Fnew28.5%Divergence58.3%Coverage
very high divergence (28.5%) in 23 regions · up to 28.5% in the other 9
coverage lowers this row to F
Tier breakdown
Tier 1 · Core 20.1% diverges100.0% coveredTier 2 · Complete 4.5% diverges11.4% covered 472 unsupportedTier 3 · Strict 47.8% diverges62.6% covered 329 unsupported -
Grade Fdiverged 0.3 percentage points more44.7%Divergence67.6%Coverage
severe divergence (44.7%) in all 32 regions
Tier breakdown
Tier 1 · Core 46.9% diverges100.0% coveredTier 2 · Complete 11.1% diverges23.6% covered 407 unsupportedTier 3 · Strict 63.7% diverges75.5% covered 216 unsupported -
Grade not scorednot scored this run3 of 1,920got no answer
Tier breakdown
Tier 1 · Core 8.3% diverges100.0% coveredTier 2 · Complete not scored3 of 533 got no answer 61 unsupportedTier 3 · Strict 50.0% diverges98.0% covered 18 unsupported -
Grade not scorednot scored this run1 of 1,920got no answer
Tier breakdown
Tier 1 · Core 0.8% diverges100.0% coveredTier 2 · Complete 2.3% diverges24.4% covered 403 unsupportedTier 3 · Strict not scored1 of 880 got no answer 216 unsupported -
Grade not scorednew603 of 1,920got no answer
Tier breakdown
Tier 1 · Core not scored24 of 507 got no answerTier 2 · Complete not scored351 of 533 got no answer 101 unsupportedTier 3 · Strict not scored228 of 880 got no answer 18 unsupported