How to run it
Every way Ministack is distributed, each linking to the project's own documentation for it. These are the project's claims, not the suite's measurements.
Needs: Python 3.10+ for the pip route; DynamoDB runs in-process, so no Docker
Per-region divergence
The headline is 21 regions, measured against 21 of 33 observed regions, and scored in ap-east-1. Real DynamoDB doesn't behave identically in every region, so a target can match some regions more closely than others. Each figure below is divergence in those regions, best first. eu-west-2 is marked as the historical baseline, and a region that couldn't be resolved this sweep is flagged rather than counted as a disagreement.
Fully supports
Operation areas this target implements completely - every test passes, nothing skipped. Areas it only partly gets wrong are in “Where it falls short” below; the ones it only partly attempts are in “What it doesn’t attempt”.
- batchGetItem tier 1
- createTable tier 1
- deleteItem tier 1
- deleteTable tier 1
- describeTable tier 1
- getItem tier 1
- listTables tier 1
- updateTable tier 1
- account tier 2
- backups tier 2
- contributorInsights tier 2
- export tier 2
- kinesis tier 2
- resourcePolicy tier 2
- streams tier 2
- tags tier 2
- ttl tier 2
- legacy-api tier 3
Divergence and coverage over time
Both are plotted because neither reads correctly alone. Divergence falls when a target stops attempting an operation it used to get wrong, so a fall in the first plot is only an improvement if the second one holds.
Divergence
Coverage
By operation
Every operation this target implements, with how much of it diverges and how much of it it covers, on the same two axes as the figures above. The counts are fails over the operation's whole size. full, partial, failing, unsupported. The matrix compares operations across targets.
Tier 1 - Core
- batchGetItem 0/15 0.0% 100.0% batchGetItem: supported, diverges on 0.0% of it, covers 100.0% (15 pass)
- batchWriteItem 1/18 5.6% 100.0% batchWriteItem: partially supported, diverges on 5.6% of it, covers 100.0% (17 pass, 1 fail)
- createTable 0/30 0.0% 100.0% createTable: supported, diverges on 0.0% of it, covers 100.0% (30 pass)
- deleteItem 0/13 0.0% 100.0% deleteItem: supported, diverges on 0.0% of it, covers 100.0% (13 pass)
- deleteTable 0/3 0.0% 100.0% deleteTable: supported, diverges on 0.0% of it, covers 100.0% (3 pass)
- describeTable 0/4 0.0% 100.0% describeTable: supported, diverges on 0.0% of it, covers 100.0% (4 pass)
- getItem 0/39 0.0% 100.0% getItem: supported, diverges on 0.0% of it, covers 100.0% (39 pass)
- listTables 0/5 0.0% 100.0% listTables: supported, diverges on 0.0% of it, covers 100.0% (5 pass)
- putItem 14/129 10.9% 100.0% putItem: partially supported, diverges on 10.9% of it, covers 100.0% (115 pass, 14 fail)
- query 7/92 7.6% 100.0% query: partially supported, diverges on 7.6% of it, covers 100.0% (85 pass, 7 fail)
- scan 5/56 8.9% 100.0% scan: partially supported, diverges on 8.9% of it, covers 100.0% (51 pass, 5 fail)
- updateItem 1/70 1.4% 100.0% updateItem: partially supported, diverges on 1.4% of it, covers 100.0% (69 pass, 1 fail)
- updateTable 0/15 0.0% 100.0% updateTable: supported, diverges on 0.0% of it, covers 100.0% (15 pass)
Tier 2 - Complete
- account 0/2 0.0% 100.0% account: supported, diverges on 0.0% of it, covers 100.0% (2 pass)
- backups 0/5 0.0% 100.0% backups: supported, diverges on 0.0% of it, covers 100.0% (5 pass)
- contributorInsights 0/2 0.0% 100.0% contributorInsights: supported, diverges on 0.0% of it, covers 100.0% (2 pass)
- export 0/2 0.0% 100.0% export: supported, diverges on 0.0% of it, covers 100.0% (2 pass)
- kinesis 0/1 0.0% 100.0% kinesis: supported, diverges on 0.0% of it, covers 100.0% (1 pass)
- partiql 25/76 32.9% 100.0% partiql: partially supported, diverges on 32.9% of it, covers 100.0% (51 pass, 25 fail)
- resourcePolicy 0/2 0.0% 100.0% resourcePolicy: supported, diverges on 0.0% of it, covers 100.0% (2 pass)
- streams 0/18 0.0% 100.0% streams: supported, diverges on 0.0% of it, covers 100.0% (18 pass)
- tags 0/8 0.0% 100.0% tags: supported, diverges on 0.0% of it, covers 100.0% (8 pass)
- transactions 7/62 11.3% 100.0% transactions: partially supported, diverges on 11.3% of it, covers 100.0% (55 pass, 7 fail)
- ttl 0/7 0.0% 100.0% ttl: supported, diverges on 0.0% of it, covers 100.0% (7 pass)
- updateTable 2/14 14.3% 100.0% updateTable: partially supported, diverges on 14.3% of it, covers 100.0% (12 pass, 2 fail)
- vectorSearch 0/28 · 28 skip n/a 0.0% vectorSearch: unsupported, implements none of it (28 skip)
Tier 3 - Strict
- error-messages 53/174 · 16 skip 30.5% 90.8% error-messages: partially supported, diverges on 30.5% of it, covers 90.8% (105 pass, 53 fail, 16 skip)
- legacy-api 0/42 0.0% 100.0% legacy-api: supported, diverges on 0.0% of it, covers 100.0% (42 pass)
- limits 10/93 10.8% 100.0% limits: partially supported, diverges on 10.8% of it, covers 100.0% (83 pass, 10 fail)
- validation-ordering 1/31 3.2% 100.0% validation-ordering: partially supported, diverges on 3.2% of it, covers 100.0% (30 pass, 1 fail)
Where it falls short
Operations Ministack implements and answers differently from real DynamoDB, biggest gap first. This is what its divergence figure counts. Open one for the exact tests, or see the conformance suite.
-
error-messages Tier 3 53 diverging 16 not attempted
View these tests in the suite →- BatchGetItem - exact error messages empty RequestItems: full required-parameter error source:26
- BatchGetItem - exact error messages mixing ProjectionExpression on one table and AttributesToGet on another is rejected source:164
- BatchGetItem - ProjectionExpression rejection messages duplicate paths (a, a) reject even when the key matches no item source:193
- BatchGetItem - ProjectionExpression rejection messages two distinct aliases resolving to one attribute (#a, #b -> a): rejected on the resolved names source:206
- BatchGetItem - ProjectionExpression rejection messages overlapping parent and child paths (a, a.b): full overlap message source:219
- BatchGetItem - ProjectionExpression rejection messages a bad projection on one table entry rejects the whole batch despite a clean entry on another source:248
- CreateTable - exact error messages PAY_PER_REQUEST with ProvisionedThroughput: full conflict message source:254
- CreateTable - exact error messages GSI INCLUDE projection without NonKeyAttributes: full missing-attributes message source:284
- CreateTable - exact error messages LSI INCLUDE projection without NonKeyAttributes: full missing-attributes message source:323
- CreateTable - exact error messages StreamSpecification StreamEnabled:false with a StreamViewType: full conflict message source:347
- GetItem - ProjectionExpression rejection messages raw duplicate paths (a, a): full overlap message with the path on both sides source:119
- GetItem - ProjectionExpression rejection messages same alias twice (#a, #a): full overlap message source:132
- GetItem - ProjectionExpression rejection messages two distinct aliases resolving to one attribute (#a, #b -> a): rejected on the resolved names source:150
- GetItem - ProjectionExpression rejection messages raw path plus alias for the same attribute (a, #a): full overlap message source:163
- GetItem - ProjectionExpression rejection messages raw parent and child paths (a, a.b): full overlap message source:176
- GetItem - ProjectionExpression rejection messages aliased parent and child paths (#a, #a.#b): full overlap message source:189
- GetItem - ProjectionExpression rejection messages child before parent (a.b, a): rejected with the paths in request order source:202
- GetItem - ProjectionExpression rejection messages cross-alias overlap (#x, #y.#b with #x and #y -> a): rejected on the resolved paths source:215
- GetItem - ProjectionExpression rejection messages deep overlap (a, a.b.c): full overlap message with the three-element path source:228
- GetItem - ProjectionExpression rejection messages list parent and list index (l, l[0]): rejected as an overlap source:245
- BatchWriteItem - index key error messages wrong-typed index key: full type-mismatch message source:54
- BatchWriteItem - index key error messages non-scalar index key: full type-mismatch message source:54
- BatchWriteItem - index key error messages empty-string index key: full secondary-index-key message source:54
- BatchWriteItem - index key error messages empty-binary index key: full secondary-index-key message source:54
- PutItem - index key error messages empty-binary index key value: full secondary-index-key message source:126
- UpdateItem - index key error messages empty-binary index key value: full secondary-index-key message source:150
- PartiQL - exact error messages DELETE RETURNING MODIFIED OLD * - exact message source:48
- PartiQL - exact error messages DELETE RETURNING ALL NEW * - exact message source:63
- PartiQL - exact error messages DELETE RETURNING MODIFIED NEW * - exact message source:78
- PartiQL - exact error messages ExecuteTransaction with a RETURNING member - exact message source:95
- PutItem - exact error messages duplicate zero-length members in BS: full duplicates error source:246
- Query - exact error messages Select SPECIFIC_ATTRIBUTES without ProjectionExpression: full required-projection message source:214
- Query - ProjectionExpression rejection messages duplicate paths (a, a) reject even when the key condition matches no partition source:250
- Query - ProjectionExpression rejection messages two distinct aliases resolving to one attribute (#a, #b -> a): rejected on the resolved names source:263
- Query - ProjectionExpression rejection messages overlapping parent and child paths (a, a.b) reject even when the key condition matches no partition source:276
- Scan - exact error messages Select SPECIFIC_ATTRIBUTES without ProjectionExpression: full required-projection message source:184
- Scan - ProjectionExpression rejection messages duplicate paths (a, a) reject even when the filter matches nothing source:218
- Scan - ProjectionExpression rejection messages two distinct aliases resolving to one attribute (#a, #b -> a): rejected on the resolved names source:231
- Scan - ProjectionExpression rejection messages overlapping parent and child paths (a, a.b) reject even when the filter matches nothing source:244
- TransactWriteItems - index key error messages Put wrong-typed index key: cancelled with full ValidationError reason source:54
- TransactWriteItems - index key error messages Put non-scalar index key: cancelled with full ValidationError reason source:54
- TransactWriteItems - index key error messages Update wrong-typed index key: cancelled with full ValidationError reason source:54
- TransactWriteItems - index key error messages Update non-scalar index key: cancelled with full ValidationError reason source:54
- TransactWriteItems - index key error messages Put empty-string index key: top-level ValidationException source:75
- TransactWriteItems - index key error messages Update empty-string index key: top-level ValidationException source:75
- TransactWriteItems - index key error messages Put empty-binary index key: top-level ValidationException source:75
- TransactWriteItems - index key error messages Update empty-binary index key: top-level ValidationException source:75
- TransactWriteItems - exact error messages Update wrong-typed Key: cancelled with schema-mismatch reason source:264
- TransactWriteItems - exact error messages Update non-scalar Key: cancelled with schema-mismatch reason source:264
- TransactWriteItems - exact error messages Delete wrong-typed Key: cancelled with schema-mismatch reason source:264
- TransactWriteItems - exact error messages Delete non-scalar Key: cancelled with schema-mismatch reason source:264
- TransactWriteItems - exact error messages ConditionCheck wrong-typed Key: cancelled with schema-mismatch reason source:264
- TransactWriteItems - exact error messages ConditionCheck non-scalar Key: cancelled with schema-mismatch reason source:264
- SearchVectors - exact error messages TopK above the maximum
- SearchVectors - exact error messages missing SearchConditionExpression against a HASH-schema index
- SearchVectors - exact error messages non-equality comparator on the HASH element
- SearchVectors - exact error messages non-equality comparator on an INLINE_FILTER element
- SearchVectors - exact error messages query vector dimension mismatch
- SearchVectors - exact error messages condition attribute outside the SearchSchema
- SearchVectors - exact error messages L-wrapped search vector
- CreateTable vector indexes - exact error messages vector index on a provisioned table
- CreateTable vector indexes - exact error messages SearchSchema attribute missing from AttributeDefinitions
- CreateTable vector indexes - exact error messages Dimensions above the 4096 maximum
- CreateTable vector indexes - exact error messages more vector indexes than the per-table limit
- CreateTable vector indexes - exact error messages two indexes on one attribute with different Dimensions
- PutItem vector writes - exact error messages vector with the wrong dimension count
- PutItem vector writes - exact error messages vector element that is not a number, naming the element
- PutItem vector writes - exact error messages vector attribute that is not a list
- PutItem vector writes - exact error messages empty string in the SearchSchema HASH attribute
-
partiql Tier 2 25 diverging
View these tests in the suite →- BatchExecuteStatement - PartiQL omits Item when a member UPDATE produces an empty MODIFIED projection source:235
- BatchExecuteStatement - PartiQL surfaces an invalid RETURNING variant on a member DELETE as a per-statement error source:259
- BatchExecuteStatement - PartiQL surfaces a malformed member statement as a per-statement ValidationError without failing the batch source:290
- ExecuteStatement - PartiQL SELECT with nested map path source:270
- ExecuteStatement - PartiQL DELETE rejects RETURNING MODIFIED OLD * (only ALL OLD * is valid on DELETE) source:637
- ExecuteStatement - PartiQL DELETE rejects RETURNING MODIFIED NEW * (only ALL OLD * is valid on DELETE) source:659
- ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED NEW * over a nested path returns only the changed leaf source:823
- ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED OLD * over a nested path returns only the changed leaf at its old value source:841
- ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED NEW * on a list index returns only the changed element source:940
- ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED OLD * on a list index returns only the prior element source:964
- ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED NEW * on a non-zero list index returns only the changed element source:998
- ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED OLD * on a non-zero list index returns the prior element source:1020
- ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED NEW * over multiple list indices returns a dense pack of the changed elements source:1044
- ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED OLD * over multiple list indices returns the prior elements densely source:1066
- ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED NEW * packs changed indices in ascending index order, not statement order source:1090
- ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED OLD * packs changed indices in ascending index order, not statement order source:1114
- ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED NEW * on an out-of-range list index (append) returns an empty Items list source:1136
- ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED NEW * appending at exactly the list length returns the appended element source:1181
- ExecuteStatement - PartiQL UPDATE REMOVE RETURNING MODIFIED OLD * on a list index returns the removed element at its old value source:1221
- ExecuteStatement - PartiQL UPDATE REMOVE RETURNING MODIFIED NEW * on a list index returns the shifted element, not an empty list source:1249
- ExecuteStatement - PartiQL UPDATE REMOVE RETURNING MODIFIED NEW * on the last list index returns an empty Items list source:1281
- ExecuteStatement - PartiQL UPDATE REMOVE RETURNING MODIFIED OLD * on the last list index returns the removed element source:1299
- ExecuteStatement - PartiQL UPDATE SET on a list index of an absent attribute is rejected, not auto-created source:1317
- ExecuteTransaction - PartiQL idempotent replay under the same ClientRequestToken does not double-apply source:189
- ExecuteTransaction - PartiQL rejects a RETURNING clause inside a transaction statement source:207
-
putItem Tier 1 14 diverging
View these tests in the suite →- PutItem - index write capacity reports the GSI write beside the table write and sums them source:44
- PutItem - index write capacity charges a sparse write nothing for the indexes it misses source:43
- PutItem - index write capacity reports the LSI write and folds it into the total source:44
- PutItem - index write capacity reports both arms when a write touches both index kinds source:43
- PutItem - index write capacity charges an identical overwrite no index writes at all source:43
- UpdateItem - index write-capacity ladder charges nothing on the index for a non-projected attribute source:44
- UpdateItem - index write-capacity ladder charges one index write for a projected non-key attribute source:43
- UpdateItem - index write-capacity ladder charges two index writes for a key change: delete plus insert source:43
- UpdateItem - index write-capacity ladder charges one index write to remove the key: delete only source:43
- UpdateItem - index write-capacity ladder walks the same ladder on the LSI key source:43
- DeleteItem - index write capacity charges one write per index the item occupied source:43
- DeleteItem - index write capacity charges only the table write for a sparse item source:43
- BatchWriteItem - index write capacity reports the per-table entry with the arms the batch actually touched source:43
- PutItem - index key validation rejects an empty-string index key attribute source:46
-
limits Tier 3 10 diverging
View these tests in the suite →- Expression size limit (4KB) - UpdateExpression rejects an UpdateExpression over the 4096-byte limit source:130
- Expression size limit (4KB) - ConditionExpression rejects a ConditionExpression over the 4096-byte limit source:161
- Expression size limit (4KB) - Query FilterExpression rejects a Query FilterExpression over the 4096-byte limit source:204
- Expression size limit (4KB) - Scan FilterExpression rejects a Scan FilterExpression over the 4096-byte limit source:239
- Expression size limit (4KB) - ProjectionExpression rejects a ProjectionExpression over the 4096-byte limit source:278
- Key-value length limits rejects an over-limit partition key on the read path source:31
- Nesting depth - 32-level document limit rejects a stored attribute nested 32 levels (leaf at level 33) source:51
- Nesting depth - 32-level document limit rejects a 32-level ExpressionAttributeValue with ValidationException source:99
- Response size limit (1MB) Query returns LastEvaluatedKey when response would exceed 1MB source:70
- Response size limit (1MB) Scan returns LastEvaluatedKey when response would exceed 1MB source:108
-
query Tier 1 7 diverging
View these tests in the suite →- ConsumedCapacity - INDEXES on a secondary-index Query Query with INDEXES returns per-index breakdown source:66
- ConsumedCapacity - INDEXES on a secondary-index Query Query with INDEXES returns per-index capacity breakdown source:85
- Query - KeyConditionExpression semantics rejects a nested path on a key attribute source:54
- Query - Select / ProjectionExpression rejections Select ALL_ATTRIBUTES with ProjectionExpression is rejected source:141
- Query - Select / ProjectionExpression rejections Select ALL_PROJECTED_ATTRIBUTES without an IndexName is rejected source:159
- Query - Select / ProjectionExpression rejections Select COUNT with ProjectionExpression is rejected source:176
- Query - Select / ProjectionExpression rejections Select ALL_PROJECTED_ATTRIBUTES with ProjectionExpression and no IndexName is rejected source:195
-
transactions Tier 2 7 diverging
View these tests in the suite →- TransactWriteItems - ConsumedCapacity: conditional, check, replay, cancel reports write capacity on the first call and read capacity on a same-token replay source:1417
- TransactWriteItems - index key validation Put with a wrong-typed index key cancels with a ValidationError reason source:60
- TransactWriteItems - index key validation Put with a non-scalar index key cancels with a ValidationError reason source:60
- TransactWriteItems - index key validation Update setting a wrong-typed index key cancels with a ValidationError reason source:60
- TransactWriteItems - index key validation Update setting a non-scalar index key cancels with a ValidationError reason source:60
- TransactWriteItems - index key validation Put with an empty-string index key is a top-level ValidationException source:117
- TransactWriteItems - index key validation Update setting an empty-string index key is a top-level ValidationException source:136
-
scan Tier 1 5 diverging
View these tests in the suite →- Scan - GSI sparse composite GSI: an item missing the GSI range key is not scanned source:169
- Scan - Select / ProjectionExpression rejections Select ALL_ATTRIBUTES with ProjectionExpression is rejected source:72
- Scan - Select / ProjectionExpression rejections Select ALL_PROJECTED_ATTRIBUTES without an IndexName is rejected source:87
- Scan - Select / ProjectionExpression rejections Select COUNT with ProjectionExpression is rejected source:101
- Scan - Select / ProjectionExpression rejections Select ALL_PROJECTED_ATTRIBUTES with ProjectionExpression and no IndexName is rejected source:117
-
updateTable Tier 2 2 diverging
View these tests in the suite →- UpdateTable - add GSI accepts a conflicting redeclaration of an existing key attribute and keeps the stored type source:475
- UpdateTable - GSI validation rejects a GSI keyed on a table attribute omitted from the request AttributeDefinitions source:745
-
batchWriteItem Tier 1 1 diverging
View these tests in the suite →- BatchWriteItem - index key validation rejects an empty-string index key value source:58
-
updateItem Tier 1 1 diverging
View these tests in the suite →- UpdateItem - ReturnValues granularity UPDATED_NEW on a nested SET returns only the changed fragment source:962
-
validation-ordering Tier 3 1 diverging
View these tests in the suite →- UpdateItem - validation ordering rejects invalid ReturnValues (UpdateItem reports the first enum error) source:54
What it doesn't attempt
Operations Ministack declines rather than gets wrong, so none of this counts towards its divergence. It is what the gap in its 95.8% coverage is made of. Each test here skipped itself because the target's own feature probe said the operation isn't implemented, which is often a deliberate choice rather than a defect.
-
vectorSearch Tier 2 none of it 28 not attempted
View these tests in the suite →- SearchVectors - ConsumedCapacity shape reports VectorSearchRequestBytes with no classic capacity fields
- SearchVectors - ConsumedCapacity shape reports the same shape under INDEXES as under TOTAL
- SearchVectors - ConsumedCapacity shape reports nothing under NONE
- PutItem - vector write capacity shape INDEXES adds a per-index VectorWriteRequestBytes map beside the classic fields
- PutItem - vector write capacity shape TOTAL folds nothing in: no vector fields at all
- PutItem - vector write capacity shape a write not touching the vector attribute reports no vector write
- PutItem - vector write capacity shape an identical overwrite reports no vector write: replication is delta-based
- CreateTable - vector index lifecycle walks CREATING to ACTIVE and describes the index faithfully
- CreateTable - vector index lifecycle round-trips a SearchSchema of HASH and INLINE_FILTER elements
- CreateTable - vector index lifecycle accepts the 4096-dimension boundary
- SearchVectors - index readiness after CreateTable rejects retryably between ACTIVE and the first served search
- PartiQL - vector indexes are out of reach rejects a PartiQL read of a vector index
- PartiQL - vector indexes are out of reach reads the vector attribute off the base table like any other item
- SearchVectors - deterministic search behaviour COSINE: identical vector scores 0, opposite scores 2, lower is closer
- SearchVectors - deterministic search behaviour EUCLIDEAN: identical vector scores 0, straight-line distance otherwise
- SearchVectors - deterministic search behaviour DOT_PRODUCT: higher is closer and scores can be negative
- SearchVectors - deterministic search behaviour returns the identical vector as the top match
- SearchVectors - deterministic search behaviour omits the vector attribute by default and returns it when projected
- SearchVectors - deterministic search behaviour caps results at the item count with no pagination surface
- SearchVectors - deterministic search behaviour stores full precision on the base table and f32 in the index
- UpdateTable - vector index lifecycle adds an index that backfills like a GSI, enforces one online action, then deletes it
- UpdateTable - vector index lifecycle refuses to delete the table while a vector index is still being created
- CreateTable - vector index request validation rejects Dimensions of 0 at the request model layer
- CreateTable - vector index request validation rejects an index name shorter than 3 characters
- SearchVectors - request validation rejects TopK of 0 at the request model layer
- SearchVectors - request validation rejects a search against an index the table does not have
- PutItem - vector index write validation accepts a write missing the HASH attribute but leaves it unreachable through that index
- PutItem - vector index write validation removes an item from the index when its vector attribute is removed
Run history
Every percentage here is divergence, per tier and over the whole suite, so lower is better in each column. Coverage is the exception it is named as.
| Run | Gradecurrent criteria | Divergence | Movement |
|---|---|---|---|
| Grade B | 11.8% | unchanged | |
| Grade B | 11.8% | unchanged | |
|
Suite on AWS corrected the vector index readiness documentation, prompted by a write-up of the earlier guidance that drew on the suite's measurements. The ACTIVE-plus-backfilling state the old advice was built around, and which no index ever occupies, is gone from the three pages that described it. The wait now reads "Backfilling is not true" rather than "is false", so a check written literally from it fires on both creation paths instead of neither. The tutorial no longer says a search during backfill can return incomplete results. Two things the suite had measured but nobody had written down are documented as well: that DescribeTable reporting ACTIVE leads the dedicated search endpoint, and that the readiness check depending on neither status field is a real search in a retry loop. That contract is now pinned rather than described. The UpdateTable walk asserts that Backfilling true is only ever reported alongside CREATING, that an ACTIVE index reports no Backfilling field at all, and that the base table goes ACTIVE while the index is still building, which is what makes a table waiter the wrong gate for a search. The first search that succeeds has to carry every seeded item, since the backfill window answers with an error rather than a partial view. On the CreateTable path a new test runs the documented check the way an application would, and every rejection before the first served response has to be the retryable ValidationException rather than a not-found. The suite's own search wait now absorbs those two rejections and rethrows every other answer, so a fixture waiting on an index that is ACTIVE but not yet served no longer fails on the lag it was waiting out. Two files asserting exact rejection messages wait for a served search rather than for ACTIVE: "does not have the specified index" is also what a freshly ACTIVE index says, and it would otherwise stand in for whichever message the case asked for. The tutorial's other new claim, that a table cannot be deleted while a vector index is being created, has a test of its own. Only the UpdateTable path can ask it. Across three runs an index created with its table reached ACTIVE in the same 250ms poll as the table, so the table is never ACTIVE with the index still building there, and a DeleteTable during creation is refused for the table's own status rather than with the documented index wording. Adding an index to a live table opens that window about thirty seconds in. The test cancels the index afterwards instead of waiting out the backfill, which a still-creating vector index turns out to accept the way a backfilling GSI does, so it costs a minute rather than seventeen. A release now dispatches its measurement as the results bot rather than with
the workflow's own token. GitHub raises no |
Grade B | 11.8% | unchanged |
|
Suite on The board now measures the most recent release tag rather than Every board now says what produced it. A board is graded against the suite manifest and split registry as they stood at the ref it measured. Region health is the exception and is read live, so a region dropped since the tag still counts against today's cohorts. That is the one input allowed to move under a board without a new measurement, and the board carries a health date beside its measurement date to say so. Releases are cut by one workflow dispatch: it bumps the version, dates this section, installs against the bumped tree, tags, and opens a draft release, then starts the measurement. The draft publishes itself when the board carrying that version lands, which takes about three hours. eu-west-2 has crossed to the validation framework's generic constraint message
for The matching validation-ordering row is retired. Both wordings refuse an empty
A probe absent from the baseline is no longer reported as drift. Adding a probe to the capture script leaves every older baseline without it, and a scheduled red then named the new probe as the thing that had moved. The weekly cross-region capture now includes eu-west-2, so the drift lens reads its baseline and the candidate regions from one capture taken at one moment rather than comparing today's candidates against an older baseline file. A scheduled red also keeps the eu-west-2 capture its drift verdict was read from, which was previously discarded with the runner. |
Grade B | 11.9% | unchanged |
| Grade B | 11.9% | unchanged | |
|
Suite on ExtendDB's SQLite backend joins the run, built from the same release as the PostgreSQL one and held to the same TLS, SigV4 and IAM posture, so the storage engine is the only thing that differs between them. A project's other builds now sit behind a disclosure on its row. Every build is measured in full and has a row of its own with its own figures; the disclosure starts closed only when every build under it reads the same grade, divergence and coverage as the row above, and only when each of them was measured in that run: a carried row on either side opens it, and so does a run the suite declined to score. It is read from each run, so a build can start closed on one and open on the next. The README table has no disclosure to offer, so it lists every build outright. Every target in the data endpoints gains two fields.
The |
Grade B | 11.9% | unchanged |
| Grade B | 11.9% | diverged 0.6 percentage points more | |
|
Suite on Read this first if you consume the JSON. The data endpoints go from schema 2 to schema 4 in one step. Schema 3 was never published on its own, so everything on 2 crosses both steps at once. Schema 3 breaks in four ways:
Schema 4 is additive on top: each target carries its grade and the full criteria
in A score is two figures, never oneDivergence is the share of the whole suite a target answers differently from real DynamoDB. Coverage is the share it implements at all. They are reported apart and never summed, because a declined operation is discoverable in minutes and a wrong one in production. No target was re-run for the change and no pass, fail or skip moved: what changed is how the same counts are expressed. Everything else that was a percentage followed the headline down - tier figures, the per-region drilldown, the per-operation table, and the colour bands, which inverted with them. A target's history is two plots rather than one, because divergence falls when a target stops attempting something it used to get wrong, so a divergence line alone can render a withdrawal as an improvement. Every target wears a letterDivergence sets it - A under 5%, B under 15%, C under 25%, D under 35%, F beyond
These are grading criteria version 1, dated in the methodology, which carries the derivation. Where a threshold sits is a hand-picked input to a published letter, and moving one regrades targets whose results never changed, so any change to a band, the coverage weight or the A+ gate bumps the version. The suite grows from 998 tests to 1054Vector search (#125). 42 tests over the deterministic surface DynamoDB
shipped in August 2026: the index lifecycle on both creation paths, request
validation on each plane, rejection wording, write-path validation, search on a
fixture where the nearest neighbour is unambiguous, the two new capacity shapes,
and PartiQL's inability to reach a vector index. Every pinned value was
characterised against real DynamoDB in eu-west-2 before it was asserted. Two
findings worth naming: searching during a backfill is an error, which settles
which side of a contradiction in AWS's own documentation is right - three
developer-guide pages say the call fails, the tutorial page says results can be
incomplete - and an overwrite leaving the stored vector unchanged reports no
vector write capacity at all, because index replication is delta-based. Both
sides are captured in Index write costs (#124). 14 tests. The suite's only per-index capacity assertion was on a Query, so the write side - the half you get billed extra for - went unmeasured. A sub-1KB write costs one unit for the table and one for each index it lands in, and LSI units fold into the total exactly as GSI units do. Moving an item to a new GSI key costs two on that index, a delete and an insert; touching a projected attribute costs one; touching a non-projected attribute costs nothing, and the response carries no arm for that index rather than a zero. An overwrite that leaves the item unchanged reports no index cost whatsoever. The index exclusions create no indexed table (#116). Corrections
|
Grade B | 11.2% | unchanged |
| Grade B | 11.2% | diverged 4.9 percentage points less | |
| Grade C | 16.1% | unchanged | |
| Grade C | 16.1% | diverged 0.2 percentage points more | |
| Grade C | 15.9% | unchanged | |
| Grade C | 15.9% | diverged 2.1 percentage points less | |
| Grade C | 18.0% | unchanged | |
| Grade C | 18.0% | diverged 1.3 percentage points more | |
|
Suite on Grew to 982 tests, up 28, all characterised against real DynamoDB across the per-region ground truth 2.0.0 put in place. New coverage is PartiQL's RETURNING clause; the GSI lifecycle also joins the ground truth, and the sweep's drift classifier gains a converged case.
|
Grade C | 16.7% | diverged 2.3 percentage points more |
| Grade B | 14.4% | unchanged | |
| Grade B | 14.4% | unchanged | |
|
Suite on Per-region scoring lands complete. 2.0.0-pre put the scoring logic in place, comparing each target against every region's recorded answer, but the evidence half was never wired: no test recorded what a target actually answered and the classifier never read one, so a fail could not be credited to a region the target matched and the score could only ever subtract. 2.0.0 closes that loop, and the seed split runs its whole lifecycle in the same release. What changed:
|
Grade B | 14.4% | diverged 0.1 percentage points less |
| Grade B | 14.5% | unchanged | |
| Grade B | 14.5% | unchanged | |
| Grade B | 14.5% | diverged 1.1 percentage points more | |
|
Suite on The scores barely move in this release, but what they mean has changed. Until now the suite pinned one region, eu-west-2, as ground truth. That was
quietly unfair: real DynamoDB disagrees with itself in a handful of places,
and a one-region baseline takes a side without saying so. The clearest case
is the What changed:
One deliberate departure from the RFC that proposed this (#75): the RFC suggested a behaviour conforms if it matches any real region. 2.0.0 scores each target against one region at a time and headlines the best match, so a target only passes a behaviour when at least one real region does what it did, and its headline reflects one coherent region rather than a mix. Match-any scoring would have accepted an engine that combines eu-west-2's answer on one behaviour with us-east-1's on another - a deployment that exists nowhere. That is stricter than the RFC asked for, and it is deliberate. No score moves at release: the one admitted split pins eu-west-2, which is
the only region in the health record until the first sweep runs. Per-target
deltas will be published once the sweep admits more regions; the expected
movement is roughly a tenth of a percent for the six engines that match
us-east-1 on the The suite also grew to 954 tests, up 81, all characterised against real DynamoDB - the control-plane pins in eu-west-2, everything else across four regions (eu-west-2, eu-central-1, us-east-1, ap-southeast-2):
|
Grade B | 13.4% | diverged 0.1 percentage points less |
| Grade B | 13.5% | unchanged |
Showing the 24 most recent runs. The chart above covers the full history, and every run is browsable from Runs.