Skip to content

DynamoDB Local

docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBLocal.html

ff89bd48ff32 46 recorded runs

Grade C
high divergence
answers 15.1% of the suite differently from real DynamoDB
up to 15.2% in the others
93.8% of the suite implemented · 65 tests it doesn't attempt
Tier 1 · Core 7.6% diverges
100.0% covered
37 of 489 attempted
Tier 2 · Complete 14.1% diverges
78.4% covered 49 unsupported
32 of 178 attempted
Tier 3 · Strict 26.5% diverges
95.3% covered 16 unsupported
90 of 324 attempted

How to run it

Every way DynamoDB Local is distributed, each linking to the project's own documentation for it. These are the project's claims, not the suite's measurements.

Needs: JRE 17+ (bundled in the Docker route)

Per-region divergence

The headline is 25 regions, measured against 25 of 33 observed regions, and scored in af-south-1. Real DynamoDB doesn't behave identically in every region, so a target can match some regions more closely than others. Each figure below is divergence in those regions, best first. eu-west-2 is marked as the historical baseline, and a region that couldn't be resolved this sweep is flagged rather than counted as a disagreement.

15.1% 25 regions
af-south-1ap-east-1ap-northeast-1ap-northeast-3ap-south-1ap-south-2ap-southeast-2ap-southeast-3ap-southeast-4ap-southeast-5ap-southeast-6ap-southeast-7ca-central-1ca-west-1eu-central-2eu-south-1eu-south-2eu-west-1eu-west-3il-central-1mx-central-1sa-east-1us-east-2us-west-1us-west-2
15.2% 8 regions
ap-east-2ap-northeast-2ap-southeast-1eu-central-1eu-north-1eu-west-2 · baselineme-central-1us-east-1

Fully supports

Operation areas this target implements completely - every test passes, nothing skipped. Areas it only partly gets wrong are in “Where it falls short” below; the ones it only partly attempts are in “What it doesn’t attempt”.

  • batchGetItem tier 1
  • deleteItem tier 1
  • deleteTable tier 1
  • getItem tier 1
  • listTables tier 1
  • scan tier 1
  • streams tier 2
  • ttl tier 2

Divergence and coverage over time

Both are plotted because neither reads correctly alone. Divergence falls when a target stops attempting an operation it used to get wrong, so a fall in the first plot is only an improvement if the second one holds.

Divergence

DynamoDB Local divergence, 2026-03-23 to 2026-08-25 0% 13% 25% divergence - lower is better 24 Apr 24 May 9 Jun 21 Jun 27 Jun 2 Jul 15 Jul 20 Jul 26 Jul 9 Aug 17 Aug 25 Aug 15.1%
Diverging on 15.1% of the suite as of 25 Aug, at worst 17.8% on 30 Jun. Lower is better, so a falling line is a target getting closer to real DynamoDB.

Coverage

DynamoDB Local coverage, 2026-03-23 to 2026-08-25 92% 96% 100% coverage - higher is better 24 Apr 24 May 9 Jun 21 Jun 27 Jun 2 Jul 15 Jul 20 Jul 26 Jul 9 Aug 17 Aug 25 Aug 93.8%
Implementing 93.8% of the suite as of 25 Aug. Higher is better. This is the share of the suite's tests a target implements at all. A fail becoming a skip leaves both numerators over the same fixed denominator, so a fall here is matched point for point by a fall in divergence: withdrawal costs exactly as much coverage as it gains divergence.

By operation

Every operation this target implements, with how much of it diverges and how much of it it covers, on the same two axes as the figures above. The counts are fails over the operation's whole size. full, partial, failing, unsupported. The matrix compares operations across targets.

Tier 1 - Core

  • batchGetItem 0/15 0.0% 100.0% batchGetItem: supported, diverges on 0.0% of it, covers 100.0% (15 pass)
  • batchWriteItem 3/18 16.7% 100.0% batchWriteItem: partially supported, diverges on 16.7% of it, covers 100.0% (15 pass, 3 fail)
  • createTable 4/30 13.3% 100.0% createTable: partially supported, diverges on 13.3% of it, covers 100.0% (26 pass, 4 fail)
  • deleteItem 0/13 0.0% 100.0% deleteItem: supported, diverges on 0.0% of it, covers 100.0% (13 pass)
  • deleteTable 0/3 0.0% 100.0% deleteTable: supported, diverges on 0.0% of it, covers 100.0% (3 pass)
  • describeTable 1/4 25.0% 100.0% describeTable: partially supported, diverges on 25.0% of it, covers 100.0% (3 pass, 1 fail)
  • getItem 0/39 0.0% 100.0% getItem: supported, diverges on 0.0% of it, covers 100.0% (39 pass)
  • listTables 0/5 0.0% 100.0% listTables: supported, diverges on 0.0% of it, covers 100.0% (5 pass)
  • putItem 25/129 19.4% 100.0% putItem: partially supported, diverges on 19.4% of it, covers 100.0% (104 pass, 25 fail)
  • query 2/92 2.2% 100.0% query: partially supported, diverges on 2.2% of it, covers 100.0% (90 pass, 2 fail)
  • scan 0/56 0.0% 100.0% scan: supported, diverges on 0.0% of it, covers 100.0% (56 pass)
  • updateItem 1/70 1.4% 100.0% updateItem: partially supported, diverges on 1.4% of it, covers 100.0% (69 pass, 1 fail)
  • updateTable 1/15 6.7% 100.0% updateTable: partially supported, diverges on 6.7% of it, covers 100.0% (14 pass, 1 fail)

Tier 2 - Complete

  • account 0/2 · 1 skip 0.0% 50.0% account: partially supported, diverges on 0.0% of it, covers 50.0% (1 pass, 1 skip)
  • backups 0/5 · 5 skip n/a 0.0% backups: unsupported, implements none of it (5 skip)
  • contributorInsights 0/2 · 2 skip n/a 0.0% contributorInsights: unsupported, implements none of it (2 skip)
  • export 0/2 · 2 skip n/a 0.0% export: unsupported, implements none of it (2 skip)
  • kinesis 0/1 · 1 skip n/a 0.0% kinesis: unsupported, implements none of it (1 skip)
  • partiql 18/76 23.7% 100.0% partiql: partially supported, diverges on 23.7% of it, covers 100.0% (58 pass, 18 fail)
  • resourcePolicy 0/2 · 2 skip n/a 0.0% resourcePolicy: unsupported, implements none of it (2 skip)
  • streams 0/18 0.0% 100.0% streams: supported, diverges on 0.0% of it, covers 100.0% (18 pass)
  • tags 0/8 · 8 skip n/a 0.0% tags: unsupported, implements none of it (8 skip)
  • transactions 11/62 17.7% 100.0% transactions: partially supported, diverges on 17.7% of it, covers 100.0% (51 pass, 11 fail)
  • ttl 0/7 0.0% 100.0% ttl: supported, diverges on 0.0% of it, covers 100.0% (7 pass)
  • updateTable 3/14 21.4% 100.0% updateTable: partially supported, diverges on 21.4% of it, covers 100.0% (11 pass, 3 fail)
  • vectorSearch 0/28 · 28 skip n/a 0.0% vectorSearch: unsupported, implements none of it (28 skip)

Tier 3 - Strict

  • error-messages 64/174 · 16 skip 36.8% 90.8% error-messages: partially supported, diverges on 36.8% of it, covers 90.8% (94 pass, 64 fail, 16 skip)
  • legacy-api 1/42 2.4% 100.0% legacy-api: partially supported, diverges on 2.4% of it, covers 100.0% (41 pass, 1 fail)
  • limits 2/93 2.2% 100.0% limits: partially supported, diverges on 2.2% of it, covers 100.0% (91 pass, 2 fail)
  • validation-ordering 24/31 77.4% 100.0% validation-ordering: partially supported, diverges on 77.4% of it, covers 100.0% (7 pass, 24 fail)

Where it falls short

Operations DynamoDB Local implements and answers differently from real DynamoDB, biggest gap first. This is what its divergence figure counts. Open one for the exact tests, or see the conformance suite.

  • error-messages Tier 3 64 diverging 16 not attempted
    View these tests in the suite →
    • BatchGetItem - exact error messages empty RequestItems: full required-parameter error source:26
    • BatchGetItem - exact error messages > 100 keys across all tables: interpolated full error source:46
    • BatchGetItem - exact error messages non-existent table: full ResourceNotFoundException message source:69
    • BatchWriteItem - exact error messages empty RequestItems: full required-parameter error source:33
    • BatchWriteItem - exact error messages > 25 requests: anchored regex on the constraint phrase source:66
    • BatchWriteItem - exact error messages non-existent table: full ResourceNotFoundException message source:109
    • BatchWriteItem - exact error messages wrong-typed table key: full schema-mismatch message source:132
    • BatchWriteItem - exact error messages non-scalar table key: full schema-mismatch message source:132
    • CreateTable - exact error messages missing TableName: full required-parameter error source:32
    • CreateTable - exact error messages short table name (2 chars): minimum length 3 error source:52
    • CreateTable - exact error messages duplicate attribute in KeySchema source:75
    • CreateTable - exact error messages more than 2 KeySchema elements source:112
    • CreateTable - exact error messages invalid KeyType source:137
    • CreateTable - exact error messages invalid AttributeType source:158
    • CreateTable - exact error messages LSI without range key on base table source:191
    • CreateTable - exact error messages duplicate index names source:233
    • CreateTable - exact error messages PAY_PER_REQUEST with ProvisionedThroughput: full conflict message source:252
    • CreateTable - exact error messages GSI INCLUDE projection without NonKeyAttributes: full missing-attributes message source:284
    • CreateTable - exact error messages LSI INCLUDE projection without NonKeyAttributes: full missing-attributes message source:323
    • CreateTable - exact error messages StreamSpecification StreamEnabled:false with a StreamViewType: full conflict message source:347
    • DeleteItem - exact error messages non-existent table: full ResourceNotFoundException message source:29
    • DeleteItem - exact error messages malformed Key (missing range key on composite table): full schema-mismatch error source:47
    • GetItem - exact error messages non-existent table: full ResourceNotFoundException message source:32
    • GetItem - exact error messages malformed Key (missing range key on composite table): full schema-mismatch error source:50
    • BatchWriteItem - index key error messages wrong-typed index key: full type-mismatch message source:54
    • BatchWriteItem - index key error messages non-scalar index key: full type-mismatch message source:54
    • BatchWriteItem - index key error messages empty-string index key: full secondary-index-key message source:54
    • PutItem - exact error messages missing table name: full validation error string source:37
    • PutItem - exact error messages empty table name: minimum length 1 error source:59
    • PutItem - exact error messages table name too long (256 chars): maximum length 255 error source:80
    • PutItem - exact error messages table name with invalid chars: regex pattern error source:98
    • PutItem - exact error messages empty string member in NS: numeric conversion error source:198
    • Query - exact error messages missing hash key in KeyConditionExpression source:30
    • Query - exact error messages non-key attribute in KeyConditionExpression source:49
    • Query - exact error messages invalid Select value source:90
    • Query - exact error messages Limit of 0 source:110
    • Query - exact error messages malformed ExclusiveStartKey: short invalid-starting-key error source:190
    • Query - exact error messages Select SPECIFIC_ATTRIBUTES without ProjectionExpression: full required-projection message source:214
    • Scan - exact error messages Segment without TotalSegments: full required-parameter error source:30
    • Scan - exact error messages Segment >= TotalSegments: full out-of-range error source:49
    • Scan - exact error messages TotalSegments without Segment: full required-parameter error source:67
    • Scan - exact error messages Limit of 0: full minimum-value error source:85
    • Scan - exact error messages non-existent table: full ResourceNotFoundException message source:104
    • Scan - exact error messages malformed ExclusiveStartKey: long schema-mismatch error source:164
    • TransactGetItems - exact error messages empty TransactItems: full minimum-length error source:20
    • TransactGetItems - exact error messages > 100 gets: anchored regex on the constraint phrase source:45
    • TransactGetItems - exact error messages non-existent table: full ResourceNotFoundException message source:69
    • TransactGetItems - exact error messages missing key attribute: action-level ValidationError surfaces as TransactionCanceledException source:124
    • TransactWriteItems - index key error messages Put wrong-typed index key: cancelled with full ValidationError reason source:63
    • TransactWriteItems - index key error messages Put non-scalar index key: cancelled with full ValidationError reason source:54
    • TransactWriteItems - index key error messages Update wrong-typed index key: cancelled with full ValidationError reason source:63
    • TransactWriteItems - index key error messages Update non-scalar index key: cancelled with full ValidationError reason source:54
    • TransactWriteItems - index key error messages Put empty-string index key: top-level ValidationException source:77
    • TransactWriteItems - exact error messages empty TransactItems: full minimum-length error source:45
    • TransactWriteItems - exact error messages > 100 actions: anchored regex on the constraint phrase source:74
    • TransactWriteItems - exact error messages non-existent table: full ResourceNotFoundException message source:128
    • TransactWriteItems - exact error messages Put wrong-typed table key: cancelled with full ValidationError reason source:264
    • TransactWriteItems - exact error messages Put non-scalar table key: cancelled with full ValidationError reason source:255
    • TransactWriteItems - exact error messages Update wrong-typed Key: cancelled with schema-mismatch reason source:264
    • TransactWriteItems - exact error messages Update non-scalar Key: cancelled with schema-mismatch reason source:255
    • TransactWriteItems - exact error messages Delete wrong-typed Key: cancelled with schema-mismatch reason source:264
    • TransactWriteItems - exact error messages Delete non-scalar Key: cancelled with schema-mismatch reason source:255
    • TransactWriteItems - exact error messages ConditionCheck wrong-typed Key: cancelled with schema-mismatch reason source:264
    • TransactWriteItems - exact error messages ConditionCheck non-scalar Key: cancelled with schema-mismatch reason source:255
    • SearchVectors - exact error messages TopK above the maximum
    • SearchVectors - exact error messages missing SearchConditionExpression against a HASH-schema index
    • SearchVectors - exact error messages non-equality comparator on the HASH element
    • SearchVectors - exact error messages non-equality comparator on an INLINE_FILTER element
    • SearchVectors - exact error messages query vector dimension mismatch
    • SearchVectors - exact error messages condition attribute outside the SearchSchema
    • SearchVectors - exact error messages L-wrapped search vector
    • CreateTable vector indexes - exact error messages vector index on a provisioned table
    • CreateTable vector indexes - exact error messages SearchSchema attribute missing from AttributeDefinitions
    • CreateTable vector indexes - exact error messages Dimensions above the 4096 maximum
    • CreateTable vector indexes - exact error messages more vector indexes than the per-table limit
    • CreateTable vector indexes - exact error messages two indexes on one attribute with different Dimensions
    • PutItem vector writes - exact error messages vector with the wrong dimension count
    • PutItem vector writes - exact error messages vector element that is not a number, naming the element
    • PutItem vector writes - exact error messages vector attribute that is not a list
    • PutItem vector writes - exact error messages empty string in the SearchSchema HASH attribute
  • putItem Tier 1 25 diverging
    View these tests in the suite →
  • validation-ordering Tier 3 24 diverging
    View these tests in the suite →
    • Batch operations - validation ordering BatchWriteItem rejects empty RequestItems source:37
    • Batch operations - validation ordering BatchGetItem rejects empty RequestItems source:53
    • CreateTable - validation ordering empty TableName reports only tableName constraint source:23
    • CreateTable - validation ordering invalid table name pattern reports only tableName source:41
    • CreateTable - validation ordering reports invalid BillingMode and invalid KeySchema element together source:61
    • CreateTable - validation ordering rejects a duplicate attribute in KeySchema source:105
    • CreateTable - validation ordering rejects more than two KeySchema elements source:134
    • CreateTable - validation ordering rejects an invalid BillingMode on its own source:153
    • DeleteItem - validation ordering empty TableName reports only tableName constraint source:25
    • DeleteItem - validation ordering reports invalid ReturnValues and invalid ReturnConsumedCapacity together source:44
    • DeleteItem - validation ordering invalid table name pattern reports only tableName source:66
    • PutItem - validation ordering empty TableName reports tableName constraint (stops early) source:27
    • PutItem - validation ordering empty TableName with invalid ReturnValues reports only tableName source:54
    • PutItem - validation ordering reports invalid ReturnConsumedCapacity, ReturnItemCollectionMetrics, and ReturnValues together source:74
    • PutItem - validation ordering reports invalid table name pattern source:98
    • Query - validation ordering reports invalid TableName pattern source:28
    • Query - validation ordering reports invalid ReturnConsumedCapacity source:71
    • Scan - validation ordering rejects Segment without TotalSegments source:24
    • Scan - validation ordering rejects Segment >= TotalSegments source:43
    • Scan - validation ordering rejects TotalSegments without Segment source:61
    • Scan - validation ordering rejects a negative Segment source:81
    • UpdateItem - validation ordering empty TableName reports only tableName constraint source:26
    • UpdateItem - validation ordering rejects invalid ReturnValues (UpdateItem reports the first enum error) source:50
    • UpdateItem - validation ordering invalid table name pattern reports only tableName source:72
  • partiql Tier 2 18 diverging
    View these tests in the suite →
    • BatchExecuteStatement - PartiQL surfaces a malformed member statement as a per-statement ValidationError without failing the batch source:290
    • ExecuteStatement - PartiQL rejects INSERT on an existing item (INSERT is not upsert) source:160
    • ExecuteStatement - PartiQL returns a populated ConsumedCapacity block when requested source:394
    • ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED NEW * over a nested path returns only the changed leaf source:824
    • ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED OLD * over a nested path returns only the changed leaf at its old value source:844
    • ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED NEW * on a list index returns only the changed element source:941
    • ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED OLD * on a list index returns only the prior element source:967
    • ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED NEW * on a non-zero list index returns only the changed element source:999
    • ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED OLD * on a non-zero list index returns the prior element source:1023
    • ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED NEW * over multiple list indices returns a dense pack of the changed elements source:1045
    • ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED OLD * over multiple list indices returns the prior elements densely source:1069
    • ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED NEW * packs changed indices in ascending index order, not statement order source:1091
    • ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED OLD * packs changed indices in ascending index order, not statement order source:1117
    • ExecuteStatement - PartiQL UPDATE RETURNING MODIFIED NEW * appending at exactly the list length returns the appended element source:1182
    • ExecuteStatement - PartiQL UPDATE REMOVE RETURNING MODIFIED OLD * on a list index returns the removed element at its old value source:1224
    • ExecuteStatement - PartiQL UPDATE REMOVE RETURNING MODIFIED NEW * on a list index returns the shifted element, not an empty list source:1252
    • ExecuteStatement - PartiQL UPDATE REMOVE RETURNING MODIFIED OLD * on the last list index returns the removed element source:1302
    • ExecuteTransaction - PartiQL idempotent replay under the same ClientRequestToken does not double-apply source:189
  • transactions Tier 2 11 diverging
    View these tests in the suite →
    • TransactGetItems - ConsumedCapacity INDEXES breakdown includes the table read capacity units source:305
    • TransactGetItems - projection matching nothing omits Item when the projection matches no attribute on a present item source:342
    • TransactWriteItems - validation Put with a non-scalar table key cancels with a ValidationError reason source:761
    • TransactWriteItems - validation Update with a non-scalar Key cancels with a ValidationError reason source:761
    • TransactWriteItems - validation Delete with a non-scalar Key cancels with a ValidationError reason source:761
    • TransactWriteItems - validation ConditionCheck with a non-scalar Key cancels with a ValidationError reason source:761
    • TransactWriteItems - ConsumedCapacity charges 2 write capacity units per item source:1299
    • TransactWriteItems - ConsumedCapacity: conditional, check, replay, cancel a standalone ConditionCheck action costs 2 write capacity units source:1388
    • TransactWriteItems - ConsumedCapacity: conditional, check, replay, cancel reports write capacity on the first call and read capacity on a same-token replay source:1417
    • TransactWriteItems - index key validation Put with a non-scalar index key cancels with a ValidationError reason source:60
    • TransactWriteItems - index key validation Update setting a non-scalar index key cancels with a ValidationError reason source:60
  • createTable Tier 1 4 diverging
    View these tests in the suite →
    • CreateTable - TableId returns a TableId on CreateTable that matches the subsequent DescribeTable source:146
    • CreateTable - validation rejects PAY_PER_REQUEST with ProvisionedThroughput specified source:248
    • CreateTable - configuration parameters TableClass STANDARD_INFREQUENT_ACCESS round-trips source:50
    • CreateTable - configuration parameters SSESpecification with the AWS-managed key round-trips source:61
  • batchWriteItem Tier 1 3 diverging
    View these tests in the suite →
    • BatchWriteItem - index key validation rejects a non-scalar index key value source:38
    • BatchWriteItem - validation rejects a wrong-typed table key value source:99
    • BatchWriteItem - validation rejects a non-scalar table key value source:115
  • updateTable Tier 2 3 diverging
    View these tests in the suite →
    • UpdateTable - add GSI accepts a conflicting redeclaration of an existing key attribute and keeps the stored type source:454
    • UpdateTable - add GSI drops an unused AttributeDefinition supplied with a GSI add source:546
    • UpdateTable - GSI validation rejects a GSI keyed on a table attribute omitted from the request AttributeDefinitions source:745
  • query Tier 1 2 diverging
    View these tests in the suite →
    • Query - Limit + FilterExpression interaction rejects ExclusiveStartKey that does not match the table schema source:400
    • Query - KeyConditionExpression semantics rejects a nested path on a key attribute source:54
  • limits Tier 3 2 diverging
    View these tests in the suite →
    • Empty values - strings, binary, and sets empty string member in NS is rejected (not a number) source:224
    • Nesting depth - 32-level document limit rejects a 32-level ExpressionAttributeValue with ValidationException source:99
  • describeTable Tier 1 1 diverging
    View these tests in the suite →
    • DescribeTable - TableId returns a stable TableId across repeated DescribeTable calls source:44
  • updateItem Tier 1 1 diverging
    View these tests in the suite →
    • UpdateItem - ReturnValues granularity UPDATED_NEW on a nested SET returns only the changed fragment source:962
  • updateTable Tier 1 1 diverging
    View these tests in the suite →
    • UpdateTable - configuration parameters UpdateTable changes TableClass source:57
  • legacy-api Tier 3 1 diverging
    View these tests in the suite →
    • Legacy API - AttributeUpdates (legacy UpdateExpression) AttributeUpdates with a Value and no Action defaults to PUT source:279

What it doesn't attempt

Operations DynamoDB Local declines rather than gets wrong, so none of this counts towards its divergence. It is what the gap in its 93.8% coverage is made of. Each test here skipped itself because the target's own feature probe said the operation isn't implemented, which is often a deliberate choice rather than a defect.

  • vectorSearch Tier 2 none of it 28 not attempted
    View these tests in the suite →
    • SearchVectors - ConsumedCapacity shape reports VectorSearchRequestBytes with no classic capacity fields
    • SearchVectors - ConsumedCapacity shape reports the same shape under INDEXES as under TOTAL
    • SearchVectors - ConsumedCapacity shape reports nothing under NONE
    • PutItem - vector write capacity shape INDEXES adds a per-index VectorWriteRequestBytes map beside the classic fields
    • PutItem - vector write capacity shape TOTAL folds nothing in: no vector fields at all
    • PutItem - vector write capacity shape a write not touching the vector attribute reports no vector write
    • PutItem - vector write capacity shape an identical overwrite reports no vector write: replication is delta-based
    • CreateTable - vector index lifecycle walks CREATING to ACTIVE and describes the index faithfully
    • CreateTable - vector index lifecycle round-trips a SearchSchema of HASH and INLINE_FILTER elements
    • CreateTable - vector index lifecycle accepts the 4096-dimension boundary
    • SearchVectors - index readiness after CreateTable rejects retryably between ACTIVE and the first served search
    • PartiQL - vector indexes are out of reach rejects a PartiQL read of a vector index
    • PartiQL - vector indexes are out of reach reads the vector attribute off the base table like any other item
    • SearchVectors - deterministic search behaviour COSINE: identical vector scores 0, opposite scores 2, lower is closer
    • SearchVectors - deterministic search behaviour EUCLIDEAN: identical vector scores 0, straight-line distance otherwise
    • SearchVectors - deterministic search behaviour DOT_PRODUCT: higher is closer and scores can be negative
    • SearchVectors - deterministic search behaviour returns the identical vector as the top match
    • SearchVectors - deterministic search behaviour omits the vector attribute by default and returns it when projected
    • SearchVectors - deterministic search behaviour caps results at the item count with no pagination surface
    • SearchVectors - deterministic search behaviour stores full precision on the base table and f32 in the index
    • UpdateTable - vector index lifecycle adds an index that backfills like a GSI, enforces one online action, then deletes it
    • UpdateTable - vector index lifecycle refuses to delete the table while a vector index is still being created
    • CreateTable - vector index request validation rejects Dimensions of 0 at the request model layer
    • CreateTable - vector index request validation rejects an index name shorter than 3 characters
    • SearchVectors - request validation rejects TopK of 0 at the request model layer
    • SearchVectors - request validation rejects a search against an index the table does not have
    • PutItem - vector index write validation accepts a write missing the HASH attribute but leaves it unreachable through that index
    • PutItem - vector index write validation removes an item from the index when its vector attribute is removed
  • tags Tier 2 none of it 8 not attempted
    View these tests in the suite →
    • Tags - basic adds tags to a table
    • Tags - basic lists tags and verifies they match what was added
    • Tags - basic adds additional tags and verifies all tags are present
    • Tags - basic removes specific tag keys with UntagResource
    • Tags - basic verifies removed tags are gone after untag
    • Tags - basic overwrites an existing tag with the same key but different value
    • Tags - validation rejects TagResource with an invalid ARN format
    • Tags - validation rejects ListTagsOfResource with a non-existent ARN
  • backups Tier 2 none of it 5 not attempted
    View these tests in the suite →
    • Continuous backups - PITR reports PITR DISABLED by default
    • Continuous backups - PITR enabling PITR transitions PointInTimeRecoveryStatus to ENABLED
    • On-demand backups - lifecycle and restore CreateBackup → DescribeBackup → ListBackups → DeleteBackup
    • On-demand backups - lifecycle and restore RestoreTableFromBackup initiates a restore into a new table
    • On-demand backups - lifecycle and restore DescribeBackup on a deleted backup throws BackupNotFoundException
  • contributorInsights Tier 2 none of it 2 not attempted
    View these tests in the suite →
    • Contributor insights - enable/describe/list reports DISABLED by default
    • Contributor insights - enable/describe/list enabling transitions the status and lists the table
  • export Tier 2 none of it 2 not attempted
    View these tests in the suite →
    • Export and import - S3 ExportTableToPointInTime initiates an export and reports it
    • Export and import - S3 ImportTable ingests S3 data into a new table
  • resourcePolicy Tier 2 none of it 2 not attempted
    View these tests in the suite →
    • Resource policies - Put/Get/Delete GetResourcePolicy on a table with no policy throws PolicyNotFoundException
    • Resource policies - Put/Get/Delete Put then Get round-trips the policy, and Delete removes it
  • account Tier 2 partly 1 not attempted
    View these tests in the suite →
    • Account reads - DescribeLimits, DescribeEndpoints DescribeEndpoints returns at least one endpoint with an address
  • kinesis Tier 2 none of it 1 not attempted
    View these tests in the suite →
    • Kinesis streaming destination enables a streaming destination and reports it via Describe

Run history

Every percentage here is divergence, per tier and over the whole suite, so lower is better in each column. Coverage is the exception it is named as.

Run Gradecurrent criteria Divergence Movement
Grade C 15.1% unchanged
Grade C 15.1% unchanged

Suite on

AWS corrected the vector index readiness documentation, prompted by a write-up of the earlier guidance that drew on the suite's measurements. The ACTIVE-plus-backfilling state the old advice was built around, and which no index ever occupies, is gone from the three pages that described it. The wait now reads "Backfilling is not true" rather than "is false", so a check written literally from it fires on both creation paths instead of neither. The tutorial no longer says a search during backfill can return incomplete results. Two things the suite had measured but nobody had written down are documented as well: that DescribeTable reporting ACTIVE leads the dedicated search endpoint, and that the readiness check depending on neither status field is a real search in a retry loop.

That contract is now pinned rather than described. The UpdateTable walk asserts that Backfilling true is only ever reported alongside CREATING, that an ACTIVE index reports no Backfilling field at all, and that the base table goes ACTIVE while the index is still building, which is what makes a table waiter the wrong gate for a search. The first search that succeeds has to carry every seeded item, since the backfill window answers with an error rather than a partial view. On the CreateTable path a new test runs the documented check the way an application would, and every rejection before the first served response has to be the retryable ValidationException rather than a not-found.

The suite's own search wait now absorbs those two rejections and rethrows every other answer, so a fixture waiting on an index that is ACTIVE but not yet served no longer fails on the lag it was waiting out. Two files asserting exact rejection messages wait for a served search rather than for ACTIVE: "does not have the specified index" is also what a freshly ACTIVE index says, and it would otherwise stand in for whichever message the case asked for.

The tutorial's other new claim, that a table cannot be deleted while a vector index is being created, has a test of its own. Only the UpdateTable path can ask it. Across three runs an index created with its table reached ACTIVE in the same 250ms poll as the table, so the table is never ACTIVE with the index still building there, and a DeleteTable during creation is refused for the table's own status rather than with the documented index wording. Adding an index to a live table opens that window about thirty seconds in. The test cancels the index afterwards instead of waiting out the backfill, which a still-creating vector index turns out to accept the way a backfilling GSI does, so it costs a minute rather than seventeen.

A release now dispatches its measurement as the results bot rather than with the workflow's own token. GitHub raises no workflow_run event when a run started by GITHUB_TOKEN finishes, so 3.2.0 measured green for three hours and its board only landed once the results table was dispatched by hand.

 		Grade C 15.1% unchanged

Suite on

The board now measures the most recent release tag rather than main. Merging a test still runs it against real AWS, which is what validates it, but the published figures no longer move until a release moves them, so a dated changelog entry always sits behind a change in the denominator. Expect the figures to shift on this release: the board is switching from measuring main to measuring a tag, and those are different trees today.

Every board now says what produced it. results/summary.json carries a suite block naming the ref measured, its commit, the suite version at that ref, the region it ran against and when. It is additive, so schemaVersion stays at 1. The same block reaches /data/latest.json, /data/index.json and /data/runs.json, where each historical run carries its own copy, so a denominator that moved between two runs can be attributed to the release that moved it. Branch on kind: only tag is a released board.

A board is graded against the suite manifest and split registry as they stood at the ref it measured. Region health is the exception and is read live, so a region dropped since the tag still counts against today's cohorts. That is the one input allowed to move under a board without a new measurement, and the board carries a health date beside its measurement date to say so.

Releases are cut by one workflow dispatch: it bumps the version, dates this section, installs against the bumped tree, tags, and opens a draft release, then starts the measurement. The draft publishes itself when the board carrying that version lands, which takes about three hours.

eu-west-2 has crossed to the validation framework's generic constraint message for BatchGetItem with an empty RequestItems map, so the split row recording that behaviour is re-characterised against a 34-region capture taken on 2026-08-17. Eleven of the thirty-three regions that answer now return the generic message; twenty-two still return the bespoke required-parameter sentence, and me-central-1 joins the row.

The matching validation-ordering row is retired. Both wordings refuse an empty RequestItems before the map is read, which is all that tier asserts, so the assertion now matches the parameter name case-insensitively and spans both. The BatchWriteItem case beside it, which no region has moved yet, is written the same way.

A probe absent from the baseline is no longer reported as drift. Adding a probe to the capture script leaves every older baseline without it, and a scheduled red then named the new probe as the thing that had moved.

The weekly cross-region capture now includes eu-west-2, so the drift lens reads its baseline and the candidate regions from one capture taken at one moment rather than comparing today's candidates against an older baseline file. A scheduled red also keeps the eu-west-2 capture its drift verdict was read from, which was previously discarded with the runner.

 		Grade C 15.1% unchanged
Grade C 15.1% unchanged

Suite on

ExtendDB's SQLite backend joins the run, built from the same release as the PostgreSQL one and held to the same TLS, SigV4 and IAM posture, so the storage engine is the only thing that differs between them.

A project's other builds now sit behind a disclosure on its row. Every build is measured in full and has a row of its own with its own figures; the disclosure starts closed only when every build under it reads the same grade, divergence and coverage as the row above, and only when each of them was measured in that run: a carried row on either side opens it, and so does a run the suite declined to score. It is read from each run, so a build can start closed on one and open on the next. The README table has no disclosure to offer, so it lists every build outright.

Every target in the data endpoints gains two fields. collapsedIntoProject is true when the board starts that build's row closed. standsForProject says which row the board treats as a project's own, which isVariant cannot answer: on a run where a project's reference build recorded nothing, a build is promoted to stand for it and every row of that project reads isVariant: true. Both are additive, so schemaVersion is unchanged, and /data/index.json gains a projects block documenting them.

/data/index.json also gains a schema block saying what the version number entitles a consumer to: a field you already read will not change type or meaning while schemaVersion stays put, and new fields can appear at any version. The grading criteria are named there as a separate axis, since a change to the bands changes what a letter means without the schema moving.

The coverage description said a partial run was carried forward on its last clean measurement. It is not: the row stays in the run, publishes null for both figures and reads carried: false, because the target did report. Carrying forward is what happens to the baseline's unobserved lanes. Corrected, along with Dynoxide's build label, which now names its storage (native SQLite) like every other build in the registry rather than its compile target.

 		Grade C 15.1% diverged 0.1 percentage points less
Grade C 15.2% diverged 0.8 percentage points less

Suite on

Read this first if you consume the JSON. The data endpoints go from schema 2 to schema 4 in one step. Schema 3 was never published on its own, so everything on 2 crosses both steps at once. Schema 3 breaks in four ways:

  • movement.state values changed from up/down to improved/regressed. This is the one that fails silently: the old names still parse, and now mean the opposite direction.
  • movement.delta keeps its shape but is computed from divergence rather than correctness, so the sign of a delta means the opposite of what it did.
  • A tier no longer carries pct and value. It carries divergence, coverage and correctness, each with a pct and a value of its own.
  • The whole-suite correctness percentage is correctness, not total. total also names the raw test count inside counts, so the same word meant a count in one place and a percentage in another.

Schema 4 is additive on top: each target carries its grade and the full criteria in metrics.grade, every envelope gains baseline.observation (how much of the suite the live-AWS row stands on, which passes reported, and what is carried), latest.json and runs.json gain divergence, project, configuration and isVariant, results/summary.json gains regionFailures, and /data/index.json lists the split registry.

A score is two figures, never one

Divergence is the share of the whole suite a target answers differently from real DynamoDB. Coverage is the share it implements at all. They are reported apart and never summed, because a declined operation is discoverable in minutes and a wrong one in production. No target was re-run for the change and no pass, fail or skip moved: what changed is how the same counts are expressed.

Everything else that was a percentage followed the headline down - tier figures, the per-region drilldown, the per-operation table, and the colour bands, which inverted with them. A target's history is two plots rather than one, because divergence falls when a target stops attempting something it used to get wrong, so a divergence line alone can render a withdrawal as an improvement.

Every target wears a letter

Divergence sets it - A under 5%, B under 15%, C under 25%, D under 35%, F beyond

  • and coverage can only lower it, never raise it: a third of whatever a target leaves unimplemented is added to its divergence before the bands are read. A+ is exactly zero divergence at full coverage. A row says when coverage is holding its letter down, so a capped row is not mistaken for one with room above it. Real DynamoDB reads baseline rather than a letter, because grading the yardstick against itself would seat it in a band an engine had to earn its way into.

These are grading criteria version 1, dated in the methodology, which carries the derivation. Where a threshold sits is a hand-picked input to a published letter, and moving one regrades targets whose results never changed, so any change to a band, the coverage weight or the A+ gate bumps the version.

The suite grows from 998 tests to 1054

Vector search (#125). 42 tests over the deterministic surface DynamoDB shipped in August 2026: the index lifecycle on both creation paths, request validation on each plane, rejection wording, write-path validation, search on a fixture where the nearest neighbour is unambiguous, the two new capacity shapes, and PartiQL's inability to reach a vector index. Every pinned value was characterised against real DynamoDB in eu-west-2 before it was asserted. Two findings worth naming: searching during a backfill is an error, which settles which side of a contradiction in AWS's own documentation is right - three developer-guide pages say the call fails, the tutorial page says results can be incomplete - and an overwrite leaving the stored vector unchanged reports no vector write capacity at all, because index replication is delta-based. Both sides are captured in captures/2026-08-12-vector-backfill-docs.json. No emulator implements the family yet, so every target skips it and every coverage figure drops with this release while divergence is untouched. Sending the new operations needs @aws-sdk/client-dynamodb 3.1103.0 or later.

Index write costs (#124). 14 tests. The suite's only per-index capacity assertion was on a Query, so the write side - the half you get billed extra for - went unmeasured. A sub-1KB write costs one unit for the table and one for each index it lands in, and LSI units fold into the total exactly as GSI units do. Moving an item to a new GSI key costs two on that index, a delete and an insert; touching a projected attribute costs one; touching a non-projected attribute costs nothing, and the response carries no arm for that index rather than a zero. An overwrite that leaves the item unchanged reports no index cost whatsoever.

The index exclusions create no indexed table (#116). !gsi and !lsi used to select the right tests and then build the tables anyway, so an engine with no secondary-index support died in setup whatever it had asked for. Shared tables are created on demand from what the running file declared, the composite table split into indexed and plain variants, and three guards keep the declarations and the tags honest.

Corrections

  • The suite counts its own tests. "The whole suite" had meant whichever target ran the most tests, so one of the measured things was setting the denominator every figure divided by. registry/suite-manifest.json lists every test by file and full name, generated from the suite rather than inferred from a run, and CI fails if it drifts. Publishing now refuses a row whose test population disagrees with it, which catches a results file carried across a rename that keeps its old total while naming tests that no longer exist.
  • Every figure on a row comes from one region. A headline came from the target's best-matching region while its tier split and raw counts stayed on the baseline region's basis. Correctness never had to reconcile, because each tier had its own denominator; divergence is additive, so it does.
  • The per-region overlay is matched to the run it describes, rather than keyed on the date real AWS was last swept, which had put a later run's figures on an earlier run's page.
  • A methodology claim was wrong. The page said that measuring divergence over the whole suite stops an engine implementing a sliver from posting a perfect score. It doesn't: zero fails is 0.0% divergence at any coverage. What does hold is the identity the page now states - a test going from failing to skipped leaves both numerators together over the same denominator, so withdrawal costs exactly as much coverage as it gains divergence, and is disclosed rather than silent.
  • The baseline row is measured rather than pinned once real AWS has been observed across the whole suite. It runs in three passes and only the main one reached the published artefact, so the row had claimed a full suite on a run that recorded less. The passes are merged before scoring, each with its own capture date, and the row stays pinned and says so until all three report.
  • The Atom feed carries no letter on any run measured before criteria version 1 took effect. It had been rewriting each entry's summary with a grade the run never had, while leaving <updated> alone so no subscriber re-notified.
  • Badges publish the grade under a parity label, having still been publishing the correctness percentage the board retired under a conformance label. The endpoint URL is unchanged; a badge whose target has no results this run reads no data rather than disappearing, because the URL sits in other people's READMEs.
  • Two more regional splits are admitted, both BatchGetItem with an empty RequestItems map. The nesting-depth row was rewritten from a full 32-region capture, having been written from four.
  • A build of an engine nests under it rather than taking a row beside it, and maintainedByAuthor is keyed on the project, so it now reads true for the WebAssembly build. That build runs in CI like every other target, where its row had been refreshed by hand.
  • The board leads with the highest-graded engine, since the baseline moved into a panel above the standings. Where that is the board author's own engine, the conflict-of-interest disclosure sits on the card.
  • The Region column is gone. It named the cohort a target matched at its best rate, which read as breadth. The count sits beside the figure instead, and the cohort listing stays on the target page.
  • Each target lists how it is actually distributed, with the project's own page for each. These are the only claims on the board the suite does not measure, so each carries the link that backs it.
 		Grade C 15.9% diverged 0.1 percentage points less
Grade C 16.0% diverged 0.1 percentage points more
Grade C 15.9% unchanged
Grade C 15.9% unchanged
Grade C 15.9% unchanged
Grade C 15.9% unchanged
Grade C 15.9% diverged 0.9 percentage points less
Grade C 16.8% diverged 0.9 percentage points more

Suite on

Grew to 982 tests, up 28, all characterised against real DynamoDB across the per-region ground truth 2.0.0 put in place. New coverage is PartiQL's RETURNING clause; the GSI lifecycle also joins the ground truth, and the sweep's drift classifier gains a converged case.

  • PartiQL RETURNING (#103, #105): the clause pinned across every PartiQL surface. DELETE accepts only RETURNING ALL OLD * and rejects the other three variants with a 400, the message pinned; UPDATE accepts all four ALL/MODIFIED x OLD/NEW forms, where MODIFIED returns just the changed attribute and drops the key. The empty-projection edges are pinned too: a MODIFIED projection that resolves to nothing - a nested leaf, a list index, a batch statement - returns Items: [] rather than a keyed row. BatchExecuteStatement honours RETURNING; ExecuteTransaction rejects it. The non-upsert paths round it out: INSERT on an existing item, and UPDATE or DELETE behind a false predicate, fail ConditionalCheckFailed and leave the item untouched.
  • PartiQL RETURNING over list indices (#102): the MODIFIED projection shapes for a list-index SET or REMOVE, pinned against real AWS. The projection reads the literal index against the resulting list (MODIFIED NEW *) or the prior list (MODIFIED OLD *): an in-range index returns its element, an out-of-range one returns nothing, and multiple changed indices pack into a dense list in ascending index order. So a non-zero index collapses to one element, an append projects only when the index lands inside the new list, and removing the last index yields nothing under NEW while a middle REMOVE returns the shifted element. Setting an index on an absent attribute is rejected, not auto-created. In BatchExecuteStatement a member that fails to parse surfaces per-statement with the short Code: ValidationError, the same Code as an execution failure, and does not fail the batch.
  • GSI lifecycle in the ground truth (#100): the 14 UpdateTable GSI lifecycle tests are now observed by the weekly sweep and recorded as per-region ground truth. They are the one slice the gating run drops for runtime, so they were the one slice without a real-AWS answer; the sweep records them without slowing the gate.
  • Drift classification (#104): a drift where every region moves off the pinned answer at once is now classified as converged rather than moved, with the converged path covered end to end.
 		Grade C 15.9% unchanged
Grade C 15.9% unchanged
Grade C 15.9% unchanged

Suite on

Per-region scoring lands complete. 2.0.0-pre put the scoring logic in place, comparing each target against every region's recorded answer, but the evidence half was never wired: no test recorded what a target actually answered and the classifier never read one, so a fail could not be credited to a region the target matched and the score could only ever subtract. 2.0.0 closes that loop, and the seed split runs its whole lifecycle in the same release.

What changed:

  • The split tests now record what the target actually answered (src/observation-sink.ts), in the same shape the registry stores each region's answer, and the classifier carries it onto the verdict. An engine that matches a rejecting region on a split is now scored as passing in that region. Evidence only ever redeems a committed fail: a pass keeps the committed assertion's deliberate wording tolerance and is never held to the byte-exact recorded string. Committed results predate the capture, so published scores move on each target's next run, not in this change.
  • Headline ties now prefer a region the registry characterises. A region absent from every split row ties the top score by having nothing recorded about it, and the Region column must not answer "conformant to what?" with a region the suite knows nothing about. Only the label is affected; a strictly higher score still wins whatever its source.
  • The seeded { NULL: false } split closed its own loop. eu-west-2 and eu-central-1 were the last regions accepting it; by the 2026-07-17 sweep both reject it again, so every region agrees, the split is retired, and its test asserts the shared rejection. The detect, admit and reconcile path the pre-release introduced, exercised end to end within days.
  • A tooling test now asserts every tracked file is text, after a raw NUL byte used as a delimiter in one source file made grep classify the file as binary and silently skip it.
 		Grade C 15.9% diverged 0.2 percentage points less
Grade C 16.1% unchanged
Grade C 16.1% unchanged
Grade C 16.1% diverged 0.9 percentage points less

Suite on

The scores barely move in this release, but what they mean has changed.

Until now the suite pinned one region, eu-west-2, as ground truth. That was quietly unfair: real DynamoDB disagrees with itself in a handful of places, and a one-region baseline takes a side without saying so. The clearest case is the { NULL: false } attribute value - accepted and normalised to { NULL: true } in eu-west-2 and eu-central-1, rejected with a ValidationException in us-east-1 and ap-southeast-2. An engine matching us-east-1 on that behaviour was marked non-conformant for doing exactly what real DynamoDB does in Virginia. From 2.0.0, ground truth is per region.

What changed:

  • A weekly sweep runs the full suite against real DynamoDB in every commercial region and publishes per-region ground truth (.github/workflows/sweep.yml, ground-truth/). It gates nothing: PR scoring stays offline and deterministic.
  • Confirmed regional splits live in a checked-in registry (registry/splits.json), each row carrying what every named region actually returned, when, and who admitted it. Detection is automatic; admission is not. The sweep files an issue with the evidence, and only a human commits a row. The registry ships seeded with the { NULL: false } split.
  • Each target is scored against every observed region's expectations, and its published number is its best-matching region - named in the results table's new Region column, with the full per-region view in results/summary.json (a versioned, additive artefact; the per-target results/<slug>.json files are unchanged in shape).
  • A third result state, indeterminate, for a failed observation: a timeout, an exhausted throttle, a transport fault. It is excluded from both sides of the score and cannot become a split, a registry row, or a fail. An absent answer is not a different answer.
  • Region health is tracked in registry/regions.json. A region that cannot complete a sweep is published as unresolved rather than silently omitted; two consecutive misses drop it from the scored set and page a maintainer in the same act.

One deliberate departure from the RFC that proposed this (#75): the RFC suggested a behaviour conforms if it matches any real region. 2.0.0 scores each target against one region at a time and headlines the best match, so a target only passes a behaviour when at least one real region does what it did, and its headline reflects one coherent region rather than a mix. Match-any scoring would have accepted an engine that combines eu-west-2's answer on one behaviour with us-east-1's on another - a deployment that exists nowhere. That is stricter than the RFC asked for, and it is deliberate.

No score moves at release: the one admitted split pins eu-west-2, which is the only region in the health record until the first sweep runs. Per-target deltas will be published once the sweep admits more regions; the expected movement is roughly a tenth of a percent for the six engines that match us-east-1 on the { NULL: false } split.

The suite also grew to 954 tests, up 81, all characterised against real DynamoDB - the control-plane pins in eu-west-2, everything else across four regions (eu-west-2, eu-central-1, us-east-1, ap-southeast-2):

  • UpdateTable AttributeDefinitions reconciliation (#77, #78, #79): delta-fed GSI adds merge into the stored union rather than replacing it, a conflicting redeclaration of an existing key keeps the stored type, and deleting a GSI prunes only its orphaned key attributes. An unused definition on an add is silently dropped where CreateTable rejects the same shape.
  • ProjectionExpression validation (#81): duplicate paths, alias collisions and parent/child overlaps rejected identically on GetItem, Query, Scan and BatchGetItem, pinned with exact messages; legal shared-prefix projections guarded as accepted; GetItem's misfiled validation tests rehomed into Tier 3.
  • Expression-size limit (#80): every expression parameter caps at 4096 bytes, measured on the raw string before ExpressionAttributeNames substitution - 4096 accepted, 4097 rejected, on all five expression surfaces. No tracked target enforces this limit today, so the Tier 3 movement it causes is new coverage, not a regression.
  • Empty set members (#82): empty strings in an SS and zero-length members in a BS are accepted and round-trip intact through every write path, and contains() can find them; an empty NS member and duplicate empty members are rejected, with messages pinned. One existing assertion got stricter: the empty-binary round-trip now asserts byte length zero.
 		Grade C 17.1% unchanged
Grade C 17.1% unchanged

Showing the 24 most recent runs. The chart above covers the full history, and every run is browsable from Runs.