Skip to content

ExtendDB

v0.1.1 20 recorded runs

86.0%
correct, across 830 of 873 operations · 43 unsupported
Tier 1 85.0%
Tier 2 89.3%
Tier 3 86.4%

Fully supports

Operation areas this target implements completely - every test passes, nothing skipped. Areas it only partly implements, or gets wrong, are in “Where it falls short” below.

  • batchGetItem tier 1
  • deleteItem tier 1
  • deleteTable tier 1
  • describeTable tier 1
  • listTables tier 1
  • account tier 2
  • streams tier 2
  • tags tier 2
  • ttl tier 2
  • updateTable tier 2
  • legacy-api tier 3

Conformance over time

ExtendDB conformance history, 2026-05-23 to 2026-07-13 80% 90% 100% 100.0% 23 May 96.8% 24 May 94.7% 25 May 93.7% 26 May 93.8% 29 May 97.9% 9 Jun 97.9% 16 Jun 97.9% 18 Jun 97.9% 19 Jun 97.4% 21 Jun 94.4% 22 Jun 93.6% 23 Jun 92.1% 24 Jun 88.9% 27 Jun 88.9% 29 Jun 88.6% 30 Jun 86.0% 1 Jul 86.0% 2 Jul 86.0% 6 Jul 86.0% 13 Jul

Where it falls short

Operation groups with gaps in the latest run, biggest first. Unsupported means the target doesn't implement that feature at all, often by design. Open one for the exact tests, or see the conformance suite.

  • partiql Tier 2 unsupported
    View these tests in the suite →
    • BatchExecuteStatement - PartiQL batch of multiple SELECT statements
    • BatchExecuteStatement - PartiQL batch of INSERT and UPDATE statements
    • BatchExecuteStatement - PartiQL partial failure - one valid and one invalid statement
    • BatchExecuteStatement - PartiQL rejects an empty Statements array
    • ExecuteStatement - PartiQL INSERTs a new item
    • ExecuteStatement - PartiQL SELECTs an item by primary key
    • ExecuteStatement - PartiQL SELECTs with WHERE clause using comparison
    • ExecuteStatement - PartiQL UPDATEs an existing item
    • ExecuteStatement - PartiQL DELETEs an item
    • ExecuteStatement - PartiQL rejects INSERT on an existing item (INSERT is not upsert)
    • ExecuteStatement - PartiQL INSERT succeeds after DELETE of same key
    • ExecuteStatement - PartiQL SELECT returns empty results for non-matching WHERE
    • ExecuteStatement - PartiQL parameterized INSERT with ? placeholders
    • ExecuteStatement - PartiQL parameterized SELECT with ? placeholder
    • ExecuteStatement - PartiQL SELECT with nested map path
    • ExecuteStatement - PartiQL SELECT with specific attributes
    • ExecuteStatement - PartiQL SELECT with begins_with in WHERE clause
    • ExecuteStatement - PartiQL PartiQL UPDATE with SET on attribute
    • ExecuteStatement - PartiQL PartiQL UPDATE with REMOVE
    • ExecuteStatement - PartiQL returns a populated ConsumedCapacity block when requested
    • ExecuteStatement - PartiQL evaluates negated predicates (NOT begins_with, IS NOT MISSING)
    • ExecuteStatement - PartiQL DELETE with a false non-key predicate fails ConditionalCheckFailed and leaves the item
    • ExecuteStatement - PartiQL DELETE with a true non-key predicate removes the item
    • ExecuteStatement - PartiQL UPDATE with a false non-key predicate fails ConditionalCheckFailed and leaves the item
    • ExecuteStatement - PartiQL UPDATE with a true non-key predicate mutates the item
    • ExecuteStatement - PartiQL DELETE with a false NOT begins_with predicate fails ConditionalCheckFailed
    • ExecuteStatement - PartiQL DELETE with a true NOT begins_with predicate removes the item
    • ExecuteStatement - PartiQL rejects a write WHERE clause that omits the primary key
    • ExecuteStatement - PartiQL DELETE on a missing key with a non-key predicate is a silent no-op
    • ExecuteStatement - PartiQL rejects a statement with syntax error
    • ExecuteStatement - PartiQL rejects a reference to a non-existent table
    • ExecuteTransaction - PartiQL transactional INSERT and UPDATE both succeed atomically
    • ExecuteTransaction - PartiQL transaction rolls back on duplicate key INSERT
    • ExecuteTransaction - PartiQL multiple INSERTs in one transaction
    • ExecuteTransaction - PartiQL idempotent replay under the same ClientRequestToken does not double-apply
    • ExecuteTransaction - PartiQL rejects empty TransactStatements
  • error-messages Tier 3 34 failing
    View these tests in the suite →
    • BatchGetItem - exact error messages mixing ProjectionExpression on one table and AttributesToGet on another is rejected
    • BatchWriteItem - exact error messages wrong-typed index key: full type-mismatch message
    • BatchWriteItem - exact error messages non-scalar index key: full type-mismatch message
    • BatchWriteItem - exact error messages empty-string index key: full secondary-index-key message
    • BatchWriteItem - exact error messages empty-binary index key: full secondary-index-key message
    • CreateTable - exact error messages GSI INCLUDE projection without NonKeyAttributes: full missing-attributes message
    • CreateTable - exact error messages LSI INCLUDE projection without NonKeyAttributes: full missing-attributes message
    • CreateTable - exact error messages StreamSpecification StreamEnabled:false with a StreamViewType: full conflict message
    • PutItem - exact error messages NULL attr with false is accepted and normalises to NULL true
    • PutItem - exact error messages empty-binary index key value: full secondary-index-key message
    • Scan - exact error messages Select SPECIFIC_ATTRIBUTES without ProjectionExpression: full required-projection message
    • TransactWriteItems - exact error messages Put wrong-typed table key: cancelled with full ValidationError reason
    • TransactWriteItems - exact error messages Put non-scalar table key: cancelled with full ValidationError reason
    • TransactWriteItems - exact error messages Put wrong-typed index key: cancelled with full ValidationError reason
    • TransactWriteItems - exact error messages Put non-scalar index key: cancelled with full ValidationError reason
    • TransactWriteItems - exact error messages Update wrong-typed index key: cancelled with full ValidationError reason
    • TransactWriteItems - exact error messages Update non-scalar index key: cancelled with full ValidationError reason
    • TransactWriteItems - exact error messages Put empty-string index key: top-level ValidationException
    • TransactWriteItems - exact error messages Update empty-string index key: top-level ValidationException
    • TransactWriteItems - exact error messages Update empty-string Key: top-level empty-value message
    • TransactWriteItems - exact error messages Delete empty-string Key: top-level empty-value message
    • TransactWriteItems - exact error messages ConditionCheck empty-string Key: top-level empty-value message
    • TransactWriteItems - exact error messages Update wrong-typed Key: cancelled with schema-mismatch reason
    • TransactWriteItems - exact error messages Update non-scalar Key: cancelled with schema-mismatch reason
    • TransactWriteItems - exact error messages Delete wrong-typed Key: cancelled with schema-mismatch reason
    • TransactWriteItems - exact error messages Delete non-scalar Key: cancelled with schema-mismatch reason
    • TransactWriteItems - exact error messages ConditionCheck wrong-typed Key: cancelled with schema-mismatch reason
    • TransactWriteItems - exact error messages ConditionCheck non-scalar Key: cancelled with schema-mismatch reason
    • TransactWriteItems - exact error messages Update empty-binary Key: top-level empty-value message
    • TransactWriteItems - exact error messages Delete empty-binary Key: top-level empty-value message
    • TransactWriteItems - exact error messages ConditionCheck empty-binary Key: top-level empty-value message
    • TransactWriteItems - exact error messages Put empty-binary index key: top-level ValidationException
    • TransactWriteItems - exact error messages Update empty-binary index key: top-level ValidationException
    • UpdateItem - exact error messages empty-binary index key value: full secondary-index-key message
  • putItem Tier 1 29 failing
    View these tests in the suite →
    • ReturnConsumedCapacity - PutItem reports only the aggregate, no write-capacity split PutItem with TOTAL reports 1 CapacityUnit and no read/write split
    • ReturnConsumedCapacity - PutItem reports only the aggregate, no write-capacity split PutItem with INDEXES omits the split at the top level and on Table
    • PutItem - number format rejects "+e2"
    • PutItem - number format rejects "e2"
    • PutItem - number format rejects "+1+2"
    • PutItem - number format rejects "1+2"
    • PutItem - number format rejects "+1.2.3"
    • PutItem - number format rejects "1.2.3"
    • PutItem - number format rejects "++5"
    • PutItem - number format rejects "+-5"
    • PutItem - number format rejects "-+5"
    • PutItem - number format rejects "+"
    • PutItem - number format rejects "-"
    • PutItem - number format rejects "1e"
    • PutItem - number format rejects "1e+"
    • PutItem - number format rejects "."
    • PutItem - number format rejects "1.2e3.4"
    • PutItem - number format rejects "0x5"
    • PutItem - number format rejects "NaN"
    • PutItem - number format rejects "Infinity"
    • PutItem - number format rejects "1_000"
    • PutItem - number format rejects " 5"
    • PutItem - number format rejects "5 "
    • PutItem - number format rejects "1 5"
    • PutItem - validation rejects a wrong-typed index key attribute
    • PutItem - validation rejects a non-scalar index key attribute
    • PutItem - validation rejects an empty-string index key attribute
    • PutItem - validation rejects a number with a leading space
    • PutItem - validation rejects a number with a trailing space
  • transactions Tier 2 10 failing
    View these tests in the suite →
    • TransactWriteItems - validation Put with a wrong-typed index key cancels with a ValidationError reason
    • TransactWriteItems - validation Put with a non-scalar index key cancels with a ValidationError reason
    • TransactWriteItems - validation Update setting a wrong-typed index key cancels with a ValidationError reason
    • TransactWriteItems - validation Update setting a non-scalar index key cancels with a ValidationError reason
    • TransactWriteItems - validation Put with an empty-string index key is a top-level ValidationException
    • TransactWriteItems - validation Update setting an empty-string index key is a top-level ValidationException
    • TransactWriteItems - validation Update with an empty-string Key is a top-level ValidationException
    • TransactWriteItems - validation Delete with an empty-string Key is a top-level ValidationException
    • TransactWriteItems - validation ConditionCheck with an empty-string Key is a top-level ValidationException
    • TransactWriteItems - ConsumedCapacity: conditional, check, replay, cancel reports write capacity on the first call and read capacity on a same-token replay
  • getItem Tier 1 9 failing
    View these tests in the suite →
    • ConsumedCapacity - single-item ops report only the aggregate, no read/write split a strongly-consistent GetItem reports 1 CapacityUnit and no read/write split
    • ConsumedCapacity - single-item ops report only the aggregate, no read/write split an eventually-consistent GetItem reports 0.5 CapacityUnits and no read/write split
    • ConsumedCapacity - single-item ops report only the aggregate, no read/write split a Query under INDEXES omits the split at the top level and on Table
    • ConsumedCapacity - single-item ops report only the aggregate, no read/write split an UpdateItem reports 1 CapacityUnit and no read/write split
    • ConsumedCapacity - single-item ops report only the aggregate, no read/write split a DeleteItem reports 1 CapacityUnit and no read/write split
    • GetItem - projection validation and fidelity rejects overlapping projection paths (a and a.b)
    • GetItem - projection validation and fidelity rejects duplicate projection paths (a and a)
    • GetItem - projection validation and fidelity projects multiple list indices compacted and index-ordered
    • GetItem - projection validation and fidelity rejects an undefined projection name even when the scan matches nothing
  • query Tier 1 8 failing
    View these tests in the suite →
    • Query - GSI pagination across tied sort keys composes LastEvaluatedKey from base and index keys and walks every tied item once
    • Query - hash-only GSI pagination on a composite base table walks every item across pages when they share the GSI hash and base partition key
    • Query - KeyConditionExpression semantics rejects a nested path on a key attribute
    • Query - LSI pagination across tied sort keys composes LastEvaluatedKey from base and index keys and walks every tied item once
    • Query - Select / ProjectionExpression rejections Select ALL_ATTRIBUTES with ProjectionExpression is rejected
    • Query - Select / ProjectionExpression rejections Select ALL_PROJECTED_ATTRIBUTES without an IndexName is rejected
    • Query - Select / ProjectionExpression rejections Select COUNT with ProjectionExpression is rejected
    • Query - Select / ProjectionExpression rejections Select ALL_PROJECTED_ATTRIBUTES with ProjectionExpression and no IndexName is rejected
  • createTable Tier 1 7 failing
    View these tests in the suite →
    • CreateTable - validation rejects a GSI INCLUDE projection without NonKeyAttributes
    • CreateTable - validation rejects StreamSpecification with StreamEnabled false plus a StreamViewType
    • CreateTable - configuration parameters TableClass STANDARD_INFREQUENT_ACCESS round-trips
    • CreateTable - configuration parameters SSESpecification with the AWS-managed key round-trips
    • CreateTable - configuration parameters OnDemandThroughput round-trips on a PAY_PER_REQUEST table
    • CreateTable - LSI rejects an LSI INCLUDE projection without NonKeyAttributes
    • CreateTable - index and stream spec validation rejects a KEYS_ONLY GSI projection carrying NonKeyAttributes
  • scan Tier 1 7 failing
    View these tests in the suite →
    • Scan - GSI pagination across tied sort keys walks every item once across a paged GSI scan with tied sort keys
    • Scan - Select / ProjectionExpression rejections Select ALL_ATTRIBUTES with ProjectionExpression is rejected
    • Scan - Select / ProjectionExpression rejections Select ALL_PROJECTED_ATTRIBUTES without an IndexName is rejected
    • Scan - Select / ProjectionExpression rejections Select COUNT with ProjectionExpression is rejected
    • Scan - Select / ProjectionExpression rejections Select ALL_PROJECTED_ATTRIBUTES with ProjectionExpression and no IndexName is rejected
    • Scan - Select / ProjectionExpression rejections Select SPECIFIC_ATTRIBUTES without ProjectionExpression is rejected
    • Scan - validation rejects TotalSegments above the maximum
  • batchWriteItem Tier 1 3 failing
    View these tests in the suite →
    • BatchWriteItem - validation rejects a wrong-typed index key value
    • BatchWriteItem - validation rejects a non-scalar index key value
    • BatchWriteItem - validation rejects an empty-string index key value
  • updateItem Tier 1 2 failing
    View these tests in the suite →
    • UpdateItem - SET evaluation semantics a second SET clause reads the pre-update value of another attribute
    • UpdateItem - SET evaluation semantics applies parenthesised arithmetic (SET c = (c - :v))
  • updateTable Tier 1 2 failing
    View these tests in the suite →
    • UpdateTable - configuration parameters UpdateTable changes TableClass
    • UpdateTable - configuration parameters UpdateTable changes OnDemandThroughput
  • backups Tier 2 2 failing
    View these tests in the suite →
    • On-demand backups - lifecycle and restore RestoreTableFromBackup initiates a restore into a new table
    • On-demand backups - lifecycle and restore DescribeBackup on a deleted backup throws BackupNotFoundException
  • contributorInsights Tier 2 unsupported
    View these tests in the suite →
    • Contributor insights - enable/describe/list reports DISABLED by default
    • Contributor insights - enable/describe/list enabling transitions the status and lists the table
  • export Tier 2 unsupported
    View these tests in the suite →
    • Export and import - S3 ExportTableToPointInTime initiates an export and reports it
    • Export and import - S3 ImportTable ingests S3 data into a new table
  • resourcePolicy Tier 2 unsupported
    View these tests in the suite →
    • Resource policies - Put/Get/Delete GetResourcePolicy on a table with no policy throws PolicyNotFoundException
    • Resource policies - Put/Get/Delete Put then Get round-trips the policy, and Delete removes it
  • limits Tier 3 2 failing
    View these tests in the suite →
    • Key-value length limits rejects an over-limit partition key on the read path
    • Nesting depth - 32-level document limit rejects a 32-level ExpressionAttributeValue with ValidationException
  • kinesis Tier 2 unsupported
    View these tests in the suite →
    • Kinesis streaming destination enables a streaming destination and reports it via Describe
  • validation-ordering Tier 3 1 failing
    View these tests in the suite →
    • UpdateItem - validation ordering rejects invalid ReturnValues (UpdateItem reports the first enum error)

Run history

Run Total Movement
86.0% 0.0pp unchanged
86.0% 0.0pp unchanged
86.0% 0.0pp unchanged

Suite on

Grew to 873 tests, up 49, all characterised against real DynamoDB in eu-west-2. New coverage in three areas:

  • ConsumedCapacity: the transactional read/write split on a same-token replay and on ExecuteTransaction, and a correction that single-item operations report the aggregate CapacityUnits and omit the split, which is transactional-only.
  • Empty-binary key values: rejected as a top-level ValidationException on every path, including secondary-index keys and inside transactions, mirroring the empty-string rejection.
  • Expression, limit and response-shape parity: KeyConditionExpression operand and nested-path rules, ExpressionAttributeNames/Values hygiene, projection validation and list-index fidelity, reversed-bounds BETWEEN, read-path key-length and segment caps, whitespace numbers, batch unprocessed fields and cross-table projection mixing, the bare no-op upsert, multi-subpath UPDATED_NEW, filter operand ordering, hash-only-GSI pagination, and CreateTable spec validation.
 		86.0% -2.6pp fell 2.6 percentage points

Suite on

Grew to 824 tests, up 7, in two parts: two sibling-parity gaps where one half of a rule was pinned and the other was not, and the capacity accounting of conditional and idempotent transactional writes. All characterised against real DynamoDB in eu-west-2.

The first covers the LSI side of the INCLUDE-projection-without-NonKeyAttributes rejection; the GSI side already had it. An LSI declared with ProjectionType INCLUDE and no NonKeyAttributes is rejected as a ValidationException in tier1 and pinned to the exact message in tier3, the same wording the GSI case returns.

The second pins the Query message for Select SPECIFIC_ATTRIBUTES with no ProjectionExpression, which Scan already had. Query and Scan enforce the same rule but word it differently: Query wraps the phrase in the "1 validation error detected:" envelope, Scan returns it bare.

The transaction cases settle what a conditional TransactWriteItems actually costs. A passing condition adds no read capacity: a conditional write bills the same 2 WCU per sub-1KB item as an unconditional one, and a standalone ConditionCheck costs 2 WCU, billed as write not read. Idempotent replay splits the accounting - the first call reports 2 write capacity units, a same-token replay within the window reports 2 read capacity units for re-reading the stored result. A failing condition cancels the transaction, and the response carries no ConsumedCapacity at all. Answers #27.

 		88.6% -0.3pp fell 0.3 percentage points
88.9% 0.0pp unchanged
88.9% -3.2pp fell 3.2 percentage points

Suite on

Grew to 762 tests, up 18, covering a malformed value in the lookup Key of a TransactWriteItems Update, Delete, or ConditionCheck - the path a Put item key does not take. Captured across four regions (eu-west-2, us-east-1, ap-southeast-2, eu-central-1), where every string was identical, so they pin exactly. An empty-string Key surfaces as a top-level ValidationException with the same message a Put item key gives; a wrong-typed or non-scalar Key cancels with a ValidationError reason carrying "The provided key element does not match the schema" - the key-only form, not the "Type mismatch for key" message the item-key path returns. The same run confirmed the BatchWriteItem table-key schema-mismatch message is region-invariant.

 		92.1% -1.5pp fell 1.5 percentage points

Suite on

Grew to 744 tests, up 8, pinning the Select / ProjectionExpression rules on Query and Scan. A ProjectionExpression is only valid with Select SPECIFIC_ATTRIBUTES, and ALL_PROJECTED_ATTRIBUTES is only valid with an IndexName; real DynamoDB rejects both with a ValidationException before reading anything. The cases span ALL_ATTRIBUTES, COUNT, and ALL_PROJECTED_ATTRIBUTES, including the request that breaks both rules at once, where AWS reports the ProjectionExpression one. They assert the contractual phrase, so they hold whether or not the engine carries the wrapper AWS adds on Query but not Scan.

 		93.6% -0.8pp fell 0.8 percentage points

Suite on

Grew to 736 tests, up 30, covering what TransactWriteItems and BatchWriteItem do with an item whose key value is malformed - the wrong type, non-scalar, or an empty string - across both table and index keys. PutItem already covered this; the transactional and batch paths covered none of it.

Characterising it against real AWS turned up a split worth pinning. An empty-string key value is rejected by up-front input validation, so even inside a transaction it surfaces as a top-level ValidationException. A wrong-typed or non-scalar key value is caught while the transaction runs, so it comes back as a TransactionCanceledException carrying a ValidationError reason rather than a top-level error. BatchWriteItem has no cancellation path, so every variant there is a plain ValidationException. The tests pin both halves, which catches two opposite mistakes: an engine that wraps the empty-string case as a cancellation, and one that surfaces the type-mismatch case as a top-level error.

 		94.4% -3.1pp fell 3.1 percentage points

Suite on

Grew to 706 tests, up 7, tightening secondary-index behaviour in Tier 1. Query and Scan on a GSI or LSI now assert sparse membership: an item that omits the index key stays off the index but remains on the base table. And PutItem now rejects an item whose GSI or LSI key value is the wrong type, non-scalar, or an empty string while the base-table keys are valid, holding an index key to its declared scalar type the same way a table key is held.

 		97.4% -0.4pp fell 0.4 percentage points
97.9% 0.0pp unchanged
97.9% 0.0pp unchanged
97.9% 0.0pp unchanged

Suite on

Real DynamoDB in eu-west-2 reworded a chunk of its validation errors, and the Tier 3 error-message tests moved with it. They now assert the contract the error carries - its type, the field it objects to, and the constraint - rather than the exact prose, because AWS varies the wrapper, the echoed input value and the field casing from one region to the next. A four-region capture in June found eu-west-2 and eu-central-1 on the new wording and us-east-1 and ap-southeast-2 still on the old, so the line between contract and cosmetic is drawn from what is invariant across all four.

This moves some Tier 3 numbers. Targets that were only ever marked down for wording DynamoDB itself renders inconsistently now pass those checks, so their Tier 3 scores rise: the suite has stopped counting a cosmetic difference as a behavioural one. Genuine behavioural divergences are still pinned exactly, for example PutItem with a { NULL: false } attribute, which DynamoDB now accepts in eu-west-2 and normalises to { NULL: true }.

 		97.9% +4.1pp rose 4.1 percentage points
93.8% +0.1pp rose 0.1 percentage points

Suite on

Grew to 684 tests with a control-plane and table-configuration sweep: the CreateTable/UpdateTable config parameters and the secondary control-plane operations (limits, backups and PITR, exports and imports, Kinesis, resource policies, contributor insights), each characterised against real AWS and probe-skipped where a target doesn't implement it.

The published percentage changed with it. It now measures correctness over implemented operations, Pass / (Pass + Fail), so skips no longer count against the score. A skip is honest scope; a fail is a bug.

 		93.7% -1.1pp fell 1.1 percentage points
94.7% -2.1pp fell 2.1 percentage points

Suite on

Grew to 625 tests, up 24 on the previous run: eleven more in Tier 1 and thirteen more in Tier 3, tightening coverage of core operations and the strict edge cases.

 		96.8% -3.2pp fell 3.2 percentage points

Suite on

ExtendDB joined the run as a target. The suite itself held steady at 601 tests.

 		100.0% first run