The effects of not maintaining consistency with DynamoDB

When working with DynamoDB I found that one of the main challenges is how to maintain consistency when multiple processes are accessing the database. I wrote about different scenarios such as keeping accurate counts, implementing foreign keys, and enforcing state in referenced items. All of these are taking advantage of DynamoDB's transaction feature with conditional checks.

Related

How to maintain database consistency in DynamoDB

How to use conditions in write transactions to guarantee a consistent state

But I got curious: what is the effect of not implementing a condition check properly? Is it easy to exploit the code that fails to check a field when another process might change the data in parallel?

My initial intuition said that it should be fairly hard to do. DynamoDB boasts single-digit millisecond performance, so what's the chance that I can make two processes run at the same time?

It turns out, it's quite easy to trigger an inconsistent condition.

Test setup

I chose the textbook example of race conditions: applying a coupon multiple times. While it sounds blatant, a similar problem took down a Bitcoin exchange and is actually a problem that affects SQL databases as well.

Here, I have a table with a coupon code and whether it has been used or not:

The API has an endpoint to apply a coupon: /apply/<coupon>. This gives back a 200 response if the coupon was applied, 400 if not.

For repeated testing, there is also an endpoint to create new coupons: /create. This creates a new item in the table and returns the generated value.

Applying the coupon

The apply code is straightforward: it first gets the current value, checks if the coupon is still valid, then sets its used status:

const item = await dynamodb.send(new GetItemCommand({
	TableName: process.env.TABLE,
	Key: {
		coupon: {
			S: coupon,
		},
	},
}));
if (item.Item?.used?.BOOL === false) {
	await dynamodb.send(new UpdateItemCommand({
		TableName: process.env.TABLE,
		Key: {
			coupon: {S: coupon}
		},
		UpdateExpression: "SET #used = :used",
		ExpressionAttributeNames: {
			"#used": "used",
		},
		ExpressionAttributeValues: {
			":used": {BOOL: true},
		},
	}));
	return {
		statusCode: 200,
	};
}
return {
	statusCode: 400,
};

Tester

The tester code creates a coupon using the /create endpoint:

const createRes = await fetch(url + "create/");
if (!createRes.ok) {
	throw new Error(createRes);
}
const couponValue = (await createRes.json()).coupon;

Then using 3 concurrent requests it tries to apply the same value:

return await Promise.all([0, 1, 2].map(async (idx) => {
	const applyRes = await fetch(url + "apply/" + couponValue);
	if (!applyRes.ok) {
		return false
	}
	return true;
}));

By seeing how many of the three values are true we can see whether the coupon was applied or not.

Results

Even with this very simple setup I could replicate multiple uses 4 out of 10 times:

[
	[ true, false, false ],
	[ false, false, true ],
	[ false, true, true ],
	[ true, false, false ],
	[ true, true, true ],
	[ false, false, true ],
	[ false, false, true ],
	[ true, true, true ],
	[ true, true, true ],
	[ true, false, false ]
]

Fix

The fix is rather simple after the root cause is identified: add a condition that only sets the used to true if its current value is false. DynamoDB guarantees that the check and the set will be done atomically so that no coupon will be applied twice.

The UpdateItem:

await dynamodb.send(new UpdateItemCommand({
	TableName: process.env.TABLE,
	Key: {
		coupon: {S: coupon}
	},
	UpdateExpression: "SET #used = :used",
	ExpressionAttributeNames: {
		"#used": "used",
	},
	ExpressionAttributeValues: {
		":used": {BOOL: true},
		":false": {BOOL: false},
	},
	ConditionExpression: "#used = :false",
}));

With this change, the results show consistency:

[
	[ false, false, true ],
	[ false, true, false ],
	[ true, false, false ],
	[ false, false, true ],
	[ false, true, false ],
	[ false, true, false ],
	[ false, false, true ],
	[ false, false, true ],
	[ false, true, false ],
	[ false, true, false ]
]