Yes I had to migrate 6 million records from DynamoDB to MongoDB

Asanka Nissanka
codeburst
Published in
5 min readNov 6, 2018

--

Image Ref : lynda.com

Why Migrate ?

Before going into details some of you might wonder why on earth we need to migrate from DynamoDB to MongoDB in the first place, since both are well performing, stable and reliable NoSQL database engines. However both have their pros and cons based on the use case we have. In my case following are the highlighted features of MongoDB that lead me to make the decision to migrate.

  • Powerful query engine (specially the aggregation pipeline)
  • No throughput limits (DynamoDB’s provisioned throughput was a real pain)
  • Friendly pricing scheme offered by MongoDB Atlas

You can read a more detailed comparison of DynamoDB vs MongoDB from here.

Migration Options

So now the decision has been made. Woooh! Wait!! The fun part is yet to come :) The main challenge is not making the decision but migrating the data we had been collecting on DynamoDB for about 3 years. It’s not that big but I would say big enough to make the migration task painful. The largest table had a storage size of 10GB.

There isn’t a natively supported way from either of the database engines (atleast by the time I was writing this article) to migrate data directly, but using the export and import options provided by each database engine we can use the following methods to migrate data.

  • Use the Export to .csv option from the AWS Console and import to MongoDB using mongoimport command (You can get more details on this from this blog post). If you are using a client like MongoDB Compass, you can do this from the GUI itself.
  • Use the AWS Data Pipeline to export the DynamoDB table to S3 using AWS EMR, you can use the predefined template named Export DynamoDB table to S3 for this (For detailed steps follow this tutorial). Then you can import the data to MongoDB using the mongoimport command.

Apart from the above methods, the other straight forward method is,

  • Write a program to read the table using the DynamoDB scan operation and write the data using the MongoDB query operation.

--

--