Serverless development with Node.js, AWS Lambda and MongoDB Atlas

Share this article

Serverless Development with Node.js AWS Lambda and MongoDB Atlas

This article was originally published on mongoDB. Thank you for supporting the partners who make SitePoint possible.

The developer landscape has dramatically changed in recent years. It used to be fairly common for us developers to run all of our tools (databases, web servers, development IDEs…) on our own machines, but cloud services such as GitHub, MongoDB Atlas and AWS Lambda are drastically changing the game. They make it increasingly easier for developers to write and run code anywhere and on any device with no (or very few) dependencies.

A few years ago, if you crashed your machine, lost it or simply ran out of power, it would have probably taken you a few days before you got a new machine back up and running with everything you need properly set up and configured the way it previously was.

With developer tools in the cloud, you can now switch from one laptop to another with minimal disruption. However, it doesn’t mean everything is rosy. Writing and debugging code in the cloud is still challenging; as developers, we know that having a local development environment, although more lightweight, is still very valuable.

And that’s exactly what I’ll try to show you in this blog post: how to easily integrate an AWS Lambda Node.js function with a MongoDB database hosted in MongoDB Atlas, the DBaaS (database as a service) for MongoDB. More specifically, we’ll write a simple Lambda function that creates a single document in a collection stored in a MongoDB Atlas database. I’ll guide you through this tutorial step-by-step, and you should be done with it in less than an hour.

Let’s start with the necessary requirements to get you up and running:

  1. An Amazon Web Services account available with a user having administrative access to the IAM and Lambda services. If you don’t have one yet, sign up for a free AWS account.
  2. A local machine with Node.js (I told you we wouldn’t get rid of local dev environments so easily…). We will use Mac OS X in the tutorial below but it should be relatively easy to perform the same tasks on Windows or Linux.
  3. A MongoDB Atlas cluster alive and kicking. If you don’t have one yet, sign up for a free MongoDB Atlas account and create a cluster in just a few clicks. You can even try our M0, free cluster tier, perfect for small-scale development projects!).

Now that you know about the requirements, let’s talk about the specific steps we’ll take to write, test and deploy our Lambda function:

  1. MongoDB Atlas is by default secure, but as application developers, there are steps we should take to ensure that our app complies with least privilege access best practices. Namely, we’ll fine-tune permissions by creating a MongoDB Atlas database user with only read/write access to our app database.
  2. We will set up a Node.js project on our local machine, and we’ll make sure we test our lambda code locally end-to-end before deploying it to Amazon Web Services.
  3. We will then create our AWS Lambda function and upload our Node.js project to initialize it.
  4. Last but not least, we will make some modifications to our Lambda function to encrypt some sensitive data (such as the MongoDB Atlas connection string) and decrypt it from the function code.

A Short Note About VPC Peering

I’m not delving into the details of setting up VPC Peering between our MongoDB Atlas cluster and AWS Lambda for 2 reasons: 1) we already have a detailed VPC Peering documentation page and a VPC Peering in Atlas post that I highly recommend and 2) M0 clusters (which I used to build that demo) don’t support VPC Peering.

Here’s what happens if you don’t set up VPC Peering though:

  1. You will have to add the infamous 0.0.0.0/0 CIDR block to your MongoDB Atlas cluster IP Whitelist because you won’t know which IP address AWS Lambda is using to make calls to your Atlas database.
  2. You will be charged for the bandwidth usage between your Lambda function and your Atlas cluster.

If you’re only trying to get this demo code to write, these 2 caveats are probably fine, but if you’re planning to deploy a production-ready Lambda-Atlas integration, setting up VPC Peering is a security best practice we highly recommend. M0 is our current free offering; check out our MongoDB Atlas pricing page for the full range of available instance sizes.

As a reminder, for development environments and low traffic websites, M0, M10 and M20 instance sizes should be fine. However, for production environments that support high traffic applications or large datasets, M30 or larger instances sizes are recommended.

Setting up Security in Your MongoDB Atlas Cluster

Making sure that your application complies with least privilege access policies is crucial to protect your data from nefarious threats. This is why we will set up a specific database user that will only have read/write access to our travel database. Let’s see how to achieve this in MongoDB Atlas:

On the Clusters page, select the Security tab, and press the Add New User button

Clusters Lambda user

In the User Privileges section, select the link. This allows us to assign read/write on a specific database, not any database.

user privileges

You will then have the option to assign more fine-grained access control privileges:

Access Control

In the Select Role dropdown list, select readWrite and fill out the Database field with the name of the database you’ll use to store documents. I have chosen to name it travel.

Select Roles

In the Password section, use the Autogenerate Secure Password button (and make a note of the generated password) or set a password of your liking. Then press the Add User button to confirm this user creation.

Let’s grab the cluster connection string while we’re at it since we’ll need it to connect to our MongoDB Atlas database in our Lambda code:

Assuming you already created a MongoDB Atlas cluster, press the Connect button next to your cluster:

Connect cluster

Copy the URI Connection String value and store it safely in a text document. We’ll need it later in our code, along with the password you just set.

URI Connection String

Additionally, if you aren’t using VPC Peering, navigate to the IP Whitelist tab and add the 0.0.0.0/0 CIDR block or press the Allow access from anywhere button. As a reminder, this setting is strongly NOT recommended for production use and potentially leaves your MongoDB Atlas cluster vulnerable to malicious attacks.

Add whitelist entry

Create a Local Node.js Project

Though Lambda functions are supported in multiple languages, I have chosen to use Node.js thanks to the growing popularity of JavaScript as a versatile programming language and the tremendous success of the MEAN and MERN stacks (acronyms for MongoDB, Express.js, Angular/React, Node.js – check out Andrew Morgan’s excellent developer-focused blog series on this topic). Plus, to be honest, I love the fact it’s an interpreted, lightweight language which doesn’t require heavy development tools and compilers.

Time to write some code now, so let’s go ahead and use Node.js as our language of choice for our Lambda function.

Start by creating a folder such as lambda-atlas-create-doc

mkdir lambda-atlas-create-doc 
&& cd lambda-atlas-create-doc

Next, run the following command from a Terminal console to initialize our project with a package.json file

npm init

You’ll be prompted to configure a few fields. I’ll leave them to your creativity but note that I chose to set the entry point to app.js (instead of the default index.js) so you might want to do so as well.

We’ll need to use the MongoDB Node.js driver so that we can connect to our MongoDB database (on Atlas) from our Lambda function, so let’s go ahead and install it by running the following command from our project root:

npm install mongodb --save

We’ll also want to write and test our Lambda function locally to speed up development and ease debugging, since instantiating a lambda function every single time in Amazon Web Services isn’t particularly fast (and debugging is virtually non-existent, unless you’re a fan of the console.log() function). I’ve chosen to use the lambda-local package because it provides support for environment variables (which we’ll use later):

(sudo) npm install lambda-local -g

Create an app.js file. This will be the file that contains our lambda function:

touch app.js

Now that you have imported all of the required dependencies and created the Lambda code file, open the app.js file in your code editor of choice (Atom, Sublime Text, Visual Studio Code…) and initialize it with the following piece of code:

'use strict'

var MongoClient = require('mongodb').MongoClient;

let atlas_connection_uri;
let cachedDb = null;

exports.handler = (event, context, callback) => {
  var uri = process.env['MONGODB_ATLAS_CLUSTER_URI'];
    
  if (atlas_connection_uri != null) {
    processEvent(event, context, callback);
  } 
  else {
    atlas_connection_uri = uri;
    console.log('the Atlas connection string is ' + atlas_connection_uri);
    processEvent(event, context, callback);
  } 
};

function processEvent(event, context, callback) {
  console.log('Calling MongoDB Atlas from AWS Lambda with event: ' + JSON.stringify(event));
}

Let’s pause a bit and comment the code above, since you might have noticed a few peculiar constructs:

  • The file is written exactly as the Lambda code Amazon Web Services expects (e.g. with an “exports.handler” function). This is because we’re using lambda-local to test our lambda function locally, which conveniently lets us write our code exactly the way AWS Lambda expects it. More about this in a minute.
  • We are declaring the MongoDB Node.js driver that will help us connect to and query our MongoDB database.
  • Note also that we are declaring a cachedDb object OUTSIDE of the handler function. As the name suggests, it’s an object that we plan to cache for the duration of the underlying container AWS Lambda instantiates for our function. This allows us to save some precious milliseconds (and even seconds) to create a database connection between Lambda and MongoDB Atlas. For more information, please read my follow-up blog post on how to optimize Lambda performance with MongoDB Atlas.
  • We are using an environment variable called MONGODB_ATLAS_CLUSTER_URI to pass the uri connection string of our Atlas database, mainly for security purposes: we obviously don’t want to hardcode this uri in our function code, along with very sensitive information such as the username and password we use. Since AWS Lambda supports environment variables since November 2016 (as the lambda-local NPM package does), we would be remiss not to use them.
  • The function code looks a bit convoluted with the seemingly useless if-else statement and the processEvent function but it will all become clear when we add decryption routines using AWS Key Management Service (KMS). Indeed, not only do we want to store our MongoDB Atlas connection string in an environment variable, but we also want to encrypt it (using AWS KMS) since it contains highly sensitive data (note that you might incur charges when you use AWS KMS even if you have a free AWS account).

Now that we’re done with the code comments, let’s create an event.json file (in the root project directory) and fill it with the following data:

{
  "address" : {
    "street" : "2 Avenue",
    "zipcode" : "10075",
    "building" : "1480",
    "coord" : [ -73.9557413, 40.7720266 ]
  },
  "borough" : "Manhattan",
  "cuisine" : "Italian",
  "grades" : [
    {
      "date" : "2014-10-01T00:00:00Z",
      "grade" : "A",
      "score" : 11
    },
    {
      "date" : "2014-01-16T00:00:00Z",
      "grade" : "B",
      "score" : 17
    }
  ],
 "name" : "Vella",
 "restaurant_id" : "41704620"
}

(in case you’re wondering, that JSON file is what we’ll send to MongoDB Atlas to create our BSON document)

Next, make sure that you’re set up properly by running the following command in a Terminal console:

lambda-local -l app.js -e event.json -E {\"MONGODB_ATLAS_CLUSTER_URI\":\"mongodb://lambdauser:$PASSWORD@lambdademo-shard-00-00-7xh42.mongodb.net:27017\,lambdademo-shard-00-01-7xh42.mongodb.net:27017\,lambdademo-shard-00-02-7xh42.mongodb.net:27017/$DATABASE?ssl=true\&replicaSet=lambdademo-shard-0\&authSource=admin\"}

If you want to test it with your own cluster URI Connection String (as I’m sure you do), don’t forget to escape the double quotes, commas and ampersand characters in the E parameter, otherwise lambda-local will throw an error (you should also replace the $PASSWORD and $DATABASE keywords with your own values).

After you run it locally, you should get the following console output:

Console output

If you get an error, check your connection string and the double quotes/commas/ampersand escaping (as noted above).

Now, let’s get down to the meat of our function code by customizing the processEvent() function and adding a createDoc() function:

function processEvent(event, context, callback) {
  console.log('Calling MongoDB Atlas from AWS Lambda with event: ' + JSON.stringify(event));
  var jsonContents = JSON.parse(JSON.stringify(event));
    
  //date conversion for grades array
  if(jsonContents.grades != null) {
    for(var i = 0, len=jsonContents.grades.length; i < len; i++) {
      //use the following line if you want to preserve the original dates
      //jsonContents.grades[i].date = new Date(jsonContents.grades[i].date);
            
      //the following line assigns the current date so we can more easily differentiate between similar records
      jsonContents.grades[i].date = new Date();
  }
}
    
//the following line is critical for performance reasons to allow re-use of database connections across calls to this Lambda function and avoid closing the database connection. The first call to this lambda function takes about 5 seconds to complete, while subsequent, close calls will only take a few hundred milliseconds.
context.callbackWaitsForEmptyEventLoop = false;
    
try {
  if (cachedDb == null) {
    console.log('=> connecting to database');
    MongoClient.connect(atlas_connection_uri, function (err, db) {
      cachedDb = db;
        return createDoc(db, jsonContents, callback);
      });
    }
    else {
      createDoc(cachedDb, jsonContents, callback);
    }
  }
  catch (err) {
    console.error('an error occurred', err);
  }
}

function createDoc (db, json, callback) {
  db.collection('restaurants').insertOne( json, function(err, result) {
    if(err!=null) {
      console.error("an error occurred in createDoc", err);
      callback(null, JSON.stringify(err));
    }
    else {
      console.log("Kudos! You just created an entry into the restaurants collection with id: " + result.insertedId);
      callback(null, "SUCCESS");
    }
    //we don't need to close the connection thanks to context.callbackWaitsForEmptyEventLoop = false (above)
   //this will let our function re-use the connection on the next called (if it  can re-use the same Lambda container)
     //db.close();
  });
};

Note how easy it is to connect to a MongoDB Atlas database and insert a document, as well as the small piece of code I added to translate JSON dates (formatted as ISO-compliant strings) into real JavaScript dates that MongoDB can store as BSON dates.

You might also have noticed my performance optimization comments and the call to context.callbackWaitsForEmptyEventLoop = false. If you’re interested in understanding what they mean (and I think you should!), please refer to my follow-up blog post on how to optimize Lambda performance with MongoDB Atlas.

You’re now ready to fully test your Lambda function locally. Use the same lambda-local command as before and hopefully you’ll get a nice “Kudos” success message:

Console output

If all went well on your local machine, let’s publish our local Node.js project as a new Lambda function!

Create the Lambda Function

The first step we’ll want to take is to zip our Node.js project, since we won’t write the Lambda code function in the Lambda code editor. Instead, we’ll choose the zip upload method to get our code pushed to AWS Lambda.

I’ve used the zip command line tool in a Terminal console, but any method works (as long as you zip the files inside the top folder, not the top folder itself!) :

zip -r archive.zip node_modules/ app.js package.json

Next, sign in to the AWS Console and navigate to the IAM Roles page and create a role (such as LambdaBasicExecRole) with the AWSLambdaBasicExecutionRole permission policy:

AWS Lambda Basic Execution Role

Let’s navigate to the AWS Lambda page now. Click on Get Started Now (if you’ve never created a Lambda function) or on the Create a Lambda function button. We’re not going to use any blueprint and won’t configure any trigger either, so select Configure function directly in the left navigation bar:

AWS Lambda Configure

In the Configure function page, enter a Name for your function (such as MongoDB_Atlas_CreateDoc). The runtime is automatically set to Node.js 4.3, which is perfect for us, since that’s the language we’ll use. In the Code entry type list, select Upload a .ZIP file, as shown in the screenshot below:

Configure function

Click on the Upload button and select the zipped Node.js project file you previously created.

In the Lambda function handler and role section, modify the Handler field value to app.handler (why? here’s a hint: I’ve used an app.js file, not an index.js file for my Lambda function code…) and choose the existing LambdaBasicExecRole role we just created:

Lambda function handler

In the Advanced Settings section, you might want to increase the Timeout value to 5 or 10 seconds, but that’s always something you can adjust later on. Leave the VPC and KMS key fields to their default value (unless you want to use a VPC and/or a KMS key) and press Next.

Last, review your Lambda function and press Create function at the bottom. Congratulations, your Lambda function is live and you should see a page similar to the following screenshot:

Lambda create function

But do you remember our use of environment variables? Now is the time to configure them and use the AWS Key Management Service to secure them!

Configure and Secure Your Lambda Environment Variables

Scroll down in the Code tab of your Lambda function and create an environment variable with the following properties:

Name Value
MONGODB_ATLAS_CLUSTER_URI YOUR_ATLAS_CLUSTER_URI_VALUE
Environment variables

At this point, you could press the Save and test button at the top of the page, but for additional (and recommended) security, we’ll encrypt that connection string.

Check the Enable encryption helpers check box and if you already created an encryption key, select it (otherwise, you might have to create one – it’s fairly easy):

Encription key

Next, select the Encrypt button for the MONGODB_ATLAS_CLUSTER_URI variable:

Select encrypt

Back in the inline code editor, add the following line at the top:

const AWS = require('aws-sdk');

and replace the contents of the “else” statement in the “exports.handler” method with the following code:

const kms = new AWS.KMS();
  kms.decrypt({ CiphertextBlob: new Buffer(uri, 'base64') }, (err, data) => {
  if (err) {
    console.log('Decrypt error:', err);
    return callback(err);
  }
  atlas_connection_uri = data.Plaintext.toString('ascii');
  processEvent(event, context, callback);
});

(hopefully the convoluted code we originally wrote makes sense now!)

If you want to check the whole function code I’ve used, check out the following Gist. And for the Git fans, the full Node.js project source code is also available on GitHub.

Now press the Save and test button and in the Input test event text editor, paste the content of our event.json file:

Input test event

Scroll and press the Save and test button.

If you configured everything properly, you should receive the following success message in the Lambda Log output:

Lambda log output

Kudos! You can savor your success a few minutes before reading on.

What’s Next?

I hope this AWS Lambda-MongoDB Atlas integration tutorial provides you with the right steps for getting started in your first Lambda project. You should now be able to write and test a Lambda function locally and store sensitive data (such as your MongoDB Atlas connection string) securely in AWS KMS.

So what can you do next?

And of course, don’t hesitate to ask us any questions or leave your feedback in a comment below. Happy coding!

Enjoyed this post? Replay our webinar where we have an interactive tutorial on serverless architectures with AWS Lambda.

Frequently Asked Questions (FAQs) on Serverless Development with Node.js, AWS Lambda, and MongoDB Atlas

What are the benefits of using AWS Lambda for serverless development?

AWS Lambda is a highly beneficial tool for serverless development. It allows developers to run their code without having to manage servers. This means that you can focus on writing your code and let AWS Lambda handle the infrastructure. It automatically scales your applications in response to incoming traffic and you only pay for the compute time you consume. This makes it a cost-effective solution for businesses of all sizes. Additionally, AWS Lambda supports multiple programming languages, including Node.js, making it a versatile choice for developers.

How does MongoDB Atlas integrate with AWS Lambda?

MongoDB Atlas integrates seamlessly with AWS Lambda. It provides a fully managed database service that automates time-consuming administration tasks such as hardware provisioning, database setup, patching, and backups. With MongoDB Atlas, you can easily trigger AWS Lambda functions in response to database events like inserts, updates, or deletes. This allows you to create powerful, real-time, serverless applications.

What are the steps to set up a serverless application with Node.js, AWS Lambda, and MongoDB Atlas?

Setting up a serverless application with Node.js, AWS Lambda, and MongoDB Atlas involves several steps. First, you need to set up your AWS Lambda function. This involves writing your code in Node.js and uploading it to AWS Lambda. Next, you need to configure your MongoDB Atlas cluster. This involves creating a new cluster, configuring your IP whitelist, and creating a database user. Finally, you need to connect your AWS Lambda function to your MongoDB Atlas cluster. This involves configuring your Lambda function to use the MongoDB Atlas connection string.

How can I troubleshoot issues with my serverless application?

Troubleshooting issues with your serverless application can be done using various tools and techniques. AWS Lambda provides detailed logs of your function executions, which can help you identify any errors or issues. MongoDB Atlas also provides comprehensive monitoring and alerting features, allowing you to track the performance of your database and receive alerts for any potential issues. Additionally, using good coding practices and thoroughly testing your application can help prevent issues from arising in the first place.

What are the security considerations when using AWS Lambda and MongoDB Atlas?

Security is a crucial aspect of any application, and serverless applications are no exception. AWS Lambda provides several security features, including AWS Identity and Access Management (IAM) for access control, encryption in transit and at rest, and VPC support for network isolation. MongoDB Atlas also provides robust security features, including IP whitelisting, database auditing, and encryption at rest and in transit. It’s important to properly configure these security features to protect your application and data.

How can I optimize the performance of my serverless application?

Optimizing the performance of your serverless application involves several strategies. These include properly configuring your AWS Lambda function for optimal performance, using efficient code, and optimizing your MongoDB Atlas database. AWS Lambda allows you to allocate memory to your function, which also proportionally allocates CPU power, network bandwidth, and disk I/O. MongoDB Atlas provides performance optimization features such as automated indexing and performance advisor recommendations.

Can I use other programming languages with AWS Lambda and MongoDB Atlas?

Yes, AWS Lambda supports several programming languages, including Node.js, Python, Java, Go, and .NET. MongoDB Atlas can be used with any programming language that has a MongoDB driver. This makes it a versatile solution for serverless development.

How does serverless architecture impact the cost of my application?

Serverless architecture can significantly reduce the cost of your application. With AWS Lambda, you only pay for the compute time you consume, and there is no charge when your code is not running. MongoDB Atlas provides a variety of pricing options, including a free tier, allowing you to choose the best option for your needs.

How can I migrate my existing application to a serverless architecture?

Migrating your existing application to a serverless architecture involves several steps. First, you need to refactor your application to be compatible with AWS Lambda and MongoDB Atlas. This may involve rewriting your code in a supported programming language and modifying your database schema. Next, you need to set up your AWS Lambda function and MongoDB Atlas cluster. Finally, you need to test your application thoroughly to ensure it works correctly in the new architecture.

What are the limitations of using AWS Lambda and MongoDB Atlas for serverless development?

While AWS Lambda and MongoDB Atlas provide many benefits for serverless development, there are some limitations to be aware of. AWS Lambda has limits on the amount of compute and storage resources that can be used, and functions have a maximum execution time. MongoDB Atlas also has limits on the size of your database and the number of connections. However, these limits are generally high enough to accommodate most applications.

Raphael LondnerRaphael Londner
View Author

Raphael Londner is a Principal Developer Advocate at MongoDB, focused on cloud technologies such as Amazon Web Services, Microsoft Azure and Google Cloud Engine. Previously he was a developer advocate at Okta as well as a startup entrepreneur in the identity management space.

javascriptlambdaLearn-Node-JSmariapmongodbnode.jssponsored
Share this article
Read Next
Get the freshest news and resources for developers, designers and digital creators in your inbox each week