Intro to Infrastructure as Code with Twilio (Part 2)

Intro to Infrastructure as Code with Twilio (Part 2)

In the first part of this series, we covered the basic principles of Infrastructure as Code (IaC) and created and deployed our first script. In this part of the series, we will see how IaC can help with different phases of development.

Prerequisites

In order to follow along, you need:

• A Twilio account (which you can try for free)

• The Twilio CLI

• The Pulumi CLI

• A Pulumi account (which you can also try out for free)

• An Azure DevOps account (you will need an org, a project, and a repo within a project) or a GitHub account 

You also need the Twilio CLI Infra plugin. You can install it using the following command: 

$ twilio plugins:install twilio-infra-as-code/plugin-twilio-infra

If you've previously installed the infra plugin, make sure you are using version 0.2.0 or greater. To check which version of the plugin you are running, use the following command: 

$ twilio plugins

You should see the following output:

plugin-twilio-infra 0.2.0

Handle multiple staging environments

In a typical development cycle, developers are requested to test their implementation in a staging environment and then deploy them to a production environment. In this section, we will see how to work with environments and IaC.

Let’s then start with the testing phase: as a developer, your peer developer may ask you to review their code. And one of the things you may want to do is deploy the code to your own testing environment to make sure that the code is robust. In a traditional environment, that would mean making sure that all the testing environments for all developers are aligned, which could be quite challenging especially when new features are introduced.

IaC can facilitate that, since the environment configuration is checked in along with the code. Let’s see how that works, by running through the following steps:  

• Check out the code and environment configuration
• Define a new test environment (or use one previously defined)
• Deploy the resources and perform tests
• Switch to a new environment (e.g. production)
• Deploy the resources to production

In Twilio, a staging environment is usually mapped to a Twilio project / account (you can read more details on Getting Started with Twilio Projects and Subaccounts). In order to follow along with this example, you will need two Twilio accounts: one for testing and one for production.

Let's begin by downloading the code (which we'll assume is written by another developer for this exercise). You can fetch the example code by cloning this repo, or reusing the code you created in part 1 of this series.

Before starting, let’s install the dependencies:

$ npm install

Create a new environment

Once you have the code locally, let’s make sure we are using the right Twilio project for the testing environment, by running:

$ twilio profiles:list

If you are not using the right profile, switch to the profile you want to use by running:

$ twilio profiles:use <profile_name>

Let’s now use the Twilio CLI to create a new environment, called test:

$ twilio infra:environment:new test

If everything runs successfully, you should see the following output:

🎉 New environment created. When you are ready, use

 $ twilio infra:deploy

to deploy it to your Twilio project

If you look in the root directory, you will now see a new file called .twilio-infra. If you open this file, you should see something like the following:

{
   "ACxxxxxxxxxxxxxxxxxxxxxxxxxxx":{
      "environment":"test",
      "deployed":false
   }
}

This file keeps track of the mapping between Twilio project environments and their deployment status information.

You can now go ahead and deploy your Twilio resources using:

$ twilio infra:deploy

If you look at the .twilio-infra file again, you will see that the deployed attribute has changed to true, to reflect that the project has now been deployed.

Deploy to production

Once you have performed your tests in the staging environment, you are ready to deploy to production. In this specific case, we don’t have a production environment set for the project, so we need to create one. In real world environments, production is typically tracked on a shared storage environment (e.g. the Pulumi backend, more on this later).

The first step before creating a new environment is to switch to a different project. If a Twilio project is already associated with a previously deployed environment, then the CLI plugin will prevent a new environment from being defined on the same project.

So let’s use the following command to select a new Twilio project which we will use for our production deployment:

$ twilio profiles:use <profile_name>

You can now use the twilio infra:environment:new command to create a new environment (let’s call it prod) and twilio infra:deploy to deploy that to your Twilio production project.

Switch between environments

If you now want to switch back to the test environment, you can use the following command: 

$ twilio infra:environment:set test

Make sure you also select the correct Twilio CLI profile before making any deployments. 

If you are unsure which environment is currently set, you can check by running:

$ twilio infra:environment:get

Use IaC in a DevOps environment

So far we have been using IaC with Twilio in a local environment. But most of the time you will be using a DevOps tool to implement your CI/CD pipelines. In this post we are going to use Azure DevOps, but you can easily adapt this example to use other products. 

Create a new project using cloud backend storage

Since we are going to use a cloud service for running our pipeline, we need to have a cloud service to store the IaC configuration metadata. In this example we are going to use the Pulumi service for our backend. If you haven't done that already, head to the Pulumi signup page to try it for free.

If you were previously using local storage for Pulumi, use the following command to log out:

$ pulumi logout

Once you're logged out, use the following command to log in to the Pulumi backend storage: 

$ pulumi login

Follow the steps to authorize your computer to log in to your Pulumi account from the CLI. 

Let's now initialize a new project. Create a new empty folder (e.g. my-second-iac-project) and inside it type the following command: 

$ twilio infra:new

After following the prompts and initializing the project, you should see your new project in the Projects tab of the Pulumi dashboard:

Edit the index.js file and insert the following code:

This is just a simplified version of the code we created in part 1. Feel free to reuse the full code you wrote before.

We are now ready to create our pipeline.  In this post, we are going to present two integrations: 

• Azure DevOps pipeline
• GitHub Actions 

Both platforms have a free tier, but in the case of Azure DevOps, you will have to enable billing in your account in order to use the pipeline.  

Create a new Pulumi access token 

Both platforms are executing Pulumi commands from a cloud runner. In order to keep track of the deployed resources, the Pulumi CLI needs to access the Pulumi backend. To allow that, we need to create an access token using the procedure below:

• Go to the Pulumi Access Tokens page 
• Click on the button "Create new token"
• Give it a name (e.g. pipeline) and click "Create Token"
• Copy the token and save it in a safe place.

We are going to use this access token later on to configure our pipeline. 

Create a new pipeline in Azure DevOps

To sign up for Azure DevOps, head to the Azure DevOps main page and click on "Start free". After creating your free account in Azure DevOps, create a new project called twilio-infra-blog. Azure DevOps will create a new repo for you, accessible from the Repo icons in the left side bar. You can now follow the instructions to upload the files through git or you can use the "Initialize" button to initialize the repo and upload the files manually. Whichever you choose, make sure to upload / commit the following files to the git repository in Azure DevOps: 

• Pulumi.yml
• index.js
• package.json

Pulumi provides a task extension for Azure DevOps that you can use to run your Pulumi apps as a build and release task. Install this extension on your org by clicking on the "Get it" button on the Pulumi Task Extension page. Since Pulumi already provides this Task extension, we'll skip using the Twilio CLI. 

Create a pipeline

Now we need to instruct Azure DevOps on what to do with the Pulumi app:

• Navigate to the Pipelines group in the Azure DevOps left nav and click on "Create Pipeline"
• Select "Azure Repos Git" as the location of your code
• Select the repository you just created (e.g. twilio-infra-blog)
• Select "Node.js" as your pipeline configuration.

A new file azure-pipeline.yml will be created. Replace the content of that file with the following: 

Let's look at the tasks defined in this files: 

• First task, it's using the task Npm to install all the dependencies defined in package.json
• The second task is using the Pulumi Task Extension to run the up command. This is the equivalent of the twilio infra:deploy command used in our previous deployment examples. In this command, we also specify the following: 
  • stack: here you define the name of what we called the environment at the beginning of this article
  • condition: this is the condition that will trigger this pipeline to run. In particular, we are running this step both when we commit something to the main branch (IndividualCI) or when we trigger the job manually (Manual)

Set the variables

Now, we need to define some variables that will be used by the pipeline. On the top right corner, click on "Variables" and add the following variables. Make sure to select "Keep this value secret" before saving the variable. 

• PULUMI_ACCESS_TOKEN: this is the token you created in the Pulumi dashboard 
• TWILIO_ACCOUNT_SID: this is the SID of the Twilio account you want to deploy your resources to (make sure this account doesn't contain any resources created through the same file)
• TWILIO_AUTH_TOKEN: this is the auth token of the Twilio account specified above

Execute your pipeline 

Now that you have defined the steps and configured the environment variables, you can execute your pipeline. Click on "Save and run" on the top right corner of the page. In the page that opens, click on the running pipelines to see the progress. At the end, you should see something like the screenshot below: 

And if you open your Twilio account, you will see that the workflow has been created. 

As a test, you can now try to update the resources in the index.js file(by changing the name of the worker as we did in part 1 for example), and see how the pipeline executes automatically once you commit the file, and the resources on your Twilio account change accordingly.

Create a new pipeline with GitHub Actions

To start, sign in to your GitHub account and create a new repository. To follow along, you can use either the web interface or git to add the following files to the repository: 

• Pulumi.yaml
• index.js
• package.json

Now in the same repo create a folder called .github followed by a subfolder called workflows. Then in the workflows directory, create a file called push.yaml within that folder. The content of the file should be as follows: 

This "Update" workflow will be triggered when a new commit is pushed to the main branch. See Workflow Syntax for GitHub Actions to learn more about triggers and branches.

Now commit this file and open the "Settings" tab of the GitHub repository. From the left nav, click on Secrets and add the following variables: 

• PULUMI_ACCESS_TOKEN: this is the token you created in the Pulumi dashboard 
• TWILIO_ACCOUNT_SID: this is the SID of the Twilio account you want to deploy your resources to (make sure this account doesn't contain any resources created through the same file)
• TWILIO_AUTH_TOKEN: this is the auth token of the Twilio account specified above

Now you are ready to execute the actions in your GitHub workflow. Click on the "Actions" tab of the repo. You may see that you have a failed worfklow on the list. Click on it and then click on Re-run jobs > Re-run all jobs. Another way to test the workflow is to add another test file, commit, and then push it to the main branch. You should see another workflow run added to the queue.

Takeaways

• You can think of your Twilio Project as a twilio infra environment. The Twilio CLI makes it easy for you to switch between your staging and production environments.
• You have different options for managing your IaC resources: use local storage and deploy and manage your resources on your Twilio account using the Twilio infra CLI plugin, or use the Pulumi backend to manage your infrastructure resources.
• Pulumi offers integration with various platforms for CI/CD pipelines: in this post we only saw two of them. Check out Pulumi's Continuous Delivery guides for integration with other platforms.