ASP.NET Core with GitOps: Orchestrating Your Containers with Kubernetes

Kubernetes can be used to deploy, scale, and manage containers. In this article, Mircea Oprea builds on the previous example in the series to show you how deploy your contains in the Elastic Kubernetes Service in AWS.

The series so far:

In the previous tutorial, you learned how to write and deploy your Infrastructure as Code using AWS CloudFormation. This allowed you to launch several EC2 instances that run your Docker image.

The next step towards automatization is to have an orchestration system for your containers for deploying, scaling and managing them. For this series, you will be using AWS EKS (Elastic Kubernetes Service), but there are alternatives – even inside AWS- which will be presented during this article.

Understanding Kubernetes and EKS

Kubernetes is an open-source system designed by Google; as stated on their website, it is a “portable, extensible, open-source platform for managing containerized workloads and services.” This means that you can use Kubernetes to deploy and handle your containers (in this case, Docker containers) automatically.

Kubernetes works by having multiple nodes – basically, machines, whether they are virtual or physical – which handle running the pods – a container, or perhaps a group of containers. In this series, a node will be represented by an EC2 instance, while each pod will be a Docker container.

While bringing a lot of benefits to your infrastructure, running a Kubernetes cluster proves to be quite difficult. One of the services that can ease your work is Amazon’s Elastic Kubernetes Service. It has alternatives on the other big cloud providers – Google Kubernetes Engine (GKE) and Azure Kubernetes Service (AKS). Choosing between them can depend on multiple factors, such as the current knowledge of your team, costs, and where your infrastructure is currently deployed. If you are already running a large number of servers, databases, and VMs on Azure, for example, switching to Google Cloud Platform just because of Kubernetes would not make much sense.

The biggest drawback of EKS is probably its cost – as opposed to its competitors, which provide free services. It requires $0.2/hour/cluster, in addition to the costs of running the rest of your infrastructure, such as your EC2 instances and your databases.

What About Elastic Container Service (ECS)?

As mentioned earlier, there are alternatives to Kubernetes even inside AWS. ECS is an orchestration service that is native to AWS, provides better integration with other services from the platform, and is free of charge. You still pay for your instances and other services, but the orchestration capabilities are provided for free.

You can find many comparisons between ECS and EKS, and the choice should be made based on your organization’s needs; you will find ECS to be easier to learn, and you might find that it solves your problems, so you don’t have to dive into Kubernetes. One of the GitHub Actions most popular pipelines is deploying your application to ECS, so you can have your container orchestration ready faster than you think.

However, since this series is aiming to teach GitOps principles and practices, rather than platform-native solutions, the rest of the tutorial will focus on EKS.

Installing kubectl

Kubectl is a command-line interface (CLI) used for working with Kubernetes clusters. Even though you are using EKS, you need Kubectl to configure the cluster, set the image that is going to be deployed, and debug the process.

To install kubectl on your system, you can check out the guide on the Kubernetes website. After you finish the installation, verify the major and minor versions by running kubectl version. These are important when creating the EKS cluster, as the cluster and kubectl version must be within one minor version of each other.

You might also get the following message after checking the version: “The connection to the server localhost:8080 was refused – did you specify the right host or port?”. This happens because kubectl defaults the connection to localhost:8080; you will get to change this later when connecting to the EKS cluster.

Creating an EKS Cluster

The first step towards having Kubernetes deploy and manage your containers is to create an EKS cluster in your Infrastructure as Code. Most of the other resources will remain unchanged: you still need an Auto Scaling Group, a Launch Configuration and their underlying infrastructure. The main change will happen in the Launch Configuration, where instead of running your Docker container manually, you will ask the EKS cluster to handle that for you.

The EKS cluster requires an IAM role to be able to manage the resources. You can choose to add the role in your IaC file, but since it’s more of a “static” resource, creating it manually from the AWS console is not wrong. To do this, search for IAM in the Find services box:

Select IAM, likely the first option. Then, from the left side of the screen, choose Roles and click the Create role button:

From the list of services that appears, search for EKS and click it, then click the Next: Permissions button.

You should see the policies that are attached to the role:

Continue to click the Next buttons until you are asked to choose a role name. Pick something that will be intuitive, such as eks-console-access, and click the Create role button.

You can then search for the role in the table that opens up by using the name you just picked, and you should see a page with the details of your role:

Copy the Role ARN value for later use. It will be needed when creating the EKS cluster.

One more policy should be attached to the role. Click Attach policies and search for the AmazonSSMFullAccess policy. It will be needed for pulling the image ID later. Select the policy and click Attach policy.

Now the three policies should be attached to the role.

It is now time to open your infrastructure.yaml file (from the previous article) and make the necessary modifications. You will create the EKS and get the nodes to attach to it. As in the previous tutorial, this can be a process of trial and error, but a good starting point is AWS’s template. It is very generic, but it is the best way to learn precisely what is needed, and you can adapt it to your own needs.

To understand the changes that are going to be made to the infrastructure file, you need to understand resource tagging. Generally, tags allow you to manage and search for your resources more efficiently. In the case of EKS, however, tags are necessary for your nodes to attach to the cluster.

Because you are creating an EKS cluster, change the Description in the top of the file:

Next, a new security group needs to be created for the EKS. This is done to be able to manage the cluster independently from the nodes, as well as to avoid future circular dependencies in the infrastructure:

This security group allows communication on port 443, as well as any port between 1025 and 65535. You can find more information about which ports are necessary and recommended for your cluster in the AWS documentation.

Next, you can create the EKS cluster. This requires the ARN of the role you created earlier, the security group, as well as a list of subnets where your worker nodes will be hosted. The latest version supported on AWS right now is 1.14, but you can use any of the other supported versions.

Now that you have the cluster ready, you can tag the related resources so that the nodes can attach to the cluster. The security group used by the nodes (MainSecurityGroup, in this case), as well as the Auto Scaling Group and the instances inside it, need a tag with the key kubernetes.io/cluster/{eks-id} and the value owned. Replace the MainSecurityGroup from your file with this code:

Replace the AutoScalingGroup in your file with this code:

For the Auto Scaling Group, you can also see the PropagateAtLaunch property being set to the tag. This will ensure that the tag will also be applied to any instance inside the group.

Although you already have an IAM role for the cluster, creating another one for the launch configuration and for accessing the cluster through the console can be a good idea. This role would need other policies than the one you created earlier:

The image ID is also changed, as AWS provides a list of image IDs that are required to be used when working with EKS. You can find the list here. You should use it to find the right ID for your situation, as they are divided by Kubernetes versions, as well as AWS regions. Be sure to match the version of kubectl that you installed and the version specified in the EKS section of the file. After finding the right image for you, it is possible to pull the latest version from AWS automatically. To do this, create a parameter (add to the Parameter section) in the file with the image ID name as the default value:

The final step is to link the cluster to the Launch Configuration; you need to specify the AMI and execute a Shell script instead of running the Docker container. Replace the LaunchConfigration section with this code:

The bootstrap.sh script is used to add the instances into the cluster; the next script, /opt/aws/bin/cfn-signal will signal when the instances are created and ready to be used.

Once you finish this, you can delete the LoadBalancer, Listener, ListenerRule and TargetGroup sections from your file. They no longer provide any value: the EKS cluster will take the role of a load balancer. You can find the final version of the file here.

Because of the IAM role that you added to the stack, you need to let CloudFormation know that it is allowed to create such resources. To do this, add the --capabilities flag to your update or create command from the previous article to create the stack. Run this in the command or terminal window replacing the Docker <username> and <password>:

Updating or recreating the stack will take some time at this point, because of the EKS. Once it is finished, you should see the three instances being created in your EC2 dashboard. However, none of them is running any application at this point, as you have not told the cluster what image to run.

Managing the Containers

In your AWS Console, go to the CloudFormation dashboard, select your stack and go to the Resources tab. You need two values from here – the ARN of the role that was created from the file, and the ID of the cluster.

For the ARN, search for the InstanceRole resource and click the link in the Physical ID column:

The summary page for the role will open, and you can copy the ARN from there:

 

For the cluster, just search for the EKS resource and copy the Physical ID column:

Before being able to interact with the cluster from EKS, you need to create a file that will tell AWS you are allowed to do so. Create a file called aws-auth-cm.yaml and paste the following content into it. Replace <ARN> with the value copied earlier.

The first step for interacting with the cluster via CLI is to run the following command, to tell AWS which cluster you are going to work with. Replace <region> with the AWS region you are working in, such as eu-central-1, and <eks-id> with the physical ID of your cluster.

You should receive an output similar to this:

If you receive an error like “Invalid choice: ‘eks’”, it is because your awscli version is outdated. Update your package and make sure your version is at least 1.15.32.

Next, you will need to apply the AWS authentication file to the cluster, so you have access to its resources:

This should output:

Note that the path to the aws executable must be in your PATH variable for this command to work if you are running Windows.

From this point, you can start managing your cluster. A good start is to check if the nodes were created correctly. If you followed the tutorial, you should see three nodes after running the command:

It should output something similar to this:

If you cannot see any nodes, several things could have gone wrong:

  • Run sudo kubectl version and make sure the client and server versions are the same (or one minor version apart);
  • Make sure you are pulling the correct AMI for the version you are using;
  • Check the aws-auth-cm.yaml file and make sure you have the correct role ARN; also check the indentation of the file, because you will not receive any errors;

The next step is to create a deployment file. The deployment file tells the cluster what image to deploy, how many containers it should create, and what method to use when updating them. The file creates a rolling update, meaning that new pods are added gradually, as old ones are destroyed, allowing your application to continue serving requests. You can read more about all the options here.

The file looks like the code below. Be sure to replace <username> with your own Docker username:

If you wish to deploy another image, you will change the -image line. Make sure the containerPort is also the one on which you expect your application to use.

Save this to a file called cluster-deployment.yaml and apply it to the cluster:

This command will create three pods, one running on each node. You can check this by running either one of the following commands:e

You are now ready to expose your application through a cluster load balancer:

To get the load balancer’s IP, run:

Copy the External-IP value from the output:

It might take a couple of minutes for the application to deploy, but you can append the standard/api/test to the external IP to test that everything is working as intended. If the app is running, you’ll see this message:

If at any point, you need to update the application served by the cluster, you can update the Docker image and then run the following command replacing your Docker <username>:

Debugging the Cluster

Working with Kubernetes can be a tedious process, with many things going wrong along the way.

One of the most common problems is the nodes not attaching to the cluster. This is most likely due to the wrong configuration in the infrastructure: check the launch configuration’s AMI, the Kubernetes version from EKS, and make sure the resources are correctly tagged.

If the nodes and pods are created, but the application is not running, the describe command can reveal a lot of information. You can run it on the pods, as well as the svc, to discover any issues, such as scheduling, or the image not being pulled:

Cleaning Up

Before deleting the stack, you need to delete the deployment and the svc. Since the deployment creates a load balancer, it will prevent you from deleting the stack as long as it exists. The command order is:

What is Next?

With Kubernetes, you are now able to easily manage a large number of containers with just a few commands. You can scale up or down, update the application that is being served, and diagnose any problems that appear along the way.

The next and final tutorial will teach you how to automate this process by using a CI/CD pipeline: any new commit to the master branch will result in the Docker image being pushed to the remote hub, and then updated in your cluster.