Why I’m not switching to AWS Fargate from EC2-backed ECS
Fargate gains you a lot so you don’t need to manage any of the underlying EC2 instances running your Docker containers. Scale forever and you’ll never need to add or remove any instances. That’s great! Should be a no-brainer; if you’re going to go with a managed solution — might as well dive into the deep end.
However, I have always found that every ounce of control you give up comes with a cost. So for that reason, I can’t bring myself in my current position to switch over to Fargate from EC2-backed ECS.
I’m currently supporting legacy infrastructure that’s built on EC2 instances that we manage through chef. We have some networking voodoo to make sure we can test our staging environment with full support from our CDN to ensure staging is as close to production as possible. This means we have a single external-facing IP that’s exposed and very much locked down to keep all external traffic out of non-production environments except for our CDN. These EC2 instances are now tightly coupled with our microservices running in ECS.
However, this means our networking is “special.” All IPs that are not internal are automatically blocked. If you’re on the VPN or a machine within our VPC, everything works as expected — you get automatically routed to internal IP addresses and our configured AWS Security Groups work as expected.
When we deployed our containers onto ECS, we started noticing errors in our non-production environments. Turns out, when trying to resolve our non-production IP addresses, the external IP address was being resolved instead of internal because of our non-AWS DNS provider.
So the one, singular, most powerful reason I can’t move to fargate right now — extraHosts and dnsServers. These are two configurable settings in task definitions that are supported in EC2-backed ECS, but NOT in Fargate (see docs here).
Therefor, I can’t just tell a container — when you mean to go to nonproduction.tripping.com, use this private IP address so that you’re not blocked by the AWS Security Group. I also can’t tell that container to use my own configured DNS.
I’m not willing to open up our security group to anything I don’t need to. There are probably other things we could do to our containers we submit to ECR to fix the issue — but I don’t believe in adding any unnecessary complexity into our containers would add anything that 4 extra lines in our task definition doesn’t get us more elegantly.
So for now, I’ll stick with the networking flexibility that EC2-backed ECS gives me, and my task definitions will have a small bit of extra configuration:
{
"containerDefinitions": [{
"cpu": 2048,
"memory": 1024,
"image": "123.dkr.ecr.example:latest",
"extraHosts": [{
"hostname": "nonprod.tripping.com",
"ipAddress": "10.20.0.125"
}],
"name": "example"
}],
}
