Last year, I joined Travelopia to lead the WebApp development teams. My early days were focused on meeting the product tech team and learning about the technology solutions they were using in their projects. During this discovery phase, I quickly identified a major challenge faced by the teams - infrastructure management. The team was heavily reliant on an external team responsible for managing AWS resources, which made the process complex and inflexible. Specifically, if the team needed a new service, they had to rely on the external team to make the necessary AWS resources available. This approach did not align with our agile requirements, which meant we needed to explore more efficient and flexible options for managing our infrastructure.
The tech stack that the product tech teams were using was quite simple: Backend services + SPA, and the backend services were dockerized. However, the deployment process using docker was not straightforward. The traditional approach was cumbersome, involving the manual creation of EC2 instances, installing Docker, creating a bash script on the server, and configuring an SSH connection from CI to the server.
My plan was to have a simpler approach, like, CI pushes a docker image to the registry and the server automatically pulls the image and deploys it. Also, I wanted the product tech team to own this and not rely on external team. During my Volkswagen days, we kind of easily accomplished this using K8s, ArgoCD & Azure. But for small projects, its a very complicated stack. An alternate option was to use AWS App Runner or Digital ocean Apps but here you don’t get full control over the compute configuration.
I ultimately settled on ECS Fargate. It allowed for easy scaling, configurable memory and CPU usage, and easy management of logs. It was also well-connected to other AWS services, making it a good fit for our needs.
When it came to managing infrastructure as code (IaC), I decided not to use Terraform or CloudFormation to avoid introducing a steep learning curve for the product tech team. Although these tools are powerful, they can be complex and difficult to manage, making them less suitable for smaller teams or projects. Instead, I chose to use AWS CDK, which allowed the team to define infrastructure as code in a language they were already familiar with. This made infrastructure management more accessible and easier to manage.
eg:
class AppStack {
async createRds(): void {
new rds.DatabaseInstance(scope, "AppNameDb", {
allocatedStorage: 100,
instanceIdentifier: "app-name-db",
instanceType: ec2.InstanceType.of(ec2.InstanceClass.M5, ec2.InstanceSize.LARGE),
vpc,
...someMoreConfig,
});
}
}Overall, with the combination of CDK and ECS Fargate, the team was able to manage the infrastructure without being dependent on external teams, making the development process smoother and more efficient.
As we embarked on the journey to improve our infrastructure management, we knew that simply writing CDK scripts was not the answer. While our team was capable of writing CDK code, we needed to ensure that our network and security configurations were correct. To do so, we collaborated with the platform engineering team to review our resources and make necessary changes. This process was implemented in one project and was then used as a reference for other teams to make the required changes in their CDK files based on their project infrastructure requirements.
In conclusion, managing infrastructure is a crucial aspect of modern web application development. In my experience as Head of WebApp Development at Travelopia, I’ve learned that infrastructure management can be a major challenge for teams, especially if they rely on external teams to manage their resources. However, with the right tools and approach, it’s possible to overcome these challenges and achieve more efficient and flexible infrastructure management.
In our case, we found that combining AWS CDK with ECS Fargate was the ideal solution for our team. This combination allowed us to scale easily, configure memory and CPU usage, manage logs efficiently, and leverage other AWS services as needed. Additionally, AWS CDK enabled us to manage our infrastructure as code using a language that our team was already familiar with. This approach reduced the learning curve and enabled our team to contribute to infrastructure management easily.
Overall, I’m confident that our experience can serve as a useful case study for other teams looking to improve their infrastructure management processes. By taking a thoughtful and strategic approach, teams can overcome infrastructure challenges and achieve greater agility, efficiency, and flexibility in their web application development efforts.