How to begin your automation journey with IaC

Automation is one of the key pillars for a successfull cloud journey. When it comes to the available tools you will find out that it´s quite hard to find a fitting tool for your use case.

What is IaC?

Infrastructure as Code (IaC) has become quite prominent in the cloud community. It allows us to describe resources in a simplified, text-based fashion. Most tools rely on known data formats like yaml or json and a tool-specific syntax. Let´s take AWS cloudformation as an example. Cloudformation allows you to describe your AWS resources a yaml or json file. It has a tool specific syntax that allows you to specify for example metadata, parameters, mappings, resources, conditions, and outputs. The syntax also allows you to do some basic functions like logical operations (and|or), flow control (if|else), and more. However, due to limitations in the functions (no support for loops) of the tools many vendors decided to deliver an extension called cloud development kit (cdk). A cdk allows you to use supported programming languages like python or typescript as an additional layer of abstraction. This enables you to solve nontrivial requirements in your setup in a more efficient way. Typically the cdk relies on the underlying IaC tool itself for its deployment. IaC tools typically follow up a declarative approach of describing resources. This means that your code doesn´t necessarily need to follow a specific deployment order. It doesn´t matter where in the code a resource is described - the tool itself understands dependencies between resources and will take that burden off you when deploying the infrastructure. This works out fine for most of the use cases.

Why should you invest time to learn IaC when you can configure resources via the AWS Console?

A lot of my peers are afraid that IaC will stop them from being productive. And that´s true to some extent. When I think back on how long it took me to deploy a simple lambda function using terraform I can understand the frustration of a lot of my peers in the IT community. Especially in the prototyping phase or in the early phase of a project the "manual" approach seems to be faster. From my experience, the lack of productivity with IaC tools has the following roots:

• It takes time to understand and use IaC Tools

• When done correctly IaC Tools are giving you a bunch of goodies that are very good for productive services, but not necessary during the prototyping phase

I am convinced that IaC is a must for cloud deployments and that once you´ve inherited my two points from above you will change your mind. With every project you spin up with IaC you will gain more insights/code snippets and best practices which are allowing you to become faster in the deployment. When you need to scale your resources you will quickly find out that the investment in the initial setup will return tenfold. With respect to the second statement, this is a list of advantages when deploying your infrastructure as code:

• Eliminate human error

• Use the same format and apply naming/tagging convention by default

• Keep your resources under control - you know exactly to which project your resource belongs and why you need it. This will help you in keeping your environment clean

• Gain flexibility: Migrations / Disaster recoveries / deprovisioning of IaC-based resources are faster / less error prune and more reliant than manual deployments

• Apply least privilege: None of my productive workloads can be changed by human beings. Only the IaC framework and the services itself allow interaction with protected resources.

A Comparison of different IaC Tools for AWS

When starting your cloud journey with IaC you will find different tools. Each tool has its advantages and disadvantages and has its own use cases where it can shine. The tools I can recommend for AWS are described below:

Cloudformation

Cloudformation is the IaC Tool provided by AWS itself. It has pretty good integration with other AWS Services like the AWS Service Catalog, Governance (powered by AWS Control Tower), and Cloudformation Stacksets (for multi-account deployments). You have to describe your code in one cloudformation template file and deploy it via the AWS Console or AWS-CLI. In my opinion, the format of cloudformation is pretty hard to read and understand. You can do a lot, but it´s really hard to read and understand cloudformation templates. Cloudformation is pretty strict when it comes to the right indentation and format. A little mistake can result in validation errors. Your template is going to be deployed in one go and you may suffer when you face an error. If done carefully and tested cloudformation is a great way to deploy your infrastructure. However, I would not recommend writing cloudformation templates on your own if it is not absolutely necessary. I recommend using the AWS CDK which is easier to write and less error-prone.

My experience with cloudformation is that it scales great for multi-account deployments and together with other tools like AWS CDK, the serverless framework or AWS SAM. It should be used when you plan to implement self service automations with the AWS Service Catalog.

Terraform

Terraform is a great tool which has integrates with dozens of providers such as AWS, Kubernetes, GCP and Azure. In comparison to Cloudformation and most alternatives to cloudformation this solution uses REST integration with AWS. Terraform will use your current AWS session to retrieve your current deployment base and provision or change resources. The state of your IaC-defined resources is stored in a state file either locally or on an S3 Bucket. In my opinion terraform provides better drift detection and better deployment features than cloudformation. The targeting function allows you to pinpoint specific parts of your deployment. Terraform allows you to build modules and structure your code in multiple files. For day 1 actions you can remove resources from terraforms management, import resources or taint resources which results in better overall control of your deployment base. The downside of terraform is that you have to spend way more time to spin it up in a proper way and to provide a secure deployment. In order to scale your terraform deployments within a team of DevOps engineers you need to invest some time to provide your state file in S3 instead of a local file. Terraform is developed by Hashicorp and is dependent on the AWS-CLI. It could be that you cannot utilize the latest AWS Calls or run into a bug that cannot be resolved by yourself. My experience is that terraforms provider for AWS is maintained pretty good and new features are available either shortly after the official launch or within a few weeks. Terraform comes with it´s own CDK for advanced purposes. I personally do not recommend to use the CDK as your default solution - additional business logic in your IaC can result in complex setups which are difficult to read/understand.

Terraform is a tool for professionals and can be used across different cloud providers. You need to invest more time in order to understand the product and make a proper use of it. Terraforms flexibility and ability to support modules is a blessing and curse.

Serverless, AWS SAM and Amplify

Next to the "standard" IaC tools you will find multiple tools with a specific focus. Those tools are strong in specific domains and will help you to describe your infrastructure with a simpler approach. Examples of such tools are the serverless framework, AWS SAM or AWS Amplify. I´ve started to build my first microservices based on the serverless framework. It allowed me to integrate an AWS API Gateway with Lambda and DynamoDB in a very elegant way and with minimal effort. In comparison to native cloudformation or terraform the lines of code needed for the deployment and the complexity of my IaC Code was less. The serverless framework uses Cloudformation to deploy your resources under the hood. AWS SAM is comparable to the serverless framework and my choice #1 for Lambda deployments. You can debug, test and deploy lambdas (and other serverless components) locally without depending on the AWS Cloud as your platform provider. AWS SAM has pretty good integration with CodeDeploy and will help you to ensure a smooth transition of lambda function for updates or migrations in your productive environments.

Amplify is another AWS Product to simplify the deployment of mobile apps and Single Page Applications. In my opinion, the service has a very specific and limited scope and has stagnated over the years. There are some nice features like GraphQL support (with Appsync) and frontend integration with some backend AWS Services next to the typical candidates like API Gateway and Appsync. I´ve used the product for one of my setups and would recommend to rather go with one of the two above tools for your application development since I had a lot of difficulties getting everything up and running. In addition, I couldn´t understand the GraphQL integration which relies on DynamoDB and uses a relational data model in the backend. In my humble opinion, this is no best practice for NoSQL databases and GraphQL doesn´t scale very well with dynamodb for bigger deployments. Amplify comes with its own console and authentication mechanisms to interact with AWS.

If you are deploying serverless applications I strongly recommend to have a look into AWS SAM (or the Serverless framework). It will save you a lot of time in comparison with terraform, cloudformation or the AWS CDK. The latter tools are not designed to have complex integrations with migrations/updates in your deployment. You can also combine different IaC Tools in order to make use of their strenght´s and weaknesses.

Other Tools (Ansible)

Next to the mentioned tools you will be able to find additional tools like Ansible which are also able to manage resources. However, from my experience, the tools above are the de facto standard for AWS. I´ve also worked with teams in the past which wanted to use Ansible for their cloud environment as they already had the knowledge and used ansible also for the automation of their VMWare infrastructure. My personal recommendation for AWS deployments is still to go with the standard tools which are designed for IaC. Ansible can do the job, but is designed to configure systems - which has some relations to classical IaC, but with a different scope. You can use Ansible for your day2 activities to configure for example containerized workloads - when doing so keep in mind that the AWS Systems Manager also has a lot of capabilities that are helping you to get the job done (especially when working with EC2 machines).

Conclusion

Start learning IaC now. I recommend you learn the basics of cloudformation and terraform first. Go with the vendor-provided documentation and do not get distracted by third-party websites. You should be able to read and understand the setup of these two tools. Depending on your use case try to focus on one product and start building domain knowledge. Since cloudformation is pretty hard to write by hand I recommend you make use of the AWS CDK for your deployments. In comparison, terraform comes with so many features which should allow you to do most of your deployments without the need for the Terraform CDK for AWS. When you start with terraform in your production environment or with teams make sure to properly secure and scale your deployment. In my case terraform deploys all resources via an AWS CodePipeline. The whole setup is secured so that only codebuild is able to interact with my state file in the production environment.