Cost-Driven Infrastructure Development With Infracost

Cost-Driven Infrastructure Development With Infracost

Estimate costs related to infrastructure as code changes

Guillaume Vincent
Guillaume Vincent

Cloud computing is awesome for growing and scaling up infrastructure quickly to follow product needs. However, this elasticity and flexibility come at a cost. Reviewing your bill daily is crucial to avoid unpleasant surprises at the end of the month. Cloud providers offer tools (here are some examples for AWS) and good practices to help you saving money.  

Most of the time, infrastructure as code (IaC) is employed to maintain and facilitate the evolution of this underlying. Like any other type of language, it should follow the same good practices and pass the code review step with other contributors. As mentioned before, cloud infrastructure has a cost, and invoicing is not always highlighted at this stage. Contributors have to refer to external resources and make estimations themselves. It is not automated and may be cumbersome over time.

In this article, I present you Infracost which is a command-line tool (CLI) that may permit you to include the cost dimension during the infrastructure as code development. As well as test-driven development, why not doing cost-driven development in the same logic? Through these lines, you'll know how to estimate the cost of your full infrastructure or the changes you want to introduce. Finally, we'll see how to integrate it in GitHub action to include cost estimation in your pull requests.

Cost-Driven Infrastructure Development

Presentation of Infracost

"Infracost shows cloud cost estimates for infrastructure-as-code projects such as Terraform. It helps DevOps, SRE and developers to quickly see a cost breakdown and compare different options upfront." https://github.com/infracost/infracost

Infracost is a binary CLI tool and so quickly installable on multiple OS :

$ brew install terracost
Installation of Infracost using brew on MacOSX

For now, it supports only Terraform but another infrastructure as code tools seems planned to be supported (Pulumi, CloudFormation).

Once Infracost is installed, you need to register to receive a free API key :

$ infracost register
Please enter your name and email address to get an API key.
See our FAQ (https://www.infracost.io/docs/faq) for more details.
Name: Guillaume Vincent
Email: [email protected]
Thank you Guillaume Vincent!
Your API key is: xxxxxxxxxxxxxxxxxxxxxxxxx
Success: Your API key has been saved to /Users/gvincent/.config/infracost/credentials.yml
You can now run infracost breakdown --path=... and point to your Terraform directory or JSON/plan file.
Registration of Infracost to receive an API key

After registration, the API key is saved in  ~/.config/infracost/credentials.yml.

For the following given example of Infracost usage, I am using a Terraform mini-project explaining how to associate elastic IP addresses to a network load-balancer.

Estimate full breakdown of costs

Infracost can be executed to have the full monthly breakdown of costs :

$ infracost breakdown --path terraform_nlb_static_eips
Detected Terraform directory at terraform_nlb_static_eips
  ✔ Running terraform init
  ✔ Running terraform plan
  ✔ Running terraform show
✔ Calculating monthly cost estimate
Project: terraform_nlb_static_eips
Name                                                      Monthly Qty  Unit            Monthly Cost
aws_autoscaling_group.webserver
 └─ aws_launch_configuration.webserver
    ├─ Instance usage (Linux/UNIX, on-demand, t2.micro)              0  hours                  $0.00
    ├─ EC2 detailed monitoring                                       0  metrics                $0.00
    └─ root_block_device
       └─ Storage (general purpose SSD, gp2)                         0  GB-months              $0.00
aws_eip.nlb
 └─ IP address (if unused)                                         730  hours                  $3.65
aws_lb.this
 ├─ Network load balancer                                          730  hours                 $16.42
 └─ Load balancer capacity units                         Cost depends on usage: $0.006 per LCU-hours
PROJECT TOTAL                                                                                $20.07
Registration of Infracost to receive an API key

You can play modifying the code and re-run the previous command to have an updated cost estimation of the new version.

Diff monthly costs between the current and planned state

Terraform updates, deletes, or creates resources saves the infrastructure state in a file. When you deploy a new version, Terraform is able to detect what needs to be updated, removed, created following your changes. Infracost relies on this state file to show you the cost impact of the current change from the plan.

To illustrate to you that, I  deployed the initial code version using terraform apply. Then I scaled out the current EC2 instance type from t2.micro to m4.large:

$ git --no-pager diff                                                                                                                             
diff --git a/launch-configuration.tf b/launch-configuration.tf
index db22265..bcf90c7 100644
--- a/launch-configuration.tf
+++ b/launch-configuration.tf
@@ -1,7 +1,7 @@
 resource "aws_launch_configuration" "webserver" {
   name_prefix                 = "${local.name_prefix}_webserver"
   image_id                    = data.aws_ami.ubuntu.image_id
-  instance_type               = "t2.micro"
+  instance_type               = "m4.large"
   security_groups             = [
  aws_security_group.public.id]
   user_data                   = data.template_cloudinit_config.this.rendered
Git diff showing the upgrade of instance_type which will occur higher cost

Then I launch Infracost command with diff argument :

$ infracost diff --path .                                                                                                                                                                           
Detected Terraform directory at .
  ✔ Running terraform plan
  ✔ Running terraform show
✔ Calculating monthly cost estimate
Project: .
~ aws_autoscaling_group.webserver
  +$64.53 ($11.37 -> $75.90)
~ aws_launch_configuration.webserver
- Instance usage (Linux/UNIX, on-demand, t2.micro)
          -$8.47
+ Instance usage (Linux/UNIX, on-demand, m4.large)
          +$73.00
Monthly cost change for .
Amount:  +$64.53 ($27.79 -> $92.32)
Percent: +232%
The output of Infracost showing the cost estimation of the instance type upgrade

The diff usage is perfect to be added in complement of a code review to justify and argue the cost of a change.

Refine cost estimation with usage file

You can feed Infracost with a usage file to refine the cost estimation :

version: 0.1
resource_usage:
  aws_autoscaling_group.webserver:
    instances: 15 # Number of instances in the autoscaling group.
    operating_system: linux # Override the operating system of the instance, can be: linux, windows, suse, rhel.
    reserved_instance_type: standard # Offering class for Reserved Instances. Can be: convertible, standard.
    reserved_instance_term: 1_year # Term for Reserved Instances. Can be: 1_year, 3_year.
    reserved_instance_payment_option: no_upfront # Payment option for Reserved Instances. Can be: no_upfront, partial_upfront, all_upfront.
    # Only applicable when T2 credit_specification is set to unlimited or T3 & T4 instance types are used within a launch template,  or T3 & T4 instance types are used in a launch configuration.
    monthly_cpu_credit_hrs: 350 # Number of hours in the month where the instance is expected to burst.
    vcpu_count: 2 # Number of the vCPUs for the instance type.

  aws_lb.this:
    new_connections: 10000    # Number of newly established connections per second on average.
    active_connections: 10000 # Number of active connections per minute on average.
    processed_bytes_gb: 1000  # The number of bytes processed by the load balancer for HTTP(S) requests and responses in GB.
    rule_evaluations: 10000   # The product of number of rules processed by the load balancer and the request rate.

This file is added to the command arguments :

$ infracost breakdown --path . --usage-file=infracost-usage-example.yml
                                                                                                                               Detected Terraform directory at .
  ✔ Running terraform plan
  ✔ Running terraform show
✔ Calculating monthly cost estimate
Project: .
Name                                                   Monthly Qty  Unit       Monthly Cost
aws_autoscaling_group.webserver
 └─ aws_launch_configuration.webserver
    ├─ Instance usage (Linux/UNIX, reserved, m4.large)       10,950  hours           $678.90
    ├─ EC2 detailed monitoring                                  105  metrics          $31.50
    └─ root_block_device
       └─ Storage (general purpose SSD, gp2)                    120  GB-months        $12.00
aws_eip.nlb
 └─ IP address (if unused)                                      730  hours             $3.65
aws_lb.this
 ├─ Network load balancer                                       730  hours            $16.42
 └─ Load balancer capacity units                              1,000  LCU-hours         $6.00
PROJECT TOTAL                                                                       $748.48
The output of Infracost showing the cost estimation with the usage file

Report

You can generate reports to share cost estimation in different formats (HTML, JSON, etc..) :

$ infracost breakdown --path . --format html > report.html
Generation of an HTML cost-estimation report
An HTML cost-estimation report generated by Infracost
An HTML cost-estimation report generated by Infracost

Cost-Estimation Review With GitHub Action

You can integrate Infracost with a lot of CI/CD solutions. Here we’re going to focus on Github Action because it is easy to reuse existing actions from the marketplace. In the workflow, we’re going to use the following actions :

The configuration of the workflow is configured in the terraform git project in.github/workflows/infracost.yml:

on:
  pull_request:
    paths:
    - '**.tf'
    - '**.tfvars'
    - '**.tfvars.json'
jobs:
  infracost:
    runs-on: ubuntu-latest
    name: Show infracost diff
    steps:
    - name: Check out repository
      uses: actions/[email protected]

    - name: Configure AWS Credentials
      uses: aws-actions/[email protected]
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: us-east-1

    - name: "Install terraform"
      uses: hashicorp/[email protected]

    - name: "Terraform init"
      id: init
      run: terraform init
      working-directory: .

    - name: "Terraform plan"
      id: plan
      run: terraform plan -out plan.tfplan
      working-directory: .

    - name: "Terraform show"
      id: show
      run: terraform show -json plan.tfplan
      working-directory: .

    - name: "Save Plan JSON"
      run: echo '${{ steps.show.outputs.stdout }}' > plan.json # Do not change

    - name: Run infracost diff
      uses: infracost/[email protected]
      env:
        INFRACOST_API_KEY: ${{ secrets.INFRACOST_API_KEY }}
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
      with:
        entrypoint: /scripts/ci/diff.sh # Do not change
        path: .
        usage_file: infracost-usage.yml
GitHub action configuration
Credentials needed by the Github action workflow stored in a vault
Credentials needed by the Github action workflow stored in a vault

I committed the previous instance type change to a branch named dev and opened a pull request. This triggers the steps of the workflow :

Creation of the GitHub pull request
Creation of the GitHub pull request
Execution of the GitHub action workflow steps
Execution of the GitHub action workflow steps

When the workflow executed ends, the Infracost output is visible inside the pull request :

The infracost output in the pull request
The infracost output in the pull request

Resources

Assign Static Ip to AWS Load Balancers
What type of load-balancer should I use? How to do with Terraform?
Cloud cost estimates for Terraform in pull requests | Infracost
Infracost shows cloud cost estimates for Terraform projects. It integrates into pull requests and allows developers and DevOps to see cost breakdowns and compare options upfront.
infracost/infracost
Cloud cost estimates for Terraform in your CLI and pull requests 💰📉 - infracost/infracost
Features • GitHub Actions
Easily build, package, release, update, and deploy your project in any language—on GitHub or any external system—without having to run code yourself.
infracost/infracost-gh-action
GitHub Action for Infracost. Shows cloud cost estimates for Terraform in pull requests. - infracost/infracost-gh-action
aws-actions/configure-aws-credentials
Configure AWS credential environment variables for use in other GitHub Actions. - aws-actions/configure-aws-credentials
Infrastructure as CodeCI/CD

Guillaume Vincent

DevOps Engineer & AWS Certified Solution Architect. Cloud enthusiast and automation addict