Extend Terraform Interactions With Data Sources

Extend Terraform Interactions With Data Sources

See how to interconnect Terraform to your REST APIs

Guillaume Vincent
Guillaume Vincent

Terraform is an infrastructure tool supporting plenty of cloud providers. Sometimes, you need to extend Terraform with your own third parties. You can also create create your own provider but it may be complex if you need a simple glue. In this article, we're going to use data sources to interconnect with a REST API.

What Is Data Source?

A data source fetches and computes data from a remote source. It has attributes that you may reuse in another resources. This creates an implicit dependency between the data source and the resource :

# Find the latest available AMI that is tagged with Component = web
data "aws_ami" "web" {
  filter {
    name   = "state"
    values = ["available"]
  }

  filter {
    name   = "tag:Component"
    values = ["web"]
  }

  most_recent = true
}
resource "aws_instance" "web" {
  ami           = data.aws_ami.web.id
  instance_type = "t1.micro"
}

They are useful to reduce coupling between modules in your project. You don't have to pass new inputs each time to the module and create dependencies. It is also great to hide the complexity to the module's users.

Mocking REST API

We're not creating an application here but we'll use JSONPlaceHolder to mock it. The fake backend serves JSON todo tasks :

$ curl https://jsonplaceholder.typicode.com/todos/1
{
  "userId": 1,
  "id": 1,
  "title": "delectus aut autem",
  "completed": false
}

In the next sections, we'll see how we can fetch data using data sources.

HTTP Data Source

The HTTP data source makes an HTTP request to a given URL. It exports information about the response :

When executing an apply the JSON body is visible in the outputs :

$ terraform init
$ terraform apply
data.http.this: Refreshing state... [id=https://jsonplaceholder.typicode.com/todos/1]
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
Outputs:
todo = {
  "userId": 1,
  "id": 1,
  "title": "delectus aut autem",
  "completed": false
}

This data source is simple to use but it doesn't support authentication or different protocol than HTTP. We'll see another data source as a workaround for these use cases.

External Data Source

“The external data source allows an external program implementing a specific protocol (defined below) to act as a data source, exposing arbitrary data for use elsewhere in the Terraform configuration.” From Terraform official documentation

The data source needs to have the path of the program. The external program accepts a JSON object from stdin stored in the query parameter. It can return a JSON object as output with all values as strings. The expected return code for a successful program result is zero :

data "external" "example" {
  program = ["python", "${path.module}/example-data-source.py"]

  query = {
    # arbitrary map from strings to strings, passed
    # to the external program as the data query.
    id = "abc123"
  }
}

We’ll see two ways of implementing a python program with the following terraform file :

Raw python program

The program collects a JSON object from the query parameter. It uses the nested information to make an HTTP GET call to get the todo tasks. The values of the todo task are cast into string types. By default, the script exits on zero exit code except if something bad occurs.

The program is testable using the command-line :

$ echo '{"id": 1}' | python fetch_todo.py
{"userId": "1", "id": "1", "title": "delectus aut autem", "completed": "False"}

Terraform-external-data package

If you’re looking for a more abstractive way you may use the terraform-external-data package. It provides a function decorator dealing with what we’ve seen in the raw approach :

Resources

Data Sources - Configuration Language - Terraform by HashiCorp
Data sources allow data to be fetched or computed for use elsewhere in Terraform configuration.
JSONPlaceholder - Free Fake REST API
terraform-external-data
Provides a decorator that implements terraform’s external program protocol for data sources.
Infrastructure as Code

Guillaume Vincent

DevOps Engineer & AWS Certified Solution Architect. Cloud enthusiast and automation addict