Environment Variable Management with Terraform and Google Cloud Secrets Manager

Motivation and Introduction

In recent years we had the opportunity to work with a lot of different technologies and tools. One of the most important ones is Terraform. Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. In those years we also kept struggling with the same issues over and over again: when a new developer joins the project, we need to exchange the environment configuration.

We usually store our configuration in the environment (according to the 12 Factor App), which enables us to run the same service in different cloud environments (e.g. staging, test, production). The values are stored in Terraform / Terraform Cloud and either set in the runtime or "baked" into the Docker image.

For local developer environments, there's little tooling:

• commit the entire .env plaintext in the repository
• share an .env file via Slack/Email
• have an encrypted .env.encrypted in the repository and use a tool like SOPS to encrypt/decrypt
• have an .env.example file and secrets in LastPass or some other password manager
• ...

None of those options is great, especially when it comes to updating values and rotating keys.

One possible improvement to this situation it the use of Google Cloud (or AWS) Secrets Manager. In most cases, the secrets and tokens needed on the developer machine are very similar to the "development" or "staging" cloud environment, so why not just synchronize from there?

We use Terraform to initialize the versions for each secret. As the configuration for google_secret_manager_secret is a little verbose, we decided to go with a local for_each:

locals {
  vars = {
    "auth0_domain" = {
      secret_id   = "auth0-domain",
      secret_data = var.auth0_domain
    },
    "auth0_audience" = {
      secret_id   = "auth0-audience",
      secret_data = var.auth0_audience
    },
    "auth0_app_client_id" = {
      secret_id   = "auth0-app-client-id"
      secret_data = auth0_client.app.client_id
    },
    "auth0_app_client_secret" = {
      secret_id   = "auth0-app-client-secret"
      secret_data = auth0_client.app.client_secret
    }
  }
}

resource "google_secret_manager_secret" "s" {
  for_each = local.vars

  project   = var.project
  secret_id = each.value.secret_id

  replication {
    automatic = true
  }
}

resource "google_secret_manager_secret_version" "v" {
  for_each = local.vars

  secret = google_secret_manager_secret.s[each.key].id

  secret_data = each.value.secret_data
}

After running this, we can see the secrets in the Google Cloud Console:

These can be used in other Terraform service configurations. In order to use them locally, we can use envsync. It's a simple tool that fetches secrets from Google Cloud Secrets Manager and writes them to an .env file.

npm install -g envsync
npx envsync

An .env.example file is required to define the variables that should be synchronized, as well as a keyfile.json and a gcp project id.

Conclusion

This is a first step towards a better developer experience with environment variables. We're still testing a few edge cases, but we're happy with the results so far.

If you have any questions or comments, please reach out on Twitter or join the discussion on GitHub.