How to handle a set of TF blocks that are required on multiple modules

Gruntwork Customers
1

So, my first major project properly using Terragrunt is coming together.
Advanced apologies for the long ramble. I’m writing this at the end of the day and my brain cells are in the single digits.

TLDR: How should I handle a set of TF blocks that are identical across multiple modules? Should I just persist the generated files rather than them being discarded in the TG cache…?

I’m just going through and abstracting stuff. Like the title says, I have a bunch of vars that are identically required on multiple modules. I already have these vars being assigned/populated generically and programmatically (not sure what the correct TF/TG nomenclature is), but my problem is abstractly handling the actual variable and data blocks.

I’m using the data blocks to avoid accidental manipulation of those resources by it being a Terragrunt dependency. But please tell me if I’m being over-cautious with this. I’m still an IaC newbie.
However, to keep the TF DRY, I’m fairly sure there should be a way to handle repeated vars (or TF blocks of any type) that is better than something like symlinking the file.

Initially, I thought generating the shared vars in their own file would work. However, that means the modules can’t be used in isolation.
For more context, m`y implementation of this looks something like this:

# common.hcl
generate "common_vars" {
  path      = "common.var.gen.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
variable "gcp_region" {}
variable "gcp_zone" {}
# ----------------------------------------------------------------
variable "org_id" {}

data "google_organization" "org" {
  organization = var.org_id
}
# ----------------------------------------------------------------
variable "vpc_host_project_id" {}

data "google_project" "vpc_host" {
  project_id = var.vpc_host_project_id
}
EOF
}

Currently, my thinking is that rather than generating the duplicated var files in the Terragrunt cache, I could instead generate them into the Terraform modules’ source directories using relative_to/from (I can never remember which way round they are).

That way, the changed Terraform files are actually persisted in the branch on VCS and as a result the TF registry. And they will be updated in-branch (and associated Env) by Terragrunt if needed.

Or am I getting completely the wrong end of the stick and thinking cross purposes…?

2

i think using a generate block to write the files to the terraform module source directory is more trouble than it’s worth and won’t solve your problem with having to override some of those common vars on a per-module basis. there are several possible solutions here that i think would work better, presented here in increasing order of effectiveness (IMO, and varying depending on how you’re using terragrunt).

first off, whichever way you go, it makes the most sense to put this generate block of yours in a root-level terragrunt config which you then source in all your other terragrunt files (apologies if you’re already doing this) - also a good place for your provider confgs, remote state path, etc.
in the module config:

include "root_config" {
  path           = find_in_parent_folders()
  merge_strategy = "deep"
  expose         = true
}

then in the root config:

generate "common_vars" {
  path      = "common.var.gen.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
variable "gcp_region" {}
...

option one: tf file overrides

break your list of common vars out into different files grouped by purpose, then let the more specific module-level .tf file take precedence over the more generic generated .tf file.
this option would look like this: you use if_exists = skip to ensure that if the file exists in the terraform module code, then whatever you set there takes precedence over the value defined in the generate block.

assuming you’re using gcp_region and gcp_zone as part of a provider config (idk i’m unfamiliar with gcp), you’d have a generate block that looks like:

generate "provider_vars" {
  path      = "provider_vars.tf"
  if_exists = "skip"
  contents  = <<EOF
variable "gcp_region" {
    default = "whatever your default region is here, idk us-west-1 or whatever"
}
variable "gcp_zone" {
    default = "default-zone-1"
}
EOF
}

and then presumably another generate block for your provider config that makes use of these variables like:

generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite"
  contents  = <<EOF
provider "gcp" {
    region = var.gcp_region
    zone   = var.gcp_zone
}
EOF
}

then when you want to override the default provider config, you create a file like provider_vars.tf in your terraform module like:

variable "gcp_region" {
    default = "whatever your default region is here, idk us-west-1 or whatever"
}
variable "gcp_zone" {
    default = "default-zone-1"
}

this setup allows you to override some of your base variables based on what the usage of those variables is. you could take this even further by putting each base variable in its own generate block that writes its own file to the terragrunt cache, where each file is named after the single variable it contains

this option works, but only if you’ve got a lot of modules that are only being instantiated once, which isn’t very DRY.

option two: tf file overrides with provided inputs

if you’re instantiating terraform modules multiple times, and you sometimes want to override some of your base variables but sometimes don’t, you can do a conditional override based on terragrunt inputs. in your base terragrunt file:

generate "region" {
  path      = "region.tf"
  if_exists = "skip"
  contents  = <<EOF
variable "gcp_region" {
    default = var.override_region == "" ? "region-1" : var.override_region
}
EOF
}
generate "override_region" {
  path      = "override_region.tf"
  if_exists = "skip"
  contents  = <<EOF
variable "gcp_region" {
    default = var.override_region == "" ? "region-1" : var.override_region
}
EOF
}

then in the module level terragrunt file:

locals {
    region = region-2
}
inputs = {
    override_region = local.region
}

what advantage does this confer? I don’t know, i’m thinking this stuff up as i type it out. i didn’t realize until a second ago that having the variables in the generate block in option 1 means you can override the default value in the same way you override it here, via terragrunt inputs, without having to overwrite the file at all. where it gets tricky is if you want to start overriding input values programmatically, like overriding the region value based on something in the directory path (like parsing out the region based on what dir you’re in, since it’s apparently common to have a bunch of terragrunt files in region-named directories), or for a use case I’m more familiar with, which is overriding other values based on the value of one of your global vars.

if you try to go this route, you should really test it thoroughly because I’m not positive it’ll even work. i think the way terragrunt passes input variables into terraform has changed before, so theres a possibility it might change again. the way it works now might not be compatible with this way of doing things, because i think terraform generates a .tfvars file that might not take precedence over the default value set in your generate block. or it might use environment variables… there’s a doc i think on the terragrunt site but i can’t find it right now, and youll have to cross reference that with the terraform variable processing precedence info from hashicorp, and of course, your own testing.

option 3: an overrides file
combine the ternary from option two with some terragrunt builtin function magic. in the same directory as your module terragrunt.hcl, make a file like overrides.json`:

{
    "region": "region-2"
}

(make this file contain {} if you don’t want any overrides)
then in your root terragrunt.hcl:

locals {
  default_region  = "region-1"
  calculated_region = lookup(fileexists("${get_original_terragrunt_dir()}/overrides.json" ? jsondecode(file("${get_original_terragrunt_dir()}/overrides.json")) : {}), "region", local.default_region)
}
generate "region" {
  path      = "egion.tf"
  if_exists = "skip"
  contents  = <<EOF
variable "gcp_region" {
    default = "${local.calculated_region}"
}
EOF
}

and source it in your module terragrunt.hcl like in the other options:

include "root_config" {
  path = find_in_parent_folders()
}

now you can override the value for one instantiation of the module by changing the value in overrides.json, or for all instantiations of the module by creating your own region.tf in the terraform directory.

there may be a more elegant way to accomplish this where you override anything you need to override in the child terragrunt config, then the generate block gets created with the base values set in your parent terragrunt.hcl and any overrides you set in the child terragrunt.hcl by doing some fancy stuff with terragrunt’s block parsing order, but any time i try to come up with solutions that make use of the block parsing order stuff my eyes start going in two different directions.

it’s fitting that you wrote your original post trying to rub your last two brain cells together for warmth because I’ve done the same with this response. let me know more about your use case and i can help you more