Examples of 3-Tier Module Structure

Gruntwork Customers
,
1

Does anyone have publicly documented examples of using a 3 tier module structure working with terragrunt?

For context, see starting at slide 22 of https://www.slideshare.net/VadimSolovey/terraform-modules-restructured-220136382. As described in this post, your “infrastructure” repo would consume “terraform-services” which consume “terraform-resources”.

I’ve also seen people use this methodology with slightly different terms, your “instance”/“infrastructure” repo could consume a “unit” which consumes “modules”.

In the majority of the documentation that I’ve read online, this is not the method that people follow. Most examples I’ve read appear to be following a pattern of having the “infrastructure” repo call “modules”, which do a number of things, directly. The general concept is that there is almost certainly some duplication in the “modules” in this case, that could be further simplified.

I was attempting to mock this out very simply using ‘outputs’ and for whatever reason I was having a difficult time having my “units” pass variables on to my “modules”. For instance:

INFRASTRUCTURE REPO

.
├── instance.tfvars
└── terragrunt.hcl

instance.tfvars:

myvars = {
  foo = "bar"
  bar = "baz"
  baz = "foo"
}

terragrunt.hcl

terraform {
  source = "git@github.com:example/terraform-units.git//outputs?ref=master"
}

include {
  path = find_in_parent_folders()
}

UNITS REPO (or terraform-services)
main.tf:

terraform {
  backend "s3" {}
}

variable myvars {}

module "outputs" {
  source = "git@github.com:example/terraform-modules.git//outputs?ref=master"

  myvars = var.myvars
}

MODULES REPO (or terraform-modules)
main.tf:

terraform {
  backend "s3" {}
}

variable myvars {}

output "myvars" {
  value       = var.myvars
  description = "myvars"
}

MAIN TERRAGRUNT.HCL

remote_state {
  backend = "s3"
  config = {
    bucket = "some-bucket"
    key = "state/${path_relative_to_include()}/terraform.tfstate"
    region = "some-where"
    encrypt = true
  }
}

terraform {
  extra_arguments "-var-file" {
    commands = get_terraform_commands_that_need_vars()

    optional_var_files = [
      "${get_terragrunt_dir()}/instance.tfvars"
    ]
  }
}

Like I mentioned, I’ve seen it work, I just don’t recall exactly what the magic piece I’m missing was in the “units” or “terraform-services” piece was that allowed me to receive the values in the module. When I run this particular example, I get nothing. If I call the module directly from the infrastructure repo, instead of the ‘unit’, I see the proper output.

In reality, what I want/intend to do in the ‘units’, is do a deep merge of all of the maps, where there may be duplicates (for instance, I may have account.tfvars and region.tfvars), and take the key/value specified further down in the stack.

2

As it turns out, this does work if I’m doing something more complex, though for some reason the outputs do not show when I’m doing just outputs and calling the ‘module’ via a ‘unit’.

I modified my ‘units’ main.tf to look something like:

terraform {
  backend "s3" {}
}

variable global {}
variable region {}
variable service {}
variable instance {}

locals {
  combined = merge(var.global, var.region, var.service, var.instance)
}

module "outputs" {
  source = "git@github.com:example/terraform-modules.git//outputs?ref=master"

  myvars = local.combined
}

Which, again, works fine and I can see the result if I’m creating an actual resource, but when I’m only generating outputs, this “works”, but does not print the output like it would if I’m calling the module directly.

3

To take this one step further and do at least two layers of merging (e.g. if you’re specifying tags in multiple places), I’ve done the following for the time being:

locals {
  // Note that the can() function was only introduced in tf v0.12.20.
  instance_merged = {
    for k, v in var.instance:
      k => can(keys(v)) ? merge(
        lookup(var.global, k, {}),
        lookup(var.region, k, {}),
        lookup(var.service, k, {}),
        v
      ) : v
  }
  service_merged = {
    for k, v in var.service:
      k => can(keys(v)) ? merge(
        lookup(var.global, k, {}),
        lookup(var.region, k, {}),
        v
      ) : v
  }
  region_merged = {
    for k, v in var.region:
      k => can(keys(v)) ? merge(
        lookup(var.global, k, {}),
        v
      ) : v
  }
  inputs = merge(var.global, local.region_merged, local.service_merged, local.instance_merged)
}

It goes without saying that you have to be passing in these variables, which I am with a slightly longer terragrunt.hcl in the main directory.

I’m sure there are betters ways and would enjoy seeing how others are doing deeper map merging.

4

Answer:

I needed to define an output in the ‘unit’:

output {
  value = module.outputs
}
5

I’d like to recommend a few of Gruntwork’s resources that discuss how we typically organize infrastructure code. This blog post details how to keep Terraform code DRY, including a discussion of a sensible repository split. You can also check out the example infrastructure-live repo and the example infrastructure-modules repo.

As for your questions on deep merge patterns, you might want to join in on the GitHub issue that is discussing this very topic.