Variables with compound structure - any workaround?

I’ve got resources that I need to describe via variables in order to populate a template. If I need to create many similar resources, I inevitably want to put that description within a list, so that I can iterate over the list via count and cause a single resource declaration to create n resources. Doing this seems to be entirely broken in terraform. When I have a single resource, I just declare a variable for each parameter that may change from environment to environment, and then I have a separate vars.tf file in each environment if I’m going to have multipe parallel environments. That’s simple enough. But if I have multiple similar resources in the same environnment, how do you manage that problem? It’s fine, if less than optimal, if there’s a static number and I just have to define separate variables for each instance - assuming the number is small. But as soon as the number of resources is dynamic, how does one accomplish this?

You can’t have a list of maps. Because none of the functions for extracting values from lists or maps will work on a compound variable. You can’t reasonably have m lists instead of a map for each resource with m keys, both because that would be INCREDIBLY error prone, requiring me to maintain multiple lists AND ensure that updates to a list never accidentally re-order any of the elements, since I cannot refer to them by name.

You can’t have n maps of m keys, because n may be a large number and I would have to refer to each map with a different variable name, so no way to use a single resource declaration with count to access multiple maps - unless you can put the maps in a list, which you cannot do if you want to get any values back out.

But more to the point, if the resource is at all complex, the odds of the set of descriptive values being able to exist in a flat structure are pretty much nil, so you actually need separate lists and maps for each nested element, and that would quickly become impossible to manage.

I asked on the terraform google group, and the response was to use an external tool to RENDER a template dynamically with each resource explicitly listed and all variables already populated so that HCL doesn’t have to do any work. But that is a hugely problematic kludge. Not only does it mean that I have to pre-render my templates - something I probably cannot do when using terragrunt because my template rendering is all too likely to be dependent on outputs of other modules/templates that are executed first, but that’s also an incredibly painful extra step when my vars and templates are separated into separate repos when following terragrunt best-practices.

So does anyone have a solution that works in a terragrunt context? I find it mindboggling that terraform doesn’t support it’s own configuration language in any meaningful way - why define a configuration language that supports nested structures if you can’t write functions/operators which are able to interact with those nested structures? It’s hard to think of any other language out there which uses a square bracket for dereferencing a nested structure which does not support chaining the damn things together - my_list_of_maps[index][“key”] is not valid syntax in HCL, even though my_list[index] is valid and my_map[“key”] is valid, for some incredibly stupid reason. It’s honestly a little difficult to understand how you would even write a parser that does NOT understand that syntax, since any recursive traversal of a parse tree ought to function correctly when it sees that. Never mind that literally every other language in widespread use does exactly that.

Is there some way to have terragrunt run an external tool to render a template prior to running terraform, so that I can at least avoid a manual pre-render step in the template/modules repo?

I can’t possibly be the first person to want to do something like this. I find it mindboggling that there isn’t a standard mechanism for doing this within a tool intended to do the job that terraform is (apparently) intended to do.

Hi Sam,

I think the following could work for you if I understand your intent…

variable "list_of_maps" {
   type = "list"
   default = [
      {
          field = "value"
      },
      {
          field = "value"
      }
   ]
}

Then in you could do…

"${lookup(var.list_of_maps[index], "field")}"

Matt

Unfortunately, that only works for one level of nesting. As soon as you go
deeper than that, there is no way to access anything. In my case, one of
the values in the map is a list.

Even worse, it’s that second layer that also makes it impossible to do the
ugly thing and just create parallel lists and iterate over all of them
together. That’s ugly and error-prone, but it works. But as soon as you
have another layer, it just becomes impossible to manage, since you can’t
generate variable names dynamically, and the only reasonable way to do
flatten everything is to have a naming convention and then concoct variable
names on the fly. However, while you can create map keys dynamically, you
can’t really do that with variable names. Fortunately, the fact that you
can use [] operator to access one nested layer allows me to put lists in a
map and use dynamically generated map keys to get what I’m after. But it is
WAY ugly and would be totally error-prone without machine- generating the
maps.

The terraform group suggested rendering the entire template out of json and
JavaScript via jsonnet, removing the need for resources with a count and
compound variables to support them, since you can just render the same
resource repeatedly into the template based on a json structure. My take on
that was that if I have to write an infrastructure management management
tool, I’d just as soon write code that talks to the AWS sdk directly and
forget about the app in the middle which is only adding to the workload.
It’s no more difficult to write code to generate terraform templates as it
is to write code to generate cloud formation templates or to just talk to
the sdk, so why add terrafrom’s complexity to the mix if it is just making
things more difficult (and it definitely does. I’ve spent much more time
bouncing off problems like this in terraform than I would Have spent just
hand-coding scripts to manage our infrastructure). In truth, terragrunt
already acts as infrastructure management-management tool, so unless I want
to somehow automate terragrunt in yet another layer, I would have to add
the functionality for rendering a template to terragrunt or else the
template couldn’t have any dependencies on other module and template
outputs so that I could render it independently of terragrunt and its
copying of files into a temp repo in order to combine vars and templates.

At the moment, my plan is to write a script to generate the canonical
nested structure (in json) as a bunch of simple lists and maps in a tfvars
file that I can generate before running any terraform/terragrunt commands
within a repo. Generating flattened vars from a canonical nested structure
that doesn’t have dependencies on list ordering seems like the only way to
manage things without risking really nasty errors if anything ever gets out
of sync. But it’s a really ugly solution that I’m not very happy about
having to implement.

This is a massive deficiency in terraform which, to be honest and when
coupled to all of the other issues I’ve been having, makes me regret
choosing it as a platform. I’m committed now, just because I’ve put so much
work into it already, not to mention paying for gruntworks library, but I
definitely regret the choice. And the problems are all terraform-specific,
not related to problems in the sdk of providers I’m using or gruntworks’
library.

The problem, for me, is not so much that terraform has a deficiency, but
that they don’t even recognize the seriousness of that deficiency, and
honestly seem to think that forcing people to write a tool to generate the
input for their tool is sufficient workaround to allow them to ignore the
problem. And really, why do they need a custom configuration language at
all in this day and age? What is HCL bringing to the table that json, yml,
xml, or a custom DSL in any number of existing languages wouldn’t solve
with a lot more competence?

In hindsight, I should have understood that if terragrunt has to exist to
make terraform development and infrastructure management a reasonable
proposition for larger teams, the underlying tool is simply not yet mature
enough to warrant adopting. If I could get all my time back, I’d drop
terraform in a heartbeat and start over, at this point.

1 Like

Great points on what Terraform is having now and that it is lacking functionality required for managing a large number of similar resources.

I want to share some of my points from my point of view.

  1. Yes, jsonnet is one of the most recommended solutions for this at the moment. Terraform core team has a solid understanding of such limitation and working on fixing it to support more complex data structures in HCL. For the time being, code generation tools are the main option.
  2. Terragrunt does orchestration of Terraform easier in a way that configuration is DRYer, but I would love to have hooks implemented in Terragrunt so that it generates Terraform code using jsonnet (for the time being) on demand.
  3. I personally find tooling which is required for full-featured infrastructure management rather challenging. For big projects, I have to glue at least Terraform, Terragrunt, Makefile, shell, jsonnet, awspec, CI/CD pipeline. I know that there will not be a silver bullet, just thinking that we (community) are on the right track!

Best regards,
Anton.