Errors when including a dependency output in state path

Gruntwork Customers
,
1

I have a terragrunt setup where instead of having multiple directories for each region in which resources are deployed, the applying region and account are passed in as env variables, which are consumed in the root terragrunt config, used to determine state path, then passed to the module with a generate block. I’m getting some strange errors with this setup that i think are related to an update from 0.31 to 0.35. the issue I’m seeing is, i think, related to the optimizations in Optimize dependency output retrieval · gruntwork-io/terragrunt@0bf8185 · GitHub , which does seem to be consistent with the fact that that commit is part of version 0.35 and didn’t exist in my previous version (i think lol). anyway the problem is like this: if i have module A that depends on an output from module B, and module B was last applied in us-west-2 account 123, but now I’m trying to apply module A in us-east-1 account 456, then in the plan for the apply of module A, I’ll see that the dependency output from module B is for resources created in us-west-2 account 123 instead of the region and account I want. this didn’t used to happen previously, so i went looking in the terragrunt source to figure out why and found this comment that says terragrunt will try to use the remote statefile to get dependency outputs instead of the results of a terraform output within the module as long as there are no dependency outputs in the state path. it kinda makes sense why this is problematic for my setup, since I’m using region and account inputs taken from the env as part of the state path, and i figure terragrunt is going to the terragrunt cache to determine where the remote state file lives for the dependent module instead of figuring it out by calculating the remote state path with what I’ve put in the remote state configuration. all good i figured, that comment indicates that if i just put a dependency output in the state path, then terragrunt will do a full terraform output like before in order to get dependency outputs and will therefore start respecting my account and region inputs again like before i upgraded. I made a new terrragrunt module that does nothing except outputs a string that was previously hardcoded into the state path, so now i technically have a dependency output in there and should see terragrunt using terraform output results to get dependency outputs, and i do. it even seems like it fixes my problem i had before? only issue is that i see some errors that don’t make a ton of sense:

ERRO[0003] Could not convert include to the execution context to evaluate additional locals  prefix=[/Users/me/repo/infrastructure/deployment/global/module1] 
ERRO[0003] Encountered error while evaluating locals.    prefix=[/Users/me/repo/infrastructure/deployment/global/module1] 
ERRO[0003] Could not convert include to the execution context to evaluate additional locals  prefix=[/Users/me/repo/infrastructure/deployment/account/module2 
ERRO[0003] Encountered error while evaluating locals.    prefix=[/Users/me/repo/infrastructure/deployment/account/module2]

and so on for all my dependencies. thing is though, just before those errors i see something like this in the debug log:

time=2021-12-19T18:20:17-08:00 level=debug msg=Evaluated 6 locals (remaining 0): var1, var2, var3, var4, var5, var6, prefix=[/Users/me/repo/infrastructure/deployment/account/module2]

comprising all the locals I set in the top level terragrunt config being sourced like:

include "global_configs" {
  path           = find_in_parent_folders()
  merge_strategy = "deep"
  expose         = true
}

i tried messing with merge strategy to no avail. the fallback to terraform output is working at least:

time=2021-12-19T18:20:17-08:00 level=debug msg=Could not parse remote_state block from target config /Users/me/repo/blahblah
time=2021-12-19T18:20:17-08:00 level=debug msg=Falling back to terragrunt output.

is there some kind of workaround anyone can think of? can i just ignore these errors? if they’re ignoreable, is it possible to silence them somehow? I’d rather not just grep out errors matching this pattern cuz there might be a legit error in there sometimes.

long term, would it make sense to submit a PR changing the dependency output optimization to recalculate the remote state path for each dependency instead of using whatever’s precalculated in the cache? I assume this isn’t currently happening, but i haven’t looked too much into the code yet. and if I’m right about where terragrunt is getting its information about where remote state lives, would it be possible to use a pre-commit hook to force recalculation of the remote state path within dependencies? seems like a bit of a long shot but yeah.

would greatly appreciate any input from anyone who might know more about the internals of this part of terragrunt than i do

2

Hi @spacerainbow000

It’s a bit difficult to debug this without seeing the entire terragrunt.hcl config of both the child and parent, as the error is likely in how and where you are referencing one of the parent included configs.

That said, you should be able to force Terragrunt to always call terraform output with a new initialization if you set disable_dependency_optimization = true on your remote_state block. This makes sense for your setup where the remote state configuration is dynamic, which is not a usual operating mode of terragrunt (we typically recommend keeping a separate terragrunt.hcl file and folder for each region-account deployment). Try that out, and see if it resolves your issue.

Yori

3

disable_dependency_optimization = true is exactly what i needed, thanks so much. now the question is, how do i get the benefits of dependency output collection optimization again? I’d like to be able to use this feature since it makes the terragrunt output much cleaner (no init & terraform output logs written to console) not to mention making the run faster. how much of an overhaul would it be to change the terragrunt code so that the state path is calculated for each dependency the way it’s calculated for the target module before dependency outputs are collected? I haven’t looked around in the terragrunt internals very much yet, but if it’s within the realm of feasibility and doesn’t introduce any serious tradeoffs in terms of performance, then I’m willing to take a crack at it myself and then open a PR against the project i guess

on a mostly unrelated note, the dynamic remote state path thing is a workaround for the relatively large maintenance overhead produced by having different terragrunt files per region. I’m deploying pretty much the same set of modules in every region, so if i were to have one directory per region, i would have like 20 directories containing the same set of files. then say i change the input variables on the associated terraform module, I’d have to go and make the same change to the terragrunt inputs block for all 20 copies of the terragrunt module that calls that terraform module. I’m deploying to a bunch of accounts too, so there’s a bunch of duplication there too. i quickly discovered that doing this is completely unmaintainable, so I created a directory structure with three subdirectories, global, account, and region to indicate module scope, then inside each there’s just one copy of each of the terragrunt modules that needs to be applied at that scope. account and region are consumed via environment variables (passed in by a wrapper script) in the root level terragrunt.hcl config, which then uses those values to calculate a state path broken out like:

- global
  * global level modules in a directory
- - account A
    * account level modules in a directory
- - - region 1
      * region level modules in a directory
- - - region 2
      * etc
- - account B
- - - region 1
- - - region 2
...

and passes in region and account into the terraform module itself via a generate block.

this solution isn’t perfect, but I’ve got it pretty dialed in and the whole ‘scope’ thing works pretty well considering how hacky it is. I see stuff in the docs and on the forums about how yall recommend a separate directory for each region being deployed to, but I don’t see much discussion about what to do about all the essentially identical terragrunt files this implies (assuming it’s the same stuff getting deployed to all those regions). I know terragrunt has a feature that lets you consume another hcl file and merge its config into the current file; is the official recommendation for the scenario I’ve described to have a deployment-template directory with the ‘real’ terragrunt code in it, minus the region/account variable, then a deployment directory where the terragrunt files source the appropriate deployment-template hcl file for the module being deployed, merge its config to the current file, then add input variables for region and account? I could see that working, but it’s not without tradeoffs same as the calculated state path thing I’m doing right now. is there a better way that I haven’t thought of?