Support full reconstruction of HCL from output dictionary

[weaversam8]

This is a follow up to #169, to more fully address the use case discussed in #23.

Along the way, I also found and fixed #171, #172, and #173. (To view those changes in isolation, please look at the diffs of the commits linked to those issues specifically.)

I also believe that this supersedes @zabacad's PR #143 to support nested interpolation. (Sorry Ian! I didn't see your PR and recreated your work - but we landed on similar approaches!)

I'm submitting this PR as a draft right now, because while this is technically working, there are a few more things I'd like to add. Namely:

• Better documentation about how to use this feature
• Better whitespace and formatting embedded in the AST generated from the dictionary
  • (this draft implementation produces syntactically equivalent output but it's very ugly)
• Helper functions to control the formatting of the generated output by embedding metadata in the Python dictionary
• Better error messages when parsing an invalid dictionary
• More tests for all of the above.

This PR adds:

• New API methods:
  • reverse_transform(hcl_dict) which returns an AST generated from a Python dictionary
• Lark grammar changes:
  • Changes to handle nested string interpolation (similar to Allow expressions in string interpolation #143)
  • Renamed several terminals to make them easier to reference when reconstructing
• BREAKING: changes to the output of the DictTransformer class to ensure it generates valid HCL expressions. (See HCL interpolations within dict output contain single quotes #172, I hope it's OK to include this since we just did a major version bump.)
• hcl2.reconstructor changes:
  • I added the HCLReverseTransformer class which aims to reverse the process that hcl2.transformer.DictTransformer takes (the bread and butter of this PR)
  • I overhauled the whitespace/formatting handling that I implemented in Support full reconstruction of HCL from parse tree #169 to make those decisions based on individual tokens/rules, rather than simply looking at the characters only
• Tests:
  • I added a simple test for HCL escape sequences for Terraform escape sequences are not processed when parsing strings #171
  • I updated the test files for the fixes I made for HCL interpolations within dict output contain single quotes #172 (see BREAKING note above)
  • I added a test for deeply nested string interpolation (Lark grammar does not support interpolations nested more than 2 times #173)
  • I modified test_load to print the full diff when a file does not match its expected JSON
  • I added test_reconstruct_dict to ensure that the HCL reconstructed from a Python dictionary is syntactically equivalent (whitespace checks will be added to this file soon.)

Still a draft PR, as described above, but open to hear any comments or suggestions!

[weaversam8]

OK, I think this is ready for a proper review. Since the draft, I added:

• More extensive documentation on how to use these features
• A rudimentary hcl2.Builder class allowing users to create fresh HCL documents from Python (the reason why this is necessary is described in the docs I added)
• Better formatting defaults on output from hcl2.reverse_transform(...).
• Tests for the hcl2.Builder class.

[weaversam8]

@kkozik-amplify - still reviewing this? any questions I can answer about my changes?

[kkozik-amplify]

@weaversam8 Yeah, need some more time. In the meantime - could you look into the errors that Codacy Analysis PR check reported?

[Nfsaavedra]

Hi!

While trying to use the code in this branch I found that when parsing the following file:

locals {
  tags = {
    application = "Modernisation Platform: ${terraform.workspace}"
  }
}

The program gets stuck in a regex match. I am almost sure it is related to the changes to better support string interpolation (that's why I'm using this branch).

[Nfsaavedra]

Suggested change

	STRING_LIT : "\"" (STRING_CHARS)* "\""
	STRING_CHARS : /(?:(?!\${)([^"\\]|\\.))+/+ // any character except '"' unless inside a interpolation string
	STRING_LIT : "\"" STRING_CHARS? "\""
	STRING_CHARS : /(?:(?!\${)([^"\\]|\\.))+/ // any character except '"'

This seems to fix the problem, but I did not run the tests to check it. It does parse this file correctly tho.