Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support full reconstruction of HCL from parse tree #169

Merged

Conversation

weaversam8
Copy link
Contributor

Hi there, this closes #23. Many folks were asking for the ability to convert the parsed files back into HCL. This PR does a big part of the heavy lifting required to enable that functionality.

Currently the ultimate output of this library is a dictionary transformed from the underlying AST generated by Lark. I did not attempt to reverse the transformation process that this library takes while traversing the AST. (Maybe a later PR.) I did, however, use Lark's experimental "reconstruction" functionality to provide a way to transform the AST back into valid HCL. As such, if developers are willing to manipulate the AST instead of the final output dictionary, reversing the parse back into HCL is now possible with this PR.

A high level overview of the changes in this PR:

  • I added new API methods:
    • parse(file) and parses(string) return an AST (Lark.Tree) after parsing an HCL2 file
    • writes(ast) writes an AST back to a valid HCL2 string
    • transform(ast) allows a user to convert the AST to a dictionary using the existing DictTransformer class, like the load and loads methods already do under the hood.
  • I modified the Lark grammar to capture whitespace within newline, comment, and = tokens, which doesn't change how those rules are ultimately parsed
  • I made a few small changes to DictTransformer in hcl2.transformer to ensure it worked with the tweaks I made to the grammar
  • I implemented hcl2.reconstructor, which rebuilds the parser without a cache, implements the Reconstructor, and contains logic to add whitespace to the correct tokens when reassembling the file
  • I added new tests in test/unit/test_reconstruct.py to test my new functionality.
    • test_write_terraform ensures that every HCL2 file produces an identical dict when parsed, even after reconstruction
    • test_write_terraform_exact ensures that the reconstructed HCL2 exactly matches the input HCL2, down to the individual character and whitespace level.
      • Note: some files are hardcoded within this test method to be skipped, as the reconstruction doesn't perfectly reproduce them in its output. This is OK, because the output is still identical when parsed. This behaviour is verified by the above test_write_terraform test, which has no such exceptions.

I have verified that all existing tests are passing, and overall test coverage is improved by this PR. Please let me know if you have any questions, or suggested changes.

@weaversam8 weaversam8 requested a review from a team as a code owner October 2, 2024 19:41
@kkozik-amplify kkozik-amplify self-assigned this Oct 4, 2024
@rddimon rddimon added the enhancement New feature or request label Oct 4, 2024
hcl2/hcl2.lark Outdated Show resolved Hide resolved
.gitignore Outdated Show resolved Hide resolved
@kkozik-amplify
Copy link
Collaborator

@weaversam8 Hi, thank you for contributing! The PR looks good - we will release this today with 5.0.0 tag

@kkozik-amplify kkozik-amplify merged commit 1dff4d7 into amplify-education:main Oct 7, 2024
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support python hcl2 object to hcl2 text
3 participants