A wrapper for Terraform with support for hooks and environments.
Offers a command-line helper tool, tf
YleTf requires Terraform which can be installed according to their instructions. On MacOS (and OSX) you can use Homebrew. You can also easily install and manage multiple versions of Terraform with homebrew-terraforms.
YleTf supports Terraform versions from 0.11 on. Tested versions range from 0.11 to 1.0.
YleTf requires Ruby 2.6 or newer. Tested versions range from 2.6 to 3.0.
YleTf can be installed as a gem:
$ gem install yle_tfThe syntax to run YleTf is much like vanilla Terraform. The main difference is that you must include desired environment as the first argument:
tf <environment> <command> [<args>]
For example:
$ tf test plan
$ tf prod apply
$ tf stage destroy
For a full list of available options, run tf --help.
When tf is executed, it creates a temporary directory where your project is copied and initialized.
The following is a really basic example on what you need in order to use YleTf. The project is pretty much the same as the example project in Terraform documentation but with the added support for environments. Introduction to hooks is in it's own section.
The root of your project directory will look like this:
.
├── envs
│ ├── prod.tfvars
│ └── test.tfvars
├── main.tf
├── tf.yaml
└── variables.tf
And here are the contents of the files:
provider "aws" {
region = "${var.region}"
}
resource "aws_instance" "example" {
ami = "${var.ami}"
instance_type = "t2.micro"
tags {
Name = "example-${var.env}"
}
}Please note that the AMI identifier changes based on your desired OS and region. For more information see AWS documentation. You can also use the aws_ami data source.
variable "region" {
description = "The AWS region for the resources"
default = "eu-west-1"
}
variable "env" {
# passed by `tf`
description = "The environment"
}
variable "ami" {
description = "The AMI to use"
default = "ami-d7b9a2b1"
}
variable "instance_type" {
description = "Instance type to use"
default = "t2.micro"
}Let's use a bigger instance type in prod. So we override the default value with envs/prod.tfvars:
instance_type = "t2.medium"This file is empty, so it won't override any values set in variables.tf. The file must anyway exist to tell tf that this is a valid environment.
backend:
type: file
terraform:
version_requirement: "~> 0.10.5"With all the above in order, there's nothing more to do than to try and run the commands:
First plan:
$ tf test plan
INFO: Symlinking state to '/usr/local/src/yle_tf/examples/examples_test.tfstate'
INFO: Initializing Terraform
INFO: Running `terraform plan`
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
+ aws_instance.example
id: <computed>
ami: "ami-d7b9a2b1"
associate_public_ip_address: <computed>
availability_zone: <computed>
ebs_block_device.#: <computed>
ephemeral_block_device.#: <computed>
instance_state: <computed>
instance_type: "t2.micro"
ipv6_address_count: <computed>
ipv6_addresses.#: <computed>
key_name: <computed>
network_interface.#: <computed>
network_interface_id: <computed>
placement_group: <computed>
primary_network_interface_id: <computed>
private_dns: <computed>
private_ip: <computed>
public_dns: <computed>
public_ip: <computed>
root_block_device.#: <computed>
security_groups.#: <computed>
source_dest_check: "true"
subnet_id: <computed>
tags.%: "1"
tags.Name: "example-test"
tenancy: <computed>
volume_tags.%: <computed>
vpc_security_group_ids.#: <computed>
Plan: 1 to add, 0 to change, 0 to destroy.
------------------------------------------------------------------------
Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.
Then apply:
$ tf test apply
INFO: Symlinking state to '/usr/local/src/yle_tf/examples/examples_test.tfstate'
INFO: Initializing Terraform
INFO: Running `terraform apply`
aws_instance.example: Creating...
ami: "" => "ami-d7b9a2b1"
associate_public_ip_address: "" => "<computed>"
availability_zone: "" => "<computed>"
ebs_block_device.#: "" => "<computed>"
ephemeral_block_device.#: "" => "<computed>"
instance_state: "" => "<computed>"
instance_type: "" => "t2.micro"
ipv6_address_count: "" => "<computed>"
ipv6_addresses.#: "" => "<computed>"
key_name: "" => "<computed>"
network_interface.#: "" => "<computed>"
network_interface_id: "" => "<computed>"
placement_group: "" => "<computed>"
primary_network_interface_id: "" => "<computed>"
private_dns: "" => "<computed>"
private_ip: "" => "<computed>"
public_dns: "" => "<computed>"
public_ip: "" => "<computed>"
root_block_device.#: "" => "<computed>"
security_groups.#: "" => "<computed>"
source_dest_check: "" => "true"
subnet_id: "" => "<computed>"
tags.%: "" => "1"
tags.Name: "" => "example-test"
tenancy: "" => "<computed>"
volume_tags.%: "" => "<computed>"
vpc_security_group_ids.#: "" => "<computed>"
aws_instance.example: Still creating... (10s elapsed)
aws_instance.example: Still creating... (20s elapsed)
aws_instance.example: Creation complete after 22s (ID: i-0d3c42fbaba89b000)
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
And when you're done using the infrastructure, destroy:
$ tf test destroy
INFO: Symlinking state to '/usr/local/src/yle_tf/examples/examples_test.tfstate'
INFO: Initializing Terraform
INFO: Running `terraform destroy`
aws_instance.example: Refreshing state... (ID: i-0d3c42fbaba89b000)
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
- destroy
Terraform will perform the following actions:
- aws_instance.example
Plan: 0 to add, 0 to change, 1 to destroy.
Do you really want to destroy?
Terraform will destroy all your managed infrastructure, as shown above.
There is no undo. Only 'yes' will be accepted to confirm.
Enter a value: yes
aws_instance.example: Destroying... (ID: i-0d3c42fbaba89b000)
aws_instance.example: Still destroying... (ID: i-0d3c42fbaba89b000, 10s elapsed)
aws_instance.example: Still destroying... (ID: i-0d3c42fbaba89b000, 20s elapsed)
aws_instance.example: Still destroying... (ID: i-0d3c42fbaba89b000, 30s elapsed)
aws_instance.example: Still destroying... (ID: i-0d3c42fbaba89b000, 40s elapsed)
aws_instance.example: Still destroying... (ID: i-0d3c42fbaba89b000, 50s elapsed)
aws_instance.example: Still destroying... (ID: i-0d3c42fbaba89b000, 1m0s elapsed)
aws_instance.example: Destruction complete after 1m1s
Destroy complete! Resources: 1 destroyed.
Now that we've tried that everything works in the test environment, let's do the same in production.
We'll run the commands just like with test:
$ tf prod plan
< SNIP >
$ tf prod apply
< SNIP >
$ tf prod destroy
< SNIP >
Note that the instance type is now t2.medium.
Here be dragons that configure your project. The tools searches for tf.yaml's up your path all the way to root /. Configs made in subdirectories override those made upper in the path. This makes it easy to define common settings without having to edit tf.yaml in every project.
The configuration value are evaluated with ERB. This is useful for example to form tfstate file name based on the module name and environment. The variables available are:
env- The environmentmodule- The (directory) name of the root modulemodule_dir- The absolute path of the module directory
By default the following configuration options are supported:
hooks- Pre and Post hooks.backend- Configuration of remote state location.terraform- Terraform specific configuration.
There are cases when it would be beneficial to run a task at the same time as Terrafrom, but building native support for that would be quite cumbersome. The support for hooks was build into YleTf having those cases in mind.
Essentially hooks are just scripts and small applications to extend the functionality of Terraform. Hooks can be either
Real world use cases for hooks include at least the following:
- Automatically register ACM certificates and link them to desired resources
- Automatically generate DNS record resources that are managed by separate configuration
- Package lambda applications for deployment via Terraform
- Manage SES authorisations and rules
- Modify parameters not yet supported by Terraform
Currently two kinds of hooks are supported:
pre- Hooks that run before Terraform execution.post- Hooks that run after Terraform execution.
For local hooks, add a directory called tf_hooks to your tf project root. You also need a folder to determine whether the hook is run before or after the execution of Terraform. The folders are pre and/or post:
.
├── envs
│ ├── prod.tfvars
│ └── test.tfvars
├── main.tf
├── tf.yaml
├── tf_hooks
│ ├── pre
│ │ └── pre-hook.sh
│ └── post
│ │ └── post-hook.rb
└── variables.tf
For example, let's say you wish to have a variable current_git_hash and you want to populate it with a value of the latest git commit hash. The hook could be something like this:
#!/bin/bash
set -eu
current_hash() {
CURRENT_GIT_HASH="$(git rev-parse HEAD)"
cat <<EOF > current_git_hash.tf.json
{
"variable": {
"current_git_hash": {
"value": "$CURRENT_GIT_HASH"
}
}
}
EOF
}
# Execute only for commands `plan` and `apply`
case "$TF_COMMAND" in
plan|apply)
current_hash
;;
*)
;;
esacPlease note that the script in the examples directory is intentionally without the execute bit.
Once you set the hook script executable (chmod +x tf_hooks/pre/get_current_hash.sh) and run tf with either plan or apply, you'll have the following file during runtime:
{
"variable": {
"current_git_hash": {
"value": "c6a02baf0597e55d7698f78d70299ca4a65776cd"
}
}
}Naturally the actual value varies according to your repository.
Please note that files generated into the temporary working directory while running hooks are removed once the execution has ended.
Hooks can also be stored in common git repositories. This is a handy way to avoid duplication in codebase.
To use a hook from git just add your script to a suitable repository (making sure it's still executable) and add following config to your tf.yaml:
hooks:
pre:
- description: "Get your current git commit hash"
source: "[email protected]:your-org/tf-hooks.git//git-hash/get_current_hash.sh"
vars:
defaults:
MESSAGE: "You can also define environment variables to your hooks"
test:
MESSAGE: "And the variables can also be environment spesific"Configure where Terraform remote state is stored. See the Terraform documentation for backends and especially types for backend specific configuration options.
Note: the local backend type is not supported due to YleTf initialization process and use of temporary directory. Use the default file type instead, which makes a symlink to the state file (by default in the working directory).
type: <type> # Backend type where the Terraform state is stored, e.g. `file` (local file), `s3`, `swift`.
<type>: # Backend type specific options
<option>: <value> # that map directly to the Terraform's ones.
# [...]Default backend configuration:
backend:
type: file
file:
path: "<%= @module %>_<%= @env %>.tfstate" # Relative or absolute path to a file
s3:
key: "<%= @module %>_<%= @env %>.tfstate" # Path of the state file in the bucketExample S3 backend configuration:
backend:
type: s3
s3:
region: eu-west-1
# Use the sanitized directory name right after the "src/" directory as part of the bucket name.
bucket: 'terraform-state-<%= @module_dir.match(%r{.*/src/([^/]+)/})[1].downcase.tr("^a-z0-9", "-") %>-<%= @env %>'
encrypt: trueExperimental encryption of the file backend (by default using sops, so maybe add .sops.yaml to configure it):
backend:
type: file
file:
encrypt: true
## The default commands:
#encrypt_command: sops --encrypt --input-type binary --output-type binary --output "{{TO}}" "{{FROM}}"
#decrypt_command: sops --decrypt --input-type binary --output-type binary --output "{{TO}}" "{{FROM}}"version_requirement- The version requirement of Terraform in ruby gem syntax.
terraform:
version_requirement: "~> 0.9.11"version_requirement- The version requirement of YleTf in ruby gem syntax.
yle_tf:
version_requirement: ">= 1.2"Available plugins are listed on the wiki page
After checking out the repo, run bundle update to install and update the dependencies. Then, run bundle exec rake spec to run the tests.
To install this gem onto your local machine, run bundle exec rake install.
Bug reports and pull requests are welcome on GitHub at https://github.com/Yleisradio/yle_tf. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.
The gem is available as open source under the terms of the MIT License.