Integrating CI/CD Pipelines with Terraform Enterprise – Know Your Options

In my previous article, I demonstrated the basic steps for connecting your Jenkins CI/CD pipelines to Terraform Enterprise (TFE), so that they can spin up/down virtual machines as needed for integration/deployment testing.  To keep the demonstration nice and concise, we used the simplest option for connecting to TFE to start a Workspace Run: the familiar “terraform” executable.  Depending on your needs, however, there are other connectivity options that may be a better fit.  In this article, we’ll take a deeper dive into all of the options for starting Workspace Runs in TFE, with an emphasis on the pro’s and con’s of each.

We’ll be covering:

  1. Terraform executable (again, but in greater detail this time)
  2. TFE-CLI
  3. VCS webhook
  4. REST API

Terraform Executable

As previously stated, I chose the Terraform executable for the previous article’s Jenkins-to-TFE demonstration for its immediate familiarity to everyone who’s ever used open source Terraform.  I always like it when my Jenkins pipelines can run the exact same commands that folks are already familiar with running manually – I find that it makes pipelines that much easier for everyone to read and troubleshoot.  There are, however, some downsides to using the Terraform executable with TFE.  Probably the main downside at the time of this article is that its “remote backend” for properly connecting to TFE is not officially supported for production environments.  I suspect that it may gain official production support when 0.12 comes out, but only time will tell. You can check the official documentation page to see if the “Preview Release” disclaimer at the top of the page is still present.  That said, I have contacted HashiCorp Support for issues I ran into while using the TFE remote backend, and they were very responsive and did get my issues resolved.  Side note, there is also a legacy “Atlas” backed for connecting to TFE, but it negates a lot of the benefits of using TFE, so I don’t generally recommend it.

Here’s a summary of the Terraform executable’s pro’s and con’s:

  • Pros
    • Simple experience, exactly like on your desktop.
    • HashiCorp Support will help with issues (even though TFE remote backend is officially still in tech preview).
  • Cons
    • Not yet recommended for “Production” by HashiCorp, as of 2/2019.
    • SSL cert on private TFE installations must be perfect, or the executable won’t connect.

TFE-CLI

If you happened to click on the official documentation page of the Terraform executable’s remote provider, the “Preview Release” disclaimer that I mentioned states that that the TFE-CLI tool is a supported alternative for production Workspaces.  The TFE-CLI tool is a shell script with an interesting history.  I recently had a ticket open with HashiCorp support for an issue I was seeing while using TFE-CLI (it turned out to be unrelated to TFE-CLI), and the author of the tool ended up joining the ticket to help, which I thought was pretty cool! He mentioned that the tool was actually created to mirror the functionality of the “terraform push” command used by a previous iteration of the Terraform Enterprise product (aka “Atlas”). His original intention was to create a drop-in replacement for the old command, to help Atlas customers make a seamless transition to the newer TFE product.   However, he’s since extended the functionality and reorganized his script code into a subcommand structure that is easy to extend, if you’re so inclined.  It also serves as a great example of how one can use the TFE API.  That said, he did indicate that newer tools, like the remote provider for TFE, are HashiCorp’s future direction, so most folks should go that route unless they have a use case that demands TFE-CLI.

It’s also important to note that while TFE-CLI will start a Workspace run, it doesn’t monitor the progress of that run to see if it concludes with a success or failure. 

UPDATE!  TFE-CLI’s author, Brent Woodruff, posted a clarification in the Comments, below.  TFE-CLI does indeed have a -poll parameter to wait for the run to complete, but it doesn’t currently work in conjunction with TFE’s Auto-Apply feature, which is required for most CD/CD pipelines.  Brent has just filed an Issue on the GitHub page to track this as a feature request!  Of course I’ve already given it a thumbs-up:  https://github.com/hashicorp/tfe-cli/issues/40

In the meantime, you would need to either extend TFE-CLI or write your own script to wait for the run – both of which require diving into TFE’s REST API.

  • Pros
    • Has official support from HashiCorp for production Workspaces.
    • Great example of connecting to TFE API from a shell script. The author actually tests against 10 different Bourne shells!
    • Since it’s a shell script, if you’re troubleshooting SSL cert issues, you can easily edit it and add “-k” to the “curl” commands. Also, the author intends to add this functionality to the tool: https://github.com/hashicorp/tfe-cli/issues/41
  • Cons
    • Not future-proof – HashiCorp is moving toward other tools like the Terraform executable.
    • Does not wait for an Auto-Apply Workspace run that it submits to finish (yet). For the moment, CI/CD pipelines will require additional scripting to monitor run progress/status, retrieve state, etc.

VCS Webhook

The basic idea of using a Webhook to connect your VCS (GitHub, Bitbucket, etc.) to TFE is that every time you make a commit to a VCS repository, it fires a webhook that tells its corresponding TFE Workspace to do a run with the updated code.  Nice, eh?

The process to connect your VCS to TFE is pretty straight forward.  After that, each time you add a Workspace you’ll need to specify the repository that will house its Terraform code, and then TFE will connect to the repository and register the webhook.  So in order for your CI/CD pipeline to kick off a Workspace run in TFE, it only needs to commit some new Terraform code to the corresponding VCS repository, and the webhook will initiate the run.

That said, your CI/CD pipeline will still probably need to monitor the status of its run to make sure that it’s successful, and to know when the new infrastructure is ready to perform whatever testing your pipeline is spinning it up for.  This will require using TFE’s REST API.  Unfortunately, since your pipeline didn’t directly initiate the Workspace run (it only did a commit to the VCS repo), it doesn’t necessarily know which run to monitor.  While it can always monitor the “latest” run of a Workspace, there isn’t a simple way for a pipeline to tell if the “latest” run was initiated by itself or by a different pipeline build.  

  • Pros
    • Provides a simple, out-of-the-box integration to easily fire Workspace runs.
    • Supported by HashiCorp Support.
  • Cons
    • Tracking the progress/status of a Run is not deterministic, and could be problematic .
    • You can’t realistically combine webhooks with tool/API-based runs for the same workspace – you’ll need to choose one or the other.

REST API

TFE’s REST API is definitely the most flexible of all the connectivity options.  As stated previously, you can use it to augment the features of TFE-CLI or Webhooks.  If you’re already using a language like Python or Go to write portions of your pipeline, then adding a few more API calls might be a no-brainer.  In case you’re interested, this article is a nice easy introduction to the API. It demonstrates some of the same calls used by TFE-CLI, but in a simpler, easier-to-digest format.

In my experience so far, the REST API has been straight-forward, and easy to use.  That said, my previous love for the API has already been eclipsed by the aforementioned elegance of the Terraform executable’s new TFE remote backend.  And even though the remote backend has that “Preview Release” disclaimer at the moment, HashiCorp does indeed support it.  I haven’t tried contacting HashiCorp Support for an API issue yet, but generally speaking, software companies are able to provide much more focused support for their published tools than they are for their APIs.

  • Pros
    • Full flexibility.
    • Useful for augmenting the other options.
  • Cons
    • The most custom, development-heavy option.
    • Natural limitations to how much help HashiCorp Support can provide for an API.

Summary

In this article I’ve tried to convey that while the Terraform executable is often the best choice for CI/CD pipelines to integrate with TFE, the available options are definitely worth a look, and might just be a better fit if your needs match up with their pro’s.  I hope you found this comparison helpful!

2 comments
  1. Hi Ted! Actually, `tfe-cli` _can_ wait on runs for success or failure. Check out the `-poll` option.

    “`
    $ tfe pushconfig -name brentwoodruff/tfe_demo -poll 5
    Uploading Terraform config…
    Run run-yw8nkXKBP3PEcPPk submitted to brentwoodruff/tfe_demo using config cv-TBRqbUx5ruG3Rxft
    planning
    planned
    “`

    It won’t handle following the plan through to an apply if auto-apply is enabled, however. I also have a branch somewhere that will stream the logs back ala `terraform` with `remote`.

  2. Hey Brent! Ah yes, you did mention -poll to me before. In the past I couldn’t get it working (prob because I was using auto-apply) so I figured I was misunderstanding its purpose. Guess I should have asked for help 🙂 Thank you for the clarification!

Comments are closed.