How to build a Hashicorp Vault server using Packer and Terraform on DigitalOcean

Introduction

In this tutorial, I will guide you step-by-step on how to create an image running a pre-configured Hashicorp Vault server, using Packer to create the image, and then using Terraform to deploy the image to a DigitalOcean droplet.

DigitalOcean

DigitalOcean is an Infrastructure as a Service (IaaS) provider. It offers developers an easy-to-use, scalable solution on spinning up Virtual Machines (VMs), referred to as “droplets.” Droplets run on virtual hardware, and can be monitored, secured, and backed-up using the DigitalOcean interface.

Hashicorp Vault

Vault is a tool used for managing secrets. A secret is what you might think it alludes to — data we want to hide from outside the system. For example, it could be a password, certificate, or an API key. Vault manages storage, generation, and encryption of secrets, among other functionality.

Hashicorp Packer

Packer is an “Infrastructure as Code” automation tool used for creating machine images. It comes out of the box with support to build images using DigitalOcean.

Hashicorp Terraform

Terraform is another “Infrastructure as Code” tool, used for the provisioning and management of system infrastructure.

Prerequisites

Step 1 — Install Packer

The recommended installation method for Packer is to install via a precompiled binary.

Navigate to the /tmp directory and download the binary appropriate for your system:

$ cd /tmp
$ wget https://link_to_your_desired_binary

Unzip it to/usr/local/packer:

$ mkdir /usr/local/packer
$ unzip your_download.zip -d /usr/local/packer

Now, add it to your path. Run the following:

$ mv /usr/local/packer/packer /usr/local/bin

Verify that Packer was properly installed:

$  packer -version

The terminal should output the version you downloaded.

Step 2 — Install Terraform

The recommended installation method for Terraform is to install via a precompiled binary.

Navigate to the /tmp directory and download the binary appropriate for your system:

$ cd /tmp
$ wget https://link_to_your_desired_binary

Unzip it to/usr/local/terraform:

$ mkdir /usr/local/terraform
$ unzip your_download.zip -d /usr/local/terraform

Now, add it to your path. Run the following:

$ mv /usr/local/terraform/terraform /usr/local/bin

Verify that Terraform was properly installed:

$  terraform -version

The terminal should output the version you downloaded.

Step 3 — Create an Installation Script for Vault

Remember, one of our primary goals is to set up an image with a pre-configured Vault server. For now, let’s just worry about the installation pieces. Later, we will use Packer to handle the configuration for us.

Let’s start by creating a central location to store all of our scipts and configuration files.

$ mkdir -p digitalocean-packer-terraform/packer/vault_configs

The -p flag ensures that the digitalocean-packer-transform folder is created first, if it doesn’t already exist.

In the vault_configs directory, create a script called install_vault.sh.

Add the following to your install script. This file will later be picked up by our Packer configuration and execute it before creating the image.

#!/usr/bin/env bash

# update and install unzip
sudo apt-get update
sudo apt-get install unzip -y

# download and install vault
cd /tmp
wget https://releases.hashicorp.com/vault/1.3.1/vault_1.3.1_linux_amd64.zip
unzip vault_*.zip
sudo cp vault /usr/local/bin 

# enable autocompletion for vault flags, subcommands, and arguments
vault -autocomplete-install
complete -C /usr/local/bin/vault vault

# prevent memory from being swapped to disk without running the process as root
sudo setcap cap_ipc_lock=+ep /usr/local/bin/vault

# create the vault.d directory in /etc
sudo mkdir --parents /etc/vault.d

# move the config files to their appropriate locations
sudo mv /home/vault/vault.hcl /etc/vault.d/vault.hcl
sudo mv /home/vault/vault.service /etc/systemd/system/vault.service

# create a system user 
sudo useradd --system --home /etc/vault.d --shell /bin/false vault

# give ownership of everything in the vault.d directory to the vault user
sudo chown --recursive vault:vault /etc/vault.d

# give read/write access to the vault.hcl file
sudo chown 640 /etc/vault.d/vault.hcl

# enable and start the vault server
sudo systemctl enable vault
sudo systemctl start vault

Step 4 — Configure the Vault Server

Now, we need to configure the Vault server. Let’s create an HCL configuration file in the vault_configs directory. This file will be picked up locally by Packer and will later be used when automating the creation of our image.

With your favorite text editor, create the file:

$ vim vault.hcl

Add the following to your configuration file.

listener "tcp" {
 address     = "127.0.0.1:8200"
 tls_disable = 1
}

storage "file" {
 path = "/home/vault/data"
}
  • listener defines where Vault will listen for API requests
  • storage defines the physical back-end Vault will use

There are other options available, but for this use case these are the two primary configurations we need (both are required). For more configuration options, see here.

Step 5 — Configure Vault to Run as a Service

If we want Vault to automatically start the server on boot, we will need to configure a .service file. This will later live in /etc/systemd/system on the remote machine.

Create a file called vault.service in the vault_configs directory.

$ vim vault.service

Add the following:

[Unit]
Description="HashiCorp Vault - A tool for managing secrets"
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/vault.hcl
StartLimitIntervalSec=60
StartLimitBurst=3

[Service]
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/usr/local/bin/vault server -config=/etc/vault.d/vault.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=on-failure
RestartSec=5
TimeoutStopSec=30
StartLimitInterval=60
StartLimitIntervalSec=60
StartLimitBurst=3
LimitNOFILE=65536
LimitMEMLOCK=infinity

[Install]
WantedBy=multi-user.target

I won’t go into the details of every parameter, but you can find more information on their definitions here.

Step 6 — Define User Variables for Packer

Now that we have what we need for Vault to be installed properly, let’s move onto the Packer comoponents. Create a variables.json file in the packer directory. This file will store our global variables, which will later be referenced in another JSON file.

In the packer directory, create a file called variables.json

$ cd ..
$ vim variables.json

Define the following. Copy and paste your DigitalOcean API key accordingly (leave the quotation marks).

{
    "do_api_token": "INSERT_YOUR_DIGITAL_OCEAN_TOKEN_HERE"
}

Step 7 — Create the Template

Packer provides various builders to create a machine and generate an image from it. A builder is a Packer component that takes a JSON template file as input and outputs the desired image, based upon how we configure the template.

Let’s build out our template for the image.

$ vim template.json

To begin building our template, add the following:

{
  "variables": {
      "do_api_token": ""
  },

The variables section tells Packer what variables we have defined. Here, we have a do_api_token variable defined with a default value of an empty string.

Now, we will move onto the builders section of the template. There are various configurations available for the digitalocean builder template, which Packer provides by default. At a bare minimum, you must define: api_token, region, image, and size.

After the variables section, add:

"builders": [
    {
      "droplet_name": "vault",
      "snapshot_name": "vault",
      "type": "digitalocean",
      "ssh_username": "root",
      "api_token": "{{ user `do_api_token` }}",
      "image": "ubuntu-18-04-x64",
      "region": "nyc1",
      "size": "1gb"
    }],
  • Note that you did not have to define your API token for api_token. This is because we already created it in our variables.json file. The back-ticks indicate a reference variable, do_api_token.
  • region, image, and size come from the slugs you get from the JSON response using the DigitalOcean API. (It might be helpful to append | json_pp after your CURL requests in the terminal, which will pipe the JSON output in pretty-print format) . It makes it easier to read in the terminal. You can also use a tool like Postman, if you prefer).

Finally, we will move onto the provisioners section of our template. Provisioners can do multiple tasks, such as create users and install packages. For our example, the provisioners section will run a series of steps (in the order you define in the template).

Add the following to template.json, after the builders section:

"provisioners": [
  {
    "type": "shell",
    "inline": [
      "mkdir -p /home/vault/data"
    ]
  },
    { "type": "file",
      "source": "vault_configs/vault.service",
      "destination": "/home/vault/vault.service"
    },
    { "type": "file",
      "source": "vault_configs/vault.hcl",
      "destination": "/home/vault/vault.hcl"
    },
    {
      "type": "shell",
      "script": "vault_configs/vault_install.sh"
    }
  ]
}
  • The first provisioner is an inline command run in the terminal, and creates a directory /home/vault/data on the remote machine.
  • The second provisioner transfer the vault.service file you created in Step 5 to /home/vault/vault.service on the remote machine.
  • The third provisioner transfers the config.hcl file you created in Step 4 to /home/vault/vault.hcl on the remote machine.
  • Finally, the last provisioner tells Packer to run the vault_install.sh script you created in Step 3.

You have now finished creating the template.

Step 8 — Create the Image using Packer

Now that we have all the necessary components in place, we can run Packer to create our image/snapshot.

From the packer directory, run the following command:

$ packer validate -var-file=variables.json template.json

validate will check our JSON files for syntax and configuration errors.

You should see the following:

Template validated successfully.

We can now build the image. Run the following:

$ packer build -var-file=variables.json template.json

The -var-file flag tells Packer to set our user variables (do_api_token) to the specified values in the variables.json file.

It takes about ~1 minute to create the snapshot. You should see quite a bit of terminal output. You should see something like the following:

Selection_006

Save the ID number given to you in the console, you will need it later to use with Terraform.

You should now be able to see the image we created in your DigitalOcean Dashboard. On the left panel, select Images. You should see the “vault” image we just created, under the Snapshots tab:

Selection_007

Step 9 — Create Variable Definitions for Terraform

Similar to the variables.json file we created for Packer, we are going to define variables for Terraform to use, and later pass them via command-line.

Create a folder outside of the packer directory called terraform, and create a file called variables.tfvars:

$ mkdir -p ../terraform
$ cd ../terraform
$ vim variables.tfvars

Add the following:

do_api_token = "INSERT_YOUR_DIGITAL_OCEAN_TOKEN_HERE"
image_id = "INSERT_THE_IMAGE_ID_CREATED_BY_PACKER"

If you lost the image_id, you can use the doctl client to obtain it by running:

$ doctl compute image list-user

Step 10 — Create an Entry Point for Terraform

Now, we can create an entry point for Terraform to use to start up a Droplet from the image.

Terraform uses its own configuration language in a declarative fashion. Each .tf file describes the intended goal (in our case, to install a Vault server), rather than the steps required to reach said goal.

In the terraform directory, create a file called main.tf:

$ vim main.tf

First, we will begin by declaring our input variables, similar to how we did in Step 6 as a part of the Packer template.

Add the following to main.tf:

# Set the variable value in *.tfvars file
variable "do_api_token" {}
variable "image_id" {}

Next, we will configure the digitalocean provider.

After the variables section, add the following.

# Configure the DigitalOcean Provider to use our DO token
provider "digitalocean" {
  token = "${var.do_api_token}"
}

The provider section tells Terraform how to interact with the DigitalOcean API, and what resources to expose (we will define the resources section next).

You will notice that we don’t actually declare a value for token, it will reference do_api_token that we declared in our variables.tf file. There are also additional arguments we can use with the digitalocean provider, aside from token, however, unlike token, they are optional.

Lastly, we will define a resource for Terraform to use. A resource describes various infrastructure objects we want to use. The digitialocean provider has many resources available. For this example, I am using the digitalocean_droplet resource.

Define the resource after the provider section as follows:

# Create a new droplet from an existing image
resource "digitalocean_droplet" "web" {
  image  = "${var.image_id}"
  name   = "vault"
  region = "nyc1"
  size   = "1gb"
}

image, name, region, and size are all required arguments for the digitalocean_droplet resource. There are others available as well, you can check the documentation for more information.

Notice again that we are referencing the image_id defined in the variables.tf file.

Step 11 — Create a Droplet from our Packer Image

Now, we can use Terraform to create a Droplet from the image we created. In the terraform directory, run the following:

$ terraform init

This command will initialize a directory containing Terraform configuration files. After running the init command, you should see the following:

Selection_008

Next, let’s get a preview of the changes Terraform is going to make using the plan command.

# terraform plan -var-file=variables.tfvars

Your terminal should output something like the following:

Selection_009

Check that your resource looks as expected.

Finally, let’s tell Terraform to apply our changes:

$ terraform apply var-file=variables.tfvars

Double-checking you want to apply the changes, Terraform will verify. Type yes:

Selection_010

Your terminal will output the following:

Selection_011

You can now navigate to the DigitalOcean Control Panel and see that the Droplet was created. Select Droplets in the navigation pane:

Selection_012

Excellent. We now have a Virtual Machine with a pre-configured Vault server! You should receive an e-mail from DigitalOcean with the login information.

Step 11 — Verify that the Vault Server has Started

Using the login information that was e-mailed to you for your new Droplet, ssh into the remote machine.

$ ssh root@your.ip.address

You will be prompted with the following message:

Are you sure you want to continue connecting (yes/no)?

(Type yes).

Login with the password that was e-mailed to you. You will be prompted to change your password.

Enter the following command to check the status of our pre-configured server:

$ systemctl status vault

You should see the following:

Selection_013

Conclusion

Now that you know how to setup a pre-configured Vault server using Packer and Terraform, try building upon this example to add configurations of your own. For example, try configuring Vault to use Consul as a backend and use Terraform to apply your changes. The possibilities are endless, but most of all, have fun!

The completed project can be found on my github here.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s