cloudlad.io

Google Cloud Workload Identity with Kubernetes and Terraform

2022-03-02T00:00:00+00:00

“The only truly secure system is one that is powered off, cast in a block of concrete and sealed in a lead-lined room with armed guards.” - Gene Spafford

In this blog post, I will write about improving one’s security posture by running Google Kubernetes Engine (GKE) workloads on the Google Cloud Platform (GCP). My focus falls mainly on authentication: the section between the Identity and Access Management (IAM) service and the GKE workloads. But first, I need to set the stage.

Service Accounts, IAM Roles, and Key Pairs

Service Accounts

Service accounts are types of accounts typically used by applications or so-called workloads. With a service account, the application can authenticate to other GCP resources or APIs. It is an entity defined by GCP and resides in the IAM service. Its email address representation, e.g. sa-name@gcp-project-id.iam.gserviceaccount.com, is unique to a given GCP account.

IAM Roles

A service account is usually granted an IAM role that defines the authorization scope for other GCP resources. With that said, we can safely assume that a service account deals with authentication and the IAM role deals with authorization.

Public-Private Key Pairs

Each service account correlates with a public/private RSA key pair. Depending on how the private key is generated, it diverges into two types:

Google-managed key pairs.
User-managed key pairs.

Google-Managed Key pairs

Are managed by Google, and only the public key can be viewed and verified. The most common use of such keys is for generating short-lived credentials. They are automatically rotated and used for signing and are valid for a maximum of two weeks. Services like App Engine and Compute Engine use Google-managed key pairs. They are my preferred method (it should be everyone’s default). That is because I do not have to care about managing or storing any private keys which are considered sensitive data.

User-Managed Key pairs

Are created and obtained by the user upon request. The private key is called a “service account key” and uses the JSON data format. See Figure 1.

🚫 Never share, nor publicly expose your service account key! 🚫

{
  "type": "service_account",
  "project_id": "some-gcp-project-id",
  "private_key_id": "313eqeq21321dsadsadqwe21213wewdasdaasdws",
  "private_key": "-----BEGIN PRIVATE KEY-----\n-----END PRIVATE KEY-----\n",
  "client_email": "cloudlad@some-gcp-project-id.iam.gserviceaccount.com",
  "client_id": "30348941921832193121",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/cloudlad%40some-gcp-project-id.iam.gserviceaccount.com"
}

Figure 1. A user-managed bogus service account key.

This User-managed key by default has no expiration date. It also inherits the same permissions as the service account. Naturally, such keys are considered a security risk. The responsibility for them falls on the user. See Figure 2.

Figure 2. Places where a service account key might exist.

Why use service account keys?

One of the more common use cases for service accounts is for accessing resources in GCP, be it a Compute Engine VM or a GKE node pool. For clarification, a node pool is a group of VMs sharing the same configuration.

The pitfall here is, in the case of the GKE node pools, all of the pods on the nodes will inherit the permissions assigned to the service account. Let’s assume that application A needs read-only access to PubSub, and application B needs write access to Memorystore (Redis). The applications are both running as pods on nodes in a node pool. If we attach a service account on the node pool level, both applications will inherit the permissions from the attached service account. Both will have unnecessarily broad permissions, increasing the attack surface and risk. The example is only with two applications. A GKE cluster hosts way more, not counting the system pods.

The solution is rather obvious, do not assign permissions on the node pool level.

If I cannot do it on the node level, I would have to do it on the application level. E.g., secrets loaded directly in the pod. The secrets API object in K8s holds a small amount of sensitive data. The data gets put into a running pod or container image, either mounted or exposed as an environment variable.

With this, I am back to square one, user-generated service account keys loaded into pods.

Workload Identity

Workload Identity is a process that enables workloads to impersonate (IAM) service accounts to access GCP services. The workloads might or might not be running on GCP. In simpler terms, the workloads assume the authorization scope of a service account and use short-lived credentials that the user does not need to manage in any way. Effectively preventing the leakage of credentials and decreasing the possible attack surface. See Figure 3.

Figure 3. Simplified representation of Workload Identity.

Let me explain this in a bit more detail. By enabling Workload Identity on a cluster, a K8s DaemonSet is deployed, with a pod in the host network. That pod dubbed the GKE metadata server (MDS), runs in the kube-system namespace and intercepts all requests to the GCP metadata server from the other pods (workloads). The pods running on the host network are the exception.

The whole authentication flow consists of a few steps.

The GKE MDS requests OpenID Connect (OIDC) signed JSON Web Tokens (JWT) from the Kubernetes API server.
The GKE MDS then requests an access token for the K8s Service Account.
IAM validates the bindings and OIDC and returns an access token to the MDS.
That access token will be sent back to IAM, and a short-lived GCP service account token will be issued.
It is then returned to the GKE MDS and passed on to the workload.

Finally, the pod can use the short-lived service account token to access GCP resources. See Figure 4.

Figure 4. Workload Identity.

Practical Example In Terraform

What good is theory without the practical part?

For that, I will be using Terraform and GCP to show the GKE Workload Identity in action.

From this point on, I will assume that the reader has the following:

Terraform 1.1 or older and gcloud CLI installed.
Functional GCP credentials (hopefully not configured via a user-managed service account key), authorized for creating and managing GCP projects, including all of its underlying resources.

I’ve made a Terraform root module that creates a:

GCP project (it can also reside under an Organization folder).
VPC with private subnets.
Private GKE Cluster with Workload Identity enabled.
Node pool with a Workload Identity enabled in GKE_METADATA mode. Google recommends including this attribute, so node creation fails if Workload Identity is not enabled on the cluster.
A shell script named k8s.sh for the creation and configuration of resources inside the GKE cluster.

See Figure 5.

Figure 5. Simplified representation of the cloud resources created by the Terraform root module.

For more details about the module and its configuration, refer to the README.md.

I would have preferred to create all of the resources in one go, but unfortunately, it is not possible with Terraform’s CRUD cycle. The problem is that the K8s provider cannot authenticate against the GKE cluster in one apply run. That makes perfect sense because the GKE cluster is not present in the application stage, and the K8s provider cannot authenticate. To programmatically create all of the resources, I use the local_file provider to generate a helper bash script from a Terraform template. The bash script k8s.sh will appear in the working directory of the root module after a successful apply run. It can be overiden to any valid path in the Operating System. When Terraform apply finishes, the k8s.sh shell script will be present in the working directory. When executed, it will create the cloudlad namespace, a service account, and the annotation for the said service account. Lastly, it will start a pod with the annotated service account attached.

Because of the pod spec override with the annotated service account, the pod itself will inherit the attached role to the IAM service account from the Terraform root module. See Figure 6.

#!/usr/bin/env bash

set -ex -o pipefail -o nounset

K8S_NAMESPACE="cloudlad"
K8S_SERVICE_ACCOUNT="cloudlad"
GCP_REGION="europe-west3"
CLUSTER_NAME="cloudlad-2-ringtail"
GCP_PROJECT_ID="workload-identity-2-ringtail"
K8S_CONTEXT="gke_workload-identity-2-ringtail_europe-west3_cloudlad-2-ringtail"
GCP_SERVICE_ACCOUNT="cloudlad-2-ringtail-k8s@workload-identity-2-ringtail.iam.gserviceaccount.com"

gcloud container clusters get-credentials "$CLUSTER_NAME" --project="$GCP_PROJECT_ID" --region="$GCP_REGION"

kubectl config use-context "$K8S_CONTEXT"

kubectl create namespace "$K8S_NAMESPACE"

kubectl create serviceaccount "$K8S_SERVICE_ACCOUNT" \
    --namespace="$K8S_NAMESPACE"

kubectl annotate serviceaccount "$K8S_SERVICE_ACCOUNT" \
    --namespace="$K8S_NAMESPACE" \
    iam.gke.io/gcp-service-account="$GCP_SERVICE_ACCOUNT"

kubectl run --image=google/cloud-sdk:slim \
    --namespace="$K8S_NAMESPACE" \
    --overrides="{ 'spec': { 'serviceAccount': $K8S_SERVICE_ACCOUNT } }" \
    cloudlad -- sleep infinity

Figure 6. Helper shell script for programmatically creating the K8s resources, valid only for a single Terraform state.

When I execute the k8s.sh script, the K8s Service Account will get annotated with a key: iam.gke.io/gcp-service-account and a value: cloudlad-2-ringtail-k8s@workload-identity-2-ringtail.iam.gserviceaccount.com. The annotation enables the K8s service account to inherit the same Authorization scope from the IAM Service account. See Figure 7.

$ kubectl -n cloudlad get serviceaccounts cloudlad -o json
{
    "apiVersion": "v1",
    "kind": "ServiceAccount",
    "metadata": {
        "annotations": {
            "iam.gke.io/gcp-service-account": "cloudlad-2-ringtail-k8s@workload-identity-2-ringtail.iam.gserviceaccount.com"
        },
        "creationTimestamp": "2022-02-22T21:28:59Z",
        "name": "cloudlad",
        "namespace": "cloudlad",
        "resourceVersion": "9849",
        "uid": "f29f981f-28fc-4ca2-ba35-31dd42ff8e7a"
    },
    "secrets": [
        {
            "name": "cloudlad-token-thlzr"
        }
    ]
}

Figure 7. Service Account annotation with kubectl.

You might ask yourself, “OK, what now?” Checking whether the coupling between GCP IAM and GKE works is straightforward. The newly created pod runs indefinite sleep. And the container is started from the official GCP SDK image, meaning the gcloud CLI is readily available.

I will execute an interactive shell in the running container. Then check the available service accounts, and finally, create a PubSub topic. It is all possible because the pod inherits the roles/pubsub.admin role. See Figure 8.1.

$ kubectl exec --stdin --tty -n cloudlad cloudlad -- bash
root@cloudlad:/# curl -H "Metadata-Flavor: Google" http://169.254.169.254/computeMetadata/v1/instance/service-accounts/
cloudlad-2-ringtail-k8s@workload-identity-2-ringtail.iam.gserviceaccount.com/
default/
root@cloudlad:/# gcloud pubsub topics create cloudlad
Created topic [projects/workload-identity-2-ringtail/topics/cloudlad].

Figure 8.1 Creating a PubSub Topic with Workload Identity from a running pod.

The informational message from the gcloud CLI confirms that the creation of the PubSub topic is without issues. Checking the GCP console whether the PubSub topic is present wouldn’t hurt. See figure 8.2.

Figure 8.2 PubSub Topic created with Workload Identity in the GCP console.

And with this, I have confirmed that Workload Identity is set up and is working.

That is all. I hope that this blog post demystifies GCP’s Workload Identity a bit. As always, feel free to contact me with any ideas, info, and suggestions. Don’t forget that sharing is caring!

Terraform, The Less-Common Commands And Options

2021-09-06T00:00:00+00:00

“Once a new technology rolls over you, if you’re not part of the steamroller, you’re part of the road.” - Stewart Brand

The most frequently used Terraform commands, plan, apply, and destroy, are the first things I learned when I started using Terraform. Primarily because those commands are a crucial part of Terraform’s cycle. Over time, as Terraform and its community grew, the features of Terraform grew in parallel. Now there are a lot more commands, options, arguments, and functions. Some are not used as often, and some do not even show when invoking the help menu.

In this blog post, I want to talk about the not-so-very common commands, arguments, and use cases.

Terraform Console

This command is my current personal fav, the console command. As per documentation it is most commonly used for experimenting with expressions and functions. As an added extra, if the command is invoked in a folder with an active Terraform state, it will read it from the configured backend. Inspecting and experimenting with the state with Terraform’s console can be fun!

Getting to it is as simple as typing in the command line:

$ terraform console

Now we can start “playing” with our state, provided there is something in it that we can fiddle with.

For the sake of this blog post let’s create a main.tf file with the following contents. The file is also available in my examples repo.

locals {
  map = {
    a = 1
    b = 2
    c = 3
  }
}

resource "local_file" "private_key" {
  content  = jsonencode(local.map)
  filename = "./map.json"
}

Needless to say, executing terraform init is necessary to pull the plugins and initialize the configured backend if there is any.

After, running terraform apply against this template, a JSON encoded file with the content of the local variable map will be created in the same folder. That is easy enough.

But what if I needed a YAML encoded file? I am also not sure how it will look. I prefer JSON, but I also like to have some comments.

I can check it on the fly by firing up Terraform console in the same folder and just YAML encoding the local map variable:

$ terraform console
> yamlencode(local.map)
<<EOT
"a": 1
"b": 2
"c": 3

EOT

Voilà! We see how our file will look like if we’ve used the yamlencode function.

Why stop there, maybe I just need the value of the b key?

> yamlencode(local.map.b)
<<EOT
2
...

EOT

Swapping the keys and the values is also something that I might want?

> yamlencode({for k,v in local.map : v => k })
<<EOT
"1": "a"
"2": "b"
"3": "c"

EOT

Or just “trying” out the try function to see how it works exactly.

> try(yamlencode(local.map.d), 4)
4

Case in point. The flexibility that the terraform console provides when testing, troubleshooting, or simply playing around is immense.

Shell Autocomplete

This is one of the commands that do not appear when I invoke Terraform via the CLI. Having worked with Terraform for quite a while myself, I stumbled upon this command fairly recently. It lets you use shell tab-completion. As of writing this blog post, it works with bash or zsh shells only.

Running this will enable autocomplete for Terraform CLI. Re-reading the shell configuration or re-reloading a shell will activate it.

$ terraform -install-autocomplete

Lock Timeout

This is another very useful option, especially when running in an automated environment, such as a CI system. Imagine the scenario: we start two or more plans or apply jobs at a similar time. Naturally, Terraform will fail if it cannot get a lock for the state. With the option -lock-timeout instead of immediately failing, Terraform will retry getting the state lock for the duration specified in seconds. If Terraform cannot acquire a lock after the timeout has passed, it will fail with an error.

$ terraform apply/plan -lock-timeout=180s

Manual State Manipulation

Usually, I edit the state via Terraform CLI. Sometimes, though, I need to manually edit it. I do it via a pull, edit, and push cycle. These commands will work regardless of the type of backend configured. The most important things to remember when manually editing a state are:

Validate the JSON before pushing.
Increment the serial key by one.

“Pull-Edit-Push”

The following command will pull and write the current active state to the terraform_state.json file.

$ terraform state pull > terraform_state.json

Which has the following contents.

{
  "version": 4,
  "terraform_version": "1.0.4",
  "serial": 1,
  "lineage": "b538a863-c14e-c216-9309-e5e35f91468c",
  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "local_file",
      "name": "private_key",
      "provider": "provider[\"registry.terraform.io/hashicorp/local\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "content": "{\"a\":1,\"b\":2,\"c\":3}",
            "content_base64": null,
            "directory_permission": "0777",
            "file_permission": "0777",
            "filename": "./map.json",
            "id": "e7ec4a8f2309bdd4c4c57cb2adfb79c91a293597",
            "sensitive_content": null,
            "source": null
          },
          "sensitive_attributes": [],
          "private": "bnVsbA=="
        }
      ]
    }
  ]
}

I will remove the last element in the content parameter, and now the local state file looks like this:

{
  "version": 4,
  "terraform_version": "1.0.4",
  "serial": 2,
  "lineage": "b538a863-c14e-c216-9309-e5e35f91468c",
  "outputs": {},
  "resources": [
    {
      "mode": "managed",
      "type": "local_file",
      "name": "private_key",
      "provider": "provider[\"registry.terraform.io/hashicorp/local\"]",
      "instances": [
        {
          "schema_version": 0,
          "attributes": {
            "content": "{\"a\":1,\"b\":2}",
            "content_base64": null,
            "directory_permission": "0777",
            "file_permission": "0777",
            "filename": "./map.json",
            "id": "e7ec4a8f2309bdd4c4c57cb2adfb79c91a293597",
            "sensitive_content": null,
            "source": null
          },
          "sensitive_attributes": [],
          "private": "bnVsbA=="
        }
      ]
    }
  ]
}

After saving the file, “Pushing” the state back to its configured backend is as easy as:

$ terraform state push ./terraform_state.json

The “push” will fail if there’s something wrong with the state file. Otherwise, it shouldn’t provide any output.

Further verifying that the state edit is successful can be done by running either terraform plan or terraform apply.

The changes that Terraform will report attest to the successful manual state edit.

$ terraform apply
local_file.private_key: Refreshing state... [id=e7ec4a8f2309bdd4c4c57cb2adfb79c91a293597]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # local_file.private_key must be replaced
-/+ resource "local_file" "private_key" {
      ~ content              = jsonencode(
          ~ {
              + c = 3
                # (2 unchanged elements hidden)
            } # forces replacement
        )
      ~ id                   = "e7ec4a8f2309bdd4c4c57cb2adfb79c91a293597" -> (known after apply)
        # (3 unchanged attributes hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

local_file.private_key: Destroying... [id=e7ec4a8f2309bdd4c4c57cb2adfb79c91a293597]
local_file.private_key: Destruction complete after 0s
local_file.private_key: Creating...
local_file.private_key: Creation complete after 0s [id=e7ec4a8f2309bdd4c4c57cb2adfb79c91a293597]

Apply complete! Resources: 1 added, 0 changed, 1 destroyed.

As expected, Terraform reverts back to the state as instructed by the configuration in the main.tf template.

JSON Instead Of HCL

The use case for this capability is quite interesting. I can programmatically generate Terraform templates using the JSON syntax. The only requirement is for the filename to have the extension tf.json. Otherwise, Terraform will not pick the configuration up.

I can convert the example above to JSON like so:

{
  "locals": {
    "map": {
    "a": 1,
    "b": 2,
    "c": 3
    }
  },
  "resource": {
    "local_file": {
      "private_key": {
        "content": "jsonencode(local.map)",
        "filename": "./map.json"
      }
    }
  }
}

I would reckon it’s unusual to generate a JSON encoded file using the JSON syntax.

It is a very appealing feature I imagine is being actively used out there.

Terraform Graph

Not the most widely used Terraform command, but most definitely uncommon. It generates a DOT (graph description language) file that I can use to render an image in a well-known format, such as PNG. Running this command in the example state yields a chart description configuration.

$ terraform graph

digraph {
	compound = "true"
	newrank = "true"
	subgraph "root" {
		"[root] local_file.private_key (expand)" [label = "local_file.private_key", shape = "box"]
		"[root] provider[\"registry.terraform.io/hashicorp/local\"]" [label = "provider[\"registry.terraform.io/hashicorp/local\"]", shape = "diamond"]
		"[root] local_file.private_key (expand)" -> "[root] local.map (expand)"
		"[root] local_file.private_key (expand)" -> "[root] provider[\"registry.terraform.io/hashicorp/local\"]"
		"[root] meta.count-boundary (EachMode fixup)" -> "[root] local_file.private_key (expand)"
		"[root] provider[\"registry.terraform.io/hashicorp/local\"] (close)" -> "[root] local_file.private_key (expand)"
		"[root] root" -> "[root] meta.count-boundary (EachMode fixup)"
		"[root] root" -> "[root] provider[\"registry.terraform.io/hashicorp/local\"] (close)"
	}
}

If I “pipe” this through a filter for drawing directed graphs:

terraform graph | dot -Tpng > graph.png

I get a nice image of a graph as seen in Figure 1. (I’ve also inverted the colors with imagemagick)

Figure 1. Graph created from a DOT template.

Getting to know a specific technology or a tool is not as easy as it seems. Hopefully, this blog post will make some of the unusual Terraform options clearer and more usable.

If you enjoyed this blog post, please share and spread it. For any suggestions or questions, contact me at nikola@cloudlad.io.

IaC Pipelines With Terraform And Cloud Build

2021-04-07T00:00:00+00:00

“Today, most software exists, not to solve a problem, but to interface with other software.” — IO Angell

In this blog post I’ll be showcasing Cloud Build, a CI/CD service from Google Cloud Platform, evaluating its capabilities and how well it works with Infrastructure as Code (IaC) deployment pipelines.

Infrastructure as Code (IaC)

Chances are you’ve come across the term IaC in your career. Undoubtedly it is the default process for managing and provisioning computational resources. The capability of defining declarative code and storing it in a version control system makes it even more appealing. Not to mention that the process can be automated, tested, re-used, and reviewed. That eventually leads to decreased risk, failure, but an increase in speed, stability, and security.

Simply put, long gone are the days where one would order a plethora of parts just to build a few servers, only to realize that the rack doesn’t fit in the server hall. With IaC and modern Infrastructure as a Service (IaaS) providers, we have almost infinite computational power at our fingertips, if our finances are on par.

GitOps

I can describe GitOps as a natural evolution of IaC that leverages Git as the single source of truth. We request changes with the creation of Pull Requests, which triggers various status checks after every commit. When the changes meet the conditions set for the repository, and we have approved the Pull Request, we can merge it back into the main/master branch. This should automatically trigger an event that deploys the changes into the environments that run the workloads, Figure 1.

Figure 1. Simplified GitOps pipeline.

With my preferred IaC tool Terraform, the main/master branch should report no changes when run against the infrastructure. It should always reflect the state of the currently running resources. This way we ensure the consistency of the Git Flow, which is especially important when collaborating within a team.

Cloud Build and Terraform

But what good is a deploy pipeline without a CI/CD service? Cloud Build comes with full CI/CD capability with which we can automate Terraform.

One of the challenges of the engineering teams is avoiding the creation of a so-called “silo”. It is prominent when working with infrastructure. A “silo” happens when the generic processes that apply for engineering do not apply to IaC that might lead to the unintentional creation of a “silo”. Tools like Terraform and Cloud Build help in bridging that gap or at least solving part of it.

Security is a subject that we must not neglect. It plays a big part in the CI/CD service selection. Having owner-level permission for a Google Cloud Project is something that I want to avoid managing and sharing via hard-coded secrets.

I am about to show a setup that has a lot of moving parts. The concepts and components are listed below for clarity. Head on to the Implementation section if you are already familiar with them.

Concepts

Terraform State

A Terraform state is an object that keeps track of the managed infrastructure and configuration. It can be both local and remote. Remote states are more or less a requirement, especially when working in a team. To use a remote state, we have to solve the “Chicken or the egg” problem. E.g., If we want to put the Terraform state on a GCP storage bucket, we first need to create that bucket with Terraform and then migrate the state. Another peculiarity is state locking. Terraform automatically locks the state for all operations that could write to it, thus preventing others from running Terraform. The need for a queuing mechanism is obvious.

Principle of Least Privilege (PoLP)

PoLP is a concept in which we give an entity a minimum level of access. It should have access to only the information needed for its purpose.

Inert vs. Active Terraform States

This concept stems from the Terraform’s behaviour. Big states that manage a lot of resources are slow to plan/apply. Together with state locking and PoLP, the need for splitting the states is clear. These so-called “Inert States” change less often than the “Active States”. E.g., we control the creation of GCP projects with one state but manage the resources of said projects in other states.

Furthermore, both Inert and Active states require a different level of permissions.

This approach brings the attack surface and blast radius to a minimum, but increases the speed of operation and security.

Implementation

Figure 2. IaC pipeline with Cloud Build.

Let us have a look at the following directory structure:

.
├── projects # Applied manually, creates the GCP projects
└── workloads
    ├── prod-1549784393 # Applied automatically, creates resources in prod-1549784393 GCP project
    └── stag-3380426388 # Applied automatically, creates resources in stag-3380426388 GCP project

There are three Terraform states. The folder names under the workloads directory correspond to the names of the GCP projects inside the projects directory.

projects - Inert Terraform State

The projects folder holds the state for creating all GCP projects: cloud-build-3660853213, prod-1549784393, stag-3380426387 that must be applied manually. It requires permissions for creating GCP projects.

prod-1549784393 - Active Terraform State

The prod-1549784393 folder holds the state of the resources in the prod-1549784393 GCP project. Cloud Build runs it automatically. It requires owner permissions for the project prod-1549784393.

stag-3380426388 - Active Terraform State

The stag-3380426388 folder holds the state of the resources in the stag-3380426388 GCP project. Cloud Build runs it automatically. It requires owner permissions for the project stag-3380426388.

Cloud Build triggers

Figure 3. Cloud Build Triggers.

There are two Cloud Build triggers, dubbed pull-request-push and pull-request-merge. As the names imply, they get triggered upon creation/pushing or merging a pull request into master. pull-request-push is the development phase and pull-request-merge is the deployment phase. For more information on how to use Cloud Build triggers, consult GCP’s documentation.

Only fmt, validate, and plan run as build steps for simplicity’s sake. Nothing prevents us from adding more checks or tests, though.

pull-request-push (Development phase)

Cloud Build starts this trigger upon creation or push to a PR. It executes two steps, Terraform checks (fmt, validate) and Terraform plan (init, plan).

pull-request-merge (Deployment phase)

Cloud Build starts this trigger upon push/merge to master. It executes three steps, Terraform checks (fmt, validate), Terraform plan (init, plan) and Terraform apply. The apply is running a plan file from the previous step to make sure it executes the changes the previous step, Figure 6.

- id: 'terraform-apply'
  waitFor:
    - 'terraform-init-plan'
  name: 'alpine:3.13.3'
  volumes:
    - name: 'terraform_plan_files'
      path: '/var/terraform_plan_files'
  dir: 'terraform/iac-pipelines-with-terraform-and-cloud-build/workloads'
  entrypoint: 'sh'
  args:
    - '-c'
    - |
        set -eo pipefail

        apk add git bash curl
        git clone https://github.com/tfutils/tfenv.git /opt/.tfenv
        ln -s /opt/.tfenv/bin/* /usr/local/bin

        for folder in $(ls .); do
          echo "+-------------------------------------------+"
          echo "Applying state ${folder} "
          echo "+-------------------------------------------+"

          cd $folder
          tfenv install
          terraform init
          terraform apply \
            -auto-approve \
            "/var/terraform_plan_files/${folder}_${BUILD_ID}_${PROJECT_NUMBER}.plan"
          cd -
        done

Figure 6. Cloud Build "terraform-apply" step.

Terraform versioning

I am using Alpine Linux for the running environment. The reason for that is easier Terraform management. I constrain the minor version within the state,

terraform {
  required_version = ">= 0.14"
}

and I use tfenv to pin the patch version.

$ cat .terraform-version
0.14.9

With this approach when updating Terraform, I do not need to change the trigger configuration files and there is no additional logic to install the correct version except executing tfenv install, Figure 4.

Pre-packaging a container image is also something that should be considered.

+-------------------------------------------+
Planning state prod-1549784393 
+-------------------------------------------+
Installing Terraform v0.14.9
Downloading release tarball from https://releases.hashicorp.com/terraform/0.14.9/terraform_0.14.9_linux_amd64.zip

Downloading SHA hash file from https://releases.hashicorp.com/terraform/0.14.9/terraform_0.14.9_SHA256SUMS
No keybase install found, skipping OpenPGP signature verification
terraform_0.14.9_linux_amd64.zip: OK
Archive:  tfenv_download.HLdilD/terraform_0.14.9_linux_amd64.zip
  inflating: terraform
Installation of terraform v0.14.9 successful. To make this your default version, run 'tfenv use 0.14.9'

Figure 4. Cloud Build installing Terraform with Tfenv.

Terraform execution

In every Cloud Build step, there is a loop that goes to each Terraform state folder and execute the specified actions, Figure 5.

for folder in $(ls .); do
  echo "+-------------------------------------------+"
  echo "Fmt/Validate state ${folder} "
  echo "+-------------------------------------------+"

  cd $folder
  tfenv install
  terraform fmt -recursive -check -diff .
  terraform init -backend=false
  terraform validate .
  cd -
done

Figure 5. Cloud Build step "ftm-validate".

This is how it all looks on GitHub when working in a real-life scenario. Terraform fmt and validate were failing the checks until pushed the fixes, Figure 7.

Figure 7. GitHub checks.

Initializing the projects Terraform state

$ terraform apply -var=billing_account=<SOME_BILLING_ACCOUNT>

Comment the Cloud Build triggers. They cannot be creation until the GitHub repository is connected.
Comment the remote state configuration. We cannot have a remote state until the storage bucket is created.
Apply the Terraform state.
- This will (amongst other things):
  - Create the three projects.
  - Enable the Cloud Build API.
  - Give Cloud Build’s service account owner permissions for the projects that host the workloads prod-1549784393, and stag-3380426388.
Connect GitHub and Cloud Build assuming that the GitHub repository hosts the same directory structure and template names, Figure 8.
Uncomment the Cloud Build triggers.
Uncomment the remote state configuration.
Init and Apply Terraform.

Figure 8. GitHub Repository and Cloud Build.

Initializing the prod-1549784393, stag-3380426388 Terraform states

$ terraform apply

Comment the remote state configuration in each folder/state. We cannot have a remote state until the storage bucket is created.
Init and Apply Terraform.
Uncomment the remote state configuration.
Init and Apply Terraform.

That is it! With this, we have now successfully created an IaC pipeline with Cloud Build and GitHub.

Conclusion

Evaluating Cloud Build was a fun task and like every solution out there, it has its strengths and weaknesses.

Pros:

Secure
- If you run your workloads on GCP this would be the most secure approach because the builds do not leave Google Cloud’s “ecosystem”.
Fast
- Cleary, builds were blazingly fast even with installing third-party software during execution, Figure 9.

Cons:

File filters do not work for every event:
- Included files and Ignored files can only be specified if Push to a branch is selected as your Event. In my case, I could only specify this filter in the pull-request-merge trigger.
No interactive build steps:
- Running Terraform with the -auto-aprove attribute can be very dangerous an interactive approval step when applying in production would be very nice to have.
Cannot establish an SSH (Debug) session into the build environment.

Figure 9. Cloud Build Execution times.

That is all for now. For any suggestions or questions, feel free to contact me. If you like what I am doing, don’t forget to share this blog post.

Terraform 0.13 and Google Cloud

2020-09-29T00:00:00+00:00

If you’re going to do operations reliably, you need to make it reproducible and programmatic. - Mike Loukides

Terraform

Terraform, the de facto configuration tool for managing infrastructure nowadays has come a long way since its early release, roughly 5 years ago. After many iterations, mostly driven by vocal supporters, one of Terraform’s biggest strengths, listening to its community has proven crucial to its success.

This blog post is going to be about how I practically apply Terraform’s 0.13 features into one of the major IaaS providers Google Cloud.

The Scenario

Having worked with both AWS and GCP, I have found that as usual, there are pros and cons to both cloud providers.

One slightly simpler thing (perhaps better) than the other cloud providers is the coupling between G Suite and GCP. G Suite acts as a central identity store that we can use in GCP’s Cloud Identity and Access Management (IAM). The coupling between the users in a company and its IaaS provider simplifies the management overhead. As an example, one can use G Suite emails or email groups to assign permissions to resources within GCP. This leads to interesting combinations that clearly define responsibilities, and we benefit from heightened security because of said segregation of roles and duties.

Google Cloud has a concept of Cloud Platform Resource Hierarchy. This enables the operator to control and fine-tune ownership, access control, and inheritance. An organization is the root node of the hierarchy. Then come the folders and the projects. See Figure 1.

Figure 1: Cloud Platform Resource Hierarchy

Let us assume the following case. There is a big organization with many engineering teams that need isolated testing environments. The security team that needs to supervise the teams and their respective resources needs to have a project to deploy its tools. This is to provide support in the unlikely case of security issues that might arise.

A Folder can have many other folders and projects below it. Besides, access to the resources within the projects can be limited by Folder so that an IAM email (user, group, or a service account) can have a set of permissions for the Cloud resources living within a Folder. With some planning, the hierarchical structure of an organization can be fascinating.

For simplicity’s sake, I will limit the example with only a subset of the resources. See Figure 2.

Figure 2: Subset of resources

From this point on, the reader should be comfortable working with Terraform.

count And for_each For Modules

The count and for_each features enable us to have a systematic creation of multiple resource instances with a single configuration block. With previous Terraform versions, it was not possible to use these arguments with modules. Starting from 0.13, these powerful meta-arguments for workflows add up to the already extensive list of Terraform’s features.

Both count and for_each can be used interchangeably. I prefer for_each because the resource instances in count are identified by their index in a list. If we remove an element from the middle of the list, every item after it would require a re-creation. Read more in the official Terraform documentation

We can set a local map variable in the root module that defines the creation of our resources, bundled inside the child module. See Figure 3.

Figure 3: Local Map Variable

This Terraform configuration will create three folders and four projects. See Figure 4.

Figure 4: Folders, projects in GCP console

depends_on For Modules

Provisioning APIs is a sophisticated business. Sometimes the APIs do not behave as we expect them to, or often they lack features. Managing resources in the cloud adds to the complexity plus sometimes we require working with the resources in a specific order. The depends_on meta-argument addresses the issue.

There are two types of resource dependencies in Terraform, implicit or explicit. Implicit dependencies are handled by Terraform automatically. It analyses expressions within configuration blocks for references to other resources. As an example, a VM instance will implicitly depend on the resource defined in the network_interface setting, because it requires an IP Address to operate.

Sometimes, though, Terraform cannot resolve some dependencies on its own. In such a case, we explicitly set them with the depends_on meta-argument.

depends_on was originally available for resources only, from version 0.13 onwards it is available for modules as well.

Let us suppose that the security team in our organization wants to have an overview of all the resources that are present along with their activity. For that, a separate GCP project would suffice. The only requirement for this project is that it needs to be created before the others (and configure its tooling accordingly). From Terraform’s perspective, all the other projects depend on this project, which I dubbed “Team-X”. See Figure 5.

Figure 5: Explicit module dependency

Custom Validation Rules

Terraform’s parameters, also called input variables, have type constraints by default. Version 0.13 offers an additional feature with which I can validate the content of the input variables.

The variable block supports an additional validation configuration that has a condition parameter that checks if an expression is true or false. Its usage varies depending on the variable.

In the example below, I am validating the required input variable billing_account. Google Cloud’s billing accounts have the form of six alphanumeric characters, separated by a dash sign.

variable "billing_account" {
  description = ""
  type        = string

  validation {
    condition     = can(regex("^[[:alnum:]]{6}-[[:alnum:]]{6}-[[:alnum:]]{6}$", var.billing_account))
    error_message = "Invalid billing account, please provide a valid billing account in the following format \"XXXXX-XXXXX-XXXXX\"."
  }
}

The validation above checks for a valid Google Cloud billing account and fails if the regex doesn’t match with the provided input variable. The content in the “error_message” argument is being displayed on stdout.

Example with an invalid billing account.

$ terraform apply

Error: Invalid value for variable

  on main.tf line 66, in module "cloudlad_org_security":
  66:   billing_account = var.billing_account

Invalid billing account, please provide a valid billing account in the
following format "XXXXX-XXXXX-XXXXX".

This was checked by the validation rule at
../modules/resource_management/variables.tf:5,3-13.

Terraform’s feature list does not stop here. There are many others, but I decided to write only about the most notable ones. For a full list of features and fixes head on to Terraform’s release changelog.

I hope that this blog post helped you in understanding how a basic hierarchical setup in Google Cloud can be achieved with Terraform. Please share this post if you find it interesting and helpful.

The example module that I am using in this blog post can be found here.

Testing Infrastructure with Terraform And Terratest

2020-05-13T00:00:00+00:00

Software testing proves the existence of bugs not their absence.
- Anonymous

In this blog post, I will show you how you can test your infrastructure with Terraform and Terratest on Google Cloud, but first,

Let’s Talk About Terraform Modules

Terraform modules are reusable, composable, and testable components, which pack all of the complexity into a single versatile and modifiable unit. This so-called unit can be controlled by input variables that extend or change its usability and its use cases. There are many modules out there, some official, and some provided by the community. You can explore them at Terraform’s module registry: https://registry.terraform.io/browse/modules.

As for creating them, the examples in the registry serve as a solid starting point as well as this guideline by Hashicorp: https://www.terraform.io/docs/modules/index.html.

Why Do We Need Testing?

Testing code has always required a lot of additional work that I believe that most people dread, including myself. Without testing though, we’ve no way of telling if our code will behave the way it was intended to, especially at present, where code logic tends to get sophisticated in most of the cases. Terratest can test real infrastructure into a real environment, a single unit like Terraform module if you will, evaluating the resources, results and outputs immediately.

Setting Up The Environment

I will be using the latest versions of both golang and Terraform as of the time of writing this blog post

Golang

The instructions for installing and configuring golang are provided here: https://golang.org/doc/install

Terraform

Get Terraform either by downloading it and placing it in your “bin path”, or by using tfenv.

Vendoring dependencies

Because this example is not operating within the GOPATH, we need to specify a module name with the convention:

github.com/<YOUR_USERNAME>/<YOUR_REPO_NAME>

If you want to start from scratch instead of using my example, follow the steps below.

These are the packages I am importing.

import (
	"fmt"
	"testing"
	"time"

	"github.com/gruntwork-io/terratest/modules/gcp"
	"github.com/gruntwork-io/terratest/modules/retry"
	"github.com/gruntwork-io/terratest/modules/terraform"
)

Create a test directory in the root of the folder if it is not there. Copy or create a file that ends in _test.go, then “cd” into the “test” directory and initialize the module.

$ mkdir test
$ cd test/
$ curl -O https://raw.githubusercontent.com/parabolic/terraform_ci_cd/master/test/terraform_ci_cd_test.go
$ go mod init github.com/parabolic/terraform_ci_cd
go: creating new go.mod: module github.com/parabolic/terraform_ci_cd
$ go mod vendor

The “go mod vendor” command downloads (vendors) all of the dependent packages to the same directory.

We will end up with a folder structure like this:

test/
├── go.mod
├── go.sum
├── terraform_ci_cd_test.go
└── vendor

Google Application Credentials

As part of the last phase, we would need to provide credentials for authenticating to Google Cloud. Please follow the steps outlined in google’s documentation on how to create a service account and export its keys. I would recommend creating a separate service account only for running Terraform or Terratest.

After we’ve got the google cloud key file onto our system, we should export an environment variable that points to the location of the file. That way both Terratest and Terraform can authenticate to Google Cloud and conduct the tests.

export GOOGLE_APPLICATION_CREDENTIALS="/home/cloudlad/.gcloud/cloudlad-project.json"

Verification

Let’s do a small check-up before we move on.

Golang

$ go version
go version go1.14.1 linux/amd64

GOPATH

$ printenv GOPATH
/go

Terraform

$ terraform -v
Terraform v0.12.24

Google Application Credentials

$ printenv GOOGLE_APPLICATION_CREDENTIALS
/home/cloudlad/.gcloud/cloudlad-project.json

How It All Works Together

Now that we have all set up and verified, we are ready to begin testing.

Terratest uses golang’s testing framework, so for those familiar with it this will be straightforward. The terratest GCP module is very limited as of writing this blog post so I am unable to test a lot of other resources that GCP has to offer. For that reason, my Terraform configuration is fairly simple, which for showcasing purposes is a good thing. Hopefully, we will see more contributions to the module in the future.

If you want to find out which resources are supported by Terratest, head on to https://godoc.org/github.com/gruntwork-io/terratest/modules/gcp.

Although we can test any Terraform configuration, Terratest works the best when testing Terraform modules as single small units. That is because it is easier to create a test for small components and Terratest can control the state with input variables. Input variables serve as parameters, changing the behaviour of the module without changing its source code.

Let us take https://github.com/parabolic/terraform_ci_cd as an example repository that has a Terraform module. It creates a bucket, and an instance group with an “N” number of instances. These are the current input variables that are accepted by the this module:

Name	Description	Type	Default	Required
environment	n/a	`string`	n/a	yes
project_id	n/a	`string`	n/a	yes
instance_number	n/a	`number`	`1`	no
machine_type	n/a	`string`	`"n1-standard-1"`	no
name	n/a	`string`	`"cloudlad"`	no
region	n/a	`string`	`"europe-west4"`	no
zone	n/a	`string`	`"europe-west4-c"`	no

Here’s how the directory structure of the Terraform module looks like.

.
├── example
│   ├── main.tf
│   ├── outputs.tf
│   ├── variables.tf
│   └── versions.tf
├── LICENSE
├── main.tf
├── outputs.tf
├── README.md
├── test
│   ├── go.mod
│   ├── go.sum
│   ├── terraform_ci_cd_test.go
└── variables.tf

The “example” directory holds the Terraform templates needed to call and use the module. The source to the module is set to a local path source = "../", which points to the top-level directory, which holds the files of the actual Terraform module.

The “test” directory contains the terratest golang code, along with the “go.mod” and “go.sum” files (the vendor directory is not checked in and it will be created after vendoring the dependencies).

We can change and control any of the input variables with Terratest. As you can see every input variable is set and preferably randomized to cover all of the possible test cases.

environment := "ci"
instanceNumber := 3
name := "terratest"
environmentName := fmt.Sprintf("%s-%s", name, environment)
projectID := gcp.GetGoogleProjectIDFromEnvVar(t)
randomRegion := gcp.GetRandomRegion(t, projectID, nil, nil)
randomZone := gcp.GetRandomZoneForRegion(t, projectID, randomRegion)
// Relative path to the terraform configuration
terraformDir := "../example/"

Note the terraformDir variable at the end. This is a common pattern with which we check the validity and usability of both the module and the provided example, better known as “Eating your own dog food”.

The variables that were previously defined are passed onto the Terraform template with terraformOptions. Consequently controlling and adjusting the Terraform module to fit our needs.

terraformOptions := &terraform.Options{
  TerraformDir: terraformDir,

  Vars: map[string]interface{}{
    "environment":     environment,
    "instance_number": instanceNumber,
    "name":            name,
    "project_id":      projectID,
    "region":          randomRegion,
    "zone":            randomZone,
  },

  EnvVars: map[string]string{
    "GOOGLE_CLOUD_PROJECT": projectID,
  },
}

Going further down the test file, we see the defer statement.

// Destroy all resources in any exit case
defer terraform.Destroy(t, terraformOptions)

Defer makes sure that all the resources are destroyed no matter the exit status of the function. It is commonly used for “clean-up” purposes. In our case the defer statement will issue a non-interactive terraform destroy for every run of the function.

After the defer we specify the terraform.InitAndApply function which will execute “terraform init, terraform get, terraform apply”, with the provided input variables.

// Run terraform init and apply
terraform.InitAndApply(t, terraformOptions)

Now the testing phase starts.

Checking if the bucket exists with the gcp.AssertStorageBucketExists function.

// Check if the bucket exists
gcp.AssertStorageBucketExists(t, environmentName)

Getting the name of the instance group is done via an output, additionally validating our naming scheme.

// Get the instance group name from the output
instanceGroupName := terraform.Output(t, terraformOptions, "instance_group_name")

Lastly, we check for the number of instances present within the instance group. The test will fail if the number does not match the number we have specified with the variable instanceNumber.

// Check the instance number
retry.DoWithRetry(t, "Geting instances from, instance group", maxRetries, sleepBetweenRetries, func() (string, error) {
  instances, err := instanceGroup.GetInstancesE(t, projectID)
  if err != nil {
    return "", fmt.Errorf("Failed to get Instances: %s", err)
  }

  if len(instances) != instanceNumber {
    return "", fmt.Errorf("Expected to find exactly %d Compute Instances in Instance Group but found %d.", instanceNumber, len(instances))
  }
  return "", nil
})

We’ve covered all of the clarifications and preparations, so we can safely start our test.

go test recompiles each package along with any files with names matching the file pattern “_test.go”.*

$ cd test/
$ go test -v
=== RUN   TestTerraformGcp
=== PAUSE TestTerraformGcp
=== CONT  TestTerraformGcp
TestTerraformGcp 2020-05-10T12:21:52Z region.go:163: Looking up all GCP regions available in this account
TestTerraformGcp 2020-05-10T12:21:52Z retry.go:72: Attempting to request a Google OAuth2 token
TestTerraformGcp 2020-05-10T12:21:52Z compute.go:606: Successfully retrieved default GCP client
TestTerraformGcp 2020-05-10T12:21:52Z region.go:67: Using Region asia-east1
TestTerraformGcp 2020-05-10T12:21:52Z retry.go:72: Attempting to request a Google OAuth2 token
TestTerraformGcp 2020-05-10T12:21:52Z compute.go:606: Successfully retrieved default GCP client
TestTerraformGcp 2020-05-10T12:21:53Z region.go:148: Using Zone asia-east1-c
TestTerraformGcp 2020-05-10T12:21:53Z retry.go:72: terraform [init -upgrade=false]
....
....
....
....
TestTerraformGcp 2020-05-10T12:26:14Z command.go:168: Destroy complete! Resources: 6 destroyed.
--- PASS: TestTerraformGcp (261.97s)
PASS
ok      github.com/parabolic/terraform_ci_cd    261.982s

If our test passed, it should conclude with output similar to the one above (the output is truncated).

Conclusion

Even though the GCP module for Terratest is still in infancy, we can still use it to test out basic resources, and the more it is used the more features and better support it will have. Testing is a relevant part of the CI/CD pipeline and I hope I’ve given you a glimpse of how easy it is to start testing your infrastructure with Terraform and Terratest.

Until next time! Please share this blog post if you liked it.

Docker image security practices

2020-01-02T00:00:00+00:00

It takes 20 years to build a reputation and few minutes of cyber-incident to ruin it.
- Stephane Nappo

The state of cyber security today

Security usually means inconvenience, inconvenience brings hindrance, how do we move fast yet stay as secure as possible in this technological present. There’s no universal answer for this subject, but what we should always do at least strive, is to adhere to certain standards and principles which will reduce the risk in the unlikely case of a breach of security. Applying best security practices would be our most dependable remedy against malicious actors of any kind. As an example let’s take today’s container technologies, namely the most popular one, docker. They are fast, lightweight and immutable, but come with a tradeoff. The border that isolates the host and the container is very thin, alas thinner than the conventional bulky and slow Virtual Machines. The tradeoff, in this case, is very clear, we’ve gained enhanced utilization of resources but because the container technology shares a lot of components with the host system if we do not employ well-defined security precautions we will be putting the host system along with the other containers running on it at risk. If we can employ as many security safeguards as we can, rest assured that running a container would be more secure than the traditional Virtual Machine.

How hard is it to misspell the name of the Docker registry image URL combination and get a third-party image with possible malevolent software packaged? Or even worse, what if you mount your docker socket or run the container with full privileges?

In the unfortunate event of a security breach, there’s no guessing on what will be the scope of the potential damages. The old proverb by the Irish Novelist Samuel Lover “Better safe than sorry” is invaluable. There are so many tools out there that can help us build and create wonderful technologies, but when one thinks about it, everything is based on trust which we are placing on the creators and community of said tools to implement the best security practices. Sadly, in reality, that often turns out not to be the case.

Looking on the positive side, we can and we should do something about it, by conforming and adhering to a set of security practices that will lower or nullify the blast radius in a potential breach, which I will elaborate further in this blog post.

I will be focusing on docker mostly as it is one of the most widely used container solutions nowadays.

For the examples I am using this simple golang web server code.

package main

import (
    "fmt"
    "log"
    "net/http"
)

func handler(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hi there, I love %s!", r.URL.Path[1:])
}

func main() {
    http.HandleFunc("/", handler)
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Do not use the root user at container runtime

This is not at all hard to elaborate, since ages even with traditional computing, running services as root was and still is exceptionally bad. Security has improved vastly since, and container technologies offer isolation through kernel namespaces, in which processes running on the container are unbeknownst from processes running on the host system. Kernel namespaces have been around since July 2008, meaning that they’ve had quite a few improvements and iterations, but nothing justifies running the application as the root user.

I will point out a few examples of why this is bad.

The attack surface is vastly enlarged

The root user can do much more harm in the unlikely event of breaking out of the system during a breach. Depending on the end goal of the malicious party it can range from deletion of assets to blackmail, extortion, identity theft, invasion of privacy, fraud, part of a DDoS botnet and more.

Additional applications can be installed on the running container

Imagine we are running a container as the root user, and unfortunately, the application had an old third-party module that was unpatched. The attacker was able to exploit it, and has gained root access within the container. Kernel namespaces have us protected with isolation, nevertheless, the attacker can now install anything on the running container. E.g. a network scanning tool like nmap which can potentially map out the network and thereby giving our imaginary vile agent a wealth of valuable information which can be used against our favor.

The root user can delete any file

If we have a container that needs to persist a state on a mounted volume, in the case of a breach all the files can be deleted from the container because there are no constraints for a user with UID and GID 0.

Remediation

Specify a dedicated user in Docker

We should create and have a dedicated user that will run the application during container runtime:

ARG USERNAME="webserver"
RUN adduser -HD -s /bin/noshell ${USERNAME} ${USERNAME}
USER ${USERNAME}:${USERNAME}

Full example:

Make sure you have the golang webserver sample in the same directory as the following dockerfile

FROM golang:alpine

# All of these are arguments so that they can be overridden per use case basis.
ARG USERNAME="webserver"
ARG WORKDIR="/go/src/app"
ARG GOCACHE="/tmp/go_cache"

# -s SHELL  Login shell
# -D        Don't assign a password
# -H        Don't create home directory
RUN adduser -HD -s /bin/noshell ${USERNAME} ${USERNAME}
WORKDIR ${WORKDIR}
USER ${USERNAME}:${USERNAME}
RUN mkdir -p ${GOCACHE}
COPY . .
RUN go get -d -v . &&\
    go install -v .

CMD ["app]

Now lets build and start the webserver.

docker build -t golang_webserver . && docker run --name golang_webserver -p 8080:8080 golang_webserver

Spawning a shell inside the running container

$ docker exec -it golang_webserver sh

and checking the running processes gives us the following output:

/go/src/app $ ps aux
PID   USER     TIME  COMMAND
    1 webserve  0:00 app
   26 webserve  0:00 sh
   31 webserve  0:00 ps aux

The $ ps output clearly shows that the simple golang web server is running as a non-root user that we have previosly created and specified.

Be vary when mounting the docker socket

Many examples on the internet casually mention that the docker socket should be mounted inside the container in particular installation instructions, which I find very alarming considering this is the main entry point for the docker API. The container that has access to the docker socket /var/run/docker.sock can control all the running containers on the host and as an added bonus the host is most probably compromised because the Docker daemon runs as root by default.

We can see that by querying the docker socket and getting the list of images on our system(a running docker daemon, curl and jq are required).

$ curl --unix-socket /var/run/docker.sock http://localhost/images/json | jq

This will give us all the images that are present on the host where the docker daemon is running currently.

Do not run containers with full container capabilities also known as the privileged flag

Running a container with the --privileged flag gives all capabilities to it plus it lifts all the limitations enforced by the cgroup controller. E.g. the container can now read the host’s /dev and /proc folder. It has “super capabilities” that will allow it to control the host’s devices, processes and kernel parameters. In combination with the container process running as root, the damage can be disastrous.

Do not pass secrets into arguments or enviroment variables during build time

There is a common misconception/anti-pattern regarding the ARG parameter within Dockerfiles, at least I have made the grave mistake thinking that passing a secret with the ARG parameter won’t be persisted to the final image. The Docker documentation somewhat implies that when using arguments they will be present only during build time.

The following excerpt is taken from https://docs.docker.com/engine/reference/commandline/build/

Set build-time variables (–build-arg)
You can use ENV instructions in a Dockerfile to define variable values. These values persist in the built image. However, often persistence is not what you want. Users want to specify variables differently depending on which host they build an image on.

For the ENV parameter it is very clear that it will be persisted in the final image. Yet it turns out that for the ARG the behavior is identical.

Consider the following example:

FROM golang:alpine

ARG USERNAME="webserver"
ARG WORKDIR="/go/src/app"
ARG GOCACHE="/tmp/go_cache"
ARG API_SECRET_KEY
# -s SHELL Login shell
# -D       Don't assign a password
# -H       Don't create home directory
RUN adduser -HD -s /bin/noshell ${USERNAME} ${USERNAME}
WORKDIR ${WORKDIR}
USER ${USERNAME}:${USERNAME}
COPY . .
RUN go get -d -v . &&\ 
    go install -v .
CMD ["app"]

What happens here is that we are passing an argument in the container build stage, which we would think it will only be present in the build stage. Unfortunately it will be persisted, and packed in the image.

Let’s reproduce this behavior.

$ docker build -t build_arg_example --build-arg API_SECRET_KEY="VERY_SECRET_KEY" .

Checking the layers of the freshly built image we can see that the “very secret key” is present on the second line from the output.

$ docker history build_arg_example:latest
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
fd4fb8c4d4d9        31 seconds ago      /bin/sh -c #(nop)  CMD ["app"]                  0B
b59927c7a0ea        32 seconds ago      |4 API_SECRET_KEY=VERY_SECRET_KEY GOCACHE=/t…   7.46MB
afd082c6f60d        34 seconds ago      /bin/sh -c #(nop) COPY dir:2511c5768dd12dce0…   668B
6af096231a1e        36 seconds ago      /bin/sh -c #(nop)  USER webserver:webserver     0B
1560adadd916        37 seconds ago      /bin/sh -c #(nop) WORKDIR /go/src/app           0B
7ccef7155352        38 seconds ago      |4 API_SECRET_KEY=VERY_SECRET_KEY GOCACHE=/t…   4.85kB
d05b1045eb6b        39 seconds ago      /bin/sh -c #(nop)  ARG API_SECRET_KEY           0B
1ef26b6f7772        3 days ago          /bin/sh -c #(nop)  ARG GOCACHE=/tmp/go_cache    0B
a09dbe8447f7        3 days ago          /bin/sh -c #(nop)  ARG WORKDIR=/go/src/app      0B
5ba770850e29        3 days ago          /bin/sh -c #(nop)  ARG USERNAME=webserver       0B
69cf534c966a        2 weeks ago         /bin/sh -c #(nop) WORKDIR /go                   0B
<missing>           2 weeks ago         /bin/sh -c mkdir -p "$GOPATH/src" "$GOPATH/b…   0B
<missing>           2 weeks ago         /bin/sh -c #(nop)  ENV PATH=/go/bin:/usr/loc…   0B
<missing>           2 weeks ago         /bin/sh -c #(nop)  ENV GOPATH=/go               0B
<missing>           2 weeks ago         /bin/sh -c set -eux;  apk add --no-cache --v…   353MB
<missing>           2 weeks ago         /bin/sh -c #(nop)  ENV GOLANG_VERSION=1.13.5    0B
<missing>           2 months ago        /bin/sh -c [ ! -e /etc/nsswitch.conf ] && ec…   17B
<missing>           2 months ago        /bin/sh -c apk add --no-cache   ca-certifica…   551kB
<missing>           2 months ago        /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>           2 months ago        /bin/sh -c #(nop) ADD file:fe1f09249227e2da2…   5.55MB

Remediation

Buildkit, Multi Stage Builds

There are multiple methods on how can we remove secrets from the docker image during the build. I will only describe the ones that I have trialed and have been using for a while in a production-ready setup.

Buildkit

This is the latest iteration of the docker build process. According to docker, it is the “much-needed overhaul of the build architecture”. I prefer this method the most because it truly is an enhacement over the traditional docker build.

Enabling it requires a few extra steps.

First we need to create a file that will hold our “very secret key”:

$ echo 'VERY_SECRET_KEY' > very_secret_key.txt

Then we need to overide the default docker frontend (notice the commented line at the beggining), also expose the secret inside the Dockerfile:

# syntax = docker/dockerfile:1.0-experimental
FROM alpine
RUN --mount=type=secret,id=very_secret_key cat /run/secrets/very_secret_key

Lastly we need to build the imagel and mount the secret during the build process:

$ DOCKER_BUILDKIT=1 docker build --no-cache --progress=plain --secret id=very_secret_key,src=very_secret_key.txt .

Buildkit offers many other improvements which are out of the scope for now, head on to https://docs.docker.com/develop/develop-images/build_enhancements/ to learn more.

Multi Stage Builds

Each instruction in the Dockerfile adds a layer to the image, and all of the artifacts, arguments and environment variables are present in the final image along with the aforementioned layers. With multi-stage build, we can selectively copy artifacts and erase everything else including sensitive data. This way we can have final images that are small, secure and do not contain any delicate information.

Consider the following example:

# 1st stage
FROM golang:alpine as builder

ARG USERNAME="webserver"
ARG WORKDIR="/go/src/app"
ARG GOCACHE="/tmp/go_cache"
ARG API_SECRET_KEY
# -s SHELL Login shell
# -D       Don't assign a password
# -H       Don't create home directory
RUN adduser -HD -s /bin/noshell ${USERNAME} ${USERNAME}
WORKDIR ${WORKDIR}
USER ${USERNAME}:${USERNAME}
COPY . .
RUN go get -d -v . &&\
    go install -v .

# 2nd stage
FROM alpine:latest
ARG WORKDIR="/app"
WORKDIR ${WORKDIR}
# Copy the executable from the first stage.
COPY --from=builder /go/bin/app /usr/local/bin
CMD ["app"]

What I’ve done here, is I got already built binary from the former image labeled as “builder”, to the latter, which doesn’t contain any of the artifacts nor environment variables. We can confirm this easily by building the multi-stage dockerfile and seeing the image history:

$ docker build -t multi_stage  .
$ docker history multi_stage:latest
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
d9ffea8de0f2        19 minutes ago      /bin/sh -c #(nop)  CMD ["app"]                  0B
a303bc8586b3        19 minutes ago      /bin/sh -c #(nop) COPY file:99e2ea856b1ad652…   7.45MB
dbe76a8816f2        21 minutes ago      /bin/sh -c #(nop) WORKDIR /app                  0B
96125fcf5377        21 minutes ago      /bin/sh -c #(nop)  ARG WORKDIR=/app             0B
965ea09ff2eb        2 months ago        /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>           2 months ago        /bin/sh -c #(nop) ADD file:fe1f09249227e2da2…   5.55MB

Image vulnerability scanner

It is always a good practice to check the container that is running the app for any known vulnerabilities. At present, the major cloud providers provide this ability out of the box. An image vulnerability scanner works by scanning the container images in a docker repository and reports on any found and known vulnerabilities. This can be incorporated in a CI/CD pipeline with ease. One such tool is clair.

Slim or container optimized images

Small container images contain fewer packages, fewer packages means reduced attack surface. The current and most excellent container distribution is without a doubt Alpine Linux. Most of the ready to use docker images currently offer an alpine Linux variation.

Use official container images only

Needless to say, if the image we intend to use is provided by a third-party we do not have any way to know what might be installed on it, coupled with a mounted filesystem or docker socket the potential for harm ascends.

Conclusion

There’s no all-in-one solution for being secure, what it takes is persistence, discipline, and above all cybersecurity culture!

If you enjoyed this blog post, or you found it helpful I’d be very grateful if you’d help by sharing it.

Over and out.

Multi-Region Kubernetes - EKS and Terraform 0.12

2019-09-10T00:00:00+00:00

Any technology you adopt today is a technology you’re going to have to troubleshoot tomorrow. - Anonymous

But why EKS and Kubernetes at all?

Kubernetes is the hot stuff these days. Without a doubt, some of the benefits that it offers are impressive. It is a portable, extensible, open-source platform for managing containerized applications with declarative configuration and automation.

The following list is only a small subset of the features it provides:

Scaling
Deployment strategies
Container balancing
Service Discovery
Resource monitoring and logging
Self-healing

It all looks appealing and very promising, but the ultimate question is, do we really need it?

Apart from the obvious benefit of having an all-in-one standardized solution that we can run our containers on, it is quite difficult to say. Unfortunately for most of the companies, it is probably not necessary. Of course, there is always the minority, companies that are required to operate under strict compliance policies, like having their hardware in-house in multiple data centers. Besides that, the times where we run one (or a few) applications per server are long gone. The intricacy of working in a team especially a large one supplements the difficulty, which by itself is one of the hardest hurdles to overcome, and we all know that that there is no perfect, out-of-the-box solution.

Having said that, running and managing a Kubernetes cluster can be a daunting job, but as with the other tech tools at hand, combined with the strive in finding the “sweet spot” between outsourcing them and creating or maintaining the smallest possible subset of them, Kubernetes ultimately improves stability, reliability, speed, and costs.

So let us explore how easy is to get a Multi-Region EKS cluster setup up and running.

The judgment of sacrificing either speed or stability is the bane of the engineering world.

The Amazon Elastic Container Service for Kubernetes tries to ease some of the sophistication that comes with Kubernetes with it’s offering, by extracting and managing one of the most important components of it, the Control Plane.

It has the following additional features:

High availability
Control Plane logging to AWS CloudWatch
Managed updates
Backup
Integration with other AWS services
Service mesh
Fine-Grained IAM Roles for Service Accounts
Container Storage Interface (CSI)

One of the finest features of EKS is that it is a certified Kubernetes conformant, meaning that the applications running on it are fully compatible with any standard Kubernetes environment and vice versa.

To determine whether EKS is the right tool for the job we would need to get it working. For this, I will demonstrate how we can run a Kubernetes cluster on AWS EKS in two separate regions by using the latest iteration of Terraform as our Infrastructure as Code/Software tool.

First let us see how AWS has managed to implement Kubernetes, with a simplified overview of its architecture. The EKS API is exposed via a Network Load Balancer on which we can orchestrate the cluster with kubectl and where the worker nodes connect to. Communication between the control plane and the worker nodes is done through an ENI interface.

Before we begin with the technicalities some considerations need to be taken into account:

DNS

The VPC needs to have DNS hostname and DNS resolution support enabled.

IAM

If we would like to have some of the resources in Kubernetes exposed via Route53 with a tool like ExternalDNS then we shouldn’t forget about setting the correct IAM permissions (they are out of scope for this blog post).

Networking

Private-only: Everything runs in a private subnet and Kubernetes cannot create internet-facing load balancers for your pods.
Public-only: Everything runs in a public subnet, including your worker nodes.

Tagging Requirements

The following tags should be set on the respective resources so that Kubernetes can discover them.

Subnet

kubernetes.io/cluster/<cluster-name> = shared

VPC

kubernetes.io/cluster/<cluster-name> = shared

Worker Nodes

kubernetes.io/cluster/<cluster-name> = owned

Terraform

Please make sure you have the latest version of terraform installed on your system or any version bigger than 0.12.
The terraform examples can be found here, which will be needed for the creation of the clusters and the accompanying resources.

Now that we have that out of the way and we have gained an overview on how AWS has put Kubernetes into operation, we can safely head on to creating two clusters in two separate regions.

Time for some fun with terraform.

I will be using terraform workspaces with which I can conveniently use single set of templates within a single backend and with that I can create clusters in as many regions as I want just by using one single configuration set.

By creating a local variable local.region I have implemented a simple check that detects whether the workspace is default one, which will subsequently stop terraform plan/apply. Setting a bogus region will render this check useless but plan/apply will fail nevertheless.

locals {
  region = terraform.workspace == "default" ? "!!!Please set the terraform workspace to a valid region!!!" : terraform.workspace
}

Then in turn I am using the local variable local.region in the aws provider configuration.

provider "aws" {
  version = "~> 2.27.0"

  profile = local.profile
  region  = local.region
}

Now, before jumping on creating the clusters, we need to make sure we have the desired workspaces created since they will represent the regions. If we do not have them set, we should do it now.

I will be operating in the regions “eu-west-1” and “eu-west-2”

The terraform commands should be executed in the folder where the terraform templates are

$ terraform workspace new eu-west-1
Created and switched to workspace "eu-west-1"!

You're now on a new, empty workspace. Workspaces isolate their state,
so if you run "terraform plan" Terraform will not see any existing state
for this configuration.

After we have the workspaces created we can list them and verify which one we are working on (denoted with an asterisk).

terraform workspace list
  default
* eu-west-1
  eu-west-2

With the workspace select we can conveniently switch between workspaces.

terraform workspace select eu-west-2
Switched to workspace "eu-west-2".

With terraform workspaces we are using the same set of templates without duplication and we are sort of “branching out” by changing the workspace which is considered as separate terraform state, consequently spliting the infrastructure into two separate regions.

And one more thing before applying the terraform configuration, please make sure you have replaced the following local variables so that they represent your local setup:

locals {
  profile     = "testbed"
  allowed_ips = ["0.0.0.0/0"]
  key_name    = ""
}

We can now finally apply terraform, by running:

terraform apply

Initially, I wanted to include the details for configuring kubectl, but because it is a lot of information and there already is more than enough, I have decided that it should stay out of the scope for this blog post. From this point on I will assume that kubectl is set up and working properly.

After terraform apply has finished we will see the resources created in the AWS console, but there is still some work left to connect the workers to the EKS control plane. What we can do at this point is, to validate if the cluster and kubectl are working as intended, and the simplest confirmation that we can do is, kubectl cluster-info which should give us output similar to this.

$ kubectl cluster-info
Kubernetes master is running at https://5E80D2AFE1D478990138A7FCE4038714.yl4.eu-west-1.eks.amazonaws.com
CoreDNS is running at https://5E80D2AFE1D478990138A7FCE4038714.yl4.eu-west-1.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Unfortunately, we are not there, yet.

The EKS service does not provide a cluster-level API parameter or resource to automatically configure the workers to join the EKS control plane via AWS IAM authentication, so we would need to get the ConfigMaps from terraform’s config_map_aws_auth output and apply it with kubectl apply -f

That output will contain the following yaml data:

Please note that in the place of <AWS_ACCOUNT_ID> you should see your current AWS account id

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::<AWS_ACCOUNT_ID>:role/eks-worker-eu-west-1-test-cloudlad
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes

After applying the ConfigMap we will see the nodes connected to the EKS master.

$ kubectl apply -f config_map_aws_auth.yaml
configmap/aws-auth created

We can confirm with the command kubectl get nodes that the worker nodes are connected to the EKS master.

$ kubectl get nodes
NAME                                        STATUS   ROLES    AGE   VERSION
ip-10-0-102-59.eu-west-1.compute.internal   Ready    <none>   26s   v1.13.8-eks-cd3eb0
ip-10-0-103-79.eu-west-1.compute.internal   Ready    <none>   25s   v1.13.8-eks-cd3eb0

For the sake of completeness here is an example of cluster-info and the nodes connected to the EKS cluster in the eu-west-2 region

$ kubectl cluster-info
Kubernetes master is running at https://D89FEC781B28CA9338FC8C2A13006773.sk1.eu-west-2.eks.amazonaws.com
CoreDNS is running at https://D89FEC781B28CA9338FC8C2A13006773.sk1.eu-west-2.eks.amazonaws.com/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

$ kubectl get nodes
NAME                                         STATUS     ROLES    AGE   VERSION
ip-10-0-102-88.eu-west-2.compute.internal    NotReady   <none>   4s    v1.13.8-eks-cd3eb0
ip-10-0-103-232.eu-west-2.compute.internal   NotReady   <none>   10s   v1.13.8-eks-cd3eb0

In a nutshell, we have successfully created two identical EKS clusters in two separate regions, and we were able to connect the worker nodes with the control plane and to verify that everything is functioning well. In the current state, we are ready to take the next steps e.g. deploying a sample app on Kubernetes but that alone deserves a separate blog post.

If you find this blog post useful and interesting please spread the word by sharing. For any suggestions and proposals feel free to contact me. Thank you for reading!

For those interested in some more details about the terraform implementation please read on.

Terraform configuration details

VPC

For the creation of the VPC, I have settled on using the offical terraform module https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/2.12.0.

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "2.12.0"

  name = module.label.id
  cidr = local.vpc_cidr

  azs             = ["${local.region}a", "${local.region}b", "${local.region}c"]
  private_subnets = local.private_subnets_cidrs
  public_subnets  = local.public_subnets_cidrs

  enable_nat_gateway = true
  enable_vpn_gateway = true

  # For the EKS workers.
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = module.label_shared.tags
}

Cloudwatch agent as a terraform module

2019-05-19T00:00:00+00:00

TL;DR: I do not have time for reading this please take me to the examples

Let me kick start this blog post with the notation of a reusable pattern in engineering, which reminds me of a particularly amazing book that I have read recently. I find the following excerpt exceptional:

‘The purpose of a pattern is to provide general advice or structure to guide your design.’

Designing Distributed Systems: Patterns and Paradigms for Scalable, Reliable Services, by Brendan Burns, 2018

How I understand it which is also described further in the book, when designing distributed systems, instead of learning by failing (though inevitable most of the times) we can use pre-defined and reusable patterns that will help us in creating a robust and stable architecture. This doesn’t mean that the same approach works exclusively in IT. As an example try cooking a meal without a recipe. Being an enthusiast and practitioner of Infrastructure as code/software, I will be showcasing this paradigm with terraform modules.

Putting the big picture aside I am going to focus on one small bit, server monitoring on AWS’s EC2 service. One of the first ones that got created back in 2006 when the IaaS model was still in its infancy.

If you have worked with AWS before or you are still using it, it is very likely you have used the EC2 web console to spin up an instance. After requesting and getting an instance you will notice that EC2 provides server monitoring for the running instance(s) but it is by default, maybe by design as well, very basic.

To overcome the problem with the rudimentary monitoring we usually fall back to employing third-party services that give us more complete and comprehensive insights. Most of the times they do the job very well, but as usual, there are some drawbacks.

Familiarity. We have to leave the the AWS web console and venture into the unknown third-party GUIs which are generally satisfactory, yet as a user that is habituated to the former, it can feel a bit disorderly. A lot of companies use more than one server monitoring provider and that is the moment when it becomes very trying.
User management. When using a third-party service for server monitoring we need to somehow manage the users. I am not speaking about whether they offer SAML integration or not, but about the hindrance of managing users and Role-based access control.
Alerting. It is much easier to create alarms in CloudWatch where both the API and the features are very complete.
Infrastructure as Code/Software. The last but not the least, not all third-party server monitoring providers can be managed via an IaC tool.

You can see what I am aiming at right? CloudWatch Agent, a daemon that can collect system-level, custom metrics (using StatsD and collectd), logs both from EC2 and on-premise instances and dispatch them to CloudWatch. We can have all of the server monitoring metrics in one place and deployable as a reusable terraform module.

Click here for more information about the CloudWatch Agent.

But what is a terraform module?

This is how hashicorp describes terraform modules.

A module is a container for multiple resources that are used together.

Aside from that modules are reusable, versionable, testable, and callable from within other modules. They should be as small as possible to have a small blast radius in case of an error and to address the edge cases.

Terraform supports various source types for getting the source code of the modules.

module "cloudwatch_agent" {
  source = "git::https://github.com/cloudposse/terraform-aws-cloudwatch-agent?ref=0.2.0"

  name      = "cloudwatch_agent"
  namespace = "eg"
  stage     = "dev"

  aggregation_dimensions      = ["AutoScalingGroupName", "InstanceId"]
  disk_resources              = ["/", "/mnt", "/home"]
  metrics_collection_interval = "60"
  metrics_config              = "advanced"
}

So how does this particular module work?

The CloudWatch terraform module currently supports collecting metrics and can only be applied on AWS/EC2, however if you have an awesome idea for an improvement or are in need an additional feature, you can either make a contribution or open an issue or here: https://github.com/cloudposse/terraform-aws-cloudwatch-agent/

Packing it all together into a reusable module has its hardships. Let us start with the uncomplicated step, installing the CloudWatch Agent package by using cloud-init in order to detect the underlyin OS utilizing /etc/os-release, a systemd configuration file. This will download, install, and run the daemon.

runcmd:
  - |
    . /etc/os-release
    case $NAME in
      "Amazon Linux") echo "Installing the cloudwatch agent for Amazon Linux."
        curl -O https://s3.amazonaws.com/amazoncloudwatch-agent/amazon_linux/amd64/latest/amazon-cloudwatch-agent.rpm
        rpm -U ./amazon-cloudwatch-agent.rpm
        ;;
      Centos) echo "Installing the cloudwatch agent for Centos Linux."
        curl -O https://s3.amazonaws.com/amazoncloudwatch-agent/centos/amd64/latest/amazon-cloudwatch-agent.rpm
        rpm -U ./amazon-cloudwatch-agent.rpm
        ;;
      Debian) echo "Installing the cloudwatch agent for Debian Linux."
        curl -O https://s3.amazonaws.com/amazoncloudwatch-agent/debian/amd64/latest/amazon-cloudwatch-agent.deb
        dpkg -i -E ./amazon-cloudwatch-agent.deb
        ;;
      Redhat) echo "Installing the cloudwatch agent for Redhat Linux."
        curl -O https://s3.amazonaws.com/amazoncloudwatch-agent/redhat/amd64/latest/amazon-cloudwatch-agent.rpm
        rpm -U ./amazon-cloudwatch-agent.rpm
        ;;
      Suse) echo "Installing the cloudwatch agent for Suse Linux."
        curl -O https://s3.amazonaws.com/amazoncloudwatch-agent/suse/amd64/latest/amazon-cloudwatch-agent.rpm
        rpm -U ./amazon-cloudwatch-agent.rpm
        ;;
      Ubuntu) echo "Installing the cloudwatch agent for Ubuntu Linux."
        curl -O https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
        dpkg -i -E ./amazon-cloudwatch-agent.deb
        ;;
      *)
        echo "Operating system not supported. Please refer to the official documents for more info https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-on-first-instance.html"
    esac
  - /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:/etc/cloudwatch_agent.json -s

At the very end, the command for starting the daemon references a specific configuration file file:/etc/cloudwatch_agent.json. It defines which sets of metrics and detail levels will be monitored from the OS. The AWS CloudWatch Agent has three levels of granularity detail, Basic, Standard, and Advanced. For convenience and simplicity, I have decided to use a pre-generated configuration file which has only the Standard and Advanced detail levels. I have generated the configuration file with using the Wizard.

Switching between the metrics granularity is done by passing the metrics_config parameter when invoking the module:

module "cloudwatch_agent" {
  source = "git::https://github.com/cloudposse/terraform-aws-cloudwatch-agent?ref=0.2.0"

  name      = "cloudwatch_agent"
  namespace = "eg"
  stage     = "dev"

  metrics_config = "advanced" # Accepts only standard or advanced.
}

Inside the main.tf within the module the metrics_config parameter will be used to either load the standard or the advanced configuration file with terraform conditionals.

data "template_file" "cloud_init_cloudwatch_agent" {
  template = "${file("${path.module}/templates/cloud_init.yaml")}"

  vars {
    cloudwatch_agent_configuration = "${var.metrics_config == "standard" ? base64encode(data.template_file.cloudwatch_agent_configuration_standard.rendered) : base64encode(data.template_file.cloudwatch_agent_configuration_advanced.rendered)}"
  }
}

For the EC2 instance to be able to send any metrics to CloudWatch we need to set the IAM permissions accordingly. The module exports both the role and the JSON Policy. Depending on our use case we can select between the two.

Here’s an example with using the role:

module "cloudwatch_agent" {
  source = "git::https://github.com/cloudposse/terraform-aws-cloudwatch-agent?ref=0.2.0"

  name      = "cloudwatch_agent"
  namespace = "eg"
  stage     = "dev"

  metrics_config = "advanced" # Accepts only standard or advanced.
}

resource "aws_iam_instance_profile" "cloudwatch_agent" {
  name_prefix = "cloudwatch_agent"
  role        = "${module.cloudwatch_agent.role_name}" # The exported role from the module.
}

Or using the JSON policy:

module "cloudwatch_agent" {
  source = "git::https://github.com/cloudposse/terraform-aws-cloudwatch-agent?ref=0.2.0"

  name      = "cloudwatch_agent"
  namespace = "eg"
  stage     = "dev"

  metrics_config = "advanced" # Accepts only standard or advanced.
}
resource "aws_iam_role_policy" "cloudwatch_agent" {
  name   = "cloudwatch_agent"
  policy = "${module.cloudwatch_agent.iam_policy_document}" # The exported JSON policy from the module.
  role   = "${aws_iam_role.ec2.id}"
}

This will get our CloudWatch agent up and running on our EC2 instance. For the observant eye, there is a serious flaw with cloud-init in the current setup. What will happen if we have already an existing cloud-init configuration? We should somehow make it work with the cloud-init configuration from the module. For this reason, the Mime Multi Part Archive capability from cloud-init was created which in terraform is a data source called template_cloudinit_config. The module takes care of the intricate Mime Multi Part Archive. The only change that needs to be done is adding the attribute userdata_part_content when calling the module, meaning that we are going to pass our cloud-init configuration and in return, we will get a Mime Multi Part Archive cloud-init which can be used in the launch_configuration:

module "cloudwatch_agent" {
  source = "git::https://github.com/cloudposse/terraform-aws-cloudwatch-agent?ref=0.2.0"

  name      = "cloudwatch_agent"
  namespace = "eg"
  stage     = "dev"

  metrics_config        = "advanced"
  userdata_part_content = "${data.template_file.cloud-init.rendered}" # This is where we pass the cloud-init to the module.
}

data "template_file" "cloud-init" {
  template = "${file("${path.module}/cloud-init.yml")}"
}

resource "aws_launch_configuration" "multipart" {
  name_prefix          = "cloudwatch_agent"
  image_id             = "${data.aws_ami.ecs-optimized.id}"
  iam_instance_profile = "${aws_iam_instance_profile.cloudwatch_agent.name}"
  instance_type        = "t2.micro"
  user_data_base64     = "${module.cloudwatch_agent.user_data}" # Multipart cloud-init from the module gzipped and base64encoded.
  security_groups      = ["${aws_security_group.ec2.id}"]
  key_name             = "${var.ssh_key_pair}"

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_cloudwatch_metric_alarm" "mem_available_percent_alert" {
  alarm_description = "Available memory for ${aws_autoscaling_group.ecs_cluster.name} is below 20 percent."

  alarm_name = "${aws_autoscaling_group.ecs_cluster.name}-mem-available-percent-alert"

  metric_name         = "mem_available_percent" # The metric name as reported by the CloudWatch Agent.
  namespace           = "CWAgent" # The namespace for the CloudWatch Agent.
  comparison_operator = "LessThanOrEqualToThreshold"
  evaluation_periods  = "1"
  period              = "60"
  statistic           = "Minimum"
  threshold           = "20"
  treat_missing_data  = "notBreaching"

  dimensions = {
    AutoScalingGroupName = "${aws_autoscaling_group.ecs_cluster.name}" # The name of the Autoscaling Group.
  }

  alarm_actions = [
    "${aws_autoscaling_policy.scale_up.arn}",
  ]
}

In the end we should have our metrics aggregated and shown in CloudWatch.

It is worth mentioning that the module provides further tailoring via the following parameters:

aggregation_dimensions parameter which defaults to ["InstanceId", "AutoScalingGroupName"] will aggreggate metrics so that alerting can be done within an autoscaling group.
metrics_collection_interval which defailts to 60, specifies how often to collect the cpu metrics and if it’s below 60 seconds then AWS will bill those metrics as high-resolution ones.
disk_resources which defaults to /, Specifies an array of disk mount points. This field limits CloudWatch to collect metrics from only the listed mount points. You can specify * as the value to collect metrics from all mount points.

Click here for more information about all configuration parameters.

That is all for now. If you find this blog post useful and interesting please spread the word by sharing. For any suggestions and proposals feel free to contact me.

Docker persistence for local development

2019-01-08T00:00:00+00:00

Containers and data persistence

In this blog post we will be exploring the possibilities to manage, persist and share data between a host and a container by utilizing a simple pattern dubbed “host based data persistence”.

Before I start diving into technicalities I reckon it is better to clarify a few things so that readers from all levels are on par with what I will be explaining.

The following excerpt is taken from the docker documentation:

By default all files created inside a container are stored on a writable container layer.

But what does this actually mean? Taking into account that the docker layers reside on the host system under /var/lib/docker/{storage-driver-name}, where storage-driver-name can be any of the supported storage drivers for docker, it means that all the data that gets copied or written inside the container will not persist. At least not in the way we expect it to. Furthermore, the fact that it is stored somewhere on the host system makes it even harder to manage or migrate. This behavior somewhat explains why we should not run applications that have a state on a production environment in docker unless we absolutely know what we are doing. But that does not stop us from doing so on a testing environment, let’s say like a local dev setup.

Read on to see how I have implemented this pattern in docker.

Question time!

Q: When a file gets created by the process that runs inside the container, which UID and GID will it get assigned?

A: It will have the UID and GID of the user that is set with the USER instruction in the Dockerfile or 0 (the root user) if the USER instruction is not specified.

Obstacle

As you might have noticed by now this blog platform runs with Jekyll on GitHub. Because Jekyll is a gem and behaves like an ordinary ruby application it is the perfect candidate for running in a container. I will not bore you with all the details, but if I have piqued your interest you can have a look at the repository for the complete setup https://github.com/parabolic/parabolic.github.io.

Let us assume that we already have our nice little blog dockerized and in order to see how it looks like, we need to have a look at it before sending it on its merry journey to the GitHub cloud. Jekyll has a nice feature that regenerates the site when files are modified. The argument in question is --watch. That is very pleasant and helpful but with a simple docker setup where we copy the needed files upon build time inside the container, anything that we change on the host after the container is started is not going to be updated to the container layer that we wrote the data to, consequently rendering Jekyll’s regeneration feature rather useless.

This is easily solvable by mounting a volume from the root level of the project to the running container and Jekyll will be able to detect the changes done from the host, but we will soon find out that all the files that were created by the dockerized Jekyll app on the host system will be owned by the root user or any other user rather that the one that we are logged into our system with. This is normal behavior but it makes matters very inconvenient by having to chown the files back to our current system user.

Note:

The solution that I am about to explain can be used for other web applications too. If we want our changes to be conveyed to the running application inside the container immediately and with the proper UID and GID set, we should have a similar implementation.

Solution

It is apparent that we would need to somehow sync the correct UID and GID between the host and the container, otherwise we would need to manually change the ownership back to our system user with the chown command. In order to do so, some extra work needs to be done. I am usually running and building docker with docker-compose which will help in leveraging the final result.

Firstly we need to somehow tell the container about our current UID and GID and since docker-compose will be controlling the lifecycle of the container we need to inject said UID and GID from it. It turns out that with docker-compose.yml you can read environment variables which will be passed to the container as arguments (can be used with environment variables as well).

args:
  - UID=${UID}
  - GID=${GID}

We should make sure we are passing those arguments in the Dockefile along with defining the USER instruction.

FROM ruby:2.5.3-alpine3.8

LABEL maintainer.url="https://github.com/parabolic"
LABEL description="A dockerized jekyll that regenerates the site on file change."
LABEL maintainer.name="Nikola Velkovski"

ARG UID
ARG GID

ENV APP_HOME="/app"

RUN apk add --update \
    build-base

WORKDIR ${APP_HOME}

COPY Gemfile* ./

RUN bundle install -j "$(getconf _NPROCESSORS_ONLN)"

# Run as the user with the UID and GID from the arguments.
USER ${UID}:${GID}

Before running the container with docker-compose we need to set the environment variables by sourcing the following bash script docker_env.

export UID="$(id -u)"
export GID="$(id -g)"

And finally, the current directory will be mounted to the working directory inside the container at run time, thus eliminating a layer in the Dockerfile where we copy the code on build time.

volumes:
  - .:/app

After we have set and confirmed all of the configuration above we can just simply execute:

source docker_env && docker-compose up --build

If everything went fine we should see something similar appear on stdoud.

blog_1  | Configuration file: /app/_config.yml
blog_1  |             Source: /app
blog_1  |        Destination: /app/_site
blog_1  |  Incremental build: disabled. Enable with --incremental
blog_1  |       Generating...
blog_1  |        Jekyll Feed: Generating feed for posts
blog_1  |                     done in 0.562 seconds.
blog_1  |  Auto-regeneration: enabled for '/app'
blog_1  |     Server address: http://0.0.0.0:5000
blog_1  |   Server running... press ctrl-c to stop.

This means that the application is running locally and if we open http://localhost:5000 we will see our Jekyll blog.

Some of you might have noticed that I have not created a dedicated user inside the container. I have just instructed the container to use my system UID and GID (which are both 1000). This is because I have found that docker has no problems running any application with a UID and GID that do not exist inside it. We can see this by running the docker image used in this blog post. Looking at the users on the container, it clearly shows that a user with UID and GID 1000 is not present.

$ docker run -it ruby:2.5.3-alpine3.8 sh

/ # cat /etc/passwd | sort -n -t: -k3
root:x:0:0:root:/root:/bin/ash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
adm:x:3:4:adm:/var/adm:/sbin/nologin
lp:x:4:7:lp:/var/spool/lpd:/sbin/nologin
sync:x:5:0:sync:/sbin:/bin/sync
shutdown:x:6:0:shutdown:/sbin:/sbin/shutdown
halt:x:7:0:halt:/sbin:/sbin/halt
mail:x:8:12:mail:/var/spool/mail:/sbin/nologin
news:x:9:13:news:/usr/lib/news:/sbin/nologin
uucp:x:10:14:uucp:/var/spool/uucppublic:/sbin/nologin
operator:x:11:0:operator:/root:/bin/sh
man:x:13:15:man:/usr/man:/sbin/nologin
postmaster:x:14:12:postmaster:/var/spool/mail:/sbin/nologin
cron:x:16:16:cron:/var/spool/cron:/sbin/nologin
ftp:x:21:21::/var/lib/ftp:/sbin/nologin
sshd:x:22:22:sshd:/dev/null:/sbin/nologin
at:x:25:25:at:/var/spool/cron/atjobs:/sbin/nologin
squid:x:31:31:Squid:/var/cache/squid:/sbin/nologin
xfs:x:33:33:X Font Server:/etc/X11/fs:/sbin/nologin
games:x:35:35:games:/usr/games:/sbin/nologin
postgres:x:70:70::/var/lib/postgresql:/bin/sh
cyrus:x:85:12::/usr/cyrus:/sbin/nologin
vpopmail:x:89:89::/var/vpopmail:/sbin/nologin
ntp:x:123:123:NTP:/var/empty:/sbin/nologin
smmsp:x:209:209:smmsp:/var/spool/mqueue:/sbin/nologin
guest:x:405:100:guest:/dev/null:/sbin/nologin
nobody:x:65534:65534:nobody:/:/sbin/nologin
/ #

Conclusion

With the solution outlined above we have successfully implemented the host data persistence pattern. We have also heightened our security, solely because the user hasn’t been created in the first place and therefore it has no home directory, no shell or anything else that might get copied from /etc/skel, that is if we do not take extra precaution whilst creating it. As an added bonus we have our application running as a non root user which is one of the simplest and straightforward security practices for docker.

All the configuration files that are mentioned above can be found here https://github.com/parabolic/parabolic.github.io.

That is all for now. If you find this blog post helpful or if you think it can be improved in anyway, I would kindly ask you to provide your feedback and comments! Also please do not forget to share it if you like it!