Manage Kubernetes Resources via Terraform

Kubernetes (K8S) is an open-source workload scheduler with focus on containerized applications. You can use the Terraform Kubernetes provider to interact with resources supported by Kubernetes.

In this tutorial, you will learn how to interact with Kubernetes using Terraform, by scheduling and exposing a NGINX deployment on a Kubernetes cluster.

The final Terraform configuration files used in this tutorial can be found in the Deploy NGINX on Kubernetes via Terraform GitHub repository.

»Why deploy with Terraform?

While you could use kubectl or similar CLI-based tools to manage your Kubernetes resources, using Terraform has the following benefits:

• Unified Workflow - If you are already provisioning Kubernetes clusters with Terraform, use the same configuration language to deploy your applications into your cluster.

• Full Lifecycle Management - Terraform doesn't only create resources, it updates, and deletes tracked resources without requiring you to inspect the API to identify those resources.

• Graph of Relationships - Terraform understands dependency relationships between resources. For example, if a Persistent Volume Claim claims space from a particular Persistent Volume, Terraform won't attempt to create the claim if it fails to create the volume.

»Prerequisites

The tutorial assumes some basic familiarity with Kubernetes and kubectl.

It also assumes that you are familiar with the usual Terraform plan/apply workflow. If you're new to Terraform itself, refer first to the Getting Started tutorial.

For this tutorial, you will need an existing Kubernetes cluster. If you don't have a Kubernetes cluster, you can use kind to provision a local Kubernetes cluster or provision one on a cloud provider.

$ brew install kind

$ brew install kind

$ curl https://raw.githubusercontent.com/hashicorp/learn-terraform-deploy-nginx-kubernetes-provider/master/kind-config.yaml --output kind-config.yaml

$ curl https://raw.githubusercontent.com/hashicorp/learn-terraform-deploy-nginx-kubernetes-provider/master/kind-config.yaml --output kind-config.yaml

$ kind create cluster --name terraform-learn --config kind-config.yaml
Creating cluster "terraform-learn" ...
 ✓ Ensuring node image (kindest/node:v1.17.0) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-terraform-learn"
You can now use your cluster with:

kubectl cluster-info --context kind-terraform-learn

Have a nice day! 👋

$ kind create cluster --name terraform-learn --config kind-config.yamlCreating cluster "terraform-learn" ... ✓ Ensuring node image (kindest/node:v1.17.0) 🖼 ✓ Preparing nodes 📦 ✓ Writing configuration 📜 ✓ Starting control-plane 🕹️ ✓ Installing CNI 🔌 ✓ Installing StorageClass 💾Set kubectl context to "kind-terraform-learn"You can now use your cluster with:
kubectl cluster-info --context kind-terraform-learn
Have a nice day! 👋

$ kind get clusters
terraform-learn

$ kind get clustersterraform-learn

$ kubectl cluster-info --context kind-terraform-learn
Kubernetes master is running at https://127.0.0.1:32769
KubeDNS is running at https://127.0.0.1:32769/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

$ kubectl cluster-info --context kind-terraform-learnKubernetes master is running at https://127.0.0.1:32769KubeDNS is running at https://127.0.0.1:32769/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

»Configure the provider

Before you can schedule any Kubernetes services using Terraform, you need to configure the Terraform Kubernetes provider.

There are many ways to configure the Kubernetes provider. We recommend them in the following order (most recommended first, least recommended last):

1. Use cloud-specific auth plugins (for example, eks get-token, az get-token, gcloud config)
2. Use oauth2 token
3. Use TLS certificate credentials
4. Use kubeconfig file by setting both config_path and config_context
5. Use username and password (HTTP Basic Authorization)

Follow the instructions in the kind or cloud provider tabs to configure the provider to target a specific Kubernetes cluster. The cloud provider tabs will configure the Kubernetes provider using cloud-specific auth tokens.

Create a directory named learn-terraform-deploy-nginx-kubernetes.

$ mkdir learn-terraform-deploy-nginx-kubernetes

$ mkdir learn-terraform-deploy-nginx-kubernetes

Then, navigate into it

$ cd learn-terraform-deploy-nginx-kubernetes

$ cd learn-terraform-deploy-nginx-kubernetes

terraform {
  required_providers {
    kubernetes = {
      source = "hashicorp/kubernetes"
    }
  }
}

variable "host" {
  type = string
}

variable "client_certificate" {
  type = string
}

variable "client_key" {
  type = string
}

variable "cluster_ca_certificate" {
  type = string
}

provider "kubernetes" {
  host = var.host

  client_certificate     = base64decode(var.client_certificate)
  client_key             = base64decode(var.client_key)
  cluster_ca_certificate = base64decode(var.cluster_ca_certificate)
}

terraform {  required_providers {    kubernetes = {      source = "hashicorp/kubernetes"    }  }}
variable "host" {  type = string}
variable "client_certificate" {  type = string}
variable "client_key" {  type = string}
variable "cluster_ca_certificate" {  type = string}
provider "kubernetes" {  host = var.host
  client_certificate     = base64decode(var.client_certificate)  client_key             = base64decode(var.client_key)  cluster_ca_certificate = base64decode(var.cluster_ca_certificate)}

$ kubectl config view --minify --flatten --context=kind-terraform-learn
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRU...
    server: https://127.0.0.1:32768
  name: kind-terraform-learn
contexts:
- context:
    cluster: kind-terraform-learn
    user: kind-terraform-learn
  name: kind-terraform-learn
current-context: kind-terraform-learn
kind: Config
preferences: {}
users:
- name: kind-terraform-learn
  user:
    client-certificate-data: LS0tLS1CRU...
    client-key-data: LS0tLS1CRU...

$ kubectl config view --minify --flatten --context=kind-terraform-learnapiVersion: v1clusters:- cluster:    certificate-authority-data: LS0tLS1CRU...    server: https://127.0.0.1:32768  name: kind-terraform-learncontexts:- context:    cluster: kind-terraform-learn    user: kind-terraform-learn  name: kind-terraform-learncurrent-context: kind-terraform-learnkind: Configpreferences: {}users:- name: kind-terraform-learn  user:    client-certificate-data: LS0tLS1CRU...    client-key-data: LS0tLS1CRU...

# terraform.tfvars

host                   = "https://127.0.0.1:32768"
client_certificate     = "LS0tLS1CRUdJTiB..."
client_key             = "LS0tLS1CRUdJTiB..."
cluster_ca_certificate = "LS0tLS1CRUdJTiB..."

# terraform.tfvars
host                   = "https://127.0.0.1:32768"client_certificate     = "LS0tLS1CRUdJTiB..."client_key             = "LS0tLS1CRUdJTiB..."cluster_ca_certificate = "LS0tLS1CRUdJTiB..."

After configuring the provider, run terraform init to download the latest version and initialize your Terraform workspace.

$ terraform init

$ terraform init

»Schedule a deployment

Add the following to your kubernetes.tf file. This Terraform configuration will schedule a NGINX deployment with two replicas on your Kubernetes cluster, internally exposing port 80 (HTTP).

resource "kubernetes_deployment" "nginx" {
  metadata {
    name = "scalable-nginx-example"
    labels = {
      App = "ScalableNginxExample"
    }
  }

  spec {
    replicas = 2
    selector {
      match_labels = {
        App = "ScalableNginxExample"
      }
    }
    template {
      metadata {
        labels = {
          App = "ScalableNginxExample"
        }
      }
      spec {
        container {
          image = "nginx:1.7.8"
          name  = "example"

          port {
            container_port = 80
          }

          resources {
            limits = {
              cpu    = "0.5"
              memory = "512Mi"
            }
            requests = {
              cpu    = "250m"
              memory = "50Mi"
            }
          }
        }
      }
    }
  }
}

resource "kubernetes_deployment" "nginx" {  metadata {    name = "scalable-nginx-example"    labels = {      App = "ScalableNginxExample"    }  }
  spec {    replicas = 2    selector {      match_labels = {        App = "ScalableNginxExample"      }    }    template {      metadata {        labels = {          App = "ScalableNginxExample"        }      }      spec {        container {          image = "nginx:1.7.8"          name  = "example"
          port {            container_port = 80          }
          resources {            limits = {              cpu    = "0.5"              memory = "512Mi"            }            requests = {              cpu    = "250m"              memory = "50Mi"            }          }        }      }    }  }}

You may notice the similarities between the Terraform configuration and Kubernetes configuration YAML file.

Apply the configuration to schedule the NGINX deployment. Remember to confirm your apply with a yes.

$ terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are
indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # kubernetes_deployment.nginx will be created
  + resource "kubernetes_deployment" "nginx" {
      ## ...
    }

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

kubernetes_deployment.nginx: Creating...
kubernetes_deployment.nginx: Still creating... [10s elapsed]
kubernetes_deployment.nginx: Still creating... [20s elapsed]
kubernetes_deployment.nginx: Creation complete after 26s [id=default/scalable-nginx-example]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

$ terraform apply
Terraform used the selected providers to generate the following execution plan. Resource actions areindicated with the following symbols:  + create
Terraform will perform the following actions:
  # kubernetes_deployment.nginx will be created  + resource "kubernetes_deployment" "nginx" {      ## ...    }
Plan: 1 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?  Terraform will perform the actions described above.  Only 'yes' will be accepted to approve.
  Enter a value: yes
kubernetes_deployment.nginx: Creating...kubernetes_deployment.nginx: Still creating... [10s elapsed]kubernetes_deployment.nginx: Still creating... [20s elapsed]kubernetes_deployment.nginx: Creation complete after 26s [id=default/scalable-nginx-example]
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Once the apply is complete, verify the NGINX deployment is running.

$ kubectl get deployments
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
scalable-nginx-example   2/2     2            2           15s

$ kubectl get deploymentsNAME                     READY   UP-TO-DATE   AVAILABLE   AGEscalable-nginx-example   2/2     2            2           15s

»Schedule a Service

There are multiple Kubernetes services you can use to expose your NGINX to users.

If your Kubernetes cluster is hosted locally on kind, you will expose your NGINX instance via NodePort to access your instance. This exposes the service on each node's IP at a static port, allowing you to access the service from outside the cluster at <NodeIP>:<NodePort>.

If your Kubernetes cluster is hosted on a cloud provider, you will expose your NGINX instance via LoadBalancer to access your instance. This exposes the service externally using a cloud provider's load balancer.

Notice how the Kubernetes Service resource block dynamically assigns the selector to the Deployment's label. This avoids common bugs due to mismatched service label selectors.

resource "kubernetes_service" "nginx" {
  metadata {
    name = "nginx-example"
  }
  spec {
    selector = {
      App = kubernetes_deployment.nginx.spec.0.template.0.metadata[0].labels.App
    }
    port {
      node_port   = 30201
      port        = 80
      target_port = 80
    }

    type = "NodePort"
  }
}

resource "kubernetes_service" "nginx" {  metadata {    name = "nginx-example"  }  spec {    selector = {      App = kubernetes_deployment.nginx.spec.0.template.0.metadata[0].labels.App    }    port {      node_port   = 30201      port        = 80      target_port = 80    }
    type = "NodePort"  }}

$ terraform apply
kubernetes_deployment.nginx: Refreshing state... [id=default/scalable-nginx-example]

## ...

Plan: 1 to add, 0 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

kubernetes_service.nginx: Creating...
kubernetes_service.nginx: Creation complete after 0s [id=default/nginx-example]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

$ terraform applykubernetes_deployment.nginx: Refreshing state... [id=default/scalable-nginx-example]
## ...
Plan: 1 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?  Terraform will perform the actions described above.  Only 'yes' will be accepted to approve.
  Enter a value: yes
kubernetes_service.nginx: Creating...kubernetes_service.nginx: Creation complete after 0s [id=default/nginx-example]
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

$ kubectl get services
NAME            TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
kubernetes      ClusterIP   10.96.0.1     <none>        443/TCP        2m53s
nginx-example   NodePort    10.96.55.64   <none>        80:30201/TCP   76s

$ kubectl get servicesNAME            TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGEkubernetes      ClusterIP   10.96.0.1     <none>        443/TCP        2m53snginx-example   NodePort    10.96.55.64   <none>        80:30201/TCP   76s

»Scale the deployment

You can scale your deployment by increasing the replicas field in your configuration. Change the number of replicas in your Kubernetes deployment from 2 to 4.

resource "kubernetes_deployment" "nginx" {
  ## ...

  spec {
    replicas = 4

    ## ...
  }

  ## ...
}

resource "kubernetes_deployment" "nginx" {  ## ...
  spec {    replicas = 4
    ## ...  }
  ## ...}

Apply the change to scale your deployment. Remember to confirm your apply with a yes.

$ terraform apply
kubernetes_deployment.nginx: Refreshing state... [id=default/scalable-nginx-example]
kubernetes_service.nginx: Refreshing state... [id=default/nginx-example]

## ...

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

kubernetes_deployment.nginx: Modifying... [id=default/scalable-nginx-example]
kubernetes_deployment.nginx: Modifications complete after 0s [id=default/scalable-nginx-example]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

$ terraform applykubernetes_deployment.nginx: Refreshing state... [id=default/scalable-nginx-example]kubernetes_service.nginx: Refreshing state... [id=default/nginx-example]
## ...
Plan: 0 to add, 1 to change, 0 to destroy.
Do you want to perform these actions?  Terraform will perform the actions described above.  Only 'yes' will be accepted to approve.
  Enter a value: yes
kubernetes_deployment.nginx: Modifying... [id=default/scalable-nginx-example]kubernetes_deployment.nginx: Modifications complete after 0s [id=default/scalable-nginx-example]
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

Once the apply is complete, verify the NGINX deployment has four replicas.

$ kubectl get deployments
NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
scalable-nginx-example   4/4     4            4           4m48s

$ kubectl get deploymentsNAME                     READY   UP-TO-DATE   AVAILABLE   AGEscalable-nginx-example   4/4     4            4           4m48s

»Clean up your workspace

Remember to destroy any resources you created once you're done with this tutorial.

Running terraform destroy will de-provision the NGINX deployment and service you created in this tutorial. Remember to confirm your destroy with a yes.

$ terraform destroy
kubernetes_deployment.nginx: Refreshing state... [id=default/scalable-nginx-example]
kubernetes_service.nginx: Refreshing state... [id=default/nginx-example]

## ...

Plan: 0 to add, 0 to change, 2 to destroy.

Do you really want to destroy all resources?
  Terraform will destroy all your managed infrastructure, as shown above.
  There is no undo. Only 'yes' will be accepted to confirm.

  Enter a value: yes

kubernetes_service.nginx: Destroying... [id=default/nginx-example]
kubernetes_service.nginx: Destruction complete after 0s
kubernetes_deployment.nginx: Destroying... [id=default/scalable-nginx-example]
kubernetes_deployment.nginx: Destruction complete after 0s

Destroy complete! Resources: 2 destroyed.

$ terraform destroykubernetes_deployment.nginx: Refreshing state... [id=default/scalable-nginx-example]kubernetes_service.nginx: Refreshing state... [id=default/nginx-example]
## ...
Plan: 0 to add, 0 to change, 2 to destroy.
Do you really want to destroy all resources?  Terraform will destroy all your managed infrastructure, as shown above.  There is no undo. Only 'yes' will be accepted to confirm.
  Enter a value: yes
kubernetes_service.nginx: Destroying... [id=default/nginx-example]kubernetes_service.nginx: Destruction complete after 0skubernetes_deployment.nginx: Destroying... [id=default/scalable-nginx-example]kubernetes_deployment.nginx: Destruction complete after 0s
Destroy complete! Resources: 2 destroyed.

If you are using a kind Kubernetes cluster, run the following command to delete it.

$ kind delete cluster --name terraform-learn

$ kind delete cluster --name terraform-learn

»Next steps

In this tutorial, you configured the Terraform Kubernetes provider and used it to schedule, expose and scale an NGINX instance.

To discover additional capabilities, visit the Terraform Kubernetes Provider Registry Documentation Page.

For a more in-depth Kubernetes example, Deploy Consul and Vault on a Kubernetes Cluster using Run Triggers (this tutorial is GKE based).