Vault with TLS running Kubernetes

Introduccion

This guide aims to help you deploy HashiCorp Vault on a Kubernetes cluster, secured by TLS certificates created using Let’s Encrypt. The Vault will be publicly accessible via an Ingress (without using Service NodePort).

Prerequisites

• A Kubernetes cluster (Version used here is 1.22)
• Kubernetes ingress NGINX controller (Github link)
• AWS KMS key for auto-sealing (AWS KMS key creation guide). Ensure the Kubernetes cluster has the appropriate permissions to access the AWS account.
• certbot (let’s encrypt) installed in your computer
• openssl installed in your computer (CLI)
• Helm installed in your computer

Create the Namespace

We’ll use a specific namespace for Vault:

kubectl create namespace vault

Generating certificates using CertBot (Let’s Encrypt)

We need three files: CA, cert, and key. We’ll use Certbot to generate these certificates for internal and external communication.

Generate the certs

sudo certbot certonly --manual --preferred-challenges=dns --email YOUR_EMAIL_HERE --server https://acme-v02.api.letsencrypt.org/directory --agree-tos -d "*.YOUR_DOMAIN_HERE"

If you want to know more about this process there plenty guides on internet about this, so I’wont go into details…

This command generates files in /etc/letsencrypt/live/<your root domain>/, we will use fullchain.pem and privkey.pem

Split the Fullchain Certificate

The fullchain.pem contains two certificates. Split this file into the CA certificate and the certificate itself.

How can know which is which?

Usually the last certificate within the fullchain it’s the CA, extract it to a file (ca.crt) and then you can be sure using the following command

openssl x509 -in ca.crt -text -noout

and look for the following line

CA:TRUE, pathlen:0

the other certificate will say CA:FALSE

You should now have three files:

• ca.crt
• vault.crt
• vault.key

Create the secret

kubectl create secret generic vault-server-tls \
    --from-file=vault.key=vault.key \
    --from-file=vault.crt=vault.crt \
    --from-file=vault.ca=ca.crt

Configure Kubernetes Ingress NGINX controller

we’ll use an specific option to use ingress for our vault, the official documentation use a NodePort to expose the service but we will use an ingress, this can be useful for clusters using private nodes.

if you are using helm to install it add the following value, enable-ssl-passthrough: true (more details here)

controller:
    extraArgs:
        enable-ssl-passthrough: true

I won’t go into details about kubernetes ingress NGINX because it depends of different things and I don’t want to mess up with your current implementation, if you need to install it, just follow a simple installation using the extraArgs mentioned before

How to know if the ssl-passthrough is being used by my NGINX?

the pod of nginx should have the following argument defined in the kubernetes manifest --enable-ssl-passthrough=true

Configure CoreDNS

Since we’re using a public CA, we need to add rewrite rules to CoreDNS to use our custom domain.

Add the following lines to your coredns Configmap

rewrite stop {
    name regex (.*)-ns-(.*)\.YOUR-DOMAIN-HERE\.$ {1}.{2}.svc.cluster.local
    answer name (.*)\.(.*)\.svc\.cluster\.local  {1}-ns-{2}.YOUR-DOMAIN-HERE.
}

Your ConfigMap should look like this:

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        rewrite stop {
          name regex (.*)-ns-(.*)\.YOUR-DOMAIN-HERE\.$ {1}.{2}.svc.cluster.local
          answer name (.*)\.(.*)\.svc\.cluster\.local  {1}-ns-{2}.YOUR-DOMAIN-HERE.
        }
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        hosts /etc/coredns/NodeHosts {
          ttl 60
          reload 15s
          fallthrough
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    import /etc/coredns/custom/*.server
  NodeHosts: |
  ...

Installing Vault using Helm

We will install Vault using Helm

Add the repository of vault

helm repo add hashicorp https://helm.releases.hashicorp.com
helm repo update

Create a values.yaml file for the Helm chart:

# values.yaml

global:
  enabled: true
  tlsDisable: false
  resources:
    requests:
      memory: 256Mi
      cpu: 250m
    limits:
      memory: 256Mi
      cpu: 250m

server:
  image:
    repository: "hashicorp/vault"
  readinessProbe:
    enabled: true
    path: "/v1/sys/health?standbyok=true&sealedcode=204&uninitcode=204"
  livenessProbe:
    enabled: true
    path: "/v1/sys/health?standbyok=true"
    initialDelaySeconds: 60
  extraEnvironmentVars:
    VAULT_CACERT: /vault/userconfig/vault-server-tls/vault.ca
  volumes:
    - name: userconfig-vault-server-tls
      secret:
        defaultMode: 420
        secretName: vault-server-tls
  volumeMounts:
    - mountPath: /vault/userconfig/vault-server-tls
      name: userconfig-vault-server-tls
      readOnly: true
  auditStorage:
    enabled: true
  standalone:
    enabled: false

  # Run Vault in "HA" mode.
  ha:
    enabled: true
    replicas: 2
    raft:
      enabled: true
      setNodeId: true

      config: |
        ui = true
        cluster_name = "vault-integrated-storage"
        listener "tcp" {
          address = "[::]:8200"
          cluster_address = "[::]:8201"
          tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
          tls_key_file = "/vault/userconfig/vault-server-tls/vault.key"
        }

        seal "awskms" {
          region     = "us-east-2"
          kms_key_id = "arn:aws:kms:us-east-2:xxxxxxxx:key/xxxxxxx-xxxx-xxxx-xxxx-xxxxxxx"
        }

        storage "raft" {
          path = "/vault/data"
          retry_join {
            leader_api_addr = "https://vault-0.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }
          retry_join {
            leader_api_addr = "https://vault-1.vault-internal:8200"
            leader_ca_cert_file = "/vault/userconfig/vault-server-tls/vault.ca"
            leader_client_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
            leader_client_key_file = "/vault/userconfig/vault-server-tls/vault.key"
          }
          autopilot {
            server_stabilization_time = "10s"
            last_contact_threshold = "10s"
            min_quorum = 5
            cleanup_dead_servers = false
            dead_server_last_contact_threshold = "10m"
            max_trailing_logs = 1000
            disable_upgrade_migration = false
          }
        }

# Vault UI
ui:
  enabled: true
  serviceType: "ClusterIP"
  externalPort: 8200

More information on the chart values can be found here

Install Vault

helm install vault hashicorp/vault --namespace vault -f values.yaml

Initialize Vault

Once Vault is running in your cluster, initialize it with the following command. This command will save all the keys in cluster-keys.json, which you should keep secure.

kubectl exec vault-0 -- vault operator init -recovery-shares=1 -recovery-threshold=1 -format=json > cluster-keys.json

if the leader it’s the pod vault-1 then point to that pod, also you can do this using UI.

Create a Kubernetes Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
  labels:
    app: vault
    name: vault
  name: vault
  namespace: vault
spec:
  ingressClassName: nginx
  rules:
  - host: vault.YOUR-DOMAIN-HERE
      http:
      paths:
      - backend:
          service:
            name: vault-ui
            port:
              number: 8200
        path: /
        pathType: Prefix

Considerations

• The certificates created by Certbot are valid only for 3 months. Renewal should be done manually for this implementation.