Hetzner Cloud - Production K8s instance with Terraform
Hetzner currently doesn't provide a managed K8s service. This isn't a problem because hetzner exposes a nice API that we can use in conjunction with Terraform to structure our infrastructure as code.
Tools we'll be using:
Terraform
Hetzner Cloud - Controller Manager
The Hetzner cloud controller manager integrates your K8s cluster with the Hetzner Cloud API. This allows us to do things like use Hetzner cloud private networks for our traffic, and also use Hetzner Cloud load balances with Kubernetes services.
Hetzner Cloud - Controller Manager - Github Repo
Hetzner Cloud - CSI Driver
This is a Container Storage Interface driver for Hetzner Cloud enabling you to use ReadWriteOnce Volumes within Kubernetes & other Container Orchestrators. This requires Kubernetes ^1.19.
Hetzner Cloud - CSI Driver - Github Repo
First off - We need:
A Hetzner Cloud Project
Go to Hetzner Cloud - Projects Page, and create a new project.
Hetzner Cloud - API Key
To Generate a new Token
- Go into your project.
- Click
securityon the side panel (looks like a key.) - Click on
API tokens. - Click Generate API token
- Give your token a name.
- Ensure you give your token Read & Write permissions.
- Ensure to keep a copy of this safe, and away from any version control.
To Use the Token
Terraform automatically uses and loads environment variables in the running system that are prefixed with: `TF_VAR_`. If they are declared within the variables file.
Therefore we can run the following command to set this new needed enviroment variable in our unix based system.
export TF_VAR_HCLOUD_API_TOKEN="YOUR HETZNER CLOUD API TOKEN"Save this token somewhere safe for later, we will need it to set up Hetzner's Container Storage Interface Driver.
Terraform - Provider Setup โก
Creating a Project Folder.
First off, create project folder. For this article we will refer to all paths as relative to this folder. Therefor we will refer to the root project folder as: `.`
Creating a Folder for Terraform
Then within the project folder let's create a folder called `./terraform/`.
This is where we will create all of our terraform code to provision our infrastructure.
Establishing our Provider
The first file to create within our terraform directory is `provider.tf`.
This is where we can define how terraform should communicate with Hetzner as a provider.
provider "hcloud" {
token = "${var.HCLOUD_API_TOKEN}"
}
terraform {
required_providers {
hcloud = {
source = "hetznercloud/hcloud"
}
}
required_version = ">= 0.14"
}./terraform/provider.tf
In this file we have:
- Created our
provideras"hcloud", and set our provider token to a variable calledHCLOUD_API_TOKEN. - We then set our new provider as one of the projects required providers.
- Within this we set the source of the providers plugin, and its required version.
Creating a place for Variables to live. ๐
Before using any variables in terraform, we need to declare them.
Lets create a `variables.tf` file within our terraform directory, to declare any variables we need to use.
variable "HCLOUD_API_TOKEN" {
type = string
sensitive = true
}./terraform/variables.tf
Setting the optional sensitive property to true ensures that this variable will not show up in any logs or outputs.
Terraform - Project Initialization ๐ก
Next we can change directory into the ./terraform directory, and initialize terraform.
This will allow terraform to download the required provider plugins.
Change into the directory:
cd terraformInitialize Terraform:
terraform initTerraform - Building Infrastructure Code ๐๏ธ
We can now create a new `main.tf` file that will hold the majority of our infrastructure code.
The Jump-Server โคด๏ธ
Let's start by defining a jump server that we will use as a base to manage all the other servers in the project.
resource "hcloud_server" "jump-server" {
name = "jump-server"
image = "debian-12"
server_type = "cax11"
datacenter = "hel1-dc2"
ssh_keys = ["My SSH KEY"]
public_net {
ipv4_enabled = true
ipv6_enabled = true
}
network {
network_id = hcloud_network.kubernetes-node-network.id
ip = "172.16.0.100"
}
depends_on = [
hcloud_network_subnet.kubernetes-node-subnet
]
}./terraform/main.tf
Specs
For our jumpbox, we have gone for:
- Linux in the flavour of debian 12.
- A CAX11 box.
- A Datacenter in Helsinki Finland.
- A IPV4 public facing IP.
We set the SSH keys to enable secure password-less log ins.
We will define the My SSH KEY at the end of this file.Network
We also attach the server to the kubernetes-node-network network, and assign it a private IP address.
Dependencies
The depends on block ensures that the server is not provisioned until the kubernetes-node-subset has been created.
[!Warning] Ensure that this dependency is in place.
If this server is provisioned before the subnet is, terraform will try to create the subnet, which will result in a conflict after the actual subnet is provisioned.
The Kube Node ๐ฅ
Continuing on in `main.tf` we will add our nodes:
//...
resource "hcloud_server" "kube-node" {
count = 3
name = "kube-node-${count.index + 1}"
image = "debian-12"
server_type = "cpx21"
datacenter = "hel1-dc2"
ssh_keys = ["My SSH KEY"]
public_net {
ipv4_enabled = true
ipv6_enabled = true
}
network {
network_id = hcloud_network.kubernetes-node-network.id
ip = "172.16.0.10${count.index + 1}"
}
depends_on = [
hcloud_network_subnet.kubernetes-node-subnet
]
}./terraform/main.tf - continued...
This is configured very similarly to the jump box.
The Private Network ๐
Continuing on in `main.tf`, we will define the private network and subnet from which the nodes and jump box will connect and communicate with each other.
//...
resource "hcloud_network" "kubernetes-node-network" {
name = "kubernetes-node-network"
ip_range = "172.16.0.0/24"
}
resource "hcloud_network_subnet" "kubernetes-node-subnet" {
type = "cloud"
network_id = hcloud_network.kubernetes-node-network.id
network_zone = "eu-central"
ip_range = "172.16.0.0/24"
}./terraform/main.tf - continued...
The SSH KEY ๐
Continuing in `main.tf`, next we will define the SSH key referred to previously in the file.
//...
resource "hcloud_ssh_key" "default" {
name = "My SSH KEY"
public_key = "${var.MY_SSH_KEY}"
}./terraform/main.tf - continued...
Now that we have added this new variable reference, we will also need to add the new variable to the `variables.tf` file.
//...
variable "MY_SSH_KEY" {
type = string
sensitive = true
}./terraform/variables.tf
We can now create a new file to set the value of this variable: `terraform.tfvars`.
MY_SSH_KEY = "YOUR PUBLIC SSH KEY"./terraform/terraform.tfvars
[!Warning] Do not include the terraform.tfvars file in version control.Ensure that yourtfvarsfile is not included in any version control, and is added to the.gitignorefile.
Outputs
Before we deploy to Hetzner, lets define some outputs in a new file. Create the file `output.tf`.
output "kube-node-1-ip" {
value = [for node in hcloud_server.kube-node[0].network: node.ip]
}
output "kube-node-2-ip" {
value = [for node in hcloud_server.kube-node[1].network: node.ip]
}
output "kube-node-3-ip" {
value = [for node in hcloud_server.kube-node[2].network: node.ip]
}./terraform/output.tf
Terraform - Deployment ๐งฑ
Plan
First lets see what terraform plans to deploy, so that we can double check it.
terraform planOnce you are satisfied with the output of what terraform will produce we can deploy it.
Deployment
To deploy our terraform scripts run:
terraform apply --auto-approveIf you go into the project overview now on Hetzner, you should see all the servers be provisioned.
SSHKeys - Jump Server ๐
Next lets SSH into our Jump server, create a new SSH key and add it to our Terraform setup.
You can get the Public IP of our jump-server by looking at the Hetzner cloud console.
Enter the jump server
ssh root@jump.server.ipGenerate the new keys on the jump-server
ssh-keygenRetrieve the newly generated public key
cat /root/.ssh/id_rsa.pubNow we can add this value to the terraform.tfvars file:
//...
JUMP_SERVER_SSH_KEY = "THE KEY THAT WAS JUST DISPLAYED"./terraform/terraform.tfvars
And declare the new variable in the variables.tf file:
//...
resource "hcloud_ssh_key" "jump_server" {
name = "JUMP SERVER SSH KEY"
public_key = "${var.JUMP_SERVER_SSH_KEY}"
}./terraform/main.tf - continued...
Then in main.tf we can declare the ssh-key as a new resource.
//...
resource "hcloud_ssh_key" "jump_server" {
name = "JUMP SERVER SSH KEY"
public_key = "${var.JUMP_SERVER_SSH_KEY}"
}./terraform/main.tf - continued...
Then we can add it to the list of SSH keys in the kube-node resource:
./terraform/main.tf
resource "hcloud_server" "kube-node" {
//...
ssh_keys = ["My SSH KEY", "JUMP SERVER SSH KEY"]
//...
}This will allow us to log into our kube-nodes from our jump server.
Now we can redeploy our architecture with our updated SSH keys.
terraform apply --auto-approveIn the Hetzner cloud console you can also see the updated SSH keys in the security tab.
Kubernetes - Deployment ๐ข
Now that we have provisioned our infrastructure with Terraform and we have our nodes, which are accessible via the jumpbox. We are now able to deploy kubernetes upon them.
Cloning the Kubespray Repo
First thing to do is to SSH into our Jump server...
ssh root@jump.server.ipInstall Git:
apt update
apt install -y git
git --version...and create a project directory
mkdir kube-setup && cd kube-setupOnce inside the project directory, clone the Kubespray Repo:
git clone https://github.com/kubernetes-sigs/kubespray.gitInstalling Ansible on the Jump Server
Ansible is what powers Kubespray. So we will need to ensure that we have ansible installed on the jump-server.
On the jump server, in our /kube-setup directory, enter:
VENVDIR=kubespray-venv...then:
KUBESPRAYDIR=kubesprayYou may now need to install Python
apt update && apt install python3.11-venvWe can then setup a virtual environment using Python3's virtual environment module
python3 -m venv $VENVDIR
now run then following command to enter the new python virtual environment.
source $VENVDIR/bin/activate
Let's now move into the Kubespray directory we defined earlier:
cd $KUBESPRAYDIR
Then use pip install, to install ansible, along with all the other required dependancies
pip install -U -r requirements.txt
finally run...
cd .....to back out into the kube-setup directory.
Creating the Cluster Configuration ๐ง
Within the kube-setup directory, we can now create a folder to store our K8s cluster configurations
mkdir -p clusters/eu-centralHosts
Now we must create the hosts.yaml file ourselves.
all:
hosts:
kube-node-1:
ansible_host: 172.16.0.101
ip: 172.16.0.101
access_ip: 172.16.0.101
kube-node-2:
ansible_host: 172.16.0.102
ip: 172.16.0.102
access_ip: 172.16.0.102
kube-node-3:
ansible_host: 172.16.0.103
ip: 172.16.0.103
access_ip: 172.16.0.103
vars:
ansible_user: root
ansible_ssh_common_args: "-o StrictHostKeyChecking=no"
kube_network_plugin: cilium
children:
kube_control_plane:
hosts:
kube-node-1: {}
kube_node:
hosts:
kube-node-1: {}
kube-node-2: {}
kube-node-3: {}
etcd:
hosts:
kube-node-1: {}
k8s_cluster:
children:
kube_control_plane: {}
kube_node: {}./kube-setup/clusters/eu-central/hosts.yaml
Cluster Config
Now we can create a custom configuration file for our cluster.
nano clusters/eu-central/cluster-config.yamlcloud_provider: external
external_cloud_provider: hcloud
external_hcloud_cloud:
token_secret_name: hcloud-api-token
with_networks: true
service_account_name: hcloud-sa
hcloud_api_token: the-same-api-token-we-used-earlier
controller_image_tag: v1.16.0
kube_network_plugin: calico
network_id: kubernetes-node-network
unsafe_show_logs: true
# Calico in VXLAN mode (works well on Hetzner L2)
calico_ipip_mode: "Never"
calico_vxlan_mode: "Always"
calico_network_backend: "vxlan"
# Stick to IPv4 single-stack unless youโve planned dual-stack
enable_dual_stack_networks: falseclusters/eu-central/cluster-config.yaml
Explanation
- cloud_provider:
external- This shows that we are using kubespray on an external cloud provider.
- external_cloud_provider:
hcloud- This defines which cloud provider, and prompts kubespray to configure hcloud controller manager as part of the K8s setup.
- external_hcloud_cloud:
- token_secret_name:
- The secret name where the Hetzner cloud api-token will be stored.
- with_networks:
true- This will allow us to leverage the underlying Hetzner cloud private network for port traffic, this is not mandatory, but will relieve the load on our cluster.
- service_account_name:
- The service account that will be granted permissions to the kube-api enabling the hcloud-controller-manager to interact with kubernetes.
- It is not explained if this needs to be an existing thing, of if this account is generated at the point of generation. I believe it is the latter, and will be generated.
- hcloud_api_token:
- The actual project API token
- controller_image_tag:
- Ensure that the correct version manager is used, and should be cross referenced with the version of K8s that you wish to deploy.
- token_secret_name:
- kube_network_plugin:
- This plugin has been tested to work well with Hetzner cloud networks, the default
calico-networkone does not work well.
- This plugin has been tested to work well with Hetzner cloud networks, the default
- network_id:
- The value of the private network that will be used for port traffic.
Deploying K8s
Return to the kube-setup/kubespray directory, and run the following ansible playbook.
ansible-playbook -i ../clusters/eu-central/hosts.yaml -e @../clusters/eu-central/cluster-config.yaml --become --become-user=root cluster.yml
-iSpecifies the location of our inventory file-eSpecifies the location of a file containing extra variables to be used.--become --become-user=rootAllows for privilege escalation
This should take some time, but after it has completed, we should have our K8s cluster fully deployed over the nodes. It should also be up and running.
Install kubectl on the jump-server
Now we can install kubectl on the jump server.
Install kubectl
Enter the home directory
cd ~Download the latest stable version of kubectl:
curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"Give the file execute permissions
chmod +x kubectlMove the file to the users local bin file:
mv kubectl /usr/local/binRetrieve the kube config file and setup kubectl
Then we can copy the kube config file from one of our control plane nodes to the jump server.
mkdir -p /root/.kube && cd /root/.kubescp root@172.16.0.101:/etc/kubernetes/admin.conf /root/.kube/configLet's now make an edit to the file to change the IP address to our control plane node:
nano /root/.kube/config//...
server: https://172.16.0.101:6443
//...Test kubectl's connection to the cluster
kubectl get nodesYou should get the following back:
| NAME | STATUS | ROLES | AGE | VERSION |
| kube-node-1 | Ready | control-plane | 8m | v1.26.6 |
| kube-node-2 | Ready | <none> | 7m25s | v1.26.6 |
| kube-node-3 | Ready | <none> | 6m15s | v1.26.6 |
Verify HCloud-Controller-Manager Installation
We can ensure that HCloud-Controller-Manager has installed and is up and running, by running the following commands:
Ensure Pods are running:
kubectl -n kube-system get podsYou should see a pod prefixed with hcloud-cloud-controller-manager-... for each node of the cluster.
Check one of the Controller Manager Pods Logs
We can also check the logs of one of these controller manager pods, with the following command:
kubectl -n kube-system logs -f hcloud-cloud-controller-manager-abc123Look through these to ensure there are not any errors.
Deploying an Example App ๐พ
Defining the Service and Deployment Manifest
First off, let's go to the home folder on the jump-server:
cd ~Then lets create a simple service deployment yaml file
nano web-app-deploy-svc.ymlapiVersion: apps/v1
kind: Deployment
metadata:
name: web-app
labels:
app.kubernetes.io/name: web-app
name: web-app
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: web-app
template:
metadata:
labels:
app.kubernetes.io/name: web-app
spec:
containers:
- image: nginx
name: web-app
command:
- /bin/sh
- -c
- "echo 'welcome to my web app!' > /usr/share/nginx/html/index.html && nginx -g 'daemon off;'"
dnsConfig:
options:
- name: ndots
value: "2"
---
apiVersion: v1
kind: Service
metadata:
name: web-app
labels:
app.kubernetes.io/name: web-app
annotations:
load-balancer.hetzner.cloud/location: hel1
load-balancer.hetzner.cloud/disable-private-ingress: "true"
load-balancer.hetzner.cloud/use-private-ip: "true"
load-balancer.hetzner.cloud/name: "kubelb1"
spec:
selector:
app.kubernetes.io/name: web-app
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
type: LoadBalancer~/web-app-deploy-svc.yml
Notice how we have included important annotations within the Service metadata, that are important for our Hetzner load balancer setup, not using these settings correctly can result in an unhealthy load balancer:
load-balancer.hetzner.cloud/location: hel1- We set the location here to match the location where our servers are deployed.
load-balancer.hetzner.cloud/disable-private-ingress: "true"load-balancer.hetzner.cloud/use-private-ip: "true"- Both the settings above ensure that we use the private network and IP address.
load-balancer.hetzner.cloud/name: "kubelb1"- This can be useful if we have already provisioned a load balancer in the hetzner cloud console and wish to use it for multiple services. If a load balancer with this name does not already exist, it will be created. Keep in mind that deleting the service will also delete the load-balancer.
Applying the new manifest to the cluster
We can then run the following command to apply the new manifest to the cluster.
kubectl -n default apply -f web-app-deploy-svc.ymlAfter thats completed we can verify the new pods have been created with the following command.
kubectl -n default get svcWe can now see the newly created load balancer in the Hetzner cloud console.
We can now try to access that load balancers public IP with our local PC and we should get a working response.
Creating a Default Config Map for Hetzner-Cloud-Controller-Manager
It can be tedious to consistently remember to configure the load balancer annotations for every service that we provide. Also adding this to things like helm charts can become very tedious.
Instead we can create some default settings in a configuration map, so that we can always use certain annotations when deploying, without the need to explicitly defining them.
Create a directory for our new Config Map
mkdir -p ~/kube-setup/clusters/eu-central/apps/hcloud-cloud-controller-managercd ~/kube-setup/clusters/eu-central/apps/hcloud-cloud-controller-managerCreate the Config Map File
nano extra-env-cm.ymlapiVersion: v1
data:
HCLOUD_LOAD_BALANCERS_LOCATION: "hel1"
HCLOUD_LOAD_BALANCERS_DISABLE_PRIVATE_INGRESS: "true"
HCLOUD_LOAD_BALANCERS_USE_PRIVATE_IP: "true"
HCLOUD_LOAD_BALANCERS_ENABLED: ""
kind: ConfigMap
metadata:
name: hcloud-controller-manager-extra-env
namespace: kube-systemclusters/eu-central/apps/hcloud-cloud-controller-manager/extra-env-cm.yml
Applying the Config Map to our Cluster
We can now apply our new config map to our cluster:
kubectl apply -f extra-env-cm.ymlEdit the HCloud-Controller-Manager Daemonset
We now need to edit the HCloud-Controller-Manager daemonset so that it will use our newly applied config map.
kubectl -n kube-system edit ds hcloud-cloud-controller-managerAdd the following lines at the below and after (not within) the env: section:
env:
...
envFrom:
- configMapRef:
name: hcloud-controller-manager-extra-envVerify that the change has taken place.
Verify that a restart has occured.
We can ensure that the changes have taken place by ensure that the pods have just been restarted after the edit has taken place.
kubectl -n kube-system get pods | grep hcloudEnsure that the pods have returned have an uptime of a time that should be less than however long it was since you made the change to their daemonset.
Verify through description
We can also use kubectl to describe the daemonset, and ensure that the changes have taken effect.
kubectl -n kube-system describe ds hcloud-cloud-controller-managerFrom the returned response, we can ensure that the Environment Variables From: section shows the following: hcloud-controller-manager-extra-env ConfigMap Optional: false
Redeploy the web-app service - Without Explicit LB Annotations
Delete the existing service
We can now redeploy the web-app service without having to explicitly define the load-balancer annotations. First off lets delete the web app service.
kubectl -n default delete service web-appRedeploy the Web-App
Let's change the deployment script, and leave off the annotations
apiVersion: apps/v1
kind: Deployment
metadata:
name: web-app
labels:
app.kubernetes.io/name: web-app
name: web-app
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: web-app
template:
metadata:
labels:
app.kubernetes.io/name: web-app
spec:
containers:
- image: nginx
name: web-app
command:
- /bin/sh
- -c
- "echo 'welcome to my web app!' > /usr/share/nginx/html/index.html && nginx -g 'daemon off;'"
dnsConfig:
options:
- name: ndots
value: "2"
---
apiVersion: v1
kind: Service
metadata:
name: web-app
labels:
app.kubernetes.io/name: web-app
spec:
selector:
app.kubernetes.io/name: web-app
ports:
- name: http
port: 80
protocol: TCP
targetPort: 80
type: LoadBalancer~/web-app-deploy-svc.yml
And redeploy it:
kubectl -n default apply -f web-app-deploy-svc.ymlEnsure that the load balancer is still in place
kubectl -n default get svcYour should see that a Load Balancer has been created.
We can then curl the IP that is returned to test that we are getting traffic back off the public IP
curl shown.external.ip.addressContainer Storage Interface Driver For Hetzner Cloud ๐
In order to create persistent volumes for use within our Kubernetes cluster, we must make use of the Container Storage Interface Driver or CSID.
We need to do some setup to get this working.
Create a manifest folder
First off lets create a folder for the manifest to live on our jump server:
mkdir -p ~/kube-setup/clusters/eu-central/apps/csi-driver
cd ~/kube-setup/clusters/eu-central/apps/csi-driverCreate and Apply a secret manifest
We need to create a secret, that the CSID can use, that contains our hetzner cloud API token.
nano api-token-secret.ymlapiVersion: v1
kind: Secret
metadata:
name: hcloud
namespace: kube-system
stringData:
token: YOUR-HETZNER-CLOUD-PROJECT-API-TOKENclusters/eu-central/apps/csi-driver/api-token-secret.yml
Now that we have our secret, we can apply this to our cluster.
kubectl apply -f api-token-secret.ymlApply the CSID Manifest
kubectl apply -f https://raw.githubusercontent.com/hetznercloud/csi-driver/v2.3.2/deploy/kubernetes/hcloud-csi.ymlThis will install the driver onto our cluster.
Verify the CSID Installation
We can check that the CSID has installed correctly on our cluster by looking at our pods
kubectl -n kube-system get podsWe should a pod, for each of our nodes with the prefix: hcloud-csi-node-.... And a single hcloud-csi-controller-... pod.
Verify the presence of the HCloud Storage Class
kubectl get storageclassYou should see a single entry with the name hcloud-volumes (default).
Test it by applying a Service with an attached Volume
Create a new test manifest
Let's create a new manifest:
nano example-pvc-pod.ymlapiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: csi-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: hcloud-volumes #this is important
---
kind: Pod
apiVersion: v1
metadata:
name: my-csi-app
spec:
containers:
- name: my-frontend
image: busybox
volumeMounts:
- mountPath: "/data"
name: my-csi-volume
command: [ "sleep", "1000000" ]
volumes:
- name: my-csi-volume
persistentVolumeClaim:
claimName: csi-pvc~/example-pvc-pod.yml
This is a simple manifest that declares a PVC with ReadWriteOnce, and a Pod that utilizes it, whilst doing nothing but sleeping.
Apply the test manifest
kubectl apply -f example-pvc-pod.ymlVerify that the pod is running
kubectl get podsWe should see that my-csi-app is running.
Verify that the PVC has been created
kubectl get pvcYou should see a single entry come back with the name csi-pvc.
Verify that the drive has been mounted onto the pod.
First off we have to exec into the pod itself
kubectl exec -it my-csi-app -- shThis will kick us into a shell session within the my-csi-app pod.
Then we can run this command to see all of the mounted drives:
df -hWe should see there is a drive ~10Gb in size, mounted to /data
Verify that the volume has been created in the Hetzner Cloud Console
Finally we can see that the volume has been created in the Hetzner Cloud Console.
Install helm on the Jump Server ๐ค
cd ~curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3chmod 700 get_helm.sh./get_helm.shrm get_helm.shThat's it! ๐
You now have a three node K8s cluster, and a jump-box to configure them ๐