Welcome to my technical notes space

This site is a collection of personal notes gathered from my experiments and projects, mainly focused on Kubernetes and its ecosystem.

What you'll find here isn't polished tutorials or official documentation. It's snippets of config, handy commands, real-world troubleshooting, and sometimes solutions to specific issues I've run into in production environments. Some notes might be incomplete or lack context — that's expected, since they were originally written for my own use.

I decided to publish them because, in my experience, you often run into weird edge cases that aren’t documented anywhere. If these notes can save someone a few hours debugging an obscure issue or figuring out a tricky setup, then it's worth it.

Happy reading — and good luck hunting bugs.

Kind

kind-quick-start

Kubernetes

etcd

Database space exceeded

failed to update node lease, error: etcdserver: mvcc: database space exceeded

The Etcd cluster has gone into a limited operation maintenance mode, meaning that it will only accept key reads and deletes.

Possible Solution

History compaction needs to occur :

$ export ETCDCTL_API=3
$ etcdctl alarm list
$ etcdctl endpoint status --write-out="json" | egrep -o '"revision":[0-9]*' | egrep -o '[0-9]*'
143862581
$ etcdctl compact 143862581
$ etcdctl defrag
$ etcdctl alarm disarm

This operation should be done on each etcd cluster node.

Api Crashed and etcd slowness

Most of time this issue is due to a busy etcd database due to too many objects (like events) or network slowness. In this case you can see on grafana a lot of etcd leader changes. Too many Events/jobs can be caused by a cronjob running in loop pods which are looping in error also.

Possible Solution

Give Disk priority to etcd:

An etcd cluster is very sensitive to disk latencies. Since etcd must persist proposals to its log, disk activity from other processes may cause long fsync latencies. The upshot is etcd may miss heartbeats, causing request timeouts and temporary leader loss. An etcd server can sometimes stably run alongside these processes when given a high disk priority.

On Linux, etcd’s disk priority can be configured with ionice:

# best effort, highest priority
sudo ionice -c2 -n0 -p `pgrep etcd

Count the number of events in etcd database.

ETCDCTL_API=3 etcdctl get /registry --prefix --keys-only | grep /registry/events | cut -d'/' -f4 | uniq -c| sort -nr

Identify the namespace wich is causing to many events and try to purge them with kubectl

kubectl delete events --all -n <NAMESPACE>

If the api crashed it could be complicated to clean events with kubectl. In this case you can clean events directly in etcd database.

ETCDCTL_API=3 etcdctl del /registry/events/<NAMESPACE> --prefix

Compact ETCD database on each master nodes to free space.

$ export ETCDCTL_API=3
$ etcdctl endpoint status --write-out="json" | egrep -o '"revision":[0-9]*' | egrep -o '[0-9]*'
143862581
$ etcdctl compact 143862581
$ etcdctl defrag

Give time to etcd to resync all nodes.

Finally, check job and cronjob wich are creating to many events and stop them (for cronjob) and delete jobs.

kube-tuning

Debug Kubernetes

Test DNS

Test Cluster Dns using busybox pod:

kubectl exec -it busybox -n <NAMESPACE> -- nslookup kubernetes.default

Cluster-Api

capi-quick-start

The goal of this page is to provide quick commands to get started with Cluster API in under 5 minutes. For more detailed information, please refer to the official Cluster API documentation at: https://cluster-api.sigs.k8s.io/

Prerequisites:

Install Kind

Install Kind following this Link or:

[ $(uname -m) = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.27.0/kind-linux-amd64
chmod +x ./kind
sudo mv ./kind /usr/local/bin/kind

Install Kubectl

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

Install Cluster-Api

Create a Kind cluster with the kind config kind-cluster.yaml in this repo.

kind create cluster

Install Clusterctl

Clusterctl is the client to deploy cluster with capi.

curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.9.5/clusterctl-linux-amd64 -o clusterctl
sudo install -o root -g root -m 0755 clusterctl /usr/local/bin/clusterctl
clusterctl version

Install CAPI and CAPO ( ClusterAPi for Opentack)

First export this variable to enable cluster-ressource-set feature:

export EXP_CLUSTER_RESOURCE_SET=true

Now we will install clusterAPI (capi) and clusterApi Openstack Controller (capo) using clusterctl in our kind.

kubectl apply -f https://github.com/k-orc/openstack-resource-controller/releases/latest/download/install.yaml
clusterctl init --infrastructure openstack

Now your kind should look like:

ubuntu@jeff:~$ kubectl get pods -A | grep -v kube-system
NAMESPACE                           NAME                                                             READY   STATUS    RESTARTS        AGE
capi-kubeadm-bootstrap-system       capi-kubeadm-bootstrap-controller-manager-66bb86b8b8-d6jtb       1/1     Running   3 (20h ago)     5d17h
capi-kubeadm-control-plane-system   capi-kubeadm-control-plane-controller-manager-7bd59d5f69-bb69p   1/1     Running   2 (2d12h ago)   5d17h
capi-system                         capi-controller-manager-578674dd86-xhk7r                         1/1     Running   3 (20h ago)     5d17h
capo-system                         capo-controller-manager-79f47999df-w5p8k                         1/1     Running   3 (20h ago)     4d20h
cert-manager                        cert-manager-94d5c9976-pjw67                                     1/1     Running   2 (2d12h ago)   5d17h
cert-manager                        cert-manager-cainjector-6c49b5cdcc-bshqd                         1/1     Running   1 (2d12h ago)   5d17h
cert-manager                        cert-manager-webhook-595556d86b-zxm82                            1/1     Running   1 (2d12h ago)   5d17h
local-path-storage                  local-path-provisioner-7dc846544d-4tzbs                          1/1     Running   1 (2d12h ago)   5d18h
orc-system                          orc-controller-manager-df6c48588-mjdz5                           1/1     Running   3 (20h ago)     5d17h

Create your first Cluster CAPI

Manual step: ClusterIP

For the moment we don't have LBAAS on Openstack for Api-Servers Workaround: Create a port manualy on prod network on openstack console it will be your CLUSTER_API_IP

Prepare env vars for your cluster.

Manual step: secret cloud.yaml

Based on the file cloud.yaml , create your encoded secret cloud.yaml in base64.

export OPENSTACK_CLOUD_YAML_B64=$(cat cloud.yaml | base64)

# apply the secret in your kind (run once )
envsubst < secret.yaml | kubectl apply -f -

based on env_cos_mutu file create vars file for your cluster and source it:

source env_mutu_svc

Create Calico CRS deployment for your futur clusters:

# create crs
envsubst <  crs/crs-calico.yaml | kubectl apply -f -

Now create your first cluster:

Create cluster command:

# create env_mutu cluster
envsubst <  cluster-template-without-lb.yaml | kubectl apply -f -

When master are available, connect on SSH on one and go on /var/log/cloud-init-output.log. Copy/Past the configuration to configure the kubeconfig file and be able to use kubectl on this master.

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

or

export KUBECONFIG=/etc/kubernetes/admin.conf

** Check your cluster status using clusterctl **:

clusterctl describe cluster dev

NAME                                                    READY  SEVERITY  REASON  SINCE  MESSAGE
Cluster/dev                                             True                     18h
├─ClusterInfrastructure - OpenStackCluster/dev
├─ControlPlane - KubeadmControlPlane/dev-control-plane  True                     18h
│ └─3 Machines...                                       True                     18h    See dev-control-plane-5djm7, dev-control-plane-tgs4l, ...
└─Workers
  └─MachineDeployment/dev-md-0                          True                     18h
    └─6 Machines...                                     True                     18h    See dev-md-0-9bh9b-89mq9, dev-md-0-9bh9b-95k5n, ...

Delete cluster command:

# create env_mutu cluster
envsubst <  cluster-template-kubevip.yaml | kubectl delete -f -

Clean Capi in your Kind:

kubectl delete cluster mycluster -n namespace

clusterctl delete  --core cluster-api -b kubeadm -c kubeadm -i openstack

Upgrade Components:**

clusterctl upgrade plan

clusterctl upgrade apply --contract v1beta1

Notes :

creation d'un autre cluster dans kind

clusterctl generate cluster capi-quickstart --flavor development
--kubernetes-version v1.32.0
--control-plane-machine-count=1
--worker-machine-count=1
--infrastructure docker \

capi-quickstart.yaml

Migrate a legacy K8s Kubeadm Cluster to a Capi K8s kubadm Cluster

Currently, this procedure is in an experimental stage and should be thoroughly tested before being used in a production environment. It is only compatible with an external ETCD and an external LBAAS.

It is designed create cluster.x-k8s.io/secret bundle to migrate a cluster created with kubeadm to a kubeadm-based cluster managed by Cluster API. At this stage, the script has been developed specifically for the Cluster API Provider OpenStack (CAPO ).

My goal is to improve the process to transition from an external ETCD to a local ETCD on the control-plane nodes, and also to migrate from a local ETCD on legacy control-planes to a local ETCD on CAPI control-planes.

The key to this, based on my analysis, would be to force CAPI to add the first control-plane node using a kubeadm join instead of a kubeadm init.

In the case of an external ETCD, this works because the secrets and ETCD are already initialized. The kubeadm init command does not pose any issues, as the kubelet simply joins an already existing API endpoint.

Feel free to share any suggestions or ideas for improvements or future developments.

Migration Process Overview

The procedure is based on having a hybrid legacy/CAPI cluster during the migration.

It is carried out in five main steps:

  1. Retrieving the necessary secrets and configurations from the existing cluster.
  2. Preparing the Cluster API (CAPI) configuration.
  3. Importing the secrets into CAPI.
  4. Create the CAPI control-plane and CAPI worker nodes on the existing cluster.
  5. Removing the nodes the old cluster nodes.

Prerequisites

  • have a CAPI cluster CAPI Controller should have acces to the api url of the cluster to manage. (https://api.mylegacy_cluster.kubeadm)

1 - Retrieving the necessary secrets and configurations.

First, run the prepare_secrets.sh script on a control plane node, passing the name of the cluster you want to migrate as an argument. This name should match the cluster_name defined in CAPI. You can find the script Here.

The script will generate a file named ${CLUSTER_NAME}-secret-bundle.yaml.

./prepare_secrets.sh ${CLUSTER_NAME}

and get the file: ${CLUSTER_NAME}-secret-bundle.yaml

2 - Preparing the Cluster API (CAPI) configuration.

Manual step: secret cloud.yaml

Based on the file cloud.yaml , create your encoded secret cloud.yaml in base64.

export OPENSTACK_CLOUD_YAML_B64=$(cat cloud.yaml | base64)

# apply the secret in your Cluster-api cluster (run once )
envsubst < secret.yaml | kubectl apply -f -

based on env_example file create vars file for your cluster and source it:

source env_example

Now, based on the 'cluster-template-migration.yaml', generate your cluster configuration, and pay close attention to the following parameters:

For the example, I hardcoded the parameters directly in the code, but it's recommended to pass these values as cluster environment variables instead.

  • External etcd endpoints in KubeadmControlPlane section :
    ...
    clusterConfiguration:
      etcd:
        external:
          caFile: /etc/kubernetes/pki/etcd/ca.crt
          certFile: /etc/kubernetes/pki/apiserver-etcd-client.crt
          keyFile: /etc/kubernetes/pki/apiserver-etcd-client.key
          endpoints:
          - https://192.168.10.10:2379
          - https://192.168.10.11:2379
          - https://192.168.10.12:2379
    ...
  • External Api endpoints in OpenStackCluster section if you use CAPO:
    ...
    controlPlaneEndpoint:
    host: api.mydevcluster.com
    port: 443
    ...
  • Add a secutitygroup rule in OpenStackCluster section if you use CAPO to allow traffic beween old and new clusters:
  ...
  managedSecurityGroups:
    allNodesSecurityGroupRules:
    - direction: ingress
      etherType: IPv4
      name: Allow old secutirygroup
      description: "Allow all between old and new control plane and workers"
      remoteGroupID: "old-secutity-group-id"
  ...

3 - Importing the secrets into CAPI.

Apply your ${CLUSTER_NAME}-secret-bundle.yaml into you CAPI Controller Cluster:

kubectl apply -f ${CLUSTER_NAME}-secret-bundle.yaml

4 - Create the CAPI control-plane and CAPI worker nodes

# create capi  ${CLUSTER_NAME} cluster
envsubst <  cluster-template-migration.yaml | kubectl apply -f

Since the etcd database is shared between the old and new cluster, and the PKI secrets (such as TLS certificates and private keys) are identical, creating this new Kubernetes cluster will actually result in the addition of new control plane and CAPI worker nodes to the existing cluster, rather than forming a separate cluster.

Get your cluster state:

# import secret bundle
kubectl apply -f ${CLUSTER_NAME}-secret-bundle.yaml -n $NAMESPACE

# Migrate  ${CLUSTER_NAME} cluster
envsubst <  cluster-template-migration.yaml | kubectl apply -n $NAMESPACE -f

5 - Remove the old cluster nodes

Synology

Enable L2TP/IPsec Logs as debug:

vim /var/packages/VPNCenter/etc/l2tp/ipsec.conf

add or uncomment this parameters:

config setup
...
    plutodebug=all
    plutostderrlog=/var/log/pluto.log

Allow UDP 500, 1701 et 4500 on the NAS, don't NAT/transfert 1701 on router.