Working with multiple kubeconfig files

Working with multiple kubernetes clusters can be a hassle, starting with the kubeconfig file, where you have to add all your clusters configs into that file, and update it manually whenever you want to add a new cluster. That's what I thought is the only way, but then I found another way, much simpler and provides a better developer experience.

Surprisingly, there is no extra tools needed, its just a feature of kubectl, as it reads an environment variable tells it where to look for kubeconfig files.

Prepare kubeconfig files

create a new directory in my home folder, will call it clusters:

mkdir clusters

then copy our clusters kubeconfig files, this will copy kubeconfig1 and kubeconfig2 to the new directory we just created.

cp kubeconfig1 kubeconfig2 ~/clusters

For a better example we will assume that kubeconfig1 is a sandbox cluster, kubeconfig2 is a kubeconfig file for staging.

cd ~/clusters
mv kubeconfig1 sandbox.yml
mv kubeconfig2 staging.yml

result:

clusters
├── exp.yml
├── test.yml
├── dev.yml
├── production.yml
├── sandbox.yml
└── staging.yml

now we are ready to tell kubectl to use these files

Configure shell

When kubectl command is executed, it checks where to find kubeconfig files, if nothing is specified, it falls back to the default file ~/.kube/config

We can tell kubectl to use the files we have prepared, by setting the environment variable KUBECONFIG with a list (separated by a colon ":") of files, they can be anywhere, they don't need to be in the same directory even.

Open your shell and set the variable

export KUBECONFIG="~/clusters/sandbox.yml:~/clusters/staging.yml"

Now use kubectl to get list of contexts, I will use kubectx

note: the list of contexts will depend on the names used in the kubeconfig files (not the file names)

$ kubectx

exp
k8s.test.com
k8s.dev.eg
production.company.org
sandbox
staging

Make sure to set that variable every time your shell starts, you can do that by setting it in .bash_profile or .zshrc, depends on your shell.

// .bash_profile
export ...

export KUBECONFIG="~/clusters/sandbox.yml:~/clusters/staging.yml"

export ...

Tada ! thats it, you are ready to get going and add as many files as you want.

Automate detection of kubeconfig files

Remember that directory we created in the first step? We can write a tiny script that loops over the files in that directory and set the KUBECONFIG variable.

# default kubeconfig
FILES="$HOME/.kube/config"

# create clusters directory if its not created
CLUSTERS_DIR="$HOME/clusters"
echo "creating clusters dir: $CLUSTERS_DIR"
mkdir -p "$CLUSTERS_DIR"
test CLUSTERS_DIR

for cluster in `find $CLUSTERS_DIR -type f -name "*.yml"`
do
  echo "found cluster: $cluster"
  FILES="$cluster:$FILES"
done

export KUBECONFIG=$FILES

Will not go through the code details in here, but you can copy that script into a file and source that files in your .bash_profile.

That's all folks!