How to create a kubectl plugin with client-go to kill a Kubernetes Namespace

Introduction

As Kubernetes continues to dominate modern infrastructure management, tools like kubectl have become essential for interacting with Kubernetes clusters. Serving as a Swiss army knife, kubectl empowers users to execute diverse operations on their clusters. Moreover, its extensibility enables the creation of custom plugins, thereby enhancing its functionality.

In this post, I'll show you how to create a kubectl plugin using Go and client-go. The plugin will allow you to kill or delete a Kubernetes Namespace that's stuck in the Terminating state.

Let's start by understanding the problem the kubectl plugin will solve.

Namespace stuck in Terminating state

A Kubernetes Namespace is a mechanism to isolate resources or Kubernetes objects within a cluster. A Namespace will typically host Deployments, Pods, Services, Ingresses, and other Kubernetes Namespaced objects. When you delete a Kubernetes Namespace, the API server will mark it as Terminating and will start a background process to delete all the resources contained in the Namespace. Eventually, the cluster will delete the Namespace.

However, there are times when the Namespace gets stuck in the Terminating state. This might happen when a Kubernetes API extension is not available and the API server is unable to delete those resources. In this case, the Namespace will contain a finalizer that prevents the Namespace from being deleted. The following code snippet contains a Namespace with pending finalizers:

apiVersion: v1
kind: Namespace
metadata:
  name: my-namespace
spec:
   finalizers:
    - kubernetes
status:
  conditions:
  - lastTransitionTime: "2023-11-16T10:31:38Z"
    message: All content-preserving finalizers finished
    reason: ContentHasNoFinalizers
    status: "False"
    type: NamespaceFinalizersRemaining
  phase: Terminating

To be able to force the deletion of the Namespace, we need to remove the finalizers. This can be achieved by using the /finalize subresource of the Namespace API.

You can find information online on how to fix this from the command line. My colleague, Jens Reimann, even has a popular script that automates the required steps.

Since this is a recurring issue, I think that it's a good way to showcase how to create a kubectl plugin.

Creating the kubectl plugin

A kubectl plugin in essence is just a binary with the name kubectl-<plugin-name> that is available in the PATH. To create our plugin we'll need a Go project that builds a binary with the name kubectl-kill-ns. This way, we'll be able to invoke the plugin with kubectl kill ns my-stuck-namespace.

Let's start by creating a new Go module for our plugin:

mkdir kubectl-kill-ns
cd kubectl-kill-ns
go mod init github.com/marcnuri-demo/kubectl-kill-ns

If the process goes smoothly, you should have a new go.mod file with the following content:

module github.com/marcnuri-demo/kubectl-kill-ns

go 1.21.2

We'll also need to add the k8s.io/client-go dependency to our project. To do so, we'll run the following command:

go get k8s.io/client-go@v0.28.3

Since we need the plugin to have a specific name, we'll need to run the Go build command with the -o flag to specify the output file name. To make this easier, we'll add a Makefile to the project with the following content:

.PHONY: build
build:
	go build -ldflags "-s -w" -o kubectl-kill-ns ./cmd/main.go

Implementing the kill Namespace logic

We'll implement the logic to kill the Namespace in the killns package. Let's create a new killns.go file in the internal/killns directory.

The starting point for client-go is a Kubernetes cluster configuration. Since the plugin is intended to be used by a user operating from outside the cluster, we'll use the following code to automatically detect and load the user's .kube/config configuration:

package killns

import (
	"k8s.io/client-go/rest"
	"k8s.io/client-go/tools/clientcmd"
	"k8s.io/client-go/util/homedir"
)

func LoadKubeConfig() (*rest.Config, error) {
	home := homedir.HomeDir()
	kubeConfig := filepath.Join(home, ".kube", "config")
	return clientcmd.BuildConfigFromFlags("", kubeConfig)
}

The previous snippet starts by getting the user's home directory for the current system platform. Next, it builds the path to the user's .kube/config file and uses it to load the Kubernetes configuration. The function returns the loaded configuration or an error if the configuration could not be loaded.

The next part of the killns.go file contains the logic to kill the Namespace. The following code snippet contains the relevant parts for the KillNamespace function

func KillNamespace(kubeConfig *rest.Config, namespace string) {
	//... config validations
	client, err := kubernetes.NewForConfig(kubeConfig)
	//... client validation
	if errDelete := client.CoreV1().Namespaces().Delete(context.TODO(), namespace, metav1.DeleteOptions{}); errDelete != nil {
		panic(errDelete)
	}
	ns, errGet := client.CoreV1().Namespaces().Get(context.TODO(), namespace, metav1.GetOptions{})
	//... Namespace and finalizer checks
	ns.Spec.Finalizers = []corev1.FinalizerName{}
	_, errUpdate := client.CoreV1().Namespaces().Finalize(context.TODO(), ns, metav1.UpdateOptions{})
	//...
}

The logic for the function is quite simple:

• We create a Kubernetes client using the configuration loaded from the provided config.
• Next, we try to delete the Namespace.
• If the Namespace still exists after deletion, we check the Namespace for finalizers.
• Finally, if the Namespace does contain finalizers, we remove them and update them using the /finalize subresource.

Implementing the plugin entry point

The plugin entry point is the code that will be executed when the plugin is invoked. We'll create a new main.go file in the cmd directory with the following content:

package main

// imports

func main() {
	defer func() {
		if r := recover(); r != nil {
			fmt.Println("Error:", r)
			os.Exit(1)
		}
	}()
	kubeConfig, err := killns.LoadKubeConfig()
	if err != nil {
		fmt.Println(".kube/config not found", err)
		os.Exit(1)
	}
	flag.Parse()
	killns.KillNamespace(kubeConfig, flag.Arg(0))
}

The function is a simple command line application:

• It starts by deferring a function that will recover from any panic, print the error message, and exit the application.
• Next, it loads the Kubernetes configuration from the user's .kube/config file in the home directory.
• Finally, it parses the command line arguments and invokes the KillNamespace function with loaded configuration and the Namespace name retrieved from the first argument.

Building the plugin

We'll use the Makefile we created earlier to build the plugin. To do so, we'll run the following command:

make build

A new kubectl-kill-ns binary should be created in the project's root directory.

Testing the plugin

We can now test the plugin by invoking it directly from the command line:

./kubectl-kill-ns my-stuck-namespace

Since this is a kubectl plugin we can also test it by invoking it using kubectl:

PATH=$PATH:$(pwd) kubectl kill ns my-stuck-namespace

If we want to make the plugin available permanently, we can copy it to a directory in the PATH:

Conclusion

In this post, we've seen how to create a kubectl plugin using Go and client-go. The plugin allows you to kill a Kubernetes Namespace that's stuck in the Terminating state.

Custom kubectl plugins present limitless opportunities to extend Kubernetes functionality. By leveraging the techniques outlined in this article, you can create tailored solutions for diverse Kubernetes challenges.

Find the complete source code for this plugin on GitHub.