Go: Testing Kubernetes Applications with EnvTest

Introduction

When developing applications that interact with a Kubernetes cluster such as Kubernetes operators or controllers, ensuring the application logic behaves as expected through robust testing is essential. Traditional approaches include using a development cluster in a testing environment, a local standalone cluster like Minikube, or even leveraging TestContainers to spin up a local cluster on demand.

However, using a complete cluster requires a lot of resources, or it's not possible to run in a given CI/CD pipeline. On many occasions, we don't need a full cluster to test our application, especially if our application is just dealing with the declarative part of Kubernetes, by creating, updating, or deleting resources. For these scenarios, we can use controller-runtime EnvTest package to spin up a local Kubernetes API server and etcd instance.

In this post, I'll show you how to use EnvTest to test a simple Kubernetes operator. I'll also give you some practical examples by implementing tests for the kubectl kill namespace plugin I wrote in a previous post.

What is EnvTest?

EnvTest, a package from the controller-runtime project, streamlines integration testing for Kubernetes applications. It furnishes a local Kubernetes API server and an etcd instance, enabling comprehensive application testing without requiring the entire Kubernetes cluster infrastructure. This flexibility allows for swift testing in local environments or seamless integration into CI/CD pipelines, all while conserving resources.

However, it's important to note that EnvTest does not accommodate applications reliant on Kubernetes components like the controller-manager or kubelet. For instance, applications needing specific Pods to be scheduled and operational within a Kubernetes cluster are not suitable for this environment.

Setting up EnvTest

Most of the online guides rely on the EnvTest binaries to be already installed in the system. Some of them show you how to download them using a Makefile, while others don't even mention how to get them.

In this section, I will show you how to use the EnvTest package directly from your Go code, without relying on any external tools. Let's begin by creating a setupEnvTest function responsible for setting up the EnvTest environment:

func setupEnvTest() *envtest.Environment {
	// Download EnvTest binaries
	// Create a new instance of the EnvTest environment
}

The function is responsible for downloading if necessary the EnvTest binaries, and creating a new instance of the EnvTest environment for those binaries.

Let's focus on the first part, downloading the binaries.

func setupEnvTest() *envtest.Environment {
	envTestDir, err := store.DefaultStoreDir()
	if err != nil {
		panic(err)
	}
	envTest := &env.Env{
		FS:  afero.Afero{Fs: afero.NewOsFs()},
		Out: os.Stdout,
		Client: &remote.Client{
			Bucket: "kubebuilder-tools",
			Server: "storage.googleapis.com",
		},
		Platform: versions.PlatformItem{
			Platform: versions.Platform{
				OS:   runtime.GOOS,
				Arch: runtime.GOARCH,
			},
		},
		Version: versions.AnyVersion,
		Store:   store.NewAt(envTestDir),
	}
	envTest.CheckCoherence()
	workflows.Use{}.Do(envTest)
	// Create a new instance of the EnvTest environment
}

Let's dig through the code step by step:

• First, we get the default EnvTest directory using the store.DefaultStoreDir() function. This way, we can reuse any binary that might have already been downloaded by any other tests or external tools.
• Then, we create a new instance of the env.Env struct. This struct provides all the necessary information such as OS, architecture, version, and so on to download the binaries.
• Next, we validate and initialize the environment by using the envTest.CheckCoherence() function.
• Finally, we use the workflows.Use{}.Do(envTest) function to download the necessary binaries.

The next part of the function logic deals with setting up the EnvTest environment:

func setupEnvTest() *envtest.Environment {
	// Download EnvTest binaries
	versionDir := envTest.Platform.Platform.BaseName(*envTest.Version.AsConcrete())
	return &envtest.Environment{
		BinaryAssetsDirectory: filepath.Join(envTestDir, "k8s", versionDir),
	}
}

We use the envTest.Platform.Platform.BaseName(*envTest.Version.AsConcrete()) function to get the directory containing the binaries for the EnvTest version we downloaded. Then, we create a new instance of the envtest.Environment struct, passing the directory where the binaries are located.

Now that we have the setupEnvTest function ready, let's continue by seeing how to use it in our tests.

Using EnvTest

In this section, I'll show you how to use the setupEnvTest function we created in the previous section to test the kubectl kill namespace plugin I analyzed in a previous post. You can find the complete code for the tests in the killns_test.go file of the project.

Since I want to test the Namespace deletion logic, I'll leverage subtests to spin up a common EnvTest environment and then run each scenario. The following code snippet shows the setup and tear down sections of the TestKillNamespace function:

func TestKillNamespace(t *testing.T) {
	envTest := setupEnvTest()
	envTestConfig, err := envTest.Start()
	if err != nil {
		t.Errorf("Error starting test environment: %s", err)
		return
	}
	defer func() {
		if stopErr := envTest.Stop(); stopErr != nil {
			panic(stopErr)
		}
	}()
	// Test with no namespace
	// Test with existent namespace
	// Test with existent namespace with finalizer
}

Let's analyze the code step by step:

• First, we call the setupEnvTest function we created in the previous section to initialize the EnvTest environment.
• Then, we start the environment by calling the envTest.Start() function. This function returns a rest.Config struct that we can later use to initialize a Kubernetes client.
• Next, we check if there was any error while starting the environment. If there was, we fail the test and return.
• Finally, we use a defer statement to stop and tear down the environment once the test finishes. This way, we make sure that the environment is always stopped, even if the test fails.

Let's now analyze the With existent namespace with finalizer subtest scenario to see how at this point using EnvTest for Kubernetes API operations is equivalent to using a real Kubernetes cluster.

func TestKillNamespace(t *testing.T) {
	// Setup EnvTest environment
	t.Run("With existent namespace with finalizer", func(t *testing.T) {
		client, _ := kubernetes.NewForConfig(envTestConfig)
		client.CoreV1().Namespaces().Create(context.TODO(), &corev1.Namespace{
			ObjectMeta: metav1.ObjectMeta{
				Name: "finalizer",
			},
			Spec: corev1.NamespaceSpec{Finalizers: []corev1.FinalizerName{"kubernetes"}},
		}, metav1.CreateOptions{})
		KillNamespace(envTestConfig, "finalizer")
		_, err := client.CoreV1().Namespaces().Get(context.TODO(), "finalizer", metav1.GetOptions{})
		if err.Error() != "namespaces \"finalizer\" not found" {
			t.Errorf("Namespace should have been deleted, but it wasn't")
		}
	})
}

Let's go over the different parts of the code:

• First, we initialize a client-go Kubernetes Client by passing the rest.Config struct we got from the envTest.Start() function.
• Then, we create a new Namespace with a finalizer using the client (assemble/given).
• Next, we call the KillNamespace function we want to test (act/when).
• Finally, we check that the Namespace was deleted by trying to get it using the client (assert/then).

Conclusion

In this post, I showed you how to use the controller-runtime EnvTest package to test Kubernetes applications. I started by introducing EnvTest and explaining when it's suitable to use it. Then, I showed you how to set up the EnvTest environment by downloading the binaries directly from your Go code. Finally, I showed you how to use the EnvTest environment to test a Kubernetes application by implementing tests for the kubectl kill namespace plugin I wrote in a previous post.

You should now be able to use EnvTest to test your Kubernetes applications.

Find the complete source code for the examples I showed you in this post on GitHub.