Kubernetes support

Kubernetes is one of the hottest management platforms for containerized applications these days. Kubernetes lets you deploy, scale and manage your containers on the platform so you get features like auto-scaling, self-healing, service discovery and load balancing. Citrus provides interaction with the Kubernetes REST API so you can access the Kubernetes platform and its resources within a Citrus test case.

Note The Kubernetes test components in Citrus are kept in a separate Maven module. If not already done so you have to include the module as Maven dependency to your project

<dependency>
  <groupId>com.consol.citrus</groupId>
  <artifactId>citrus-kubernetes</artifactId>
  <version>2.7.1</version>
</dependency>

Citrus provides a "citrus-kubernetes" configuration namespace and schema definition for Kubernetes related components and actions. Include this namespace into your Spring configuration in order to use the Citrus Kubernetes configuration elements. The namespace URI and schema location are added to the Spring configuration XML file as follows.

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:citrus-k8s="http://www.citrusframework.org/schema/kubernetes/config"
       xsi:schemaLocation="
       http://www.springframework.org/schema/beans 
       http://www.springframework.org/schema/beans/spring-beans.xsd
       http://www.citrusframework.org/schema/kubernetes/config
       http://www.citrusframework.org/schema/kubernetes/config/citrus-kubernetes-config.xsd">

    [...]

</beans>

After that you are able to use customized Citrus XML elements in order to define the Spring beans.

Kubernetes client

Citrus operates with the Kubernetes remote REST API in order to interact with the Kubernetes platform. The Kubernetes client is defined as Spring bean component in the configuration as follows:

<citrus-k8s:client id="myK8sClient"/>

The Kubernetes client is based on the Fabric8 Java Kubernetes client implementation. Following from that the component can be configured in various ways. By default the client reads the system properties as well as environment variables for default Kubernetes settings such as:

• kubernetes.master / KUBERNETES_MASTER
• kubernetes.api.version / KUBERNETES_API_VERSION
• kubernetes.trust.certificates / KUBERNETES_TRUST_CERTIFICATES

If you set these properties in your environment the client component will automatically pick up the configuration settings. Also when using kubectl command line locally the client may automatically use the stored user authentication settings from there. For a complete list of settings and explanation of those please refer to the Fabric8 client documentation.

In case you need to set the client configuration explicitly on your environment you can also use explicit settings on the Kubernetes client component:

<citrus-k8s:client id="myK8sClient"
              url="http://localhost:8843"
              version="v1"
              username="user"
              password="s!cr!t"
              namespace="user_namespace"
              message-converter="messageConverter"
              object-mapper="objectMapper"/>

Now Citrus is able to access the Kubernetes remote API for executing commands such as list-pods, watch-services and so on. Citrus provides a set of actions that perform a Kubernetes command via REST. The results usually get validated in the Citrus test as usual.

Based on that we can execute several Kubernetes commands in a test case and validate the Json results:

Citrus supports the following Kubernetes API commands with respective test actions:

• k8s:info
• k8s:list-pods
• k8s:get-pod
• k8s:delete-pod
• k8s:list-services
• k8s:get-service
• k8s:delete-service
• k8s:list-namespaces
• k8s:list-events
• k8s:list-endpoints
• k8s:list-nodes
• k8s:list-replication-controllers
• k8s:watch-pods
• k8s:watch-services
• k8s:watch-namespaces
• k8s:watch-nodes
• k8s:watch-replication-controllers

We will discuss these commands in detail later on in this chapter. For now lets have a closer look on how to use the commands inside of a Citrus test.

Kubernetes commands in XML

We have several Citrus test actions each representing a Kubernetes command. These actions can be part of a test case where you can manage Kubernetes pods inside the test. As a prerequisite we have to enable the Kubernetes specific test actions in our XML test as follows:

<beans xmlns="http://www.springframework.org/schema/beans"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns:k8s="http://www.citrusframework.org/schema/kubernetes/testcase"
        xsi:schemaLocation="
        http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.citrusframework.org/schema/kubernetes/testcase
        http://www.citrusframework.org/schema/kubernetes/testcase/citrus-kubernetes-testcase.xsd">

    [...]

</beans>

We added a special kubernetes namespace with prefix k8s: so now we can start to add Kubernetes test actions to the test case:

XML DSL

<testcase name="KubernetesCommandIT">
    <actions>
      <k8s:info client="myK8sClient">
        <k8s:validate>
          <k8s:result>{
            "result": {
              "clientVersion": "1.4.27",
              "apiVersion": "v1",
              "kind":"Info",
              "masterUrl": "${masterUrl}",
              "namespace": "test"
            }
          }</k8s:result>
        </k8s:validate>      
      </k8s:info>

      <k8s:list-pods>
        <k8s:validate>
          <k8s:result>{
            "result": {
              "apiVersion":"v1",
              "kind":"PodList",
              "metadata":"@ignore@",
              "items":[]
            }
          }</k8s:result>
          <k8s:element path="$.result.items.size()" value="0"/>
        </k8s:validate>        
      </k8s:list-pods>
    </actions>
</testcase>

In this very simple example we first ping the Kubernetes REST API to make sure we have connectivity up and running. The info command connects the REST API and returns a list of status information of the Kubernetes client. After that we get the list of available Kubernetes pods. As a tester we might be interested in validating the command results. So wen can specify an optional k8s:result which is usually in Json format. With that we can apply the full Citrus Json validation power to the Kubernetes results. As usual we can use test variables here and ignore some values explicitly such as the metadata value. Also JsonPath expression validation and Json test message validation features in Citrus come in here to validate the results.

Kubernetes commands in Java

Up to now we have only used the Citrus XML DSL. Of course all Kubernetes commands are also available in Java DSL as the next example shows.

Java DSL

@CitrusTest
public void kubernetesTest() {
    kubernetes().info()
                .validate(new CommandResultCallback<InfoResult>() {
                    @Override
                    public void doWithCommandResult(InfoResult info, TestContext context) {
                        Assert.assertEquals(info.getApiVersion(), "v1");
                    }
            });

    kubernetes().pods()
                .list()
                .withoutLabel("running")
                .label("app", "myApp");
}

The Java DSL Kubernetes commands provide an optional CommandResultCallback that is automatically called with the unmarshalled command result object. In the example above the InfoResult model object is passed as argument to the callback. So the tester can access the command result and validate its properties with assertions.

Java 8 Lambda expressions add some syntactical sugar to the command result validation:

Java DSL

@CitrusTest
public void kubernetesTest() {
    kubernetes().info()
                .validate((info, context) -> Assert.assertEquals(info.getApiVersion(), "v1"));

    kubernetes().pods()
                .list()
                .withoutLabel("running")
                .label("app", "myApp");
}

By default Citrus tries to find a Kubernetes client component within the Citrus Spring application context. If not present Citrus will instantiate a default kubernetes client with all default settings. You can also explicitly set the kubernetes client instance when using the Java DSL Kubernetes command actions:

Java DSL

@Autowired
private KubernetesClient kubernetesClient;

@CitrusTest
public void kubernetesTest() {
    kubernetes().client(kubernetesClient)
                .info()
                .validate((info, context) -> Assert.assertEquals(info.getApiVersion(), "v1"));

    kubernetes().client(kubernetesClient)
                .pods()
                .list()
                .withoutLabel("running")
                .label("app", "myApp");
}

Info command

The info command just gets the client connection settings and provides them as a Json result to the action.

XML DSL

<k8s:info client="myK8sClient">
  <k8s:validate>
    <k8s:result>{
      "result": {
        "clientVersion": "1.4.27",
        "apiVersion": "v1",
        "kind":"Info",
        "masterUrl": "${masterUrl}",
        "namespace": "test"
      }
    }</k8s:result>
  </k8s:validate>      
</k8s:info>

Java DSL

@CitrusTest
public void infoTest() {
    kubernetes().info()
                .validate((info, context) -> Assert.assertEquals(info.getApiVersion(), "v1"));
}

List resources

We can list Kubernetes resources such as pods, services, endpoints and replication controllers. The list can be filtered by several properties such as

• label
• namespace

The test action is able to define respective filters to the list so we get only pods the match the given attributes:

XML DSL

<k8s:list-pods label="app=todo">
    <k8s:validate>
      <k8s:result>{
        "result": {
          "apiVersion":"${apiVersion}",
          "kind":"PodList",
          "metadata":"@ignore@",
          "items":"@ignore@"
        }
      }</k8s:result>
      <k8s:element path="$.result.items.size()" value="1"/>
      <k8s:element path="$..status.phase" value="Running"/>
    </k8s:validate>
</k8s:list-pods>

Java DSL

@CitrusTest
public void listPodsTest() {
    kubernetes()
        .client(k8sClient)
        .pods()
        .list()
        .label("app=todo")
        .validate("$..status.phase", "Running")
        .validate((pods, context) -> {
            Assert.assertFalse(CollectionUtils.isEmpty(pods.getResult().getItems()));
        });
}

As you can see we are able to give the pod label that is searched for in list of all pods. The list returned is validated either by giving an expected Json message or by adding JsonPath expressions with expected values to check.

In Java DSL we can add a validation result callback that is provided with the unmarshalled result object for validation. Besides label filtering we can also specify the namespace and the pod name to search for.

You can also define multiple labels as comma delimited list:

<k8s:list-services label="stage!=test,provider=fabric8" namespace="default"/>

As you can see we have combined to label filters stage!=test and provider=fabric8 on pods in namespace default. The first label filter is negated so the label stage should not be test here.

List nodes and namespaces

Nodes and namespaces are special resources that are not filtered by their namespace as they are more global resources. The rest is pretty similar to listing pods or services. We can add filteres such as name and label.

XML DSL

<k8s:list-namespaces label="provider=citrus">
    <k8s:validate>
      <k8s:element path="$.result.items.size()" value="1"/>
    </k8s:validate>
</k8s:list-namespaces>

Java DSL

@CitrusTest
public void listPodsTest() {
    kubernetes()
        .client(k8sClient)
        .namespaces()
        .list()
        .label("provider=citrus")
        .validate((pods, context) -> {
            Assert.assertFalse(CollectionUtils.isEmpty(pods.getResult().getItems()));
        });
}

Get resources

We can get a very special Kubernetes resource such as a pod or service for detailed validation of that resource. We need to specify a resource name in order to select the resource from list of available resources in Kubernetes.

XML DSL

<k8s:get-pod name="citrus_pod">
    <k8s:validate>
      <k8s:result>{
      "result": {
        "apiVersion":"${apiVersion}",
        "kind":"Pod",
        "metadata": {
            "annotations":"@ignore@",
            "creationTimestamp":"@ignore@",
            "finalizers":[],
            "generateName":"@startsWith('hello-minikube-')@",
            "labels":{
              "pod-template-hash":"@ignore@",
              "run":"hello-minikube"
            },
            "name":"${podName}",
            "namespace":"default",
            "ownerReferences":"@ignore@",
            "resourceVersion":"@ignore@",
            "selfLink":"/api/${apiVersion}/namespaces/default/pods/${podName}",
            "uid":"@ignore@"
        },
        "spec": {
          "containers": [{
            "args":[],
            "command":[],
            "env":[],
            "image":"gcr.io/google_containers/echoserver:1.4",
            "imagePullPolicy":"IfNotPresent",
            "name":"hello-minikube",
            "ports":[{
              "containerPort":8080,
              "protocol":"TCP"
            }],
            "resources":{},
            "terminationMessagePath":"/dev/termination-log",
            "volumeMounts":"@ignore@"
          }],
          "dnsPolicy":"ClusterFirst",
          "imagePullSecrets":"@ignore@",
          "nodeName":"minikube",
          "restartPolicy":"Always",
          "securityContext":"@ignore@",
          "serviceAccount":"default",
          "serviceAccountName":"default",
          "terminationGracePeriodSeconds":30,
          "volumes":"@ignore@"
        },
        "status": "@ignore@"
      }
      }</k8s:result>
      <k8s:element path="$..status.phase" value="Running"/>
    </k8s:validate>
</k8s:get-pod>

Java DSL

@CitrusTest
public void getPodsTest() {
    kubernetes()
        .client(k8sClient)
        .pods()
        .get("citrus_pod")
        .validate("$..status.phase", "Running")
        .validate((pod, context) -> {
            Assert.assertEquals(pods.getResult().getStatus().getPhase(), "Running");
        });
}

As you can see we are able get the complete pod information from Kubernetes. The result is validated with Json message validator in Citrus. This means we can use @ignore@ as well as test variables and JsonPath expressions.

Create resources

We can create new Kubernetes resources within a Citrus test. This is very important in case we need to setup new pods or services for the test run. You can create new resources by giving a .yml file holding all information how to create the new resource. See the following sample YAML for a new pod and service:

kind: Pod
apiVersion: v1
metadata:
  name: hello-jetty-${randomId}
  namespace: default
  selfLink: /api/v1/namespaces/default/pods/hello-jetty-${randomId}
  uid: citrus:randomUUID()
  labels:
    server: hello-jetty
spec:
  containers:
    - name: hello-jetty
      image: jetty:9.3
      imagePullPolicy: IfNotPresent
      ports:
        - containerPort: 8080
          protocol: TCP
  restartPolicy: Always
  terminationGracePeriodSeconds: 30
  dnsPolicy: ClusterFirst
  serviceAccountName: default
  serviceAccount: default
  nodeName: minikube

This YAML file specifies a new resource of kind Pod. We define the metadata as well as all containers that are part of this pod. The container is build from jetty:9.3 Docker image that should be pulled automatically from Docker Hub registry. We also expose port 8080 as containerPort so the upcoming service configuration can provide this port to clients as Kubernetes service.

kind: Service
apiVersion: v1
metadata:
  name: hello-jetty
  namespace: default
  selfLink: /api/v1/namespaces/default/services/hello-jetty
  uid: citrus:randomUUID()
  labels:
    service: hello-jetty
spec:
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      nodePort: 31citrus:randomNumber(3)
  selector:
    server: hello-jetty
  type: NodePort
  sessionAffinity: None

The service resource maps the port 8080 and selects all pods with label server=hello-jetty. This makes the jetty container available to clients. The service type is NodePort which means that clients outside of Kubernetes are also able to access the service by using the dynamic port nodePort=31xxx. We can use Citrus functions such as randomNumber in the YAML files.

In the test case we can use these YAML files to create the resources in Kubernetes:

XML DSL

<k8s:create-pod namespace="default">
  <k8s:template file="classpath:templates/hello-jetty-pod.yml"/>
</k8s:create-pod>

<k8s:create-service namespace="default">
  <k8s:template file="classpath:templates/hello-jetty-service.yml"/>
</k8s:create-service>

Java DSL

@CitrusTest
public void createPodsTest() {
    kubernetes()
        .pods()
        .create(new ClassPathResource("templates/hello-jetty-pod.yml"))
        .namespace("default");

    kubernetes()
        .services()
        .create(new ClassPathResource("templates/hello-jetty-service.yml"))
        .namespace("default");
}

Creating new resources may take some time to finish. Kubernetes will have to pull images, build containers and start up everything. The create action is not waiting synchronously for all that to have happened. Therefore we might add a list-pods action that waits for the new resources to appear.

<repeat-onerror-until-true condition="@assertThat('greaterThan(9)')@" auto-sleep="1000">
  <k8s:list-pods label="server=hello-jetty">
    <k8s:validate>
      <k8s:element path="$.result.items.size()" value="1"/>
      <k8s:element path="$..status.phase" value="Running"/>
    </k8s:validate>
  </k8s:list-pods>
</repeat-onerror-until-true>

With this repeat on error action we wait for the new server=hello-jetty labeled pod to be in state Running.

Delete resources

With that command we are able to delete a resource in Kubernetes. Up to now deletion of pods and services is supported. We have to give a name of the resource that we want to delete.

XML DSL

<k8s:delete-pod name="citrus_pod">
    <k8s:validate>
      <k8s:element path="$.result.success" value="true"/>
    </k8s:validate>
</k8s:delete-pod>

Java DSL

@CitrusTest
public void deletePodsTest() {
    kubernetes()
        .pods()
        .delete("citrus_pod")
        .validate((result, context) -> Assert.assertTrue(result.getResult().getSuccess()));
}

Watch resources

Note The watch operation is still in experimental state and may face severe adjustments and improvements in near future.

When using a watch command we add a subscription to change events on a Kubernetes resources. So we can watch resources such as pods, services for future changes. Each change on that resource triggers a new watch event result that we can expect and validate.

XML DSL

<k8s:watch-pods label="provider=citrus">
    <k8s:validate>
      <k8s:element path="$.action" value="DELETED"/>
    </k8s:validate>
</k8s:watch-pods>

Java DSL

@CitrusTest
public void listPodsTest() {
    kubernetes()
        .pods()
        .watch()
        .label("provider=citrus")
        .validate((watchEvent, context) -> {
            Assert.assertFalse(watchEvent.hasError());
            Assert.assertEquals(((WatchEventResult) watchEvent).getAction(), Watcher.Action.DELETED);
        });
}

Note The watch command may be triggered several times for multiple changes on the respective Kubernetes resource. The watch action will always handle one single event result. The first event trigger is forwarded to the action validation. All further watch events on that same resource are ignored. This means that you may need multiple watch actions in your test case in case you expect multiple watch events to be triggered.

Kubernetes messaging

We have seen how to access the Kubernetes remote REST API by using special Citrus test actions in out test. As an alternative to that we can also use more generic send/receive actions in Citrus for accessing the Kubernetes API. We demonstrate this with a simple example:

XML DSL

<testcase name="KubernetesSendReceiveIT">
    <actions>
      <send endpoint="k8sClient">
        <message>
          <data>
            { "command": "info" }
          </data>
        </message>
      </send>

      <receive endpoint="k8sClient">
        <message type="json">
          <data>{
            "command": "info",
            "result": {
                "clientVersion": "1.4.27",
                "apiVersion": "v1",
                "kind":"Info",
                "masterUrl": "${masterUrl}",
                "namespace": "test"
              }
            }</data>
        </message>
      </receive>

      <echo>
        <message>List all pods</message>
      </echo>

      <send endpoint="k8sClient">
        <message>
          <data>
            { "command": "list-pods" }
          </data>
        </message>
      </send>

      <receive endpoint="k8sClient">
        <message type="json">
          <data>{
            "command": "list-pods",
            "result": {
                  "apiVersion":"v1",
                  "kind":"PodList",
                  "metadata":"@ignore@",
                  "items":[]
              }
          }</data>
          <validate path="$.result.items.size()" value="0"/>
        </message>
      </receive>
    </actions>
</testcase>

As you can see we can use the send/receive actions to call Kubernetes API commands and receive the respective results in Json format, too. This gives us the well known Json validation mechanism in Citrus in order to validate the results from Kubernetes. This way you can load Kubernetes resources verifying its state and properties. Of course JsonPath expressions also come in here in order to validate Json elements explicitly.