MonkeyLittle

Exploring Kubernetes Init Container, ConfigMap & Secrets

2017-02-22T00:00:00+00:00

At the end of my last post I created a docker image for Jenkins that extended the official docker image by disabling the setup wizard, installing the default plugins and installing maven. I skipped over:

setting up ssh for GitHub.
automating the configuration of the Jenkins job(s).
creating appropriate Jenkins slave images.

In this post I’ll describe how to set up SSH for GitHub.

Setting Up SSH for GitHub

To set up SSH for GitHub I created a ConfigMap containing the ssh config. I did this by creating the ConfigMap from a file.

.ssh/config

Host github.com
    HostName ssh.github.com
    Port 443
    User git
    IdentityFile ~/.ssh/id_jenkins_rsa

kubectl create configmap jenkins-ssh-config --from-file=.ssh/config --from-file=.ssh/known_hosts

The SSH config specifies the identity to use as ~/.ssh/id_jenkins_rsa. As the private key is something I want to secure, I will generate the key and store it as a Kubernetes secret. First, follow the instructions on GitHub to generate a new ssh key and add it to your Github account. Then you can store the private and public key as a Kubernetes secret.

kubectl create secret generic jenkins-ssh-key --from-file=id_jenkins_rsa=.ssh/id_jenkins_rsa --from-file=id_jenkins_rsa.pub=.ssh/id_jenkins_rsa.pub

Now that we have made the SSH config and keys available to Kubernetes we need to configure the Jenkins container. I’ve done this using the Kubernetes Init Container feature which is in beta as of version 1.5.

deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: jenkins
      annotations:
        pod.beta.kubernetes.io/init-containers: '[
          {
            "name": "ssh-config",
            "image": "monkeylittle/jenkins:1.0.0",
            "imagePullPolicy": "IfNotPresent",
            "command": ["bash", "-c", "
              while [ ! -e /var/jenkins_home/ssh_config -o ! -e /var/jenkins_home/ssh_key ]; do sleep 1; done;\n
              mkdir -p /var/jenkins_home/.ssh\n
              cp /var/jenkins_home/ssh_config/* /var/jenkins_home/.ssh/\n
              cp /var/jenkins_home/ssh_key/* /var/jenkins_home/.ssh/\n
              chmod 400 /var/jenkins_home/.ssh/id_jenkins*
            "],
            "volumeMounts": [
              {
                "name": "jenkins-home",
                "mountPath": "/var/jenkins_home"
              },
              {
                "name": "jenkins-ssh-config",
                "mountPath": "/var/jenkins_home/ssh_config"
              },
              {
                "name": "jenkins-ssh-key",
                "mountPath": "/var/jenkins_home/ssh_key"
              }
            ]
          }
        ]'

    spec:
      containers:
        - name: jenkins
          image: monkeylittle/jenkins:1.0.0
          env:
            - name: JAVA_OPTS
              value: -Djenkins.install.runSetupWizard=false
          ports:
            - name: http-port
              containerPort: 8080
            - name: jnlp-port
              containerPort: 50000
          volumeMounts:
            - name: jenkins-home
              mountPath: /var/jenkins_home

      volumes:
        - name: jenkins-ssh-config
          configMap:
            name: jenkins-ssh-config
        - name: jenkins-ssh-key
          secret:
            secretName: jenkins-ssh-key
        - name: jenkins-home
          emptyDir: {}

I have defined 3 volumes as part of the deployment. The first populates the volume with the contents of the jenkins-ssh-config ConfigMap, the second with the contents of the jenkins-ssh-key Secret and the third is the Jenkins home directory.

I have also defined an Init Container that:

mounts all 3 volumes.
copies the contents of jenkins-ssh-config to /var/jenkins_home/.ssh
copies the contents of jenkins-ssh-key to /var/jenkins_home/.ssh
changes the permissions of the Jenkins ssh keys to those allowed by the git client.
generates the ssh known_hosts file.

One thing of note (and this may be related to minikube itself) but the volumes may or may not be mounted at the start of the command so I needed to insert a statement tat waits until the jenkins-ssh-config and jenkins-ssh-key volumes are mounted before executing the remainder of the script.

Building a Custom Jenkins Docker Image - Part 2

2017-02-15T00:00:00+00:00

In Building a Custom Jenkins Docker Image - Part 1 we disabled the Jenkins install wizard and installed the default plugins. In part 2, I want to focus on ensuring the build tools are installed. The target end state of our build infrastructure must include the installation and configuration of:

Java
Jenkins
Jenkins plugins
Jenkins jobs (using JobDSL)
Git
Apache Maven

We can review the official Jenkins image source on the jenkinsci docker GitHub repository to understand which of these tools are already provided for. From the Dockerfile we can see the following:

FROM openjdk:8-jdk

RUN apt-get update && apt-get install -y git curl && rm -rf /var/lib/apt/lists/*

This means that the official Jenkins image is based on the openjdk:8-jdk image (Java 8 JDK is installed) and that the Git client is installed. There is nothing to suggest that Maven is installed and we can verify this by creating a new bash session in the container and listing the installed packages.

$ kubectl exec jenkins-155111175-p72qf -it -- bash
jenkins@jenkins-155111175-p72qf:/$ apt list --installed
Listing... Done
acl/now 2.2.52-2 amd64 [installed,local]
...
lsb-base/now 4.1+Debian13+nmu1 all [installed,local]
mawk/now 1.3.3-17 amd64 [installed,local]
mercurial/now 3.1.2-2+deb8u3 amd64 [installed,local]
mercurial-common/now 3.1.2-2+deb8u3 all [installed,local]
mime-support/now 3.58 all [installed,local]
mount/now 2.25.2-6 amd64 [installed,local]
multiarch-support/now 2.19-18+deb8u7 amd64 [installed,local]
ncurses-base/now 5.9+20140913-1 all [installed,local]
...
zlib1g/now 1:1.2.8.dfsg-2+b1 amd64 [installed,local]
jenkins@jenkins-155111175-p72qf:/$

To install maven we use the Docker RUN statement in the Jenkins/Dockerfile to invoke apt-get install. This requires us first to become the root user and subsequently to resume as the jenkins user.

# install Maven
USER root
RUN apt-get update && apt-get install -y maven
USER jenkins

I can now build a new Docker image:

Dockerfile

from jenkins:2.32.2

# install plugins specified in https://github.com/kohsuke/jenkins/blob/master/core/src/main/resources/jenkins/install/platform-plugins.json

# install Organisation and Administration plugins
RUN /usr/local/bin/install-plugins.sh cloudbees-folder
RUN /usr/local/bin/install-plugins.sh antisamy-markup-formatter

# install Build Features plugins
RUN /usr/local/bin/install-plugins.sh build-timeout
RUN /usr/local/bin/install-plugins.sh credentials-binding
RUN /usr/local/bin/install-plugins.sh timestamper
RUN /usr/local/bin/install-plugins.sh ws-cleanup

# install Build Tools plugins
RUN /usr/local/bin/install-plugins.sh ant
RUN /usr/local/bin/install-plugins.sh gradle

# install Pipelines and Continuous Delivery plugins
RUN /usr/local/bin/install-plugins.sh workflow-aggregator:2.0
RUN /usr/local/bin/install-plugins.sh github-organization-folder:1.6
RUN /usr/local/bin/install-plugins.sh pipeline-stage-view:2.0

# install Source Code Management plugins
RUN /usr/local/bin/install-plugins.sh git
RUN /usr/local/bin/install-plugins.sh subversion

# install Distributed Builds plugins
RUN /usr/local/bin/install-plugins.sh ssh-slaves

# install User Management and Security plugins
RUN /usr/local/bin/install-plugins.sh matrix-auth
RUN /usr/local/bin/install-plugins.sh pam-auth
RUN /usr/local/bin/install-plugins.sh ldap

# install Notifications and Publishing plugins
RUN /usr/local/bin/install-plugins.sh email-ext
RUN /usr/local/bin/install-plugins.sh mailer

# install Maven
USER root
RUN apt-get update && apt-get install -y maven
USER jenkins

docker build -t monkeylittle/jenkins:1.1.0 .

Note that I have changed the version tag.

As this is a personal exercise in learning about Kubernetes and Docker, I attempted to perform a rolling update of the deployment but this required using hostPath persistent volumes. Minikube does not yet support running containers with non root users and hostPath volumes (which are provisioned by root user). See this guthub pull request for more information.

deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      containers:
        - name: jenkins
          image: monkeylittle/jenkins:1.1.0
          env:
            - name: JAVA_OPTS
              value: -Djenkins.install.runSetupWizard=false
          ports:
            - name: http-port
              containerPort: 8080
            - name: jnlp-port
              containerPort: 50000
          volumeMounts:
            - name: jenkins-home
              mountPath: /var/jenkins_home
      volumes:
        - name: jenkins-home
          emptyDir: {}

Then recreate the pod.

$ kubectl apply -f deployment.yaml
deployment "jenkins" configured

It’s possible to verify the update by describing the pod:

$ kubectl describe pod | grep Image:
    Image:		monkeylittle/jenkins:1.1.0

To validate that maven is now installed within the container I can create a new bash session in the container and run apt list –installed.

$ kubectl exec jenkins-258199304-4jf60 -it -- bash
jenkins@jenkins-258199304-4jf60:/$ apt list --installed | grep maven

WARNING: apt does not have a stable CLI interface yet. Use with caution in scripts.

libmaven-parent-java/stable,now 21-2 all [installed,automatic]
libmaven-scm-java/stable,now 1.3-5 all [installed,automatic]
libmaven2-core-java/stable,now 2.2.1-17 all [installed,automatic]
maven/stable,now 3.0.5-3 all [installed]

You can also go ahead and create a build job via the Jenkins UI for one of your Java maven projects (i used this one). A couple of things that I skipped over:

I used the HTTPS protocol for the git repository to avoid having to add github to the known_hosts file.
I created the job manually rather than scripting the job creation.
I need to create equivalent jenkins slave images with tooling installed.

Those are a few follow up tasks I’ll get to over the next couple of weeks.

Building a Custom Jenkins Docker Image - Part 1

2017-02-14T00:00:00+00:00

Last week I spent some time learning how to utilize Kubernetes and Jenkins to form the foundation of a build infrastructure. I documented some of those learnings in the posts below:

This is a pretty good start but having built out VM based build infrastructure a number of times in the past I’m well aware that there is a long way to go before I have something I can use, manage and maintain. When working with VM’s I’ve perviously chosen to use Chef to manage and maintain the Jenkins master and slave hosts. Typically, Chef facilitated automation of the installation and configuration of:

Java
Jenkins
Jenkins plugins
Jenkins jobs (using JobDSL)
Git
Apache Maven

To achieve the same level of automation with Docker and Kubernetes, I will need to be able to perform all of the above and distribute as a set of Kubernetes resource definition files and Docker image(s). Before we start doing anything meaningful, I want to disable the setup wizard because, after all, we will be automating the Jenkins setup. To do this I modify the Kubernetes deployment resource file to specify a JAVA_OPTS environment variable.

jenkins-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      containers:
        - name: jenkins
          image: jenkins:2.32.2
          env:
            - name: JAVA_OPTS
              value: -Djenkins.install.runSetupWizard=false
          ports:
            - name: http-port
              containerPort: 8080
            - name: jnlp-port
              containerPort: 50000
          volumeMounts:
            - name: jenkins-home
              mountPath: /var/jenkins_home
      volumes:
        - name: jenkins-home
          emptyDir: {}

You can test this by creating the Kubernetes deployment and service as described in previous posts.

The next task is to replicate the installation of the recommended Jenkins plugins. To do this we must build our own docker image but first we will need to setup our local development environment.

Install the docker command line tool:

brew install docker

Configure your environment to use the Docker Host provided by minikube:

eval $(minikube docker-env)

I will create a working directory in which I’ll store source files. The name is not important but I will call it docker-library/jenkins. This is where I will maintain all my docker images (there’s ambition for you!).

In this directory I’ll create a Dockerfile. The first statement I’ll add is the FROM statement which tells Docker which image my image will be based on. Naturally I will base my image on the official Jenkins image.

from jenkins:2.32.2

The documentation for the official docker image details how to install additional plugins. I found this not to work with the latest version of the image but found more recent instructions on the associated github readme which I verified did behave as expected. I was able to install additional plugins by adding the statement below to the Dockerfile.

RUN /usr/local/bin/install-plugins.sh docker-slaves plugin-name:plugin-version

In order to mimic exactly the behavior of the install wizard I wanted to install the recommended plugins, versions and their dependencies. The wizard retrieves the recommended plugins from a json file available from the Jenkins GitHub repository.

Dockerfile

from jenkins:2.32.2

# install plugins specified in https://github.com/kohsuke/jenkins/blob/master/core/src/main/resources/jenkins/install/platform-plugins.json

# install Organisation and Administration plugins
RUN /usr/local/bin/install-plugins.sh cloudbees-folder
RUN /usr/local/bin/install-plugins.sh antisamy-markup-formatter

# install Build Features plugins
RUN /usr/local/bin/install-plugins.sh build-timeout
RUN /usr/local/bin/install-plugins.sh credentials-binding
RUN /usr/local/bin/install-plugins.sh timestamper
RUN /usr/local/bin/install-plugins.sh ws-cleanup

# install Build Tools plugins
RUN /usr/local/bin/install-plugins.sh ant
RUN /usr/local/bin/install-plugins.sh gradle

# install Pipelines and Continuous Delivery plugins
RUN /usr/local/bin/install-plugins.sh workflow-aggregator:2.0
RUN /usr/local/bin/install-plugins.sh github-organization-folder:1.6
RUN /usr/local/bin/install-plugins.sh pipeline-stage-view:2.0

# install Source Code Management plugins
RUN /usr/local/bin/install-plugins.sh git
RUN /usr/local/bin/install-plugins.sh subversion

# install Distributed Builds plugins
RUN /usr/local/bin/install-plugins.sh ssh-slaves

# install User Management and Security plugins
RUN /usr/local/bin/install-plugins.sh matrix-auth
RUN /usr/local/bin/install-plugins.sh pam-auth
RUN /usr/local/bin/install-plugins.sh ldap

# install Notifications and Publishing plugins
RUN /usr/local/bin/install-plugins.sh email-ext
RUN /usr/local/bin/install-plugins.sh mailer

Lets build the docker image specifying the image name and version tag.

$ docker build -t monkeylittle/jenkins:1.0.0 .

Learn about what happens during the build process by reading the Docker getting started guide.

Verify the image was created and uploaded to the minikube local registry as follows:

$ docker images
REPOSITORY                                            TAG                 IMAGE ID            CREATED             SIZE
monkeylittle/jenkins                                  1.0.0               443d3e0a31f8        9 minutes ago       712 MB
...

It’s important that when building the image that a tag is specified. If not specified Docker will attempt and fail to find the image on DockerHub.

Now that we have our docker image available in the minikube local docker registry we can update our kubernetes deployment resource file to specify our newly created image.

jenkins-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      containers:
        - name: jenkins
          image: monkeylittle/jenkins:1.0.0
          env:
            - name: JAVA_OPTS
              value: -Djenkins.install.runSetupWizard=false
          ports:
            - name: http-port
              containerPort: 8080
            - name: jnlp-port
              containerPort: 50000
          volumeMounts:
            - name: jenkins-home
              mountPath: /var/jenkins_home
      volumes:
        - name: jenkins-home
          emptyDir: {}

jenkins-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: jenkins
spec:
  type: NodePort
  ports:
    - port: 8080
      targetPort: 8080
  selector:
    app: jenkins

You can now create the kubernetes deployment (and service) and view the Jenkins UI in a browser to verify the default plugins have been installed. Next step is to automate the installation of the various required tools.

Auto-Scaling Jenkins with Kubernetes

2017-02-09T00:00:00+00:00

Jenkins is a great piece of software (ok, it has problems but I couldn’t imagine software development without it). But one of the challenges with maintaining a Jenkins cluster is capacity management. It’s fairly typical to start out with a single master instance. Over time the number and size of Jenkins jobs increases placing more and more demand on the server. The first fix people apply when this happens is to vertically scale the Jenkins server (In fact I recently interviewed for a position and was told their Jenkins server hardware had 40 cores and 512GB of RAM). Some of the problems with scaling vertically include:

Cost per unit of scale increases with the size of the hardware.
Complex software configuration required to support a large variety of job types.
Greater risk of ‘noisy neighbours’ impacting:
- job performance.
- server stability.

The alternatives to vertically scaling the Jenkins master are to:

Deploy multiple Jenkins masters allocated:
- per environment.
- per organisational unit.
- per product line.
Deploy statically provisioned Jenkins slaves.
Deploy dynamicaly provisioned Jenkins slaves.
Deploy multiple Jenkins masters with statically or dynamically provisioned Jenkins slaves.

There are a few things to consider when choosing how to scale your Jenkins infrastructure. If you choose to deploy multiple master you should have an efficient and effective way to manage them. At a minimum, you should use configuration management or orchestration tooling to manage the lifecycle of the instances themselves. You should also consider similar for managing plugins, jobs etc. In the past I have had great success using tools like Chef and Job DSL to manage build infrastructure.

If you choose to use slaves, consider if you should provision bloated slaves capable of performing any build Job or if you should provision specialized slaves.

Time to press on and deploy an auto-scaling Jenkins cluster with Kubernetes.

First thing we’ll do is to deploy a Jenkins master. If you haven’t already done this you can follow the instructions in the previous posts Deploying Jenkins with Kubernetes and Adding Persistent Volumes to Jenkins with Kubernetes. We do however need to make one change to the jenkins-deployment.yaml file so that the JNLP port is exposed.

jenkins-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      containers:
      - name: jenkins
        image: jenkins:2.32.2
        ports:
        - name: http-port
          containerPort: 8080
        - name: jnlp-port
          containerPort: 50000
        volumeMounts:
          - name: jenkins-home
            mountPath: /var/jenkins_home
      volumes:
        - name: jenkins-home
          emptyDir: {}

With the Jenkins master deployed we next configure Jenkins to auto-scale slave nodes.

As described in Adding Persistent Volumes to Jenkins with Kubernetes you should enter the administrator password, install suggested plugins and create an administrator user.

For Jenkins to dynamically provision Jenkins slaves as Kubernetes Pods, the Kubernetes Plugin is required. To install the Kubernetes Plugin navigate to Manage Jenkins > Manage Plugins > Available, search for the Kubernetes Plugin, check the install checkbox and press Download now and install after restart.

With the Kubernetes plugin installed it must be configured by navigating to Manage Jenkins > Configure System and scrolling to the Cloud section. First we configure the Kubernetes Section as below:

Kubernetes Plugin Configuration

To obtain the Kubernetes URL you should invoke:

$ kubectl cluster-info | grep master
Kubernetes master is running at https://192.168.99.100:8443

To obtain the Jenkins URL you first need to obtain the pod name of the Jenkins master:

$ kubectl get pods | grep ^jenkins
jenkins-2559287856-5p145             1/1       Running   0          54m

And then obtain the IP address of the pod:

$ kubectl describe pod jenkins-2559287856-5p145 | grep IP:
IP:		172.17.0.4

With these configuration entries the Jenkins Kubernetes plugin can interact with the Kubernetes API server. Next we need to configure the pod template and container for the slave so that the plugin can provision a pod.

Kubernetes Plugin Pod Template Configuration

A few things to note here:

The docker image is the standard Jenkins slave image from Jenkins CI. It is probable that you will want to use your own images for Jenkins slaves.
Being able to specify labels and docker images makes it possible to use specialized Jenkins slaves.
The working directory is specific to the docker image.

If you’ve followed along so far you should be ready to create a job to test our autoscaling behavior. Go ahead and create a new Freestyle project. You will need to set the Label Expression field to match that specified in the Pod Template configuration. I’ve also created an Execute shell build step.

Job General Configuration

Job Build Step Configuration

You are now ready to build the job. Before doing that you should watch the Kubernetes Pods. Do this by installing watch (brew install watch) and executing the watch as follows:

$ watch kubectl get pod
Every 2.0s: kubectl get pod                                                                                                                                                                                                                                                                                                            Homer.local: Thu Feb  9 17:06:24 2017

NAME                                 READY     STATUS    RESTARTS   AGE
default-jenkins-slave-11f26675ac49   2/2       Running   0          4s
jenkins-2559287856-5p145             1/1       Running   0          1h

As pointed out by Daniel Vigueras below, a better way to watch the Kubernetes Pods may be to use the command:

kubectl get pods -w

Then as your build job executes you can watch the pod being created and destroyed.

Over the past few days I’ve:

given a high level overview of Kubernetes,
demonstrated how to deploy Jenkins,
how to add persistent volumes so that the Jenkins data is durable across container restarts
and how to configure an autoscaling Jenkins cluster.

Next, I’d like to build a bespoke Jenkins image that avoids having to go through the Jenkins initialization steps. As always, feedback very welcome.

Adding Persistent Volumes to Jenkins with Kubernetes

2017-02-08T00:00:00+00:00

Having had a first attempt at Deploying Jenkins with Kubernetes I’ll next focus on incrementally making the Jenkins deployment more robust. The first thing that I’ll investigate is how I can preserve the contents of the JENKINS_HOME directory. Jenkins stores all of its important information within the JENKINS_HOME such as:

build server configuration.
build jobs.
build artifacts.
user accounts.
user installed plugins.

By default disk files in a container are ephemeral. This means that when the Jenkins container fails and is recovered (or indeed when an upgrade occurs) that the data within the JENKINS_HOME directory is lost forever. It would be desirable to have the JENKINS_HOME directory preserved across failure, recovery and upgrade processes. This can be achieved using Kubernetes Volumes.

Container Filesystem

We cannot talk about Kubernetes volumes without understanding the Docker filesystem. The Docker filesystem is composed of a number of layers starting with one or more read only image layers. When a container is created, a read/write container layer is added to the top of the stack. When a file is read from the filesystem each layer is inspected from the top of the stack to the bottom until the file is found or the bottom of the stack is reached. When a file is modified it is first copied to the top of the stack and so the next time it is read the modified file is found first.

Container Filesystem

The top of the stack is the container layer. The container layer is ephemeral so the modifications to files within the container layer have the same lifecycle as the container. That is, when the container is destroyed the modifications to the files in the container layer are lost.

We can demonstrate this using Jenkins as described in ‘Deploying Jenkins with Kubernetes’. First, create the Jenkins deployment and service:

kubectl create -f jenkins-deployment.yaml
kybectl create -f jenkins-service.yaml

In your browser, navigate to the jenkins service endpoint. You can discover this URL by running the following command:

$ echo "$(minikube ip):$(kubectl describe service jenkins | grep NodePort: | grep -Eo '[0-9]{1,5}')"
192.168.99.100:32028

You should be presented with the Unlock Jenkins screen below.

Unlock Jenkins Screen

Lets proceed to configure Jenkins. To retrieve the administrator password we need to look at the Jenkins log. First query the name of the Jenkins pod.

$ kubectl get pods
NAME                       READY     STATUS    RESTARTS   AGE
jenkins-2843131955-brz6t   1/1       Running   1          1h

Next, query the log from the Jenkins pod.

$ kubectl logs jenkins-2843131955-brz6t
Running from: /usr/share/jenkins/jenkins.war
...
*************************************************************
*************************************************************
*************************************************************

Jenkins initial setup is required. An admin user has been created and a password generated.
Please use the following password to proceed to installation:

2dda09e55aaa4052983aaf78e38eaedf

This may also be found at: /var/jenkins_home/secrets/initialAdminPassword

*************************************************************
*************************************************************
*************************************************************

Now armed with the administrator password you can proceed through the setup screens. What is happening in the background is that the modifications to the data in JENKINS_HOME is being written to the container layer of the filesystem. Next, we want to ssh onto the Kubernetes Node and kill the Jenkins docker container.

$ minikube ssh
$ docker ps
CONTAINER ID        IMAGE                                                        COMMAND                  CREATED             STATUS              PORTS               NAMES
17f7d403f26b        jenkins:2.32.2                                               "/bin/tini -- /usr/lo"   About an hour ago   Up About an hour                        
...
$ docker kill 17f7d403f26b
17f7d403f26b

If you were to run docker ps again you will notice that the Kubernetes replication controller has recreated the container and the new container has a different CONTAINER ID. If you return to Jenkins you will be presented with the Unlock Jenkins screen. What has happened is that when the container was destroyed so to was the associated container layer. When the container was recreated a new (empty) container layer was created. As a result, the previous modifications to the configuration was lost forever.

Kubernetes Volumes

Kubernetes Volumes can help preserve data across container restarts (and indeed Pod restarts depending on the type of volume used). The lifecycle of a Kubernetes Volume is bound to the same lifecycle as a Pod. A Volume (and its data) is preserved across container restarts irrespective of the nature of the restart (routine restart or due to failure). When the Pod is destroyed so too are the associated Volumes. Because the volume is associated with the Pod it is accessible to all containers within the Pod.

A volume is mounted on top of the container filesystem and so data is still read in the same way as before except that modifications will occur to the volume. Because the volume is durable across container restarts the data is preserved.

Kubernetes Volumes

Lets add a volume to our Jenkins deployment.

jenkins-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      containers:
      - name: jenkins
        image: jenkins:2.32.2
        ports:
        - containerPort: 8080
        volumeMounts:
          - name: jenkins-home
            mountPath: /var/jenkins_home
      volumes:
        - name: jenkins-home
          emptyDir: {}

The spec now defines a volume named jenkins-home. This volume is mounted within the container at the path /var/jenkins_home and so modifications to data within /var/jenkins_home are written to the volume. If we were to recreate the deployment, configure Jenkins and kill the Jenkins container as before you will not be presented with the Unlock Jenkins screen a second time.

Of course, it the Pod is destroyed so too is the volume. If you need the data to be preserved across Pod failures (node failures etc.) you will need to use remote storage such as AWS EBS or GCE Persistent Disk.

I’ve definitely learned a bit about the Docker and Kubernetes filesystem today and evolved the previous Jenkins example so that it is a little bit more robust. I didn’t like the manual setup steps needed to configure Jenkins so next I’ll walk through creating my own Jenkins image.

Deploying Jenkins with Kubernetes

2017-02-07T00:00:00+00:00

Quick on the heels of my recent overview of the Kubernetes Fundamentals I wanted to dive into deploying something familiar (Jenkins) so that I could focus in on some of the Kubernetes specifics.

Installing Kubernetes Locally

Before going any further I’ll describe my local Kubernetes environment. I’ve previously explored Kubernetes on Google Container Engine and found it effortless. I’ve also used the community provided kube-up scripts to create a local cluster but it appears that this approach has been deprecated. The recommended approach to creating a local Kubernetes environment is using MiniKube.

If you are using Mac, it is fairly trivial to install the components required for MiniKube:

brew cask install virtualbox
brew install kubectl
brew cask install minikube

Note: If you don’t have brew installed you can find the brew installation instructions here.

To start minikube simply invoke:

minikube start

At this point there is a single node Kubernates environment running within a VirtualBox VM. To validate the environment simply invoke:

kubectl cluster-info

This command will return URL’s for the Kubernetes Master, DNS service and Dashboard.

Kubernetes master is running at https://192.168.99.100:8443
KubeDNS is running at https://192.168.99.100:8443/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at https://192.168.99.100:8443/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard

Deploying Jenkins

So now we have a single node Kubernetes environment running locally it’s time to define our Jenkins deployment. A simple deployment is defined below:

jenkins-deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: jenkins
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: jenkins
    spec:
      containers:
      - name: jenkins
        image: jenkins:2.32.2
        ports:
        - containerPort: 8080

Some points of note:

The file is defining a Deployment as indicated by the kind field.
The Deployment specifyies a single replica. This ensures one and only one instance will be maintained by the Replication Controller in the event of failure.
The container image name is jenkins and version is 2.32.2
The list of ports specified within the spec are a list of ports to expose from the container on the Pods IP address.
- Jenkins running on (http) port 8080.
- The Pod exposes the port 8080 of the jenkins container.

To create the deployment execute:

kubectl create -f jenkins-deployment.yaml

To validate that creating the deployment was successful you can invoke:

kubectl get deployments

It’s worth noting that the Kubernetes command line is very consistent in its naming and syntax. You can invoke kubectl get RESOURCE for any Kubernetes resource. If all has worked as expected you will be presented with output similar to that below:

NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
jenkins   1         1         1            1           3m

We can also see that a single Pod has been created by invoking:

$ kubectl get pods
NAME                       READY     STATUS    RESTARTS   AGE
jenkins-2843131955-31rh1   1/1       Running   0          7m

Exposing Jenkins

As it stands we have a Jenkins instance deployed but it is still not accessible. The jenkins Pod has been assigned an IP address which is internal to the Kubernetes cluster. Of course it’s possible to log into the Kubernetes Node and access Jenkins from there but that’s not a very useful way to access the service.

To make Jenkins accessible outside the Kubernetes cluster the Pod needs to be exposed as a Service. With a local deployment this means creating a NodePort service type. A NodePort service type exposes a service on a port on each node in the cluster. It’s then possible to access the service given the Node IP address and the service nodePort. A simple service is defined below:

jenkins-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: jenkins
spec:
  type: NodePort
  ports:
    - port: 8080
      targetPort: 8080
  selector:
    app: jenkins

Some points of note:

The file is defining a Service as indicated by the kind field.
The Service is of type NodePort. Other options are ClusterIP (only accessible within the cluster) and LoadBalancer (IP address assigned by a cloud provider e.g. AWS Elastic IP).
The list of ports specified within the spec are a list of ports exposed by this service.
- The port is the port that will be exposed by the service.
- The target port is the port to access on the Pods targeted by this service. A port name may also be specified.
The selector specifies the selection criteria for the Pods targeted by this service.

To create the service execute:

kubectl create -f jenkins-service.yaml

To validate that creating the service was successful you can invoke:

$ kubectl get services
NAME         CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE
jenkins      10.0.0.202   <nodes>       8080:30104/TCP   3m
kubernetes   10.0.0.1     <none>        443/TCP          6h

So now we have created a deployment and service how do we access Jenkins?

From the output above we can see that the service has been exposed on port 30104. We also know that because the service is of type NodeType the service will route requests made to any node on this port to the jenkins pod. All that’s left for us is to determine the IP address of the minikube VM. Minikube have made this really simple by including a specific command that outputs the IP address of the running cluster:

$ minikube ip
192.168.99.100

Now we can access the jenkins instance at http://192.168.99.100:30104/

Jenkins Unlock Screen

So that is a bit of a world wind tour of deploying Jenkins on Kubernetes. I plan on developing this use case in future posts so as to explore Kubernetes further so stay tuned if you are finding this useful. On the other hand, if you are experienced with Kubernetes and spot any rookie mistakes I’d appreciate you drawing my attention to them using the comment section below.

Kubernetes Fundamentals

2017-02-06T00:00:00+00:00

Over the last few years, I’ve been watching with interest the container space in general and specifically the level of interest being generated by the various orchestration options available. I still remain quite sceptical about the ease with which most organizations could effectively adopt containers (and the resulting changes to development and operational practices). However, I do believe that systems like Mesos and Kubernetes will eventually become default deployment environments for all but a few. With that in mind, I’ve decided to dive into Kubernetes to get a better understanding of the opportunities and challenges ahead.

Kubernetes is a system that has emerged from Google for automating deployment, scaling and management of containerized applications. It is one of a number of orchestration tools whose emergence has coincided with the increasing popularity and maturity of container runtimes such as Docker and rkt. Other orchestration tools include Docker Swarm, Fleet and Mesos.

Each of the previously mentioned orchestration tools incorporate a number of concepts, some of which are very similar and others which are quite distinctive. This can make it quite challenging when you come to learn about one or other (and even more so when you come to compare them). In this post I will briefly describe some of the key concepts you should understand when starting out with Kubernetes.

Kubernetes Cluster

Kubernetes is a distributed system which consists of a number of connected virtual or physical machines. At a minimum a Kubernetes cluster consists of two machines; one Master and one Node. As of v1.5.1, Kubernetes state support for clusters of up to 2000 nodes with the following stipulations:

No more than 2000 nodes
No more than 60,000 total pods
No more than 120,000 total containers
No more than 100 pods per node

Kubernetes environments can of course grow beyond this scale and that is achieved through cluster federation. Federation facilitates:

Keeping resources in multiple clusters in sync.
Exposing DNS and loadbalancer information across clusters.

Kubernetes Master

The Master is responsible for coordinating all activities in the cluster. This includes deploying applications, maintaining desired state (e.g. in the event of failure), scaling applications and rolling out updates. Cluster administration is performed by invoking an API exposed by the Master either directly (if you are building tools), via the Kubernetes command line interface or the Kubernetes dashboard. The Nodes within the cluster also interact with the Master via the same API.

In a non-critical environment there may be a single master node but in environments that need to be resilient to failure multiple master nodes are required. Within the Master node etcd is used for data storage so when using multiple masters it is necessary to cluster etcd so that data is replicated across the Master nodes. It is also necessary to place a loadbalancer in front of the API server running on each Master node.

The Kubernetes Master contains a number of components for which there may only be a single instance active within the cluster. These are the scheduler and the controller-manager. For these components a leader election process occurs to ensure a single instance running at any one time. In the event of failure a new leader will be elected from the available nodes.

Kubernetes Node

The Kubernetes Node is the workhorse of the cluster. A Node is managed by the Master and can be either a virtual or physical machine. Within each node exists an operating system, a container runtime (Docker is supported with experimental support for rkt), a proxy server (kube-proxy) and a node agent (kubelet).

The kubelet executes instructions issued by the master node and provides periodic updates of node and pod health. Those instructions include locally provisioning pod resources such as containers, volumes etc. The kubelet also enables log retrieval.

The kube-proxy is a simple network proxy and loadbalancer for services on the Node.

The Node runs a process supervisor service that restarts the container runtime or kubelet in the case of failure.

Deployment

Kubernetes Deployment

A Deployment is a logical component that describes the desired state of Pods and Replica Sets. Use of a Deployment allows Kubernetes to perform richer orchestration as opposed to fine grained transactional actions. Examples of this include:

Scaling a deployment either explicitly or based on CPU utilization of existing Pods.
Performing rolling updates.
Rolling back an update to a previous version.

Given that a deployment specifies the desired state, all the detail relating to how Kubernetes converges the deployment is abstracted. This makes using deployments easier to learn than the alternative of explicitly writing instructions to achieve the desired state. It also means that Kubernetes can reason about the current state and act accordingly.

Pod

Kubernetes Pod

A Pod exists within a single Node and can consist of a number of application containers with associated volumes. Each Pod is allocated a dedicated IP address upon creation. Containers and Volumes cannot be created individually and must be created within a Pod.

Pods have no orchestrated behavior and so if you chose to deploy Pods outside a Deployment or Replication Controller you will not have any auto-recovery, auto-scaling, update or rollback behaviors.

Services

A Service provides a stable name and address for a set of Pods. Each Service gets it’s own virtual IP address which remains constant across Pod failure and recovery (Pods themselves will be assigned a new IP address).

A Service is associated with Pods via a selector which matches against labels assigned to the Pods during creation. Traffic is then routed from the Service to the Pods.

Now I’ve a high level understanding of the key Kubernetes concepts I’m in a better place to dive into some specifics.

JPA Inheritance Strategies Explained

2015-04-13T00:00:00+00:00

Last week, I was speaking to an intern who worked for me while I was at Paddy Power. He was explaining with some frustration that he had recently had the supervisor for his final year project reassigned. The guidance from the new supervisor was that the project needed some “wow”. I was not sure if this meant he needed to present the project while flanked by the Dallas Coyboys Cheerleaders so I asked for some specifics. This was the rather perplexing response.

I know Ian is using JPA and I would like to give him a little more information than he received from the above suggestion. So here is my effort at explaining the JPA inheritance strategies.

JPA Inheritance Strategies

The JPA inheritance strategies facilitate mapping of an inheritance hierarchy in 3 different ways. To demonstrate the advantages and disadvantages of each, I will use a class hierarchy comprising a Vehicle, Airplane, Bike and Car.

So you can follow or experiment with the code I’ve made it available on GitHub. Note that there are separate repositories for each class, that the Vehicle class is abstract and that each repository has a corresponding test class. I’m using Spring Boot to automatically discover the repositories, entity mappings as well as provide an in memory database for testing purposes.

I’ve enabled the Hibernate SQL logging so that we can see the table DDL as well as the structure of a findAll query executed via each of the repositories.

InheritanceType.JOINED

The JOINED inheritance strategy specifies a table for each class within the hierarchy.

Table Structure

Given the Vehicle hierarchy, there will be 4 tables created as demonstrated by the log output below:

Create Tables

[main] DEBUG org.hibernate.SQL - create table airplane (id bigint not null, primary key (id))
[main] DEBUG org.hibernate.SQL - create table bike (id bigint not null, primary key (id))
[main] DEBUG org.hibernate.SQL - create table car (id bigint not null, primary key (id))
[main] DEBUG org.hibernate.SQL - create table vehicle (id bigint not null, colour integer, primary key (id))

You will also notice that in this simple example, the properties of each class file correlate directly to columns on each table with the id duplicated across all 4 tables. The id column on the airplane, bike and car tables act as a foreign key while the id on the vehicle table is the primary key.

From this we can deduce all the advantages and disadvantages.

Changing Class Definitions is Easier

Because there is a direct correlation of class + properties to table + columns there is no duplication of the vehicle definition. As a result a single change for a class or property definition will result in a single change for a table or column definition. For example, if I add an ‘isMortorised’ property to vehicle class it will result in the addition of a ‘motorised’ column on the vehicle table.

Data Integrity at the DBMS

Another side effect of the direct mapping between class and table is that I can manage data integrity at the database layer. For example, column definitions including null constraints, foreign key constraints etc. can be managed by the DBMS (without resorting to stored procedures).

Creating an Airplane, Bike or Car is (more) Expensive

To save an Airplane requires an insert into the vehicle and airplane tables. It will also require a sequence number to be generated for the vehicle and a foreign key constraint validation to occur on the airplane table. We can see this from the log statement below:

Insert Airplane

[main] DEBUG org.hibernate.SQL - insert into vehicle (colour, id) values (?, ?)
[main] DEBUG org.hibernate.SQL - insert into airplane (id) values (?)

Retrieving an Airplane, Bike or Car is (more) Expensive

To retrieve an Airplane (or Airplanes) requires a join between vehicle and airplane.

Retrieve Airplane

[main] DEBUG org.hibernate.SQL - select airplane0_.id as id1_3_, airplane0_1_.colour as colour2_3_ from airplane airplane0_ inner join vehicle airplane0_1_ on airplane0_.id=airplane0_1_.id

Retrieving a Vehicle is (much more) Expensive

To retrieve a Vehicle (or Vehicles) requires a (left outer) join between vehicle, airplane, bike and car.

Retrieve Vehicle

[main] DEBUG org.hibernate.SQL - select vehicle0_.id as id1_3_, vehicle0_.colour as colour2_3_, case when vehicle0_1_.id is not null then 1 when vehicle0_2_.id is not null then 2 when vehicle0_3_.id is not null then 3 when vehicle0_.id is not null then 0 end as clazz_ from vehicle vehicle0_ left outer join airplane vehicle0_1_ on vehicle0_.id=vehicle0_1_.id left outer join car vehicle0_2_ on vehicle0_.id=vehicle0_2_.id left outer join bike vehicle0_3_ on vehicle0_.id=vehicle0_3_.id

InheritanceType.SINGLE_TABLE

As the name suggests, the SINGLE_TABLE inheritance strategy specifies a table for the entire class hierarchy.

Table Structure

Given the Vehicle hierarchy, there will be 1 table created as demonstrated by the log output below:

Create Tables

[main] DEBUG org.hibernate.SQL - create table vehicle (dtype varchar(31) not null, id bigint not null, colour integer, primary key (id))

You’ll notice that the first column definition does not correspond to a property from any of the class files. This is a discriminator column that allows JPA to understand the type of entity to create when it retrieves a row from this table.

Changing Class Definitions is Harder

When using the JOINED strategy I only impacted the table corresponding to the class I was modifying. With the SINGLE_TABLE strategy I am changing table that all vehicles are stored in. This may or may not be a big deal but is worth considering especially if you are storing lots of data.

Data Integrity at the DBMS

Column definitions including null constraints, foreign key constraints etc. can no longer be managed by the DBMS (without resorting to stored procedures). This can be a problem especially when a database can be access by different applications using different access layers.

Data Storage

Because a single table is stores all vehicles there will necessarily be a lot of null values in columns that do not relate to the specific type. This may or may not be a problem depending on the size of the tables and how efficient the DBMS is at storing null values.

Creating an Airplane, Bike or Car is Cheap

Because the SINGLE_TABLE strategy uses a single table, creating an Airplane, Bike or Car is a single insert executed against a single (albeit larger) table.

Insert Vehicles

[main] DEBUG org.hibernate.SQL - insert into vehicle (colour, dtype, id) values (?, 'Bike', ?)
[main] DEBUG org.hibernate.SQL - insert into vehicle (colour, dtype, id) values (?, 'Car', ?)
[main] DEBUG org.hibernate.SQL - insert into vehicle (colour, dtype, id) values (?, 'Airplane', ?)

Note that the discriminator value defaults to the class name. Be careful that whatever discriminator you use is treated efficiently by the DBMS (in terms of both storage and query efficiency. The discriminator can be changed using the JPA DiscriminatorColumn annotation.

Retrieving a Vehicle, Airplane, Bike or Car is Cheap

Again, because of the use of a single table retrieving a Vehicle, Airplane, Bike or Car (or many of same) is cheap. It is a single select statement which filters using the discriminator value if a subclass is being queried. This is the cheapest strategy for executing polymorphic queries (i.e. retrieving Vehicles).

Retrieve Vehicles

[main] DEBUG org.hibernate.SQL - select car0_.id as id2_0_, car0_.colour as colour3_0_ from vehicle car0_ where car0_.dtype='Car'
[main] DEBUG org.hibernate.SQL - select bike0_.id as id2_0_, bike0_.colour as colour3_0_ from vehicle bike0_ where bike0_.dtype='Bike'
[main] DEBUG org.hibernate.SQL - select airplane0_.id as id2_0_, airplane0_.colour as colour3_0_ from vehicle airplane0_ where airplane0_.dtype='Airplane'
[main] DEBUG org.hibernate.SQL - select vehicle0_.id as id2_0_, vehicle0_.colour as colour3_0_, vehicle0_.dtype as dtype1_0_ from vehicle vehicle0_

InheritanceType.TABLE_PER_CLASS

The TABLE_PER_CLASS strategy is really a table per concrete class strategy.

Table Structure

Given the Vehicle hierarchy, there will be 3 tables created as demonstrated by the log output below:

Create Tables

[main] DEBUG org.hibernate.SQL - create table airplane (id bigint not null, colour integer, primary key (id))
[main] DEBUG org.hibernate.SQL - create table bike (id bigint not null, colour integer, primary key (id))
[main] DEBUG org.hibernate.SQL - create table car (id bigint not null, colour integer, primary key (id))

The inherited colour property is defined in the definitions of the airplane, bike and car tables. There is no vehicle table as Vehicle is an abstract class. There is also no discriminator as data from each vehicle is stored in its own table.

Changing Class Definitions is Easier

Given that each concrete class has its own corresponding table definition, it is easier to modify these. However, if I modify the Vehicle definition I have to make corresponding modifications to the airplane, bike and car tables.

Data Integrity at the DBMS

Because each concrete class is stored in its own table the DBMS can manage null constraints, foreign key constraints etc. without resorting to stored procedures.

Creating an Airplane, Bike or Car is Cheap

Because the TABLE_PER_CLASS strategy uses a single table for each concrete class, creating an Airplane, Bike or Car is a single insert executed against a single table.

Create Vehicles

[main] DEBUG org.hibernate.SQL - insert into bike (colour, id) values (?, ?)
[main] DEBUG org.hibernate.SQL - insert into car (colour, id) values (?, ?)
[main] DEBUG org.hibernate.SQL - insert into airplane (colour, id) values (?, ?)

There is no discriminator value in this case!

Retrieving an Airplane, Bike or Car is Cheap

Again, because of the use of a single table retrieving an Airplane, Bike or Car (or many of same) is cheap. It is a single select statement.

Retrieve Airplane, Bike, Car

[main] DEBUG org.hibernate.SQL - select car0_.id as id1_3_, car0_.colour as colour2_3_ from car car0_
[main] DEBUG org.hibernate.SQL - select bike0_.id as id1_3_, bike0_.colour as colour2_3_ from bike bike0_
[main] DEBUG org.hibernate.SQL - select airplane0_.id as id1_3_, airplane0_.colour as colour2_3_ from airplane airplane0_

Retrieving a Vehicle is Expensive

The biggest disadvantage of this strategy is that polymorphic queries are expensive, requiring a union of 3 data sets (from each of the tables).

[main] DEBUG org.hibernate.SQL - select vehicle0_.id as id1_3_, vehicle0_.colour as colour2_3_, vehicle0_.clazz_ as clazz_ from ( select id, colour, 1 as clazz_ from airplane union all select id, colour, 2 as clazz_ from car union all select id, colour, 3 as clazz_ from bike ) vehicle0_

Wrapping Up

So there we have it, some of the considerations to be taken into account when selecting the JPA inheritance strategy. A lot of these will depend on the particulars of the DBMS you are using but for the most part this will provide a useful guide.

Application Performance Monitoring - Back to the Future

2014-03-26T00:00:00+00:00

In my previous post Application Monitoring - Past, Present and Future I touched on my desire to see Application Performance Monitoring (APM) vendors add log aggregation and analytics to their service offerings. I’d like to offer further suggestions into what I would like to see in future APM offerings.

Application Insight

The design and function of APM solutions is relatively simple and straightforward to implement. Typically, there exists some form of “Agent” running on the node under observation that relays data to a “Collector”. The “Collector” then stores the data on some form of storage solution. The data is then inspected and presented to a number of services including those that provide functions such as:

Administration
Alerting
Analytics
Presentation

Let’s not consider the actual data that is being collected other than say it should include all data relevant to application health. Notwithstanding the omission of log data stated earlier, APM solutions from companies such as AppDynamics and New Relic already do a good job of collecting the relevant data at a reasonably low cost (operational overhead, configuration etc.). New Relic has also recently announced the addition of a platform, Rubicon, which allows one to perform in depth analytics on all the data collected (though currently New Relic do not collect distributed tracing data.

I refer to this capability as “Application Insight” and while this space will continue to evolve it does not exactly provide the next big step forward for APM vendors.

The leaders in this space are considered to be AppDynamics and New Relic and they both provide APM. Interesting, they both define APM as application performance management while I would suggest that neither actually perform any management. They stop short at application performance monitoring.

Application Intelligence

When I consider what I want from an application performance management solution I most certainly want some level of autonomic behaviour. IBM coined the phrase “Autonomic Computing” way back in 2001 (they did bugger all with the research worth talking about but hey, that’s IBM for you!) and they defined autonomic computing as having four functional areas:

Self-configuration: Automatic configuration of components.
Self-healing: Automatic discovery and correction of faults.
Self-optimisation: Automatic monitoring and control of resources to ensure the optimal functioning with respect to the defined requirements.
Self-protection: Proactive identification and protection from arbitrary attacks.

The thrust of the concept is that systems should apply self-* principles to make intelligent runtime adaptations to the system behaviour. I call this “Application Intelligence”

Colonel John Boyd was a United States Air Force fighter pilot and consultant to the Pentagon. Boyd hypothesized that all intelligent organisms and organizations undergo a continuous cycle of interaction with their environment. Boyd breaks this cycle down to four interrelated and overlapping processes through which one cycles continuously:

Observation: the collection of data by means of the senses
Orientation: the analysis and synthesis of data to form one’s current mental perspective
Decision: the determination of a course of action based on one’s current mental perspective
Action: the physical playing-out of decisions

“Application Insight” gives us the capability to apply the Observe, Orientate and Decision process while “Application Intelligence” gives us the capability to apply the Action process. In the context of todays APM solutions, the action process is applied via the same mechanism through which application metrics are collected. That is, the same runtime code instrumentation that collects application metrics at the “edge of the node” can be used to modify the behaviour. I’ll explain this by way of a couple of examples.

The Timeout

Lets consider a very simple request from a client to a service. This request represents a client invoking a RESTful service and awaiting a response.

It is well understood that in order to prevent the client continuing to wait for a response when the service is not responding in a timely fashion, the client should implement a request timeout. The request timeout is typically set based on some expectation of service or prior knowledge of service behaviour. The process of deciding what an appropriate timeout should be can actually be quite difficult. Indeed, it may also be appropriate that the timeout change over time.

Now lets consider applying APM and the OODA loop to the problem of deciding an appropriate timeout for the service call. We already know that APM solutions gather metrics on service response times and error rates. This gives us the information to allow us to apply some basic heuristics to determine an appropriate timeout for the service call. The heuristics could be the following:

Set the timeout to some arbitrarily high value.
Observation: Collect metrics on service response time and error rates.
Orientation: Analyze the data to determine the optimal timeout.
Decision: If the dataset is statistically significant, adjust timeout.
Action: Adjust the timeout.

This provides a degree of self-optimization and therefore active management of the application performance.

Lets continue with a second example.

The Circuit Breaker

Lets consider again the simple scenario in Figure 1 and apply the concept of a Circult Breaker to service requests. During normal operations the circuit is considered closed and requests are forwarded to the service. In the event of a number of service failures, the circuit breaker trips and requests are no longer forwarded to the service. Instead, the circuit breaker automatically returns an error response. This has the effect of reducing the load on the service allowing some period of time for the service to recover (perhaps through auto-scaling). After some period of time the circuit breaker is restored and requests are once again forwarded to the service.

It comes as no surprise the applying APM and the OODA loop to the problem we can implement the Circuit Breaker. The heuristics could be the following:

Set the failure count to zero.
Observation: Record failure count within a contextual (temporal and request count) window.
Orientation: Analyze the data to determine the significance of the failure count given throughput.
Decision: If the dataset is statistically significant, trip circuit breaker.
Action: Trip circuit breaker.

This again provides a degree of self-optimization and therefore active management of the application performance.

The Handshake

Handshaking allows the service to throttle requests in order to protect itself. Based on some measure of load, the service automatically rejects requests with a response such as a 503 service unavailable if the load is beyond what the service can cater for. When load returns to normal, the service starts to accept requests once more. The heuristics could be the following:

Observation: Record load as some function of request count, latency, resource utilization and downstream service performance.
Orientation: Analyze the data to determine service health.
Decision: If health is deteriorating, reject incoming requests.
Action: Reject incoming requests.

Wrapping Up

So this stuff is not exactly novel. Michael Nygard has documented the above patterns in his book, Release It!. Netflix have implemented these patterns in their Hystrix library that they have made open source.

My take away point is that APM solutions exist where the “M” stands for monitoring. I would like an APM solution where the “M” stands for management.

Application Monitoring - Past, Present and Future

2014-03-16T00:00:00+00:00

In addition to working on PaaS and Continuous Delivery initiatives during 2013, I also had the opportunity to learn a lot about Application Performance Monitoring (APM) tools. Because I see PaaS and CD as enabling technologies for DevOps it was natural that I also look at shared tooling such as APM and log aggregation.

Application Monitoring in the Past

In the past, infrastructure monitoring was used as a proxy for the health of an application. Application health was measured in terms of CPU and memory utilisation, disk and network IO or some other infrastructure metric. Some form of translation between the behaviour of the infrastructure and the user experience took place to allow the operations team to understand when the user experience may be adversely affected by symptoms within the infrastructure.

Things changed that meant this approach to application monitoring has become a thing of the past. Applications leverage their infrastructure in very different ways. For example, look at the CPU and memory utilisation profile of a reactive (or event based) system against that of a synchronous request-response system. In one, high CPU utilisation is considered normal while in the other it may signal that the system is overloaded.

Applications have become increasingly distributed so a user experience may be dictated by the performance of a number of application services residing on different OS, using different language runtimes and storage solutions etc. Environments are no longer static and systems are capable of dynamically provisioning and un-provisioning nodes to a cluster. In summary, things just got complicated.

The increased proliferation of technologies, and the distribution and scale of today’s systems means that it has become increasingly difficult to predict system health by measuring the symptoms. We’ve got to actually talk to the patient!

Application Monitoring in the Present

I’ve looked at a lot of the APM vendors currently operating in the marketplace and have been really impressed with how application monitoring has evolved to where it is today. In my opinion, the significant features of todays APM offerings are:

Distributed transaction tracing
Application topology detection
Automatic entry and exit detection
Lossless or aggregated metric collection
Adaptive alerting thresholds
Auto-scaling notifications

Distributed Transaction Tracing

Way back in April 2010, Google published a paper on ‘Dapper, a Large-Scale Distributed Systems Tracing Infrastructure’. The subject matter is not new but people tend to listen more attentively when Google speaks. Dapper facilitates tracing service calls that traverse a graph of distributed servers or containers. The image below shows a distributed trace for a web request as captured by Zipkin, Twitters distributed tracing system.

As systems become more distributed the ability to visualise distributed call graphs is immeasurable. From the graph above I can see the touch points involved in a particular request, the latency contributed by each of these touch points and the concurrency of service invocations.

Application Topology Discovery

The collection of distributed transaction tracing data has another nice property or side effect; that is the ability to dynamically build the application topology based on the way in which services communicate with one another. Below we see an example of such a visualisation as presented by AppDynamics.

This ‘Flow Map’ is able to present a considerable amount of information in a single view. With this type of view (note: not necessarily this particular view) I’m able to see any of the following:

The services involved in responding to a request.
The latency within those services.
The latency between those services.
Variation in latency within service clusters.

Comparing how this view evolves over time I might also be able to see:

New service instances servicing requests.
Service instances not servicing requests.

Automatic Entry and Exit Detection

Distributed tracing has a notion of entry and exit detection. This is the ability to start and stop a transaction trace. The easiest way to think about this is to think about a single method trace. If I want to trace a method it is reasonable for me to expect that I must specify the fully qualified path to this method to allow the tracing tool to instrument the code (either at compile time or runtime). If this method is well known such as the JMS listener onMessage(…) method I would expect the tracing tool to do this automatically. The trace would start on method entry and stop on method exit. Expand this to consider distributed tracing and we have automatic entry and exit detection.

Turns out that identifying what to trace has quite a big overhead in a heterogeneous environment and so this automatic detection takes a significant burden off your development and operations teams. If you also consider a dynamic system with releases every day, week or month it does not take long to incur considerable spend just updating your monitoring tools.

You also want to be able to define the entry and exit detection manually though for situations where the entry and exit points are not well known.

Lossless or Aggregated Metric Collection

There is still a debate over collecting and retaining all available metrics as opposed to sampling and/or aggregating metrics. The trade-offs are quite obvious.

When sampling and/or aggregating:

The infrastructure requirements for network and storage are reduced.
Heuristics are executed on the system being observed so there may be an increased compute (CPU and memory) overhead.
When aggregating, segmentation of data is limited.
Data is lost and so rare events may not be recorded.

The point on segmentation of data is worth expanding upon. Lets consider a user making a request from their browser to a backend service. The request is tagged on the web server with a unique identifier. The APM agent then tags this request with the user identifier, browser, location etc. Each of these tags allows the tracing data to be viewed along a different dimension. If we want aggregation to result in reduced bandwidth utilisation we must aggregate on the client which in turn increases the compute overhead on the client. To limit this, you limit the amount of segmentation of the metrics data.

Limiting the segmentation of data is ok if we are creating operational dashboards from this data but becomes limiting when creating business dashboards. I’m not in any way saying this is a bad thing but that it is a trade off.

When collecting all available metrics:

The infrastructure requirements for network and storage are increased.
Overhead on the system being observed is limited to the overhead imposed by transporting the data.
All data is captured and available for extensive analytics external to the system being observed.
All data is captured so even rare event are recorded.

Adaptive Thresholds

We spoke about how automatic entry and exit detection reduces the operational overhead (or TCO in sales speak) of owning and managing a monitoring solution. One of the other duties that go along with deploying monitoring solutions is setting the high and low water marks for alerting. This can be arduous when the system topology or behaviour is changing rapidly.

Some of today’s solutions attempt to take the overhead of setting thresholds away by using heuristics on historical data. The success or otherwise of this approach is therefore based on both the quality of the heuristics and data. It could be argued that it is always better for a human to set these levels as they can have better knowledge of the system and its behaviours. It can also be argued that in reality humans are time poor and often just set watermarks to so arbitrary level that ‘seams reasonable’.

Personally, I would like to be able to set the baseline watermarks using heuristics and manually adjust where necessary. With this approach I don’t have arbitrary settings that ‘seem reasonable’ but I’m also facilitated in changing watermarks when I have more insight than the heuristics and data could possible have.

Auto-Scaling Trigger

I’ve seen auto scaling become increasingly prevalent in both public and private cloud environments (see IaaS: Making a Case for Infrastructure as a Service. Ordinarily auto scaling is triggered by infrastructure metrics like CPU and memory.

How much better would it be to base scaling triggers on the response time or throughput of the service rather than the behaviour of the infrastructure it resides upon. It is certainly a lot more intuitive if I am setting watermarks manually. If I tell you that at the 99th percentile a user receives a response in 100ms then that is quite meaningful without any knowledge of the system. If I tell you that the CPU is at 55% then it is a lot less meaningful. The CPU might be burning cycles in an event loop providing good, adequate or poor response times to the user.

Application Monitoring in the Future

The main thing that I would like to see in APM tools in the future is that they incorporate log aggregation capabilities.

Log Aggregation

If you have implemented APM and log aggregation tooling you’ll know that the infrastructure required by both is remarkably similar. There is typically an agent responsible for streaming data to some form of intermediary. There is then some for of analytics platform as well as a portal that allows you to manage all that goodness. Why or why can’t they be the same thing (I fear someone will actually answer this!)?

Here is a challenge to APM vendors: I often hear Splunk being proposed as a solution for distributed tracing and application monitoring. I’d like to see you get into the log aggregation space because I don’t want to deploy two similar infrastructures that allow me to export runtime data from my applications.