Installing Command Definitions in a Kubernetes Environment

In order to launch a container on a compute backend, the Container Service requires an admin user to first define a Command, which is a template for how to run a particular kind of container. However, those Commands are our own invented metadata; they aren’t part of any kind of external standard. As such, the compute backends don’t know anything about Commands and do not provide any tools to store them or retrieve them, like they do for the images which are used to run containers. To help Container Service users obtain the Commands they need to run containers from their images, we came up with a workaround: if the person creating the image writes the Command as a JSON string into the image’s labels, then when the user pulls the image to their compute backend the Container Service can read the labels, find the Command JSON, and define the Commands. That workaround served users well for years, when the only compute backends the Container Service supported were Docker and Docker Swarm. But we have since added Kubernetes as a supported compute backend, where this workaround does not work.

Kubernetes provides no mechanism by which the Container Service can “pull” an image locally, which is a necessary part of the steps of reading the Command JSON from the image labels. Kubernetes manages which images are pulled to each worker node and does not provide any way to inspect these images.

This documentation describes a set of workarounds enabling XNAT administrators to obtain the Command JSON from an image labels so that they can add those Commands into their Container Service.

Step 1: Getting the Command JSON List from the Image Labels

In the first step, we will read the Command JSON list from the image labels. There are two methods presented here, both of which will result in a shell variable $command_json containing the Command JSON list as a string.

That in itself is 90% of the way to the full solution. The rest is all string manipulation and getting that Command string defined in the Container Service, which is discussed in Step 2.

Method 1: Docker on User’s Machine

Required CLI tools: docker

In this method, we simply pull the image down and read the labels using docker CLI tool. This has the advantage of being simplest. But it does require time and disk space to pull the binary image files which likely will never be used to run a container, only to read a small string from the image metadata.

Open a terminal on any machine (there is no requirement that this be your XNAT machine, but it may be). You use the docker CLI in the terminal to pull the image locally and read the labels. Here's an example, where $image is the image you want to pull the labels from, like xnat/dcm2niix:latest.

docker pull $image
command_json=$(docker inspect --format='{{index .Config.Labels "org.nrg.commands"}}' $image)
echo $command_json

That will get you a JSON string containing a list of Commands, stored in the shell variable $command_json.

Method 2: Docker Hub API

CLI tools: an HTTP client (in the example we used curl), jq for manipulating JSON strings. (jq is technically optional, but highly recommended. Manipulating JSON on the CLI is vastly more difficult without it.)

If you know their image is stored on Docker Hub you can use the Docker Hub APIs to read the image labels without pulling the whole image. This is trickier than pulling the image down with the docker CLI, and it only works for sure with Docker Hub, but it has the advantage of working on a machine without docker installed and doesn’t require you to pull down a whole image that you don’t really need.

For this we can’t use the $image string as we had it before, we have to split it into what we will call the "repo" and the "version". Where before we used image=xnat/dcm2niix:latest here we will have im_repo=xnat/dcm2niix and im_version=latest.

The steps are:

1. Obtain a token which you will use to authenticate with Docker Hub. (See later note about authentication.)

2. Obtain the manifest for a particular image version; extract the digest

3. Obtain the image metadata using that digest; extract the Command JSON list from the image labels

token=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${im_repo}:pull" | jq -r '.token')
digest=$(curl -s -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Authorization: Bearer $token" "https://registry-1.docker.io/v2/${im_repo}/manifests/${im_version}" | jq '.config.digest' -r)
command_json=$(curl -s -L -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -H "Authorization: Bearer $token" "https://registry-1.docker.io/v2/${im_repo}/blobs/$digest" | jq -r '.config.Labels["org.nrg.commands"]')
echo $command_json

That will get you a JSON string containing a list of Commands, stored in the shell variable $command_json.

Docker Hub APIs

These API calls were verified to work at the time of writing. However, Docker may change its APIs at any time without warning, which could result in these steps not functioning properly.

Authenticating with Docker Hub

In the example above, we were reading the metadata from a public image. If you are attempting to read the metadata from a private image you may need to authenticate with a username and password when obtaining a token in the first step. However, we have not tested this.

Step 2: Installing the Command JSON in XNAT

Once you have the Command JSON string—the result of either Method 1 or Method 2—it isn't too tough to extract the Commands from that JSON string by hand, paste them into the XNAT UI, etc. etc. So you could choose to do that. But there is some extra convenience we could add under the right circumstances.

If you know that there is only a single Command in the image’s labels—as in the JSON string in $command_json is a list with only one element—you can pull the command out of the string using jq—(which, as before, is not installed on any machine by default but is highly recommended as it will make all these steps vastly easier).

echo $command_json | jq --compact-output '.[0]'

That will print the single command JSON string to your console. If you are on a Mac you can use the built in pbcopy CLI tool to put that string onto your local machine’s clipboard which you could then take over to the XNAT UI for defining commands and paste it in.

echo $command_json | jq --compact-output '.[0]' | pbcopy

Important note with that method, though: the command that you pull from the labels may not have the image field defined. If you’re pasting it into the XNAT UI, you may need to edit the JSON by hand to insert the key "image": "$image" or "image": "$im_repo:$im_version" (depending on what method you used and what variables you have at hand).

The smoothest experience is to instead send that Command string directly to the Container Service API. For this to work you would need to have you XNAT username and password (or an equivalent alias token) on the command line—which I’ll represent as $username and $password respectively—and your XNAT’s URL—which I will represent as $xnat. I’m also using the variable $image in this example, which you have if you used Method 1; if you used Method 2 you can define image=$im_repo:$im_version.

echo $command_json | jq --compact-output '.[0]' | curl -X POST -u $user:$pass --json @- $xnat/xapi/commands\?image=$image

Running that will extract a single Command from the JSON list stored in $command_json and define it as a new Command in the Container Service.

It is possible to do this same thing if you have multiple Commands in the JSON list, but it requires looping over the Commands and making multiple calls to the Container Service API. I leave that as an exercise for the reader.