XNAT Configuration Framework

This page is mostly a specification for the configuration framework, but is under construction to be more readable. You can find tips about creating and accessing configurations via the XNAT REST API at the bottom of this page.

nrg_config Framework

Description

nrg_config is a service that provides generic configuration text file versioning and persistence using Spring/Hibernate. The service is integrated with XNAT and make is accessible via REST.

Scope

This project includes one new framework and changes to XDAT and XNAT builder. Specifically, the nrg_config framework was created to provide a generic configuration service. XNAT's REST interface was extended with new URI's and a Restlet resource called "ConfigResource" was created. Small changes to XDAT were necessary to integrate the configuration service with XNAT.

Specs

  • A configuration is simply a text string plus some metadata. This project will not support any type of binary configuration
  • Configuration text strings can be saved by doing a PUT to an almost arbitrary URI path (See REST API spec below for examples) Place the config string in the body, content type is text/plain; charset=UTF-8 (text/xml has been tested and seems to work, too.)
  • Doing a GET to the same path will return the Configuration. A Configuration is an object (represented as JSON, XML, or HTML) that includes the uploaded script plus metadata.
  • Configurations can be "tool" and optionally, Project specific. Tool is an arbitrary text string used for general (almost namespace) organization. Project is the XNAT ProjectID to provide project specific configurations for the same tool/path combination. Configurations with no ProjectID could be considered site-wide configurations.
  • Saving a new text string to a URI that already has a configuration effectively creates a new version of the configuration. The most recent upload is the "current" configuration. Each Configuration gets a "version" number which is an incrementing integer.
  • nrg_config provides metadata for each configuration and supports version history.
  • status - enabled/disabled
  • path - location of the file
  • project - projectID to which this configuration belongs
  • xnatUser - userID of the person who saved/updated/disabled the configuration
  • reason - text field to store the "why" for history/audit
  • tool - the tool-name to which this configuration belongs
  • created - the date/time the configuration was uploaded
  • version - an int that increments (starts at 1) each time you call replaceConfig.
  • contents - the actual configuration string that was uploaded.
  • Configurations (text plus metadata) can be retrieved with a GET on the same path they were created.
  • A GET with no querystring parameters returns the Configuration object which includes metadata and contents.
  • A GET with a version=X parameter will retrieve the configuration for that path whose version number matches X
  • A GET with contents=true will return only the most current string that was uploaded
  • A GET with meta=true will return only the metadata associated with the most recently uploaded configuration.
  • A GET with action=getHistory will return a set of all configurations that have been saved to that URI ordered by createdate. This setting ignores meta and contents parameters.
  • A PUT with status=enabled or status=disabled will set the status field for the configuration.
  • Configurations are never deleted. In fact, the back end DAO does not even support delete or update. Instead, we support disable and enable methods which set "status" in the configuration's metadata. Enable/disable actions are tracked. It is up to the client to use the status appropriately. nrg_config does not act differently depending upon status.
  • If a configuration does not exist for a given path, nrg_config will return a 404 error for each request (history, getScript, tools, etc.).
  • Get History returns all metadata and content for all versions of the configuration. This effectively provides who/what/when/why for each version of the configuration.
  • Project level security is supported for configurations stored at the project level. The resource checks user.canEdit(project) on both PUT and GET. Not sure that is conceptually correct.
  • When a configuration is added, it automatically set to enabled.
  • Path, Tool and Project is case sensitive.
  • Maximum configuration contents (the string)'s length is defined by ConfigService's static member, MAX_FILE_LENGTH.
  • for now postgres limits varchar to10485760 bites or about 10MB... (see
    http://postgresql.1045698.n5.nabble.com/Maximum-size-for-char-or-varchar-with-limit-td3297362.html
    )
  • if this is too small, change Configuration.java's contents annotation to type=text (see comments in source file for correct syntax)
  • No duplicate configuration contents are stored in the database. if the contents of a configuration file does not change, do not add it to the database. In other words, there are no duplicate scripts in the database for a given tool, path, project. However, we do not share configurations between tools and paths. So, if you add the same script to two different tools, you will have a duplicate script in the database.
  • There is no support in the REST interface to set the "reason" metadata.
  • A user of this system need not use the REST interface. They may use the ConfigService API directly. They will never use the DAO's used by the service. Do note that if you use the ConfigService API, you have to supply projectID as a Long. Currently, that is implemented in the REST interface as the projectdata_info property on XnatProjectdata

REST API

Site-wide configurations

/REST/config

GET

return a list of Strings or an empty set. Each string is the name of a tool that has a configuration

format={json,xml,html}

/REST/config/{tool-name}

GET

return a list of Configurations or an empty set. This list includes all Configurations for the tool at all paths. It includes the contents, meta-data, and history.

format={json,xml,html}

/REST/config
/{tool-name}/{PATH/TO/FILE}

GET

returns one Configuration or an empty set. The Configuration represents the most recent configuration file stored at the supplied path. Use the querystring parameters specified below to control return values.

format={json,xml,html}

/REST/config
/{tool-name}/{PATH/TO/FILE}?action=getHistory

GET

Return a list of Configurations or an empty set. The list includes all versions stored at this path ordered by date added. This returns all metadata and contents for each Configuration.

format={json,xml,html}

/REST/config
/{tool-name}/{PATH/TO/FILE}?contents=true

GET

Return only the contents (no metadata) of the most recent configuration stored at this path as a String. "format" is ignored. If no configuration exists, simply return an empty string. (SHOULD THIS RETURN AN ERROR, INSTEAD?)

format ignored!

/REST/config
/{tool-name}/{PATH/TO/FILE}?meta=true

GET

Returns one Configuration or an empty set. The Configuration includes only the meta data of the most recent configuration stored at this path.

format={json,xml,html}

/REST/config
/{tool-name}/{PATH/TO/FILE}?meta=true&contents=true

GET

same as/REST/config/{tool-name}/{PATH/TO/FILE}

format={json,xml,html}

/REST/config
/{tool-name}/{PATH/TO/FILE}?version={versionID} (contents and meta optional)

GET

Return one Configuration or an empty set. The Configuration returned represents the one configuration that matches this tool, path, and version number. Version numbers can be found in the meta-data for each Configuration. Additional querystring parameters "contents" and "meta" are available. If contents=true this will return the String representation of the Configuration contents. Format is ignored. If meta=true this will return the Configuration with no contents.

format={json,xml,html}

/REST/config
/{tool-name}/{PATH/TO/FILE}

PUT

replace (or insert) a site-wide configuration at the specified path. returns 201 Created on successful creation. Returns 200 OK if you replace an existing configuration. the body of your put will be the contents of the config. meta-data is automatically set as you would expect (xnat user, path, etc.) However, status=enabled and reason=null.

 

/REST/config
/{tool-name}/{PATH/TO/FILE}?status={enabled,disabled}

PUT

update the status for the most current configuration. Returns 200 OK on success.

 

Project-specific configurations

/REST/projects/{PROJECT_ID}/config

GET

Each of these act exactly
as above except they
contain a project ID

/REST/projects/{PROJECT_ID}/config/{tool-name}

GET

 

/REST/projects/{PROJECT_ID}/config/{tool-name}/{PATH/TO/FILE}

GET

 

/REST/projects/{PROJECT_ID}/config/{tool-name}/
{PATH/TO/FILE}?action=getHistory

GET

 

/REST/projects/{PROJECT_ID}/config/{tool-name}/
{PATH/TO/FILE}?contents=true

GET

 

/REST/projects/{PROJECT_ID}/config/{tool-name}/{PATH/TO/FILE}
?meta=true

GET

 

/REST/projects/{PROJECT_ID}/config/{tool-name}/{PATH/TO/FILE}
?meta=true&contents=true

GET

 

/REST/projects/{PROJECT_ID}/config/{tool-name}/{PATH/TO/FILE}
?version={versionID} (contents and meta optional)

GET

 

/REST/projects/{PROJECT_ID}/config/{tool-name}/{PATH/TO/FILE}

PUT

 

/REST/projects/{PROJECT_ID}/config/{tool-name}/{PATH/TO/FILE}?status={enabled,disabled}

PUT

 

Implementation

Configuration Service

  • Spring/Hibernate implementation accessible via 1.6's ContextService
  • Create, Update, Disable, Enable configurations.
  • All configs are simply referenced by a path. The path can be anything after the /REST/conf/{script,status,history}/ uri.
  • store who/what/when/why in version table. who/when/what is user/timestamp/script. why is implemented as reason and will have to be passed in as an optional query param.(Not implemented)
  • simply store the script contents in a varchar field of a table.

Database Table (xnat_configs)

  • ID : (serial) used to uniquely identify this version of the the file
  • tool: (text) used to identify the tool which uses this config file
  • path: (text) used to uniquely identify this config file and associate it with the REST path, {PATH-TO-FILE}
  • project: (text) used to identify the project which owns this config file
  • contents: (text) used to store the contents fo the config file
  • created: (dateTime) - provided by AbstractHibernateEntity
  • status: (text) enabled/disabled.
  • version: (int) incrementing version of this config.
  • xnatUser: (text) user id of user who last modifed this config

Notes/Issues

  1. Why doesn't REST support "reason"?
  2. Shouldn't ?action=getHistory support ?meta=true
  3. There is no RESTful way to get a list of projects that have a configuration. Should there be?
  4. Would be nice to have user level configurations /REST/user/config
  5. Document the hell out of PUT ?status=disabled. Do not ignore the body contents. This will allow a user to PUT a disabled configuration up if they wanted to for some reason.
  6. Add content-type or "format" to the metadata so the user can know what to expect when retrieving a file. Also, make sure it shows up in the HTTP HEAD request.
  7. Would be nice to see all the "current version" configs for a project (all paths) per Lauren.
  8. API should be able to copy configs from one project to another.
  9. Would be nice if the API could support configurations in a "project group"
  10. Compile an inventory of the things that should be moved to this (e.g. validation, pipelines, anonymization) and schedule

CURL Example


curl -X PUT -u user:pass http://localhost:8080/xnat/data/projects/prj001/config/anon/script?inbody=true -d @anon.das

Where anon.das is a text file that contains:

-(0010,1040)

Be sure to append "inbody=true" on the end of the URL. That forces the content type to multipart/form-data.

Using the Configuration Services

API Calls to Access Configurations

String configSiteWide = configService.getConfigContents("toolName", "restpath/to/file");
String configForProj  = configService.getConfigContents("toolName", "restpath/to/file", 1L); // Where 1L is the project ID.

if(config == null){
    // doesn't exist
} else {
    // do your thing
}

Accessing Configurations via REST

A good REST client tool is very handy for working with the XNAT REST APIs during development and sometimes even for production instances. One used by many developers in the XNAT lab is the freely available Wiz Tools REST Client. The instructions in this section provide both general direction for calling the REST APIs, but also some specific instructions for using the Wiz Tools REST Client. You can even use command-line tools like curl to make these calls.

Creating a New Configuration

The REST URL for creating a new configuration is:

PUT http://server/data/config/toolName/path?inbody=true

Where:

  • server is the address to your XNAT server (you may also have a path on the server for /xnat or something similar)
  • toolName is the feature or tool that will use the configuration you're storing
  • path is an arbitrary path on the URL; these paths are used to distinguish multiple configurations for the same tool

You'll need to specify the contents of your configuration in the request body. Note that the format of the configuration contents is completely dependent on the needs of the tool you're configuring. The configuration service itself doesn't really care what those contents are.

To run in Wiz Tools REST Client

In the URL box, enter the URL for your configuration, e.g.

https://myServer/data/config/testConfig/path?inbody=true

On the Method tab, select PUT.
On the Body tab, select String body, then enter your configuration in the box below.
On the Auth tab, select BASIC, then enter your username and password, and make sure the Preemptive checkbox is checked.
Click the >> arrows next to the URL.

That should run and get an HTTP 201 status.

Getting an Existing Configuration

To check the contents of a configuration, you can query the database from psql or pgAdmin3:

select c.tool, c.path, c.status, c.unversioned, c.reason, d.contents from xhbm_configuration c, xhbm_configuration_data d where c.tool = 'toolName' and c.path = 'path' and c.config_data = d.id;

To run in Wiz Tools REST Client

In the URL box, enter the URL for your configuration, e.g.

https://myServer/data/config/testConfig/path

On the Method tab, select GET.
On the Auth tab, select BASIC, then enter username and password, and make sure the Preemptive checkbox is checked.
Click the >> arrows next to the URL.

If you look in the Body tab on the HTTP Response, you’ll see the returned data as JSON.

Enabling and Disabling an Existing Configuration

To enable or disable a configuration, you just need to make a PUT call to the REST URL for the configuration with the query variable status set to the state you want:

PUT http://server/data/config/toolName/path?status=[enabled|disabled]

You can check the configuration again to make sure everything is set properly. You can toggle the enabled and disabled states by calling these successively.

To run in Wiz Tools REST Client

In the URL box, enter:

https://myServer/data/config/testConfig/path?status=disabled

On the Method tab, select PUT.
On the Body tab, select None.
On the Auth tab, select BASIC, then enter username and password, and make sure the Preemptive checkbox is checked.
Click the >> arrows next to the URL.

You should get back a status of 200.

$label.name