XNAT Code Structure

XNAT is a large, open-source project with an increasing ecosystem of tools that build on top of it. This page will describe this ecosystem, then focus on describing, in detail, the "core" pieces of XNAT.

XNAT Ecosystem

To an end user, "XNAT" can refer to a collections of tools that allow for archiving and analyzing data. But from an administrator and developer's point of view, it is important to understand the borders between tools and their interactions. Many exciting possibilities have been opened up by the introduction of the Using the XNAT REST API in XNAT 1.4, which has resulted in an increase in the number of XNAT-related tools.

At its core, XNAT is a customizable web interface and API for storing and interacting with study data. XNAT depends upon XFT and XDAT, which are lower level layers responsible for translating between various data formats as well as securing and securing that data. The XNAT codebase is merged with site-specific customizations and deployed into a Apache Tomcat web container. Also siting inside the Tomcat web container is the Pipeline Engine and the xnatfs WebDav server. The Pipeline Engine handles distributing jobs to a grid as well as performing important asynchronous tasks such as moving data from the pre
archive to the archive and generating snapshots of imaging sessions. xnatfs forms a wrapper around XNAT's REST API that translates requests into the WebDav protocol, allowing client machines to mount XNAT much like a shared drive or FTP site is mounted.

Before XNAT 1.5, we also deployed DicomServer, a DICOM C-STORE service class provider (SCP). Starting with version 1.5, XNAT includes an integrated DICOM C-STORE SCP. This service can receive data from any DICOM C-STORE Service Class User (SCU), including scanners or DICOM viewers such as OsiriX or DicomBrowser. This capability is configured on the Default Settings page; some advanced features rely on configuration files in the webapp. DicomBrowser allows modification of DICOM headers and the submission of DICOM data to endpoints including DicomServer and the integrated DICOM C-STORE SCP.

The XNAT REST API exposes the XNAT imaging and related data in an easily-consumed format and allows powerful interactions with that data. The REST API makes it easy to write shell scripts that download or upload data to XNAT using curl or the XNAT Rest Client. Programs can also take advantage of the REST API using any standard HTTP library, including HttpURLConnection, HttpClient, httplib2, libcurl, and many others.

There are several excellent examples of tools that take advantage of the REST API, including XNAT DICOM Gateway, XNAT Desktop, and PyXNAT. XNAT DICOM Gateway exposes XNAT as a query-able DICOM resource, which allows clients like OsiriX to query XNAT. (Related: XNAT Gateway Setup for Osirix). XNAT Desktop allows local management of files using a rich graphical user interface. XNAT Desktop can submit files to the traditional XNAT web application using the REST API. PyXNAT, developed for use by NiPype, creates a Python representation of XNAT data by wrapping the REST API.

(Original file: xnat_parts.svg)

"Core" XNAT

The XNAT codebase is divided into four sections (XFT, XDAT, XNAT, & Customizations).

  1. XFT: eXtensible Formatting Toolkit
  2. XDAT: eXtensible Data Archive Toolkit
  3. XNAT: eXtensible Neuroimaging Archive Toolkit
  4. Customizations: Built by the site administrator for site specific customizations.

XFT: eXtensible Formatting Toolkit

Namespace: org.nrg.xft
Location: bitbucket.org/nrg/xdat_1_5, built into xdat-1.jar

XFT is the Java code which translates an XML Schema into other formats. It represents XML Schema in a Java object structure and uses that loaded structure to generate database structures, Java classes, & Javascript files. Given an XML document (or spreadsheet, database, etc) which describes the instance of some data, XFT can translate that data into any of the other supported structures. It does this translation using a generic item structure as an intermediary. XFT can load data from any supported format into this item structure and then write it out to any other format.

The intermediary item structure (org.nrg.xft.XFTItem) was created to support runtime translation of new XML Schema. The need to support new XML Schema at runtime, made the use of common XML parsing tools (which rely on the generation of Java source files and subsequent compilation) problematic. Instead, a generic structure which could process any XML document (without required java compilation) was built. In practice, the need to support new XML Schema at runtime has not been required. However, we still anticipate this as a possible need in the future. The core of the Item structure is the org.nrg.xft.XFTItem class. Its primary feature is a Hashtable which contains all of the properties for a data object (including other items), and some specification describing the type of object it represents. XFT uses its loaded XML Schema structure to identify the possible elements and fields a data object could contain. It uses this knowledge to populate and translate data objects through the XFTItem structure.

XFT generates SQL files which can be used to create a database structure for the loaded XML Schema. Once created, XFT can interact with this database to store, search and re-populate data objects from the Item Structure. It also creates a collection of Database functions for the easy manipulation of the database.

XFT generates Java files which can be used by developers to deal with the item structure through a more familiar business object API. These classes (OMs) correspond to the structure identified in the XML Schema and the generated database. They are essentially wrappers of the Item structure which provide convenient methods for interacting with data objects. They are a conveniece for developers, but are not required (in order to support runtime loading of additional schemas without additional compliation).

XFT generates Javascript files which can be used by developers to interact with data within a web browser. There is a supplied Javascript API for populating the objects from XML and translating instances back to XML.

XDAT: eXtensible Data Archive Toolkit

Namespace: org.nrg.xdat
Location: bitbucket.org/nrg/xdat_1_5, built into xdat-1.jar
Additional Files: XML, VM, JS, etc. in $XNAT_HOME/plugin-resources/webapp/xdat

XDAT is an addon to the XFT structure which provides functionality for using the features of XFT within a web application. It is dependent on the XFT structure and is bundled with the XFT API in a jar. The key features which XDAT provides are:

  • Web application structure (based on Apache Turbine)
  • Administrative structure
  • Security structure
  • Search infrastructure

XDAT uses Apache Turbine to provide a robust web application structure. It provides navigation structures as well as default reports and edit pages for core types defined in the XML Schema. It provides a rich set of Actions and Screens (from the Turbine model) for dealing with stored data in Model-View-Controller format.

XDAT defines an administrative structure (which is built off of the security.xsd). This structure allows administrators to define user accounts (org.nrg.xdat.security.XDATUser) and to indentify which elements in the schema should be browseable within the web application (org.nrg.xdat.security.ElementSecurity).

XDAT provides a structure for securing access to the data stored within XFT. The security structure functions by determining access priviledges to a data object based on some predefined field within that object (identified via XPATH style notation). So, if I have a schema which defines a data type called subject (which in turn defines properties like project, ID, age, gender, etc), I can secure access to stored subject data based on any field within that data type. So I can say that specific users (or groups of users) can access subjects with a project equal to MY_PROJECT (subject/project=MYPROJECT). Once I make this spefication, users without this permission will not be able to access data stored for this subject from the database. Understanding Data Sharing in XNAT's Security Structure.

XDAT also provides a powerful search structure for querying data (stored and derived) from the database. One complication which arises from the use of XML Schema to define your data model, is that there is no inherent way to define derived data structures. XDAT allows administrators to define display documents which will identify the key fields which users want to make available for their data types. These can include fields defined in the schema, combinations and re-formattings of those fields, or fields defined in custom SQL Views. These display documents are then used to display and search listings of data stored in the web application.

XNAT: eXtensible Neuroimaging Archive Toolkit

Namespace: org.nrg.xnat
Location: bitbucket.org/nrg/xnat_builder_1_5
Java Files: $XNAT_HOME/plugin-resources/webapp/xnat/src/java
Additional Files: XML, VM, JS, etc. in $XNAT_HOME/plugin-resources/webapp/xnat

XNAT is a customized version of XDAT (and XFT) which is tailored to the Neuroimaging community. It provides a schema (xnat.xsd) which describes the common neuroimaging structures. It also provides a rich set of features which are customized to that xnat.xsd structure incuding:

  • XNAT specific security structure
  • Image upload/download support
  • More comprehensive views of image related data

XNAT provides a specific security structure which is based on the XNAT Schema (xnat.xsd). It is preset to configure access to data based on the project structure defined here.

XNAT provides support for upload and downloading image data. XNAT is specifically geared to the support of DICOM data. More information here.

XNAT also includes pre defined actions and screens which display data stored in the XNAT structure in a more comprehensible format. This includes reports and edit pages for the project, subject, and imaging session levels. It additionally includes an integrated image viewer for the display of stored image data.

Customizations: Built by the site administrator for site specific customizations.

Files: $XNAT_HOME/projects/$YOUR_PROJECT

This is the content which has been added by individual site administrators. It can be as simple as customized versions of XNAT pages, or as complicated as new schema and all of their subsequent customizations (display documents, java files, VM templates).