XNAT Desktop: tagged upload of non-DICOM data to XNAT

Video:

Use case:

You have a project with lots of non-DICOM files stored on your hard drive, consistently organized in directories, and you want it archived in XNAT. And, you are ready to invest a some time to linking your data to XNAT types using regular expressions.

Model project

Here, we will use a model project named "OIS", which involves optical imaging of mice, and discuss steps for formatting and uploading files of this project to XNAT using XNAT Desktop.
First, we download the sample OIS subject: ftp://ftp.nrg.wustl.edu/pub/xnd/download/OIS_test_data.zip

In this project, directories for each mouse experiment are called CONV1, CONV2, ... . The directory has raw images from the optical imager saved in TIFF files, and analysis results in .mat files:

Consider this hierarchy conversion:

Writing custom naming rule

XNAT Desktop has a built-in algorithm that will execute a series of searches at each directory level according to your instructions, and assign files certain key tags that match specified regular expression patters. XND will further use these tags to put files at the appropriate place in XNAT hierarchy. The instruction set for this algorithm (called rule) is stored in an XML file. For this project, we created the custom rule called naming_ois.xml

Looking at the naming_ois.xml:

CODE

 <folder ID="root_fold" pattern=".*" treeRoot="1">
  <child>subj_dir</child>
 </folder>

Here, we describe the root folder where the rule should be applied.

CODE

 <folder ID="subj_dir" pattern="CONV.*" treeRoot="1">
  <tag name="Subject" pattern=".*" recursive="fixed"/>
  <tag name="Project" value="default" recursive="fixed"/>
  <tag name="Modality" value="CT" recursive="fixed"/>
  <child>auxiliary_file</child>
  <child>tiff_file</child>
 </folder>

The search algorithm will:

pseudo-code

CODE

For each directory at root level with arbitrary name:
	look for immediate subdirectories that start with "CONV"
		for each file in each such subdirectory:
			assign the tag "Subject" with value equal to the name of that directory
			assign the tag "Project" with value "default"
			assing the tag "Modality" with value "CT"

The rest of the XNAT tags (also called XNAT ontology tags) required for each file to be uploaded to XNAT, are assigned by the rest of the rule in the similar way, going down the directory tree and looking for directory and file name patterns.

Tagging and uploading the data

Now, we add naming_rule.xml document to XND's rule collection via View->Preferences->Ontology and rules->Add rule from file. We run the "apply custom rule"->"naming_ois" context menu command on the OIS directory, and switch to Tag View tab:

Then, we run the "upload" context menu command on the "Subject:CONV1" level. The REST calls are tracked in the XND Console:

Then, we check the XNAT experiment online:

It might take several iterations in modifying the rule to achieve desirable results, but then the upload will be automatic for the rest of your project.

Additional information

To upload additional metadata to the newly created session, the recommended method is to prepare spreadsheets that list information at each level for each experiment, and use the Spreadsheet Uploader.
For detailed tutorial for creating naming rule for OIS data, refer to this wiki page: https://flywheelio.atlassian.net/wiki/spaces/XNAT16/pages/2507803056
For an exhaustive manual on XNAT Destkop, refer to this page: full XND manual