Page tree
Skip to end of metadata
Go to start of metadata

PIPELINE_HOME is a shortcode referred to in this documentation. It stands for the location of your pipeline directory in the XNAT web app file system. This can be set in XNAT File System Settings. By default, this location is /data/home/xnat/pipeline. XNAT comes preinstalled with a series of example pipelines that can be found in this folder. Some of these example pipelines are mentioned in this documentation.

 

Creating a pipeline involves:

  • Installing the package/executable(s) that will be executed by the pipeline
  • Creating the pipeline descriptor
  • Creating resource descriptors
  • Optional - creating velocity template file, creating screen and action class
  • Modifying/creating report page(s) for the results of a pipeline

We recommend that pipelines and resource descriptors be placed in a separate folder within PIPELINE_HOME/catalog

XPATH and Pipeline

As pipeline engine uses XML documents, one can use the power of XPATH expressions to navigate through the various elements and attributes. One can thus set parameters using XPATH statements. A string contained within the caret symbol (^) is treated as XPATH expression and the engine resolves all such expressions before executing the steps.

Pipeline Schema

Pipelines are defined using a pipeline descriptor and resource descriptors. A resource descriptor describes an executable. An executable is identified by its name, its location and the arguments that it takes.

Pipeline Descriptor

PIPELINE_HOME/sample_pipelines/SampleAutoRunPipeline.xml is an instance of a pipeline descriptor.

Schema Representation of Pipeline Element

 

Element Name

Purpose

name

Name of the pipeline

description

A short description of the pipeline which is displayed to the user.

resourceRequirements

Name, Value pair of requirements for running the pipeline. This is used while scheduling jobs on the grid. Eg:

<resourceRequirements>
   <property name="DRMAA_JobTemplate_JobResource">-l arch=lx24-amd64,mem_free=1.9G</property>
</resourceRequirements>

documentation

Use this element to inform the XNAT users' about the pipeline. Set the parameters that the pipeline needs using the input-parameters. The parameter can be specified as a XPATH using schemalink or a comma separated list using csv.

xnatInfo

Set the datatype that the pipeline is applicable to using the appliesTo attribute and set the datatypes that the pipeline will create using generatesElements. E.g.

<xnatInfo appliesTo="xnat:mrSessionData">
     <generatesElements>
           <element>fs:aparcRegionAnalysis</element>
           <element>fs:asegRegionAnalysis</element>
     </generatesElements>
</xnatInfo>

outputFileNamePrefix

Pipeline engine captures the STDOUT and STDERROR when the pipeline executes. Use this element to specify the file path prefix. STDOUT will be created as .log and error as .err.

loop

A pipeline step can be executed for a list of values. Create such a list using loop element. For example:

<loop id="mpragescans" xpath="^/Pipeline/parameters/parameter[name='mprs']/values/list^"/>

mpragescans identifies the list of values specified as the xpath statement. Note that the xpath statement is contained within ^ symbol.

Using PIPELINE_LOOPON(mpragescans) will mimic a for loop over the values of the mpragescans.

Using PIPELINE_LOOPVALUE(mpragescans) will result in all values of mpragescans

For example:

<step id="0" description="Prepare Folder Structure"
workdirectory="^/Pipeline/parameters/parameter[name='workdir']/values/unique/text()^">
    <resource name="mkdir" location="commandlineTools">
         <argument id="p"/>
         <argument id="dirname">
              <value>^concat('RAW/',PIPELINE_LOOPON(mpragescans))^</value>
          </argument>
     </resource>
</step>

parameters

Use this element to specify the parameters to the pipeline inline.

steps

Use this element to specify the ordered sequence of steps that the pipeline engine should execute. A step results in call to possibly multiple executables.

step

Each step is identified by its ID attribute.

Attributes:
precondition [CONDITION]: The step is executed only if the precondition evaluates to true
workdirectory [PATH]: The directory within which the executables will be invoked
gotoStepId [STEP ID]: Like a GOTO statement, can be used to bypass the ordered sequence of steps.
awaitApprovalToProceed [true|false]: Setting this attribute to true will result in pipeline engine terminating execution. This is like a pause. The step at which the pipeline can be restarted is identified using the ID attribute.
continueOnFailure [true|false]: The pipeline engine stops executing when it encounters an exit status of non-zero value. If you want to override this behavior, set continueOnFailure=true 

step/resource

This sequence of resources specifies the ordered collection of tasks to be done. A task may be an email, executing a script etc. Each task is defined in a resource descriptor.

Attributes:
name: the name of the resource descriptor
location: path to the resource descriptor. A resource descriptor is identified using location/name. NOTE: the location attribute does not refer to the location of the executable, it refers to the location of the XML which describes the executable. The location when not absolute is relative to the PIPELINE_CATALOG_ROOT_PATH property in pipeline.config file.
ssh2*: One can execute a task remotely using the ssh2 credentials set using ssh2Host, ssh2User,ssh2Password and ssh2Identity. The data is not copied on the remote host, the assumption here is that the folder in which the data is present is mounted on both hosts - remote and the host on which the pipeline engine is executing.
pipeId: A sequence of executables can be chained within a step using the pipeId string.

step/pipelet

A pipelet is a pipeline. One can string together pipelines to create a new pipeline. Parameters are passed to the pipelet.

step/output

This element defines a collection of files which a step may create.

 

Resource Descriptor

An executable is invoked with appropriate arguments. A resource descriptor defines the executable - its location, its arguments, its output

PIPELINE_HOME/catalog/ant-tools/AntCopy.xml is an instance of a resource descriptor.

Schema Representation of Resource Element


 

Element Name

Purpose

name

Name of the executable

location

path to the executable

commandPrefix

prefix to be used before invoking executable at location/name.

input/argument

name - the argument name as used by the executable
value - value of the argument

Attributes:
id - This is the arguments ID
prefix - The prefix to be used. E.g. "-" or "--" or "/". The default value is "-"

nospace [true|false] - This attribute specifies if a space character should be present between an argument and its value
isSensitive [true|false] - This attribute when set to true, is masked in all log files

 

Specifying Parameters for a Pipeline (Parameter Descriptor)

Input parameters can be specified inline within the pipeline descriptor document or on the command prompt or as a parameter file. Specifying the parameters inline on a production pipeline is rare as parameters change with the project/experiment.

PIPELINE_HOME/sample_pipelines/Parameters.xml is an instance of a parameter descriptor document.