PIPELINE_HOME is a shortcode referred to in this documentation. It stands for the location of your pipeline directory in the XNAT web app file system. This can be set in Admin UI - File System Settings. By default, this location is
/data/home/xnat/pipeline. XNAT comes preinstalled with a series of example pipelines that can be found in this folder. Some of these example pipelines are mentioned in this documentation.
Creating a pipeline involves:
- Installing the package/executable(s) that will be executed by the pipeline
- Creating the pipeline descriptor
- Creating resource descriptors
- Optional - creating velocity template file, creating screen and action class
- Modifying/creating report page(s) for the results of a pipeline
We recommend that pipelines and resource descriptors be placed in a separate folder within PIPELINE_HOME/catalog
XPATH and Pipeline
As pipeline engine uses XML documents, one can use the power of XPATH expressions to navigate through the various elements and attributes. One can thus set parameters using XPATH statements. A string contained within the caret symbol (^) is treated as XPATH expression and the engine resolves all such expressions before executing the steps.
Pipelines are defined using a pipeline descriptor and resource descriptors. A resource descriptor describes an executable. An executable is identified by its name, its location and the arguments that it takes.
PIPELINE_HOME/sample_pipelines/SampleAutoRunPipeline.xml is an instance of a pipeline descriptor.
Schema Representation of Pipeline Element
Name of the pipeline
A short description of the pipeline which is displayed to the user.
Name, Value pair of requirements for running the pipeline. This is used while scheduling jobs on the grid. Eg:
Use this element to inform the XNAT users' about the pipeline. Set the parameters that the pipeline needs using the input-parameters. The parameter can be specified as a XPATH using schemalink or a comma separated list using csv.
Set the datatype that the pipeline is applicable to using the appliesTo attribute and set the datatypes that the pipeline will create using generatesElements. E.g.
Pipeline engine captures the STDOUT and STDERROR when the pipeline executes. Use this element to specify the file path prefix. STDOUT will be created as .log and error as .err.
A pipeline step can be executed for a list of values. Create such a list using loop element. For example:
Use this element to specify the parameters to the pipeline inline.
Use this element to specify the ordered sequence of steps that the pipeline engine should execute. A step results in call to possibly multiple executables.
Each step is identified by its ID attribute.
This sequence of resources specifies the ordered collection of tasks to be done. A task may be an email, executing a script etc. Each task is defined in a resource descriptor.
A pipelet is a pipeline. One can string together pipelines to create a new pipeline. Parameters are passed to the pipelet.
This element defines a collection of files which a step may create.
An executable is invoked with appropriate arguments. A resource descriptor defines the executable - its location, its arguments, its output
PIPELINE_HOME/catalog/ant-tools/AntCopy.xml is an instance of a resource descriptor.
Schema Representation of Resource Element
Name of the executable
path to the executable
prefix to be used before invoking executable at
Specifying Parameters for a Pipeline (Parameter Descriptor)
Input parameters can be specified inline within the pipeline descriptor document or on the command prompt or as a parameter file. Specifying the parameters inline on a production pipeline is rare as parameters change with the project/experiment.
PIPELINE_HOME/sample_pipelines/Parameters.xml is an instance of a parameter descriptor document.