Pipeline Engine

Quick Index

Get Adobe Flash player


Pipeline Engine is a Java-based framework that links sequential activities, human and computer, into a defined process flow and manages how data moves from step to step in that flow based on the results of each step. In most laboratories, some processes (or pipelines) are carried out automatically without any human intervention while others require a person to do manual steps, such as drawing a region of interest. Pipeline Engine facilitates both fully automated and semi-automated workflows. Pipelines can be executed up to a step, then notify a user(s) to perform the manual task, and finally restart the pipeline from the next step. The process flow is defined in am XML document called the pipeline descriptor and the executables are defined in a separate XML document called resource descriptors.

Pipeline engine is a standalone tool. However, when used with XNAT one can:

  • set up project-based workflows with project specific and experiment specific parameters,
  • track a pipeline and send email notifications
  • capture provenance information as the pipeline executes

Download and Installation

When you install XNAT, pipeline engine is setup for you. Pipeline engine is located in data/{XNAT_HOME}/pipeline, henceforth called PIPELINE_HOME. Pipeline engine can be located anywhere on the file system, as long as the PIPELINE_HOME folder has appropriate permissions for the user who launches tomcat and the non-default path to the PIPELINE_HOME should be set using Administer -> More Options -> Default Settings.

If you choose to setup pipeline engine outside XNAT, after downloading the engine, launch


This step will generate the file PIPELINE_HOME/pipeline.config file.

Pipeline Configuration file

PIPELINE_EMAIL_ID = admin@your-place.org
PIPELINE_SMTP_HOST = your.smtp.server
ADMIN_EMAIL = admin@your-place.org

Creating a pipeline

Please refer to the XNAT Pipeline Development Schema developer documentation.

Installing a pipeline

Only site admins can install pipelines in XNAT. Please refer to the Installing Pipelines in XNAT administrator documentation.

Once a pipeline has been installed, any project owner can add it to their project.

Running a pipeline

Pipelines can be executed in two modes viz. standalone mode using <PIPELINE_HOME>/bin/PipelineRunner or by updating XNAT as the pipeline progresses. In order to update XNAT, use PIPELINE_HOME/bin/XnatPipelineLauncher with appropriate parameters.