Skip to main content
Skip table of contents

DICOM resource survey technical overview

For directions on running DICOM resource survey and mitigation, see the user guide.

There is no REST API reference for this functionality. Instead you should open the Swagger page for your XNAT installation and find the section labeled resource-survey-api.

XNAT's DICOM resource survey and mitigation is done in two steps:

  • Surveying consists of grouping all of the files in a particular DICOM resource by SOP class and instance UIDs, calculating the expected name of the file based on the DICOM metadata, and then looking for files that have the same SOP class and instance UID as one or more other files (these are duplicates) or that don't match the expected file name (these are mismatches)
  • Mitigating consists of taking the report generated by surveying, backing up all mismatched and duplicate files, renaming mismatched files to the expected name, and renaming one of the files from duplicated instances to the expected name (if it's not already named that) and removing the rest of the duplicates

This workflow provides the opportunity to run the survey operation and inspect the resulting survey reports before any file is moved, renamed, or deleted so you'll know what files will be changed during mitigation.

Because this functionality touches critical data on the back-end, no files are actually deleted. Instead all mismatched or duplicated files are copied to a safe location in XNAT's cache folder before being restructured in the archive folder. The report generated during mitigation provides detailed information on which files were moved and where. See Mitigating below for more information.

To prevent large sessions from causing performance issues, these tasks are done in the smallest increments possible at the resource level. However, you can create and queue resource survey requests at the project level, as described in Creating and queuing resource survey requests.

This overview includes the following sections:

Survey and mitigation workflow

There are a number of steps in the survey and mitigation lifecycle. The current state of a particular resource survey request is reflected in its status and the possible statuses define this lifecycle:

  • CREATED is the first step and indicates that a resource survey request has been created for a particular resource but no other action has been taken.
  • QUEUED_FOR_SURVEY indicates that the request has been placed on XNAT's processing queue and will be surveyed as soon as a worker process becomes available. Only requests with status CREATED can be transitioned to QUEUED_FOR_SURVEY.
  • SURVEYING indicates that a worker has pulled the request from the queue and is currently surveying the files in the resource folder.  Only requests with status QUEUED_FOR_SURVEY can be transitioned to SURVEYING.
  • DIVERGENT indicates that the survey request was completed and one or more files was found to be mismatched or duplicated.
  • QUEUED_FOR_MITIGATION indicates that the request has been placed on XNAT's processing queue and will be mitigated as soon as a worker process becomes available. Only requests with status DIVERGENT can be transitioned to QUEUED_FOR_MITIGATION.
  • MITIGATING indicates that a worker has pulled the request from the queue and is currently performing mitigation based on the survey report.  Only requests with status QUEUED_FOR_MITIGATION can be transitioned to MITIGATING.
  • CONFORMING indicates that the survey request was completed and/or mitigation was performed and no files were found to be mismatched or duplicated. Requests with status SURVEYING or MITIGATING can be transitioned to CONFORMING.
  • RESOURCE_DELETED indicates that the resource with which the request is associated was deleted before the survey and mitigation workflow was completed.
  • CANCELED indicates that the request was canceled. Survey and mitigation reports for canceled requests should be disregarded, since the cancellation may have affected how these operations were done.
  • ERROR indicates that an error was encountered during processing. Survey and mitigation reports for error requests should be disregarded, since the error may have affected how these operations were done.

The last four statuses–CONFORMING, RESOURCE_DELETED, CANCELED, and ERROR–are considered "closing" statuses: a resource survey request is considered closed if one of these statuses is set and no further action will be taken based on that request. You need to create a new request if you want to re-run the survey and mitigation operations on the same resource. You can not create a new resource survey request for a particular resource unless there are no existing requests for that resource or any existing requests for that resource are closed.

Creating survey requests 

Creating a resource survey request just captures basic information about the resource:

  • The ID of the resource (from xnat_abstractresource.xnat_abstractresource_id)
  • The ID of the project containing the image session to which the resource belongs
  • The ID of the subject associated with the image session
  • The ID of the image session experiment
  • The data type of the image session
  • The ID of the scan with which the resource is associated
  • The URI for the resource catalog
  • The username of the person requesting the survey and mitigation operation
  • The time at which the request was made

At this point the request could simply sit there with no further action taken. In actual implementation, newly created resource survey requests are immediately set to QUEUED_FOR_SURVEY and added to the processing queue.

Surveying resource files 

As mentioned above, surveying is done at the resource level, and specifically on resources with the format DICOM. The first thing the worker does once it's dequeued a request to survey a resource is set the request status to SURVEYING. It then opens the catalog file for that resource. The catalog file is an XML file stored in the root resource folder that contains a list of the files contained in the DICOM resource. To survey a resource, XNAT opens this catalog, goes through the list of files, opens each file and gets the SOP class and instance UIDs, generates the expected file name using the default DICOM file namer. It then looks for files that have names that different from their expected name, as well as files that have the same SOP class and instance UIDs. In the latter case, at least one of the files will not match the expect file name.

Below is an example of a simple catalog XML (note that this has been edited for clarity and is not a fully valid catalog!):

Sample catalog XML

XML 
<cat:DCMCatalog UID="1.2.276.0.7230010.3.1.2.0.31909.1624637609.925.1">
  <cat:entries>
    <cat:entry ID="1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-5-2ypk0a.dcm" UID="1.2.840.113654.2.45.2.119792" URI="1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-5-2ypk0a.dcm" format="DICOM" instanceNumber="5" />
    <cat:entry ID="1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-24-2z4w8x.dcm" UID="1.2.840.113654.2.45.2.122818" URI="1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-24-2z4w8x.dcm" format="DICOM" instanceNumber="24" />
    <cat:entry ID="1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-12-2ylm61.dcm" UID="1.2.840.113654.2.45.2.113207" URI="1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-12-2ylm61.dcm" format="DICOM" instanceNumber="12" />
    <cat:entry ID="1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-14-2z3jcl.dcm" UID="1.2.840.113654.2.45.2.120430" URI="1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-14-2z3jcl.dcm" format="DICOM" instanceNumber="14" />
    <cat:entry ID="1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-2-2z80yc.dcm" UID="1.2.840.113654.2.45.2.127513" URI="1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-2-2z80yc.dcm" format="DICOM" instanceNumber="2" />
    <cat:entry ID="1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-23-2ypfgv.dcm" UID="1.2.840.113654.2.45.2.119159" URI="1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-23-2ypfgv.dcm" format="DICOM" instanceNumber="23" />
    <cat:entry ID="1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-21-2ylma9.dcm" UID="1.2.840.113654.2.45.2.113254" URI="1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-21-2ylma9.dcm" format="DICOM" instanceNumber="21" />
    <cat:entry ID="XNAT_01_01.MR.XNAT_01.1.25.20210315.nnpbiv.dcm" UID="1.2.840.113654.2.45.2.125188" URI="XNAT_01_01.MR.XNAT_01.1.25.20210315.nnpbiv.dcm" format="DICOM" instanceNumber="25" />
    <cat:entry ID="XNAT_01_01.MR.XNAT_01.1.21.20210315.n22d7g.dcm" UID="1.2.840.113654.2.45.2.113254" URI="XNAT_01_01.MR.XNAT_01.1.21.20210315.n22d7g.dcm" format="DICOM" instanceNumber="21" />
    <cat:entry ID="XNAT_01_01.MR.XNAT_01.1.26.20210315.o4uktl.dcm" UID="1.2.840.113654.2.45.2.130566" URI="XNAT_01_01.MR.XNAT_01.1.26.20210315.o4uktl.dcm" format="DICOM" instanceNumber="26" />
    <cat:entry ID="XNAT_01_01.MR.XNAT_01.1.24.20210315.nlvlu4.dcm" UID="1.2.840.113654.2.45.2.122818" URI="XNAT_01_01.MR.XNAT_01.1.24.20210315.nlvlu4.dcm" format="DICOM" instanceNumber="24" />
  </cat:entries>
  <cat:dimensions x="256" y="256" z="1"/>
  <cat:voxelRes x="0.9375" y="0.9375" z="8.0"/>
  <cat:orientation>Cor</cat:orientation>
</cat:DCMCatalog>

Notice that the ID and URI attributes for the <cat:entry> elements look quite different. This indicates that some of these files are potential mismatches. Generally, the files whose names comprise primarily digits and periods (e.g. 1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-5-2ypk0a.dcm) are correctly named, meaning that those files that start with XNAT_01_01.MR... are probably mismatches.

More subtly, notice that a couple of these files have the same UID (which indicates the SOP instance UID for the DICOM instance contained in the file):

  • 1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-21-2ylma9.dcm and XNAT_01_01.MR.XNAT_01.1.21.20210315.n22d7g.dcm have SOP instance UID 1.2.840.113654.2.45.2.113254
  • 1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-24-2z4w8x.dcm and XNAT_01_01.MR.XNAT_01.1.24.20210315.nlvlu4.dcm have SOP instance UID 1.2.840.113654.2.45.2.122818

This indicates that those files are probably duplicates of the same DICOM instance.

Sample survey report

JS 
{
  "resourceSurveyRequestId": 1,
  "surveyDate": 1675125126617,
  "totalEntries": 34,
  "totalUids": 9,
  "totalBadFiles": 0,
  "totalMismatchedFiles": 2,
  "totalDuplicates": 2,
  "uids": {
    "1.2.840.10008.5.1.4.1.1.4": [
      "1.2.840.113654.2.45.2.113207",
      "1.2.840.113654.2.45.2.113254",
      "1.2.840.113654.2.45.2.119159",
      "1.2.840.113654.2.45.2.119792",
      "1.2.840.113654.2.45.2.120430",
      "1.2.840.113654.2.45.2.122818",
      "1.2.840.113654.2.45.2.125188",
      "1.2.840.113654.2.45.2.127513",
      "1.2.840.113654.2.45.2.130566"
    ]
  },
  "mismatchedFiles": {
    "/data/xnat/archive/XNAT_01/arc001/XNAT_01_01_MR_01/SCANS/1/DICOM/XNAT_01_01.MR.XNAT_01.1.26.20210315.o4uktl.dcm": "1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-26-2zncre.dcm",
    "/data/xnat/archive/XNAT_01/arc001/XNAT_01_01_MR_01/SCANS/1/DICOM/XNAT_01_01.MR.XNAT_01.1.25.20210315.nnpbiv.dcm": "1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-25-2z6o6o.dcm"
  },
  "duplicates": {
    "1.2.840.10008.5.1.4.1.1.4": {
      "1.2.840.113654.2.45.2.113254": {
        "/data/xnat/archive/XNAT_01/arc001/XNAT_01_01_MR_01/SCANS/1/DICOM/1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-21-2ylma9.dcm": "1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-21-2ylma9.dcm",
        "/data/xnat/archive/XNAT_01/arc001/XNAT_01_01_MR_01/SCANS/1/DICOM/XNAT_01_01.MR.XNAT_01.1.21.20210315.n22d7g.dcm": "1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-21-2ylma9.dcm"
      },
      "1.2.840.113654.2.45.2.122818": {
        "/data/xnat/archive/XNAT_01/arc001/XNAT_01_01_MR_01/SCANS/1/DICOM/XNAT_01_01.MR.XNAT_01.1.24.20210315.nlvlu4.dcm": "1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-24-2z4w8x.dcm",
        "/data/xnat/archive/XNAT_01/arc001/XNAT_01_01_MR_01/SCANS/1/DICOM/1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-24-2z4w8x.dcm": "1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-24-2z4w8x.dcm"
      }
    }
  }
}

Once a resource folder has been surveyed, the request itself is given a status based on the results of the survey:

  • If everything is in order–i.e. all files matching the expected names with no duplicated SOP class and instance UIDs–the request status is set to CONFORMING

  • If any files are out of order–i.e. don't match the expected name or have duplicate SOP class and instance UIDs–the request status is set to DIVERGENT

Note that the survey report is saved as part of the originating resource survey request, not as a separate entity.

Another list of files that may appear in your survey report is badFiles. This will probably be fairly rare, but provides a way to let you know about files that the survey operation can't parse as DICOM. Note that, if a resource contains one or more bad files but no duplicates or mismatched files, the resource survey report status will be set to CONFORMING, since there are no mitigation operations that can be performed to rectify the issue.

Mitigating mismatched and duplicate instances

Once a resource has been surveyed and determined to be divergent, you can request mitigation for that resource (as noted earlier, only requests with status DIVERGENT can be queued for mitigation). The request status is set to QUEUED_FOR_MITIGATION and pushed to XNAT's processing queue. Once dequeued, the worker sets the request status to MITIGATING and retrieves the request's survey report. The survey report drives the operations performed during mitigation:

  • All existing files identified as mismatched or duplicates are backed up to XNAT's cache space under a folder hierarchy with the structure:
    <root cache path>/<project ID>/<experiment ID>/<request time>/<scan ID>
    The files are copied into that folder in the same relative structure in which they were stored under the resource folder (in practice, DICOM files are almost always stored directly in the top-level resource folder and are similarly backed up into the top-level mitigation backup folder). The exception is when maintainFileHistory is set to true. In that case, the mismatched or duplicated files are stored under the system's history folder (which is a subfolder named .history located under the root cache folder).
  • Mismatched files are renamed to the expected file names
  • Duplicate files are deleted with the exception of the newest file for each DICOM instance: if the remaining file's name doesn't match the expected name, it is renamed to the expected name once the other files have been removed

Once mitigation is completed, the worker creates a mitigation report, which includes:

  • The ID of the resource survey request
  • The top-level cache folder where files were backed up (unless maintainFileHistory is set to true, as noted above)
  • A map of files that were moved within the resource 
  • A map of files that were removed from the resource along with the path to where the file was backed up

The sample mitigation report below shows how this might look in practice.

Sample mitigation report

XML 
{
  "resourceSurveyRequestId": 1,
  "cachePath": "/data/xnat/cache/XNAT_01/XNAT_E00001/1675125126441/2",
  "movedFiles": {
    "/data/xnat/archive/XNAT_01/arc001/XNAT_01_01_MR_01/SCANS/1/DICOM/XNAT_01_01.MR.XNAT_01.1.24.20210315.nlvlu4.dcm": "/data/xnat/archive/XNAT_01/arc001/XNAT_01_01_MR_01/SCANS/1/DICOM/1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-24-2z4w8x.dcm",
    "/data/xnat/archive/XNAT_01/arc001/XNAT_01_01_MR_01/SCANS/1/DICOM/XNAT_01_01.MR.XNAT_01.1.29.20210315.noiyo6.dcm": "/data/xnat/archive/XNAT_01/arc001/XNAT_01_01_MR_01/SCANS/1/DICOM/1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-29-2z7h0z.dcm",
    "/data/xnat/archive/XNAT_01/arc001/XNAT_01_01_MR_01/SCANS/1/DICOM/XNAT_01_01.MR.XNAT_01.1.21.20210315.n22d7g.dcm": "/data/xnat/archive/XNAT_01/arc001/XNAT_01_01_MR_01/SCANS/1/DICOM/1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-21-2ylma9.dcm"
  },
  "removedFiles": {
    "/data/xnat/archive/XNAT_01/arc001/XNAT_01_01_MR_01/SCANS/1/DICOM/XNAT_01_01.MR.XNAT_01.1.21.20210315.n22d7g.dcm": "/data/xnat/cache/XNAT_01/XNAT_E00001/1675125126441/2/XNAT_01_01.MR.XNAT_01.1.21.20210315.n22d7g.dcm",
    "/data/xnat/archive/XNAT_01/arc001/XNAT_01_01_MR_01/SCANS/1/DICOM/1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-24-2z4w8x.dcm": "/data/xnat/cache/XNAT_01/XNAT_E00001/1675125126441/2/1.2.276.0.7230010.3.1.2.0.31909.1624637609.925-1-24-2z4w8x.dcm",
    "/data/xnat/archive/XNAT_01/arc001/XNAT_01_01_MR_01/SCANS/1/DICOM/XNAT_01_01.MR.XNAT_01.1.20.20210315.ml4ml3.dcm": "/data/xnat/cache/XNAT_01/XNAT_E00001/1675125126441/2/XNAT_01_01.MR.XNAT_01.1.20.20210315.ml4ml3.dcm"
  },
  "totalMovedFiles": 3,
  "totalRemovedFiles": 3,
  "totalFileErrors": 0
}

Sanitizing survey and mitigation reports 

One of the important reasons for implementing the DICOM resource survey and mitigation functionality is that, in some cases, the paths and names of files might include private or other sensitive information such as subject IDs or the like. The mitigation operation should take care of any possible issues with the files stored in the XNAT archive, but the paths and names of the files are stored and preserved in the survey and mitigation reports that include the starting paths and names. If you notice that these reports contain potentially sensitive information while reviewing them, use the sanitizing function to clean the reports. Sanitizing survey and mitigation reports retains the aggregate information from the reports–cache path, total number of duplicates and mismatches, etc.–but clears all properties that include file paths and other system information.

Sanitizing a report that has a survey report and is set to DIVERGENT but has not had mitigation run on it yet will have the survey report sanitized and its status set to CANCELED.


To illustrate what sanitized reports look like, look at the sample survey and mitigation reports above. If you have a request with those reports and then sanitize the request, you'd end up with something like the reports below:

Sample sanitized survey and mitigation reports

JS 
{
  "resourceSurveyRequestId": 1,
  "surveyDate": 1675125126617,
  "totalEntries": 34,
  "totalUids": 9,
  "totalBadFiles": 0,
  "totalMismatchedFiles": 2,
  "totalDuplicates": 2
}
{
  "resourceSurveyRequestId": 1,
  "cachePath": "/data/xnat/cache/XNAT_01/XNAT_E00001/1675125126441/2",
  "totalMovedFiles": 3,
  "totalRemovedFiles": 3,
  "totalFileErrors": 0
}
JavaScript errors detected

Please note, these errors can depend on your browser setup.

If this problem persists, please contact our support.

Contact Support Close