Container Service
Blog

Blog

2017-03-20 Container Service Scrum
Steve Moore posted on Mar 20, 2017

Target: CS early March 2017

This sprint is intended to end 2017.03.23

Status / Tasks


Flavin

  • CS-130,131 about to complete
  • Next items: CS-132, 134-138
  • CS-117 Add wrapper field to CommandEventMapping: Need to confer with Matt

Mark

  • Away: Holiday
  • Need followup with Flavin to stage Thursday demo
  • Will be out Wednesday

Matt

  • Away

Rick

  • Away: XNAT-Core

Will

  • Away: Spring Break
  • Completed demo item and passed off to Mark before end of 2017.03.17
  • No open items

 

2017-02-13 Container service scrum
Rick Herrick posted on Feb 13, 2017

Flavin

  • Refactored command structure to move XNAT-y stuff off to the back-end part of the command to make it a bit more independent.
  • Figure out way to break out translating XNAT object models into a more general model.
  • Use Google Autovalue library to generate models to work with.
  • Working on project configuration for container services. Adding a wrapper around config service to store and work with commands.

Rick

  • Putting out fires, no work on event services.

 

2017-02-02 Container service events discussion
Rick Herrick posted on Feb 02, 2017

Events are:

  • Noun
  • Verb

Verb is operation:

  • Import (question)
  • Create
  • Retrieve (question)
  • Update
  • Delete
  • Move
  • Archive
  • Merge (question)
  • User request (e.g. user searches for sessions and wants to launch some process on the results)

Noun is event target:

  • Identified by some criteria: ID and data type with hierarchy,1

     data types, etc. 


  • Actor data: user who made the event happen, for example, maybe other info about the event context. These could be in a map of attributes on the event, event acceptance can be calculated based on matching one or more attributes.

Each event will have a payload. How are these defined?

Many "events" aren't discrete points in time, so events also require status: started, completed, failed, etc. Unlike workflows, these aren't persisted as a unique timeline. We may need some way to associate events across an entire operation, but we can revisit later.

Definition of the noun will depend on the particular operation and context and would be defined in the event itself.

Event responders would be the way to wire to a data event. The basic event responder interface would indicate whether an event should be accepted and something about the event payload. Different base implementations for, e.g., handling events by data type, by other attributes.

This really is IFTTT.

An API to get sample events.

Footnotes
Ref Notes
1

The data type hierarchy is the full stack of inheritance. For example, an MR session is:

  • xnat:mrSessionData
  • xnat:imageSessionData
  • xnat:subjectAssessorData
  • xnat:experimentData

You could say you want an xnat:mrSessionData or an xnat:experimentData and the MR session would match either one.

1 Comment  · 
2017-01-24 Meeting Notes
Will Horton posted on Jan 24, 2017

Container service progress review meeting with Dan.

 

Development Progress Review

Progress is indeed being made. 

  • Flavin wants to refactor the structure of command definitions to make them more portable, by lumping all the XNAT-specific stuff into a discrete chunk of JSON. Other applications could ignore it. 
  • How do we want to support command versioning? Dan definitely wants it. 
    • Option 1: Natively support versioning inside XNAT, similar to Protocols. 
    • Option 2: Use an external, already-developed version control system for distributing commands (aka GitHub) and have sites / projects "slurp them up." 
  • What version of "Events" are we dependent on? 
    • ???
    • ?????
    • ????????????
  • Workflows! 
    • Ugh
    • Groan
    • *eye roll*

UI Progress Review

Reviewed these wireframes: https://collaborate.uxpin.com/5ba98d7201c1c221aa36b602c3185675a308b68a#/pages/53451616 

Notes: 

  • Add support for other types of container servers besides Docker. Aka Singularity. (This will be a short list.)
  • Add support for filtering on scan types, session attributes, etc in a project-specific command configuration. 

 

Open Questions: 

  • Should Container Service site admin settings be in their own page? Or should they be bundled in with Site Admin stuff?
    Will: They could probably be bundled in, but this may depend on how a plugin is able to add to the site admin UI. We think the standard practice will be to give each plugin its own page, or perhaps its own section of "Plugin Settings." 
  • Will site admins need to spend much (or any) time administering an image once it is imported? 
  • Should the majority of the Site Admin panel be devoted to administering commands? Yes, probably.  
  • There are two elements of sharing a command between projects: the command and the project-specific command configuration. Will we have a mechanism for sharing both easily? 


 

2017-01-09 Container Service scrum
Rick Herrick posted on Jan 09, 2017

Matt going through hackathon feedback. Looking at Jenny's matlab GUI thing. Work on I/O streaming through Spring.

Flavin knocking out tickets. Find specific Spring REST API 400 issues to figure out debug/troubleshooting steps. Check on XFT init issue.

Rick work on archive streaming image. Document that.

 

2016-12-04 Container service scrum
Rick Herrick posted on Dec 05, 2016

Rick Herrick needs to create XML import service to complete round trip with container services.

Start planning container services "alpha" or validation pass. Kick this off at XNAT developer meeting, but need to bring in real processing workflows to test it.

John Flavin work with Jenny Gurney, Mikhail Milchenko, Tim Brown to find pipelines, etc., suitable for test run configuration.

 

2016-11-18 Container service scrum
Rick Herrick posted on Nov 18, 2016

Previous scrum notes

In last scrum we discussed not having a UI for container service functionality. It sounds like Dan wants to have UI, so we may need to re-visit this:

  • We have existing wireframes, but they're a bit old. They may need to be revised and definitely need to be extended.
  • John Flavin and Will Horton will work together to define all UI tasks:
    • What wireframes do we have that are fine?
    • What wireframes do we have that need to be revised?
    • What wireframes do we not have?
  • Generate JIRA issues, specifications, and documentation for UI tasks.
  • Need the resource service and XML import service from Rick Herrick.
  • Blocked by xdat bug that breaks assigned roles. May email Tim Olsen and Jenny Gurney. But export to OVA and re-build Vagrant box.
  • John Flavin created a configuration that pulls in 3rd-party jars that aren't already in XNAT, but doesn't re-include jars in XNAT.

Matt has no real change on status. Hasn't started restart controller. Need to define all tasks required to hit end of sprint 1.

Goal

Beta features:

  • Do we need UI for beta or is REST control enough?
  • Need to define goals for beta test.

Action Items

  • Matt Kelsey define tasks for finishing up API development on container status, history, and logging. Pull those into sprint 1.
  • Start knocking those out.

Flavin

Rick

  • Start sprint.
  • Email Dan about recruiting beta participants and features.
  • Finish resource and XML import services.

 

2016-11-02 Container service status
Rick Herrick posted on Nov 02, 2016

Container service scrum board

Existing scrum board (Closed during issue migration from Pipeline project to Container Service project)

Remaining items:

  • Rick
    • Resource service  XNAT-4548 - Getting issue details... STATUS
    • XML service  XNAT-4556 - Getting issue details... STATUS
  • Matt:
    • Monitoring and managing container status: running, failed, logging.
    • History and auditing.
    • About 1/2 way through, 2/3 done, none tested.
  • Flavin:
    • There are JIRA issues in the pipeline container service project. 
    • Handling container output (waiting on Rick's XNAT tickets)
      • Resources  CS-20 - Getting issue details... STATUS
      • XML, especially assessors  CS-29 - Getting issue details... STATUS
      • Scans (probably not in the beta)  CS-8 - Getting issue details... STATUS
    • Created FreeSurfer command for practical testing of container services command resolver. Output of this isn't tested yet (see above).
    • Migrating modules to plugins so that we have data types, UI, etc.  CS-35 - Getting issue details... STATUS

Functional and integration testing is an open issue.

Beta program for container services: recruit users to adapt pipeline or processing tasks to use containers.

Requirements for Beta:

  • Figure out path forward for existing pipelines. Convert each to a container, or wrap them all inside a container
    • Migrate XML pipeline descriptors to container services commands: this may not even require "migrating" the descriptor, if we can feed the descriptor to a translator that turns into a command. Maybe automated, maybe not.
    • OR Write a guide that tells XNAT admins how to
      • create a docker image that contains their pipeline engine + deployed pipelines + dependencies, and 
      • Write commands for each pipeline

Future tasks:

  • No UI components for container services beta, just REST API. We can get UI designs from Will and contract out to UAR for implementation.
  • Custom Docker container build so that users can build containers with, e.g., XNAT pipeline engine, Freesurfer, and custom pipeline descriptor.
  • Provenance. Flywheel Exchange: worth discussing between Gunnar at Flywheel, Bennett Landman, Dan, container services.