Target: CS early March 2017
This sprint is intended to end 2017.03.23
Status / Tasks
- CS-130,131 about to complete
- Next items: CS-132, 134-138
- CS-117 Add wrapper field to CommandEventMapping: Need to confer with Matt
- Away: Holiday
- Need followup with Flavin to stage Thursday demo
- Will be out Wednesday
- Away: XNAT-Core
- Away: Spring Break
- Completed demo item and passed off to Mark before end of 2017.03.17
- No open items
- Refactored command structure to move XNAT-y stuff off to the back-end part of the command to make it a bit more independent.
- Figure out way to break out translating XNAT object models into a more general model.
- Use Google Autovalue library to generate models to work with.
- Working on project configuration for container services. Adding a wrapper around config service to store and work with commands.
- Putting out fires, no work on event services.
Verb is operation:
- User request (e.g. user searches for sessions and wants to launch some process on the results)
Noun is event target:
Identified by some criteria: ID and data type with hierarchy,1
data types, etc.
- Actor data: user who made the event happen, for example, maybe other info about the event context. These could be in a map of attributes on the event, event acceptance can be calculated based on matching one or more attributes.
Each event will have a payload. How are these defined?
Many "events" aren't discrete points in time, so events also require status: started, completed, failed, etc. Unlike workflows, these aren't persisted as a unique timeline. We may need some way to associate events across an entire operation, but we can revisit later.
Definition of the noun will depend on the particular operation and context and would be defined in the event itself.
Event responders would be the way to wire to a data event. The basic event responder interface would indicate whether an event should be accepted and something about the event payload. Different base implementations for, e.g., handling events by data type, by other attributes.
This really is IFTTT.
An API to get sample events.
The data type hierarchy is the full stack of inheritance. For example, an MR session is:
You could say you want an xnat:mrSessionData or an xnat:experimentData and the MR session would match either one.
Development Progress Review
Progress is indeed being made.
- Flavin wants to refactor the structure of command definitions to make them more portable, by lumping all the XNAT-specific stuff into a discrete chunk of JSON. Other applications could ignore it.
- How do we want to support command versioning? Dan definitely wants it.
- Option 1: Natively support versioning inside XNAT, similar to Protocols.
- Option 2: Use an external, already-developed version control system for distributing commands (aka GitHub) and have sites / projects "slurp them up."
- What version of "Events" are we dependent on?
- *eye roll*
UI Progress Review
Reviewed these wireframes: https://collaborate.uxpin.com/5ba98d7201c1c221aa36b602c3185675a308b68a#/pages/53451616
- Add support for other types of container servers besides Docker. Aka Singularity. (This will be a short list.)
- Add support for filtering on scan types, session attributes, etc in a project-specific command configuration.
- Should Container Service site admin settings be in their own page? Or should they be bundled in with Site Admin stuff?
Will: They could probably be bundled in, but this may depend on how a plugin is able to add to the site admin UI. We think the standard practice will be to give each plugin its own page, or perhaps its own section of "Plugin Settings."
- Will site admins need to spend much (or any) time administering an image once it is imported?
- Should the majority of the Site Admin panel be devoted to administering commands? Yes, probably.
- There are two elements of sharing a command between projects: the command and the project-specific command configuration. Will we have a mechanism for sharing both easily?
Matt going through hackathon feedback. Looking at Jenny's matlab GUI thing. Work on I/O streaming through Spring.
Flavin knocking out tickets. Find specific Spring REST API 400 issues to figure out debug/troubleshooting steps. Check on XFT init issue.
Rick work on archive streaming image. Document that.
Rick Herrick needs to create XML import service to complete round trip with container services.
Start planning container services "alpha" or validation pass. Kick this off at XNAT developer meeting, but need to bring in real processing workflows to test it.
In last scrum we discussed not having a UI for container service functionality. It sounds like Dan wants to have UI, so we may need to re-visit this:
- We have existing wireframes, but they're a bit old. They may need to be revised and definitely need to be extended.
- John Flavin and Will Horton will work together to define all UI tasks:
- What wireframes do we have that are fine?
- What wireframes do we have that need to be revised?
- What wireframes do we not have?
- Generate JIRA issues, specifications, and documentation for UI tasks.
- Need the resource service and XML import service from Rick Herrick.
- Blocked by xdat bug that breaks assigned roles. May email Tim Olsen and Jenny Gurney. But export to OVA and re-build Vagrant box.
- John Flavin created a configuration that pulls in 3rd-party jars that aren't already in XNAT, but doesn't re-include jars in XNAT.
Matt has no real change on status. Hasn't started restart controller. Need to define all tasks required to hit end of sprint 1.
- Do we need UI for beta or is REST control enough?
- Need to define goals for beta test.
- Matt Kelsey define tasks for finishing up API development on container status, history, and logging. Pull those into sprint 1.
- Start knocking those out.
- Send email to Will Horton, cc Rick Herrick, Matt Kelsey, for update on CS wireframes. Review wireframes and categorize as:
- Complete and ready
- Complete, needing revision
- Incomplete or missing
- Finish development once Rick Herrick delivers resource and XML import services.
- Send bug in XNAT to Tim Olsen and Jenny Gurney, cc: Rick Herrick.
- Send compileAndInclude configuration to Tim Olsen, Jenny Gurney, Mike McKay, Mohana Ramaratnam, Rick Herrick.
- Start sprint.
- Email Dan about recruiting beta participants and features.
- Finish resource and XML import services.
Existing scrum board (Closed during issue migration from Pipeline project to Container Service project)
- Monitoring and managing container status: running, failed, logging.
- History and auditing.
- About 1/2 way through, 2/3 done, none tested.
- There are JIRA issues in the
pipelinecontainer service project.
- Handling container output (waiting on Rick's XNAT tickets)
- Created FreeSurfer command for practical testing of container services command resolver. Output of this isn't tested yet (see above).
- Migrating modules to plugins so that we have data types, UI, etc. - CS-35Getting issue details... STATUS
- There are JIRA issues in the
Functional and integration testing is an open issue.
Beta program for container services: recruit users to adapt pipeline or processing tasks to use containers.
Requirements for Beta:
- Figure out path forward for existing pipelines. Convert each to a container, or wrap them all inside a container
- Migrate XML pipeline descriptors to container services commands: this may not even require "migrating" the descriptor, if we can feed the descriptor to a translator that turns into a command. Maybe automated, maybe not.
- OR Write a guide that tells XNAT admins how to
- create a docker image that contains their pipeline engine + deployed pipelines + dependencies, and
- Write commands for each pipeline
- No UI components for container services beta, just REST API. We can get UI designs from Will and contract out to UAR for implementation.
- Custom Docker container build so that users can build containers with, e.g., XNAT pipeline engine, Freesurfer, and custom pipeline descriptor.
- Provenance. Flywheel Exchange: worth discussing between Gunnar at Flywheel, Bennett Landman, Dan, container services.