Page tree
Skip to end of metadata
Go to start of metadata

Many of the sites that participate in data collection for NRG and HCP based projects have data relays that have been built at NRG.   This is an outline of what those relays are and how we manage them.   

Software

Operating System

Centos 7 - Server Desktop install

Firewall

Built in firewalld that produces the following iptables:

iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:53
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:53
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:67
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:67
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
INPUT_direct  all  --  0.0.0.0/0            0.0.0.0/0
INPUT_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0
INPUT_ZONES  all  --  0.0.0.0/0            0.0.0.0/0
DROP       all  --  0.0.0.0/0            0.0.0.0/0            ctstate INVALID
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
ACCEPT     all  --  0.0.0.0/0            192.168.122.0/24     ctstate RELATED,ESTABLISHED
ACCEPT     all  --  192.168.122.0/24     0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
FORWARD_direct  all  --  0.0.0.0/0            0.0.0.0/0
FORWARD_IN_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0
FORWARD_IN_ZONES  all  --  0.0.0.0/0            0.0.0.0/0
FORWARD_OUT_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0
FORWARD_OUT_ZONES  all  --  0.0.0.0/0            0.0.0.0/0
DROP       all  --  0.0.0.0/0            0.0.0.0/0            ctstate INVALID
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:68
OUTPUT_direct  all  --  0.0.0.0/0            0.0.0.0/0
Chain FORWARD_IN_ZONES (1 references)
target     prot opt source               destination
FWDI_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]
FWDI_scanner  all  --  0.0.0.0/0            0.0.0.0/0
FWDI_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]
Chain FORWARD_IN_ZONES_SOURCE (1 references)
target     prot opt source               destination
Chain FORWARD_OUT_ZONES (1 references)
target     prot opt source               destination
FWDO_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]
FWDO_scanner  all  --  0.0.0.0/0            0.0.0.0/0
FWDO_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]
Chain FORWARD_OUT_ZONES_SOURCE (1 references)
target     prot opt source               destination
Chain FORWARD_direct (1 references)
target     prot opt source               destination
Chain FWDI_public (2 references)
target     prot opt source               destination
FWDI_public_log  all  --  0.0.0.0/0            0.0.0.0/0
FWDI_public_deny  all  --  0.0.0.0/0            0.0.0.0/0
FWDI_public_allow  all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0
Chain FWDI_public_allow (1 references)
target     prot opt source               destination
Chain FWDI_public_deny (1 references)
target     prot opt source               destination
Chain FWDI_public_log (1 references)
target     prot opt source               destination
Chain FWDI_scanner (1 references)
target     prot opt source               destination
FWDI_scanner_log  all  --  0.0.0.0/0            0.0.0.0/0
FWDI_scanner_deny  all  --  0.0.0.0/0            0.0.0.0/0
FWDI_scanner_allow  all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
Chain FWDI_scanner_allow (1 references)
target     prot opt source               destination
Chain FWDI_scanner_deny (1 references)
target     prot opt source               destination
Chain FWDI_scanner_log (1 references)
target     prot opt source               destination
Chain FWDO_public (2 references)
target     prot opt source               destination
FWDO_public_log  all  --  0.0.0.0/0            0.0.0.0/0
FWDO_public_deny  all  --  0.0.0.0/0            0.0.0.0/0
FWDO_public_allow  all  --  0.0.0.0/0            0.0.0.0/0
Chain FWDO_public_allow (1 references)
target     prot opt source               destination
Chain FWDO_public_deny (1 references)
target     prot opt source               destination
Chain FWDO_public_log (1 references)
target     prot opt source               destination
Chain FWDO_scanner (1 references)
target     prot opt source               destination
FWDO_scanner_log  all  --  0.0.0.0/0            0.0.0.0/0
FWDO_scanner_deny  all  --  0.0.0.0/0            0.0.0.0/0
FWDO_scanner_allow  all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
Chain FWDO_scanner_allow (1 references)
target     prot opt source               destination
Chain FWDO_scanner_deny (1 references)
target     prot opt source               destination
Chain FWDO_scanner_log (1 references)
target     prot opt source               destination
Chain INPUT_ZONES (1 references)
target     prot opt source               destination
IN_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]
IN_scanner  all  --  0.0.0.0/0            0.0.0.0/0
IN_public  all  --  0.0.0.0/0            0.0.0.0/0           [goto]
Chain INPUT_ZONES_SOURCE (1 references)
target     prot opt source               destination
Chain INPUT_direct (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp flags:0x17/0x10
Chain IN_public (2 references)
target     prot opt source               destination
IN_public_log  all  --  0.0.0.0/0            0.0.0.0/0
IN_public_deny  all  --  0.0.0.0/0            0.0.0.0/0
IN_public_allow  all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0
Chain IN_public_allow (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:922 ctstate NEW
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:80 ctstate NEW
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:22 ctstate NEW
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:443 ctstate NEW
Chain IN_public_deny (1 references)
target     prot opt source               destination
Chain IN_public_log (1 references)
target     prot opt source               destination
Chain IN_scanner (1 references)
target     prot opt source               destination
IN_scanner_log  all  --  0.0.0.0/0            0.0.0.0/0
IN_scanner_deny  all  --  0.0.0.0/0            0.0.0.0/0
IN_scanner_allow  all  --  0.0.0.0/0            0.0.0.0/0
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0
Chain IN_scanner_allow (1 references)
target     prot opt source               destination
ACCEPT     icmp --  0.0.0.0/0            0.0.0.0/0
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:104 ctstate NEW
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:8104 ctstate NEW
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:80 ctstate NEW
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:443 ctstate NEW
ACCEPT     tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:873 ctstate NEW
ACCEPT     udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:873 ctstate NEW
Chain IN_scanner_deny (1 references)
target     prot opt source               destination
Chain IN_scanner_log (1 references)
target     prot opt source               destination
Chain OUTPUT_direct (1 references)
target     prot opt source               destination

 

Remote Management

Each system has preinstalled scripts that manage updates to the installed operating system and software.   These scripts are triggered via two cron jobs.  

  1. One that updates the scripts hosted publicly at Bitbucket: 
    https://bitbucket.org/xnatrelay/relaysync
  2. The other executes the script update-relay.sh which is maintained in the repository

In the coming months this will be replaced with Puppet configuration management.  The existing mechanism will be used to deploy Puppet.   

Each system has it's own account at https://nrg-owncloud.wustl.edu that is used to push log file information to NRG for troubleshooting and system health monitoring.

Where sites allow we also open SSH and HTTPs to our systems managers at WU.  

SSH private/public key authentication is the only allowed outside management of the operating system.   Site network/system administrators will be given root access once they provide a public SSH key.  

Included Software

XNAT

  1. XNAT listens on port 8104 for incoming DICOM.   
  2. After receiving a DICOM session it scrubs any potential PHI
  3. Xsync is uses HTTPS to securely to push the DICOM to the parent XNAT system at WU such as IntraDB or CNDA

Raw Data Relay

Only on select systems that are relaying raw k-space data.

  1. A script executed by a local cron job periodically queries the XNAT system to determine the UUID of sessions that need to be retrieved. 
  2. The UUID is added to a list of sessions to later be retrived
  3. At a predetermined time of day, a script pulls raw data from the MARS computer and stores it on the relay.   
  4. Upon successfully copying the raw data files to the relay they are deleted from the MARS computer.
  5. Raw data on the relay is pushed to WU via Aspera to the central repository.
  6. Once the raw data is successfully sent to WU it is deleted from the relay.

ZFS / OZMT

All file systems on the relay are ZFS and managed by OZMT  ZFS guarentees data integrity and provides snapshots to preserve data and history.

Periodic snapshots are managed by OZMT.   Periodic email reports are sent with the status.   Quota reports will be emailed by OZMT in the event the disks are filling up.

Should a disk fail an email will be sent by the ZFS event daemon.   

Information Needed Before Deploying

Required Information