NRG / HCP Pre-Built Data Relay
Many of the sites that participate in data collection for NRG and HCP based projects have data relays that have been built at NRG. This is an outline of what those relays are and how we manage them.
Software
Operating System
Centos 7 - Server Desktop install
Firewall
Built in firewalld that produces the following iptables:
iptables -L -n
CODE
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:53
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:53
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:67
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:67
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
INPUT_direct all -- 0.0.0.0/0 0.0.0.0/0
INPUT_ZONES_SOURCE all -- 0.0.0.0/0 0.0.0.0/0
INPUT_ZONES all -- 0.0.0.0/0 0.0.0.0/0
DROP all -- 0.0.0.0/0 0.0.0.0/0 ctstate INVALID
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 192.168.122.0/24 ctstate RELATED,ESTABLISHED
ACCEPT all -- 192.168.122.0/24 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-port-unreachable
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
FORWARD_direct all -- 0.0.0.0/0 0.0.0.0/0
FORWARD_IN_ZONES_SOURCE all -- 0.0.0.0/0 0.0.0.0/0
FORWARD_IN_ZONES all -- 0.0.0.0/0 0.0.0.0/0
FORWARD_OUT_ZONES_SOURCE all -- 0.0.0.0/0 0.0.0.0/0
FORWARD_OUT_ZONES all -- 0.0.0.0/0 0.0.0.0/0
DROP all -- 0.0.0.0/0 0.0.0.0/0 ctstate INVALID
REJECT all -- 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:68
OUTPUT_direct all -- 0.0.0.0/0 0.0.0.0/0
Chain FORWARD_IN_ZONES (1 references)
target prot opt source destination
FWDI_public all -- 0.0.0.0/0 0.0.0.0/0 [goto]
FWDI_scanner all -- 0.0.0.0/0 0.0.0.0/0
FWDI_public all -- 0.0.0.0/0 0.0.0.0/0 [goto]
Chain FORWARD_IN_ZONES_SOURCE (1 references)
target prot opt source destination
Chain FORWARD_OUT_ZONES (1 references)
target prot opt source destination
FWDO_public all -- 0.0.0.0/0 0.0.0.0/0 [goto]
FWDO_scanner all -- 0.0.0.0/0 0.0.0.0/0
FWDO_public all -- 0.0.0.0/0 0.0.0.0/0 [goto]
Chain FORWARD_OUT_ZONES_SOURCE (1 references)
target prot opt source destination
Chain FORWARD_direct (1 references)
target prot opt source destination
Chain FWDI_public (2 references)
target prot opt source destination
FWDI_public_log all -- 0.0.0.0/0 0.0.0.0/0
FWDI_public_deny all -- 0.0.0.0/0 0.0.0.0/0
FWDI_public_allow all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
Chain FWDI_public_allow (1 references)
target prot opt source destination
Chain FWDI_public_deny (1 references)
target prot opt source destination
Chain FWDI_public_log (1 references)
target prot opt source destination
Chain FWDI_scanner (1 references)
target prot opt source destination
FWDI_scanner_log all -- 0.0.0.0/0 0.0.0.0/0
FWDI_scanner_deny all -- 0.0.0.0/0 0.0.0.0/0
FWDI_scanner_allow all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
Chain FWDI_scanner_allow (1 references)
target prot opt source destination
Chain FWDI_scanner_deny (1 references)
target prot opt source destination
Chain FWDI_scanner_log (1 references)
target prot opt source destination
Chain FWDO_public (2 references)
target prot opt source destination
FWDO_public_log all -- 0.0.0.0/0 0.0.0.0/0
FWDO_public_deny all -- 0.0.0.0/0 0.0.0.0/0
FWDO_public_allow all -- 0.0.0.0/0 0.0.0.0/0
Chain FWDO_public_allow (1 references)
target prot opt source destination
Chain FWDO_public_deny (1 references)
target prot opt source destination
Chain FWDO_public_log (1 references)
target prot opt source destination
Chain FWDO_scanner (1 references)
target prot opt source destination
FWDO_scanner_log all -- 0.0.0.0/0 0.0.0.0/0
FWDO_scanner_deny all -- 0.0.0.0/0 0.0.0.0/0
FWDO_scanner_allow all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
Chain FWDO_scanner_allow (1 references)
target prot opt source destination
Chain FWDO_scanner_deny (1 references)
target prot opt source destination
Chain FWDO_scanner_log (1 references)
target prot opt source destination
Chain INPUT_ZONES (1 references)
target prot opt source destination
IN_public all -- 0.0.0.0/0 0.0.0.0/0 [goto]
IN_scanner all -- 0.0.0.0/0 0.0.0.0/0
IN_public all -- 0.0.0.0/0 0.0.0.0/0 [goto]
Chain INPUT_ZONES_SOURCE (1 references)
target prot opt source destination
Chain INPUT_direct (1 references)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x10
Chain IN_public (2 references)
target prot opt source destination
IN_public_log all -- 0.0.0.0/0 0.0.0.0/0
IN_public_deny all -- 0.0.0.0/0 0.0.0.0/0
IN_public_allow all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
Chain IN_public_allow (1 references)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:922 ctstate NEW
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 ctstate NEW
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 ctstate NEW
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 ctstate NEW
Chain IN_public_deny (1 references)
target prot opt source destination
Chain IN_public_log (1 references)
target prot opt source destination
Chain IN_scanner (1 references)
target prot opt source destination
IN_scanner_log all -- 0.0.0.0/0 0.0.0.0/0
IN_scanner_deny all -- 0.0.0.0/0 0.0.0.0/0
IN_scanner_allow all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
Chain IN_scanner_allow (1 references)
target prot opt source destination
ACCEPT icmp -- 0.0.0.0/0 0.0.0.0/0
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:104 ctstate NEW
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:8104 ctstate NEW
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 ctstate NEW
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 ctstate NEW
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:873 ctstate NEW
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:873 ctstate NEW
Chain IN_scanner_deny (1 references)
target prot opt source destination
Chain IN_scanner_log (1 references)
target prot opt source destination
Chain OUTPUT_direct (1 references)
target prot opt source destination
Remote Management
Each system has preinstalled scripts that manage updates to the installed operating system and software. These scripts are triggered via two cron jobs.
- One that updates the scripts hosted publicly at Bitbucket:
https://bitbucket.org/xnatrelay/relaysync - The other executes the script update-relay.sh which is maintained in the repository
In the coming months this will be replaced with Puppet configuration management. The existing mechanism will be used to deploy Puppet.
Each system has it's own account at nrg-owncloud.wustl.edu that is used to push log file information to NRG for troubleshooting and system health monitoring.
Where sites allow we also open SSH and HTTPs to our systems managers at WU.
SSH private/public key authentication is the only allowed outside management of the operating system. Site network/system administrators will be given root access once they provide a public SSH key.
Included Software
XNAT
- XNAT listens on port 8104 for incoming DICOM.
- After receiving a DICOM session it scrubs any potential PHI
- Xsync is uses HTTPS to securely to push the DICOM to the parent XNAT system at WU such as IntraDB or CNDA
Raw Data Relay
Only on select systems that are relaying raw k-space data.
- A script executed by a local cron job periodically queries the XNAT system to determine the UUID of sessions that need to be retrieved.
- The UUID is added to a list of sessions to later be retrived
- At a predetermined time of day, a script pulls raw data from the MARS computer and stores it on the relay.
- Upon successfully copying the raw data files to the relay they are deleted from the MARS computer.
- Raw data on the relay is pushed to WU via Aspera to the central repository.
- Once the raw data is successfully sent to WU it is deleted from the relay.
ZFS / OZMT
All file systems on the relay are ZFS and managed by OZMT ZFS guarentees data integrity and provides snapshots to preserve data and history.
Periodic snapshots are managed by OZMT. Periodic email reports are sent with the status. Quota reports will be emailed by OZMT in the event the disks are filling up.
Should a disk fail an email will be sent by the ZFS event daemon.