Transcription

NMR in the Cloud: NMRbox Resources for MovingNMR Data Processing and Analysis OnlineMark Maciejewski – UConn HealthNational Center for Biomolecular NMR Data Processing and AnalysisBiomedical Technology Research Resource (BTRR)NIH / NIGMS 2-P41-GM111135-06Follow uson IVAN NMR User Group – “Live at Asilomar”

NMRbox @ ENC 2021https://nmrbox.org/pages/enc-2021NMRbox poster #1046 – Monday 11 – 12:30 EDTIVAN NMR User Group – “Live at Asilomar”

Acknowledgements: NMRbox teamPIs / Co-PIsJeff Hoch - PIAdam SchuylerMichael GrykHamid EghbalniaFrank DelaglioEldon UlrichMiron LivnyIon MoraruStaffGerard WeatherbyJon WedellYulia PustovalovaKumaran BaskaranDimitri MaziukHesam DashtiColin WilburnTerry WrightMichael WilsonStephen KingSophan IvClarissa TrudellIVAN NMR User Group – “Live at Asilomar”

motivationdiscoveryplatforms & librariespersistencycomplexityreproducibilityA computational study is reproduciblewhen it provides the “completesoftware environment needed toreproduce the figures”- David Donoho, Stanford UniversityIVAN NMR User Group – “Live at Asilomar”

what is the NMRbox project all about? Platform (NMRbox VM) – TRD1 Virtual machine (VM) loaded with NMR and related softwareComputational, storage, networking, & support servicesArchive of NMR software/versions (“native” & easily installable)Significant back-end infrastructureEvolving! Data Capturing – TRD2Bayesian Analytics (metabolomics) – TRD3Driving Biological Projects – DBPsCommunity Engagement User and developer support NMRbox & External Workshops, courses, tutorials Use our platform to support your workshop / conference /courseIVAN NMR User Group – “Live at Asilomar”

NMRbox Virtual Machine (VM)Acquisition of software – 200 packages agnostic – install all software available and requestedPlatform Ubuntu 16.04 LTS, 18.04 LTS, 20.04 LTS Xubuntu – Xfce desktop environmentNMR software NMR – all flavors MNova OpenVNMRJ TopSpinscientific software SAXSCryoEMMolecular Dynamicsmolecular visualizationbioinformatics toolsMATLAB – toolboxesutility packages 15 editorsoffice toolsdrawing tools60 python libraries200 dev librariesIVAN NMR User Group – “Live at Asilomar”

what is in the NMRbox VM - softwareWhere can I see what is installed? NMRbox Software Explorer inside NMRbox VMsIVAN NMR User Group – “Live at Asilomar”

what is in the NMRbox VM - softwareWhere can I see what is installed? NMRbox website (https://nmrbox.org/software)IVAN NMR User Group – “Live at Asilomar”

access to NMRbox – educational and non-profit usersfree PaaS Platform-as-a-Serviceconnect in seconds via RealVNC / sshrun anywherepersistent jobs when disconnectedaccess to high core / high memory VMsGPU backed 3D renderingCUDA processingsecure data archival“spin-up” VMs on demand custom VMs developer VMsaccess to distributed computing downloadablelocal hardware and hypervisoroff-line accessroot permissionmust setup shared filesuser controls software installsutilizing high core / high memory VMs with several users per NMRbox VMIVAN NMR User Group – “Live at Asilomar”

connecting to NMRbox PaaSBenefits of RealVNC free viewer – works on any deviceautomatic connectionssingle sign-onpersistent sessionsfully encrypteddynamic screen resolution changeslocal printer mapped to sessionshare sessionsIVAN NMR User Group – “Live at Asilomar”

connecting via RealVNCBookmarks all users connect to port 5900 if no VNC session exists, it is started automatically VNC server maps users to their proper sessionIVAN NMR User Group – “Live at Asilomar”RealVNC login

connecting via RealVNC:443If having trouble connecting, try adding a “:443” to the hostname avoids issues with IT departments blocking outgoing hightnumbered ports (e.g. 5900)RealVNC will map to the same session whether using the default:5900 or :443IVAN NMR User Group – “Live at Asilomar”

release schedule / list of NMRbox VMsRelease cycle every two weeks; VM duration 12 weeks; LTS versions every 12 wksIVAN NMR User Group – “Live at Asilomar”

list of NMRbox production VMsOld versions of NMRbox VMs Rolling release LTS versions will remain persistent older versions have reduced computational capacity increase capability on request Old VMs powered off after Ubuntu stops security updates can be restarted upon requestIVAN NMR User Group – “Live at Asilomar”

screen resolutionSelect a pre-defineddefault or enter yourdesired Width andHeightSave setting to startupfile – Sets the defaultfor future sessionsSave to favorites –Adds a given setting tothe Favorites tabIVAN NMR User Group – “Live at Asilomar”

closing VNC connectionsMenu barCloses window –VNC session remainsrunning andprograms continue.from ssh connection – “kill-vnc-server” will killan unresponsive VNC sessionLogs you offNMRbox VM andshuts down VNCsession will fix oddwindowing issuesthat appearVNC window closes automatically after 1 hour of inactivity – session remains aliveIVAN NMR User Group – “Live at Asilomar”

NMRbox by the numbersregistered userscompute, storage, & networking 2927 users ( 50% US) 44 physical servers 47 states, 56 countries 2092 CPU cores (77 TFLOPS) 16 TB RAMprograms 50 NVIDIA GPUs (418 TFLOPS) 200 software packages 138,240 GPU CUDA cores 200M program executions 15,680 GPU Tensor cores 700 NMRPipe users! 600 TB storage object storagecommunity 36 community events 500 support tickets/year 40/100 GbE network layer 800k budget over next 4 yrs.IVAN NMR User Group – “Live at Asilomar”

NMRbox file locationsresourcelocationNMRbox accesshome folder (old)/home/nmrbox/ username /universalhome folder (new)/home/nmrbox/XXXX/ username /universalshared folder/public/universaltutorial folder/tutorial/universalfast scratch/scratch/ username /or/scratch/XXXX/ username /specific to VM; dataremoved automaticallysoftware/usr/software/specific to releaseDB resources/reboxitory/universalgroup sharing/public/group-name/universalreboxitory BMRB, PDB, NCBI non-redundant sequence DB updated monthly – old versions will persistIVAN NMR User Group – “Live at Asilomar”

collaboration groups for sharing filesIVAN NMR User Group – “Live at Asilomar”

Transfer files – secure copySEND files from local computer to NMRboxscp -r /Desktop/myProject [email protected]: /Desktop-r means recursiveFETCH files from NMRbox to local computerscp -r [email protected]: /Desktop/myProject /DesktopHint: set up password-less using-ssh-keygen-in-5-easy-steps/Filezilla (all OS’s) / Bitvise Tunnelier (Windows) GUI based scp/sftp program for all platforms https://filezilla-project.org/ Installed inside NMRbox VMsIVAN NMR User Group – “Live at Asilomar”

Transfer files - GlobusWhat is Globus?Tool to transfer files between “endpoints”Endpoints personal endpoints –setup on your computers institutional endpoints – NMRbox is a registered institutionalendpointFeatures circumvents firewalls no need to be local to either endpoint efficient transfers of large #s of files allows syncing transfers restart if interrupted new file sharing capabilityIVAN NMR User Group – “Live at Asilomar”

Transfer files - sshfsSSHFS (SSH FileSystem) uses FUSE (filesystem in user space)to create a mount point of your NMRbox home folder onyour local computerProvides drag/drop file access just like a local driveWorks with: OSX Linux Windows 10IVAN NMR User Group – “Live at Asilomar”

Getting helphttps://nmrbox.org/support FAQs Documentation (Website, NMRbox YouTube Channel,NMRbox Slack Workspace) Contact Us [email protected] Creates a Zendesk support ticketIVAN NMR User Group – “Live at Asilomar”

NMRbox – community Slack WorkspaceJoin at https://nmrbox.org/community/connectIVAN NMR User Group – “Live at Asilomar”

NMRbox - communityhttps://nmrbox.org/community Events Past workshop lecture and tutorial material Files located on NMRbox at /tutorial Publications List of publications that use NMRbox (send us more!) Connect Teams Calendar (New)IVAN NMR User Group – “Live at Asilomar”

Citing NMRbox If you utilize NMRbox, please cite and acknowledge us send us a note!IVAN NMR User Group – “Live at Asilomar”

Thank youQuestions?IVAN NMR User Group – “Live at Asilomar”

ContainersContainer applicationsare isolated from thesystem with their ownlibraries anddependenciesIVAN NMR User Group – “Live at Asilomar”

Singularity containers Advantagescomplete isolation of appcontainer single fileportable to any Linux OSno privilege escalationGPUs & X11 just workeasy mapping of filesystemsflexible interaction: executed directly multiple apps defined shell into container arbitrary commandsDisadvantages user interaction with theprogram MAY change Single binary programscan be setup to simplyrun Programs such asNMRPipe uses will needto shell into beforerunning scriptsIVAN NMR User Group – “Live at Asilomar”

Software provisioning workflowIdentify softwaredownload and archivedetermine all install stepsidentify all libraries and dependenciesbuild custom install scriptcreate APT package from automated frameworkstore in NMRbox APT repositoryprovision VMs from NMRbox APT repoIVAN NMR User Group – “Live at Asilomar”

what is in the NMRbox VM - softwareWhere can I see what is installed? directory listing of /usr/softwareIVAN NMR User Group – “Live at Asilomar”

what is in the NMRbox VM - softwareWhere can I see what is installed? release notes (https://nmrbox.org/pages/release-notes)IVAN NMR User Group – “Live at Asilomar”

connecting via RealVNCChange to yourNMRbox usernameIf red, alert NMRbox team!!IVAN NMR User Group – “Live at Asilomar”

Downloadable NMRbox VMNew downloadable VM model download a base NMRbox VM simple GUI to install NMRbox software ability to upgrade packagesMuch smaller installation size and fully customizableIVAN NMR User Group – “Live at Asilomar”

Downloadable NMRbox VM – Software install toolFirst time use – authenticate with your NMRbox accountIVAN NMR User Group – “Live at Asilomar”

Downloadable NMRbox VM – Software install toolTool will install selected software and all dependenciesIVAN NMR User Group – “Live at Asilomar”

NMRbox – managing accountForgot your password? Go to https://nmrbox.org/IVAN NMR User Group – “Live at Asilomar”

NMRbox – managing accounthttps://nmrbox.org/user-dashboardIVAN NMR User Group – “Live at Asilomar”

connect in seconds via RealVNC / ssh run anywhere persistent jobs when disconnected access to high core / high memory VMs GPU backed 3D rendering CUDA processing secure data archival “spin-up” VMs on demand custom VMs developer VMs access to distrib