Transcription

TotalView TrainingDECEMBER 2020TotalViewby Perforce2019 PerforceSoftware,Inc.TotalViewby Perforce PerforceSoftware,Inc.

LBL/NERSC Agenda Introduction (Bill) Python Debugging (Dean) TotalView Features (Bill) ReplayEngine (Dean) TotalView’s New UI (Dean) OpenMP debugging (Bill) Startup (Dean) MPI Debugging (Bill) Remote Debugging – Remote UI/RDC (Dean) Reverse Connect Feature (Bill) UI Navigation and Process Control (Dean) MemoryScape (Dean or Bill) Action Points (Dean) CUDA debugging (Bill) Examining and Editing Data (Dean) TotalView Roadmap (Bill) Advanced C and Data Debugging (Dean) Common TotalView usage questions Q&A Q&A Break2 TotalView by Perforce Perforce Software, Inc.totalview.io

Introduction

Perforce Global Footprint Customers in 80 countries. 9,000 customers worldwide. More than 250 of the Fortune 500. Customers deploying multipleproducts. 25 offices and 4 datacenters which give us global reach. Over 900 employees in 25 countries.5 TotalView by Perforce Perforce Software, Inc.totalview.io

Perforce Product PortfolioAgileManagementCode Management& CollaborationApplication Mgmt.& ComponentsAutomatedTestingHelix ALMHelix CoreAkanaHelix ndKlocworkTotalViewVisualizationSourceProIMSL6 TotalView by Perforce Perforce Software, Inc.totalview.io

TotalView Features

What is TotalView used for? Provides interactive Dynamic Analysis capabilities to help: Understand complex code Improve code quality Collaborate with team members to resolve issues faster Shorten development time Finds problems and bugs in applications including: Program crash or incorrect behavior Data issues Application memory leaks and errors Communication problems between processes and threads CUDA application analysis and debugging Contains batch and Continuous Integration capabilities to: Debug applications in an automated run/test environment8 TotalView by Perforce Perforce Software, Inc.totalview.io

TotalView Features Comprehensive C, C and Fortran multi-process/thread dynamic analysis anddebugging Thread specific breakpoints with individual thread control View thread specific stack and data View complex data types easily MPI and OpenMP HPC debugging CUDA debugging Integrated Reverse debugging Mixed Language - Python C/C debugging Memory leak detection Batch/unattended debugging Linux, macOS and UNIX More than just a tool to find bugs Understand complex code Improve developer efficiency Collaborate with team members Improve code quality Shorten development time9 TotalView by Perforce Perforce Software, Inc.totalview.io

TotalView’s New UI

TotalView’s New UI Provides a modern, dockable interface Easier to use, better workflows An architecture to grow To use: Set UI preference Or command line argumenttotalview –newUI or totalview -oldUI New UI gaps: Missing array viewer, data visualization Memory debugging not integrated No high-scale MPI support11 TotalView by Perforce Perforce Software, Inc.totalview.io

TotalView’s Dark Theme – 2019.112 TotalView by Perforce Perforce Software, Inc.totalview.io

Startup

Start Page14 TotalView by Perforce Perforce Software, Inc.totalview.io

Debugging a New Program15 TotalView by Perforce Perforce Software, Inc.totalview.io

Debugging a Parallel Program16 TotalView by Perforce Perforce Software, Inc.totalview.io

Attach to a Running Program17 TotalView by Perforce Perforce Software, Inc.totalview.io

Open a Core File or Replay Recording Session18 TotalView by Perforce Perforce Software, Inc.totalview.io

Remote DebuggingClassic UI Only Specify remote host information when launching a newdebugging session in the UI Or from the command line:totalview executable –r hostname[:port] When launching an MPI job, TotalView will automaticallyfollow the launch and establish the remote debuggingconnections19 TotalView by Perforce Perforce Software, Inc.totalview.io

Starting a Previous Debugging Session20 TotalView by Perforce Perforce Software, Inc.totalview.io

Remote Debugging

Remote Debugging Sessions Debugging on a remote HPC cluster can be a challenge Setting up the secure remote connection Launching/connecting to the target application Interactive debugging UI TotalView Remote Debugging Options TotalView Remote Display Client Conveniently setup a remote VNC connection TotalView Reverse Connect Disconnect launching the core debugger within the clusterfrom the UI on a front-end node TotalView Remote UI Run the TotalView UI on a remote client and connect to theremote TotalView debugger22 TotalView by Perforce Perforce Software, Inc.totalview.io

TotalView Remote Display Client (RDC) Offers users the ability to easily set up and operate a TotalView debug session that is running on another system Consists of three components Client – runs on local machine Server – runs on any system supported by TotalView and “invisibly” manages the secure connection between host and client Viewer – window that appears on the client system Remote Display Client is available for: Linux x86, x86-64 Windows Mac OSX23 TotalView by Perforce Perforce Software, Inc.totalview.io

TotalView Remote Display Client Free to install on as many clients as needed No license required to run the client Presents a local window that displays TotalView or MemoryScape running on the remote machine Requires SSH and X Windows on Server24 TotalView by Perforce Perforce Software, Inc.totalview.io

TotalView Remote Display Client User must provide information necessary to connect to remote host Passwords are NOT stored Information required includes: User name, public key file, other ssh informationDirectory where TotalView/MemoryScape is locatedPath and name of executable to be debuggedIf using indirect connection with host jump, each host Host name Access type (User name, public key, other ssh information) Access value Client also allows for batch submission via PBS Pro or LoadLeveler25 TotalView by Perforce Perforce Software, Inc.totalview.io

TotalView Remote Display Client26 TotalView by Perforce Perforce Software, Inc.totalview.io

Session Profile Management Connection information can be saved as a profile, including all host jumping information Multiple profiles can be generated Profiles can be exported and shared Generated profiles can be imported for use by other users27 TotalView by Perforce Perforce Software, Inc.totalview.io

Demo Remote Display Client demo (Linux)28 TotalView by Perforce Perforce Software, Inc.totalview.io

TotalView Reverse Connections The organization of modern HPC systems often makes it difficult to deploy tools such as TotalView The compute nodes in a cluster may not have access to any X libraries or X forwarding Launching a GUI on a compute node may not be possible Using the Reverse Connect feature you can start the TotalView UI on a front end node and when a job is run in thecluster it connects back to the waiting UI29 TotalView by Perforce Perforce Software, Inc.totalview.io

TotalView Remote UI (TotalView 2020.3) Combine the convenience of establishing a remote connection to a cluster and the ability to run the TotalView GUIlocally. Front-end GUI architecture does not need to match back-end target architecture (macOS front-end - Linux back-end) Secure communications Convenient saved sessions Supports reverse connections30 TotalView by Perforce Perforce Software, Inc.totalview.io

Remote UI Architecture31 TotalView by Perforce Perforce Software, Inc.totalview.io

Demo TotalView Remote UI Demo32 TotalView by Perforce Perforce Software, Inc.totalview.io

UI Navigation and Process Control

TotalView’s Default ViewsProcesses &Threads ViewLookup File orFunctionDocumentsSource ViewCall Stack ViewLocal VariablesViewData View,Command Line,Logger,Input/OutputAction Points,Replay Bookmarks34 TotalView by Perforce Perforce Software, Inc.totalview.io

Process and Threads View35 TotalView by Perforce Perforce Software, Inc.totalview.io

Source View36 TotalView by Perforce Perforce Software, Inc.totalview.io

Call Stack View and Local Variables Panel37 TotalView by Perforce Perforce Software, Inc.totalview.io

Action Points and Replay Engine Bookmarks38 TotalView by Perforce Perforce Software, Inc.totalview.io

Data View, Command Line and Logger39 TotalView by Perforce Perforce Software, Inc.totalview.io

Lookup File or Function40 TotalView by Perforce Perforce Software, Inc.totalview.io

PreferencesDisplay Settings41 TotalView by Perforce Perforce Software, Inc.totalview.io

TotalView ToolbarCommandDescriptionGoSets the thread to running until it reaches a stopping point. Often this will be a breakpoint thatyou have set, but the thread could stop for other reasons.HaltStops the thread at its current execution point.KillStops program execution. Existing breakpoints and other settings remain in effect.RestartStops program execution and restarts the program from the beginning. Existing breakpoints andother settings remain in effect. This is the same as clicking Kill followed by Go.NextMoves the thread to the next line of execution. If the line the thread was on includes one or morefunction calls, TotalView does not step into these functions but just executes them and returns.StepLike Next, except that TotalView does step into any function calls, so the thread stops at the firstline of execution of the first function call.OutIf the thread is in a block of execution, runs the thread to the first line of execution beyond thatblock.Run ToIf there is a code line selected in one of the Source views, the thread will stop at this line,assuming of course that it ever makes it there. This operates like a one-time, temporarybreakpoint.42 TotalView by Perforce Perforce Software, Inc.totalview.io

Stepping CommandsSelect Step in the toolbar. TotalView stops the program just before thefirst executable statement, the call to setjmp (context);43 TotalView by Perforce Perforce Software, Inc.totalview.io

Stepping CommandsSelect Step again to advance to the while loop on line 31, and thenselect Step again to step into the readexpr() function. (Next would stepover, or execute it).44 TotalView by Perforce Perforce Software, Inc.totalview.io

Using Set PC Resumes execution from anarbitrary point Select the line Thread- Set PC45 TotalView by Perforce Perforce Software, Inc.totalview.io

Demo TotalView UI demo (QT Threads Example)46 TotalView by Perforce Perforce Software, Inc.totalview.io

Action Points

Action PointsBreakpointEvalpointWatchpointBarrierpoint48 TotalView by Perforce Perforce Software, Inc.totalview.io

Setting Breakpoints Setting action points Single-click line number Right clicking on the linenumber and using thecontext menu Clicking a line in the sourceview then selecting theAction Points - Setbreakpoint menu option49 TotalView by Perforce Perforce Software, Inc.totalview.io

Setting Breakpoints Breakpoint- At Location Specify function name or line number If function name, TotalView sets a breakpoint atfirst executable line in the function50 TotalView by Perforce Perforce Software, Inc.totalview.io

Pending Breakpoints Useful when setting a breakpoint on a library that has not yet been loaded into memory51 TotalView by Perforce Perforce Software, Inc.totalview.io

Modifying Breakpoints Enable / Disable /Delete a breakpoint Adjust thebreakpoints width Group: Stops all running threads in all processes in the group. Process: Stops all the running threads in the process containing the thread that hit thebreakpoint Thread: Stops only the thread that first executes to this breakpoint52 TotalView by Perforce Perforce Software, Inc.totalview.io

Evalpoints53 TotalView by Perforce Perforce Software, Inc.totalview.io

Evalpoints Use Eval points to :– Include instructions that stop a process and its relatives– Test potential fixes or patches for your program– Include a goto for C or Fortran that transfers control to a linenumber in your program– Execute a TotalView function– Set the values of your program’s variables54 TotalView by Perforce Perforce Software, Inc.totalview.io

Evalpoints Examples Print the value of a variable to the command lineprintf("The value of result is %d\n", result); Skip some codegoto 63; Stop a loop after a certain number of iterationsif ( (i % 100) 0){printf("The value of i is %d\n", i); stop;};55 TotalView by Perforce Perforce Software, Inc.totalview.io

Watchpoints Watchpoints are set on a specific memory location Execution is stopped when the value stored in that memory location changes A breakpoint stops before an instruction executes. A watchpoint stops after an instruction executes56 TotalView by Perforce Perforce Software, Inc.totalview.io

Using Watchpoint Expressions TotalView has two variables that are used exclusively with watchpoint expressions: oldval: The value of the memory locations before a change is made. newval: The value of the memory locations after a change is made. Example 1if (iValue ! 42 && iValue ! 44) {iNewValue newval; iOldValue oldval; stop;} Example 2if ( oldval 0 && newval 0) stop57 TotalView by Perforce Perforce Software, Inc.totalview.io

Barrier Breakpoints Used to synchronize a group of threads or processes defined in the action point Threads or processes are held at barrierpoint until all threads or processes in the group arrive When all threads or processes arrive the barrier is satisfied and the threads or processes are released58 TotalView by Perforce Perforce Software, Inc.totalview.io

Saving BreakpointsFrom the Action Points menu select Save or Save As to save breakpointsTurn on option to save action points on exit59 TotalView by Perforce Perforce Software, Inc.totalview.io

Demo TotalView evaluation point demo (Combined) TotalView evaluation point demo (ex2)60 TotalView by Perforce Perforce Software, Inc.totalview.io

Examining and Editing Data

Call Stack and Local VariablesThe Call Stack View consists of two panels : The Call Stack panel The Local Variables panel62 TotalView by Perforce Perforce Software, Inc.totalview.io

The Local Variables PanelThe Local Variables panel displays variables for the thread of interest (TOI)63 TotalView by Perforce Perforce Software, Inc.totalview.io

The Data View PanelAdd data to the Data View using the context menu or by dragging and droppingContext menuDrag and drop64 TotalView by Perforce Perforce Software, Inc.totalview.io

The Data View PanelSelect the right arrow to display the substructures in a complex variableAny nested structures are displayed in the data view65 TotalView by Perforce Perforce Software, Inc.totalview.io

The Data View PanelDive on a single element to view individual data in the Data View66 TotalView by Perforce Perforce Software, Inc.totalview.io

The Data View PanelEnter a new expression in the Data View panel to view that dataType the expression in the [AddNew expression] fieldA new expression is addedIncrement a variable67 TotalView by Perforce Perforce Software, Inc.totalview.io

The Data View PanelDereferencing a pointerWhen you dive on a variable, it isnot dereferenced automaticallyDouble click in the Namecolumn to make it editableand dereference the pointerThe Data View displays thevariables value68 TotalView by Perforce Perforce Software, Inc.totalview.io

The Data View PanelCasting to another typeCast a variable into an array byadding the array specifierTotalView displays the array69 TotalView by Perforce Perforce Software, Inc.totalview.io

Viewing Data in FortranThe qualified subroutinename appears in the CallStack view.The qualified variable namesappear in the Local Variablepanel.70 TotalView by Perforce Perforce Software, Inc.totalview.io

Fortran Common BlocksFor each common block defined in the scope of a subroutine or function, TotalView creates anentry in that function’s common block list.The names of common block members have function scope, not global scope.If you select the function in the Call Stack view, the common blocks and their variables appear inthe Local Variables panel.71 TotalView by Perforce Perforce Software, Inc.totalview.io

Fortran User-Defined TypesTotalView displays user-defined types in the Local Variables panel, which youcan then add to the Data View for more detail72 TotalView by Perforce Perforce Software, Inc.totalview.io

Advanced C and Data Debugging

C Container Transformations TotalView transforms many of the C and STL containers including: array, forward list, tuple, map, set, vector and others.See This!Insteadof This74 TotalView by Perforce Perforce Software, Inc.totalview.io

Advanced C Support TotalView supports debugging the latest C 11/14/17 features including:– lambdas, transformations for smart pointers, auto types, R-Value references,range-based loops, strongly-typed enums, initializer lists, user defined literals75 TotalView by Perforce Perforce Software, Inc.totalview.io

Array Slicing, Striding and Filtering Slicing – reduce display to a portion of the array [lower bound:upper bound]Classic UI Only [5:10] Striding – Skip over elements [::stride] [::5], [5:10:-1] Filtering– Comparison: , ! , , , , – Range of values: [ ] low-value : [ ] high-value– IEEE values: nan, inf, denorm76 TotalView by Perforce Perforce Software, Inc.totalview.io

Viewing Array Data Easily view 2D array data in table format.77 TotalView by Perforce Perforce Software, Inc.Classic UI Onlytotalview.io

Array Statistics Easily display a set of statistics for the filtered portion of your array78 TotalView by Perforce Perforce Software, Inc.totalview.io

Visualizing Array DataClassic UI Only Visualizer creates graphic images of your program’sarray data. Visualize one or two dimensional arrays View data manually through the Window Visualizecommand on the Data Window Visualize data programmatically using the visualizefunction79 TotalView by Perforce Perforce Software, Inc.totalview.io

Dive in All Dive in All Use Dive in All to easily see each member of a data structure from an array of structures80 TotalView by Perforce Perforce Software, Inc.totalview.io

Demo TotalView STL types (Combined) TotalView dive in all demo (Combined)81 TotalView by Perforce Perforce Software, Inc.totalview.io

Python Debugging

Python in HPC Python development trends: Increased usage of Python to build applications that call out to C Provides access to High-performance routines Leverage existing algorithms and libraries Utilize advanced multi-threaded capabilities Calling between languages easily enabled using technologies such as SWIG, ctypes, Pybind, Cython, CFFI, etc Debugging mixed language applications is not easy83 TotalView by Perforce Perforce Software, Inc.totalview.io

Python debugging with TotalView Debugging one language is difficult enough Understanding the flow of execution across language barriers is hard Examining and comparing data in both languages is challenging What TotalView provides: Easy python debugging session setup Fully integrated Python and C/C call stack ”Glue” layers between the languages removed Easily examine and compare variables in Python and C Modest system requirements Utilize reverse debugging and memory debugging Support for Python 2.7 and Python 3.0 What TotalView does not provide (yet): Setting breakpoints and stepping within Python code84 TotalView by Perforce Perforce Software, Inc.totalview.io

Python without FilteringGluecode85 TotalView by Perforce Perforce Software, Inc.totalview.io

Python with filtering86 TotalView by Perforce Perforce Software, Inc.totalview.io

Demo TotalView Python / C debugging demo (test python types.py)87 TotalView by Perforce Perforce Software, Inc.totalview.io

Day 1 Q&A

Contact us Bill Burns (Senior Director of Product Development and Product Manager)bbu[email protected] Dean Stewart (Senior Sales Engineer)[email protected] [email protected] TotalView by Perforce Perforce Software, Inc.totalview.io

Reverse Debugging with Replay Engine

Reverse debugging How do you isolate an intermittent failure?– Without TotalView Set a breakpoint in code Realize you ran past the problem Re-load Set breakpoint earlier Hope it fails Keep repeating– With TotalView Set a breakpoint Start recording See failure Run backwards/forwards in context of failing execution– Reverse Debugging Re-creates the context when going backwards Focus down to a specific problem area easily Saves days in recreating a failure91 TotalView by Perforce Perforce Software, Inc.totalview.io

Recording and Playback When ReplayEngine is saving state information, it is in Record Mode The saved state information is the program’s execution history You can save the execution history at any time and reload the recording when debugging the executable in a subsequentsession Using a ReplayEngine command, ether from the Toolbar or the CLI, shifts ReplayEngine into ReplayMode Debugging commands that do not work in ReplayMode include: Changing a variable’s value Functions that alter memory Running threads asynchronously92 TotalView by Perforce Perforce Software, Inc.totalview.io

ReplayEngine ToolbarRecorda toggle that enables and disables ReplayEngine.Go Backdisplays the state that existed at the last action point. If no action point is encountered,ReplayEngine displays the state that existed at the start of its recorded history.Prevdisplays the state that existed when the previous statement executed. If that line had a functioncall, Prev skips over the call.Unstepdisplays the state that existed when the previous statement executed. If that line had a functioncall, ReplayEngine moves to the last statement in that function.Callerdisplays the state that existed before the current routine was called.Back Todisplays the program’s state for the line you select. This line must have executed prior to thecurrently displayed line.Liveshifts from replay mode to record mode. It also displays the statement that would have executedhad you not moved into ReplayMode.BookmarkSavecreates a ReplayEngine bookmark at a selected location.saves the current replay recording session to a file.93 TotalView by Perforce Perforce Software, Inc.totalview.io

Saving and Loading Execution History TotalView can save the current ReplayEngineexecution history to file at any time The saved recording can be loaded into TotalViewusing any of the following: At startup, using the same syntax as when openinga core file:totalview -newUI executable recording-file On the Start Page view by selecting Load Core Fileor Replay Recording File94 TotalView by Perforce Perforce Software, Inc.totalview.io

Replay Bookmarks Replay bookmarks mark a point in the execution of a program, allowing you to quickly jump back to that point in timeCreating a Replay BookmarkActivating a Replay Bookmark95 TotalView by Perforce Perforce Software, Inc.totalview.io

Setting Preferences for ReplayEngine You can set the following preferences for ReplayEngine the maximum amount of memory to allocate to ReplayEngine The preferred behaviour when the memory limit is reached Setting the maximum amount of memory. The default value ‘0’ specifies to limit the maximum size by available memory only.dset TV::replay history size valuee.g.dset TV::replay history size 1024M Setting the preferred behaviour. By default, the oldest history is discarded so that recording can continuedset TV::replay history mode 1 (Discard oldest history and continue recording)dset TV::replay history mode 2 (Stop the process when the buffer is full)96 TotalView by Perforce Perforce Software, Inc.totalview.io

Demo TotalView ReplayEngine Demo97 TotalView by Perforce Perforce Software, Inc.totalview.io

Debugging OpenMP Applications

TotalView support for OpenMPSome of the features TotalView supports: Source-level debugging of the original OpenMP code The ability to plant breakpoints throughout the OpenMP code, including lines that are executed in parallel Visibility of OpenMP worker threads Access to SHARED and PRIVATE variables in OpenMP PARALLEL code Access to OMP THREADPRIVATE data in code compiled by supported compilers99 TotalView by Perforce Perforce Software, Inc.totalview.io

About TotalView OpenMP Features The compiler can generate multiple outlined routines from a single parallel region. This means that a single line ofsource code can generate multiple blocks of machine code inside different functions You can’t single step into or out of a parallel region. Instead, set a breakpoint inside the parallel region and let theprocess run to it. After execution reaches the parallel region, you can single step in it OpenMP programs are multi-threaded programs, so the rules for debugging multi-threaded programs apply100 TotalView by Perforce Perforce Software, Inc.totalview.io

Demo TotalView OpenMP demo (tx omp c threadprivate)101 TotalView by Perforce Perforce Software, Inc.totalview.io

Debugging Parallel Applications

Multi-Thread and Multi-Process Debugging TotalView provides the power to Simultaneously debug many threads and processes in a single debugging session Supports MPI, fork/exec, OpenMP, pthreads, std::thread, et al Help locate deadlocks and race conditions Understand complex applications utilizing threads By Providing control of entire groups of processes, individual processes or even down to individual threads within a process Enabling thread level breakpoints and barrier controls Showing aggregated thread and process state display103 TotalView by Perforce Perforce Software, Inc.totalview.io

Starting a Parallel Program Session from the UI From New Parallel Session page select: MPI preference Number of tasks Number of nodes Starter arguments Click Start Session to save and launch104 TotalView by Perforce Perforce Software, Inc.totalview.io

Starting a Parallel Program Session from the Command LineMPIIBMStartup Commandtotalview --args poe myprog -procs 4 -rmpool 0QUADRICSIntel Linuxunder SLURMMVAPICHOpteron Linuxunder SLURMSGItotalview --args srun -n 16 -p pdebug myprogSuntotalview --args mprun myprog -np 16MPICHmpirun -np 16 -tv myprogMPICH2Intel MPItotalview --args python ‘which mpiexec’ –tvsu -np 16 myprogtotalview --args srun -n 16 -p pdebug myprogtotalview --args mpirun myprog -np 16The order of arguments and executables differs between platformsYou can use totalview --args instead of totalview starter -a105 TotalView by Perforce Perforce Software, Inc.totalview.io

Parallel Debugging Group, Process and Thread ControlSelect either Group Process Thread106 TotalView by Perforce Perforce Software, Inc.totalview.io

Call GraphClassic UI Only Quick view of program state Each call stack is a path Functions are nodes Calls are edges Labeled with the MPI rank or thread ID Construct process groups Look for outliers Dive on a node in the call graph to create a Call Graphgroup.107 TotalView by Perforce Perforce Software, Inc.totalview.io

Parallel PreferencesAttach Behavior controls if TotalView should attachto all of the processes, none or ask what to doAfter Attach Behavior controls if parallel job stops,runs or if TotalView should ask what to do108 TotalView by Perforce Perforce Software, Inc.totalview.io

Subset Attach - Control Which Processes TotalView Attaches To Debug a subset of the processes that make up the jobClassic UI Only Sometimes the user does not need to control and seeevery process to understand the behavior or id thedefect The subset can be changed at any time Can narrow, expand or shift focus Uncouples interactive performance from job size After the subset operation completes Interactive performance depends on subset size Supports the use of lightweight tools Such as LLNL’s STAT109 TotalView by Perforce Perforce Software, Inc.totalview.io

View MPI Message Queues Information visible whenever MPI rank processes arehalted Provides information from the MPI layerClassic UI Only Unexpected messages Pending Sends Pending Receives Use this info to debug Deadlock situations Load balancing May need to beenabled in the MPI library --enable-debug110 TotalView by Perforce Perforce Software, Inc.totalview.io

Message Queue GraphClassic UI Only Hangs & Deadlocks Pending Messages Receives Sends Unexpected Inspect Individual entries Patterns111 TotalView by Perforce Perforce Software, Inc.totalview.io

Find Deadlocks and Performance SinksClassic UI Only Filtering Choose messages to track Choose MPI Communicators Cycle detection112 TotalView by Perforce Perforce Software, Inc.totalview.io

Multi-Thread Debugging Techniques Multiple ID’s for threads pthread library ID – Displayed by default in TotalView OS Light Weight Process (LWP) ID TotalView thread ID – ProcessID.ThreadID, e.g. 1.3 Finding deadlocks due to mutex misuse Utilize ReplayEngine/reverse debugging Leverage watchpoints to find when mutex was acquired Set the “Open process window at breakpoint” preference on the Action Points tab To get LWP id, turn off TotalView user threads (-no user threads) TotalView normally just displays

Offers users the ability to easily set up and operate a TotalView debug session that is running on another system Consists of three components Client - runs on local machine Server - runs on any system supported by TotalView and "invisibly" manages the secure connection between host and client