Totalview® for HPC User Guide : PART IV Advanced Tools and Customization : Chapter 21 Setting Up Parallel Debugging Sessions : Debugging Cray XT Applications : Cray Linux Environment (CLE) : Special Requirements for Using ReplayEngine
Special Requirements for Using ReplayEngine
On Cray XC, XK, and XC systems, the MPIs use RDMA techniques, similar to Infiniband MPIs. When using ReplayEngine on MPI programs, certain environment variable settings must be in effect for the MPI rank processes. These settings ensure that memory mapping operations are visible to ReplayEngine. The required settings are:
MPICH_SMP_SINGLE_COPY_OFF = 1
LD_PRELOAD: Set to include a preload library, which can be found under the TotalView installation directory at toolworks/totalview.<version>/linux-x86/lib/undodb_infiniband_preload_x32.so or toolworks/totalview.<version>/linux-x86-64/lib/undodb_infiniband_preload_x64.so.
These settings may be applied with the aprun -e option. For example, to have TotalView launch an MPI program with ReplayEngine enabled, use a command similar to this:
 
totalview -replay -args aprun -n 8 \
-e MPICH_SMP_SINGLE_COPY_OFF=1 \
-e LD_PRELOAD=/<path>/undodb_infiniband_preload_x64.so \
myprogram