The second method is to experiment and define various OpenMP settings throughout your PV-WAVE application. The remainder of this chapter explains the options available to you to do your own OpenMP tuning.

You can modify certain parameters to affect the run-time behavior of OpenMP. These include:

You can set the OpenMP parameters in two ways, either at startup via environment variables or at run-time via the SET_OMP routine. The environment variables provide only the initial values of the OpenMP settings which can then be changed as often as you like in your code via the SET_OMP routine. The examples below all use C shell syntax for the environment variable examples; bash or Korn shell users should make appropriate modifications.

setenv OMP_THRESHOLD N    -or- 

WAVE> SET_OMP, Threshold=N

A threshold value is used to suppress parallelization when the number of elements in the target data is too small to justify the overhead of additional threads.

You can control the number of elements at which parallelization kicks in via the SET_OMP routine in PV-WAVE or via the environment variable OMP_THRESHOLD in the environment from which PV-WAVE is started. The parallelization threshold can vary from operation to operation by several orders of magnitude, and the most convenient way to account for this variation is by using automatic thread control (see PV-WAVE-defined OpenMP Settings (Automatic Thread Control) for more information).

setenv OMP_NUM_THREADS N    -or- 

WAVE> SET_OMP, Nthreads=N

The number of threads you wish to use may be set via either the SET_OMP routine at run-time or the OMP_NUM_THREADS environment variable at startup. If you do not have dynamic threads enabled this is the exact number of threads used in the parallel sections of the code, regardless of the machine’s load. Unless it has been set manually or automatic thread control is in effect, Nthreads defaults to the number of processors on the machine.

setenv OMP_DYNAMIC TRUE | FALSE    -or- 

WAVE> SET_OMP, Dynamic = 1 | 0

Dynamic threading allows OpenMP to adjust the number of threads invoked each time the application enters a parallel section. The set number of threads becomes the maximum thread count instead of the absolute thread count. The actual number of threads that are invoked depends on the number of available, idle processors. This improves performance, in theory, by making sure that multiple processes are not fighting over the same processor, slowing everyone down. In practice, dynamic threading has some limitations. Very lightweight processes from the OS or from a LAN can prevent many of the processors from being used for any given operation. It can be a convenient setting for a shared machine, but on some platforms can degrade the performance of a lone process.

Setting OMP_DYNAMIC to TRUE prior to starting PV-WAVE or calling SET_OMP, Dynamic=1 turns on dynamic threading. Dynamic threading is disabled by default.