CRAY-LDMS

This website contains archival information. For updates, see https://github.com/ovis-hpc/ovis-wiki/wiki

Protection Domain Tags (Cray Specific)

Note: If you are going to be using the "ugni" transport (RDMA over Gemini) you will need to run with either system (as root) or user (as user) ptags. While root CAN run using any ptag the fact that its use is unknown to ALPS could cause collisions with applications.

To see current ptags (XE/XK):

> apstat -P
PDomainID         Type    Uid   PTag     Cookie
LDMS              system      0       84 0xa9380000

To see current ptags (XC):

> apstat -P
PDomainID   Type   Uid     Cookie    Cookie2
LDMS      system     0 0x86b80000          0

To create a userspace ptag (XE/XK/XC):

apmgr pdomain -c <somenamehere>

Example (XE/XK):

> apmgr pdomain -c foo
> apstat -P
PDomainID         Type    Uid     PTag     Cookie
LDMS              system      0          84 0xa9380000
foo                     user     12345  233 0xa1230000

Note: A system administrator will have to setup system ptags and/or enable users to set up ptags in the following way.

  • In /etc/alps.conf file in sharedroot modify the following:
# Variables to enable user/system protection domains
# 
# Max # of user protection domains allowed
       pDomainMax      10
# List of system protection domains used by all applications
       pDomainIDs      sys1,sys2

Example (XC):

> apmgr pdomain -c foo
> apstat -P
PDomainID   Type   Uid     Cookie    Cookie2
LDMS      system     0 0x86b80000          0
foo             user 20596 0x86bb0000 0x86bc0000

To remove a userspace ptag (XE/XK/XC):

apmgr pdomain -r <somenamehere>

Note: The userid of the ptag being removed must match that of the user running the command or root

Set the following environment variables for either user or system ptags (example shows user ptag values) (XE/XK):

export LDMS_UGNI_PTAG 233
export LDMS_UGNI_COOKIE 0xa1230000

Set the following environment variables for either user or system ptags (example shows user ptag values) (XC):

export LDMS_UGNI_PTAG 0
export LDMS_UGNI_COOKIE 0x86bb0000

When running with user space ptags you must specify the ptag name when using aprun (XE/XK/XC):

aprun <<usual aprun args here>> -p foo ldmsd <<usual ldmsd flags here>>
or
aprun <<usual aprun args here>> -p foo ldms_ls <<usual ldms_ls flags here>>

Note: On some systems you will run aprun after a qsub -I or within a script specified in qsub or similar.

Modify your build environment to use the gnu compiler

module unload PrgEnv-XXX (if XXX is not gnu)
module load PrgEnv-gnu

Building LDMS and support libraries and headers

  • Download ovis and gpcd from github.com/ovis-hpc
  • Download libevent-2.0.x-stable from libevent.org
  • In this example source files are assumed to be in ~/Source and builds in ~/Build

Building gpcd libraries and header files

  • cp <path>/gpcd-master.zip ~/Source
  • cd ~/Source
  • unzip gpcd-master.zip
  • cd gpcd-master
  • Run autogen.sh
./autogen.sh
configure.ac:6: warning: AM_INIT_AUTOMAKE: two- and three-arguments forms are deprecated.  For more info, see: configure.ac:6: http://www.gnu.org/software/automake/manual/automake.html#Modernize-AM_005fINIT_005fAUTOMAKE-invocation

NOTE: This warning is ok. Makefile.am will soon be updated to address this warning.

  • Create config file with the following: — in this example configure.sh
../configure --prefix=<absolute path>/Build/gpcd_build
  • make configure.sh executable
chmod +x configure.sh
  • Create a build directory
mkdir build
  • cd to build directory and run configure.sh
cd build
../configure.sh
  • Build and install gpcd libs and includes
make && make install
  • Create link from libgpcdlocal.a to libgpcd.a in the build dir
  • cd ~Build/gpcd_build/lib
  • ln -s libgpcd.a libgpcdlocal.a

Building libevent-2.0

  • cp <path>/libevent-2.0.x-stable.tar.gz ~/Source
  • cd ~/Source
  • untar libevent-2.0.x-stable.tar.gz
  • cd libevent-2.0.x-stable
  • Run autogen.sh
./autogen.sh
  • Create config file with the following: — in this example configure.sh
../configure --prefix=<absolute path>/Build/libevent-2.0_build
  • make configure.sh executable
chmod +x configure.sh
  • Create a build directory
mkdir build
  • cd to build directory and run configure.sh
cd build
../configure.sh
  • Build and install libevent-2.0 libs and includes
make && make install

Building LDMS

  • cp <path>/ovis-master.zip ~/Source
  • cd ~/Source
  • unzip ovis-master.zip
  • cd ovis-master
  • Run autogen.sh
./autogen.sh
  • Create config file with the following: — in this example configure.sh
#!/bin/bash
#
# SYNOPSIS: Remove existing build directories, do the automake routine, rebuild,
#           and install everything.
#
# REMARK: This script doesn't do uninstall. If you wish to uninstall (e.g. make
# uninstall), please go into each build directories ( */build-$HOSTNAME ) and
# call make uninstall there, or just simply do the following
#     for D in */build-$HOSTNAME; do pushd $D; make uninstall; popd; done;
#
#
BUILD_PATH=<absolute path to builds>
PREFIX=$BUILD_PATH/OVIS-3.3
#
# add --enable-FEATURE here
ENABLE="--enable-ugni  \
       --enable-ldms-python \
       --enable-kgnilnd \
       --enable-lustre \
       --enable-tsampler \
       --enable-cray_power_sampler \
       --enable-cray_system_sampler \
       --enable-aries-gpcdr \
       --enable-aries_mmr"
#
# add --disable-FEATURE here
DISABLE="--disable-rpath \
        --disable-readline \
        --disable-baler \
        --disable-sos \
        --disable-mmap "
#
# libevent2 prefix
LIBEVENT_PREFIX=$BUILD_PATH/libevent-2.0_build
#
#
WITH="--with-rca=/opt/cray/rca/default --with-krca=/opt/cray/krca/default --with-cray-hss-devel=/opt/cray-hss-devel/default --with-aries-libgpcd=$BUILD_PATH/gpcd_build/lib/,$BUILD_PATH/gpcd_build/include/"
#
if [ -n "$LIBEVENT_PREFIX" ]; then
    WITH="$WITH --with-libevent=$LIBEVENT_PREFIX"
fi
#
# 
CFLAGS='-g -O0 -Wl,-z,defs -I/opt/cray/rca/default/include -DLDMSD_UPDATE_TIME'
#
# Exit immediately if a command failed
set -e
set -x
#
../configure --prefix=$PREFIX $ENABLE $DISABLE $WITH CFLAGS="$CFLAGS" LDFLAGS=$LDFLAGS CPPFLAGS=$CPPFLAGS
  • Nake configure.sh executable
chmod +x configure.sh
  • Create a build directory
mkdir build
  • Cd to build directory and run configure.sh
cd build
../configure.sh
  • Build and install ldms libs, executables, and includes
make && make install

Running LDMS

README files and sample scripts for the XC40 running Rhine Redwood reside in the ovis-master/util/sample_init_scripts/XC40_RR directory.

  • Note that the scripts must be modified, as described in the README file, to fit a particular deployment configuration.