Protection Domain Tags (Cray Specific)
Note: If you are going to be using the “ugni” transport (RDMA over Gemini) you will need to run with either system (as root) or user (as user) ptags. While root CAN run using any ptag the fact that its use is unknown to ALPS could cause collisions with applications.
To see current ptags (XE/XK):
> apstat -P PDomainID Type Uid PTag Cookie LDMS system 0 84 0xa9380000
To see current ptags (XC):
> apstat -P PDomainID Type Uid Cookie Cookie2 LDMS system 0 0x86b80000 0
To create a userspace ptag (XE/XK/XC):
apmgr pdomain -c <somenamehere>
Example (XE/XK):
> apmgr pdomain -c foo > apstat -P PDomainID Type Uid PTag Cookie LDMS system 0 84 0xa9380000 foo user 12345 233 0xa1230000
Note: A system administrator will have to setup system ptags and/or enable users to set up ptags in the following way.
- In /etc/alps.conf file in sharedroot modify the following:
# Variables to enable user/system protection domains
#
# Max # of user protection domains allowed
pDomainMax 10
# List of system protection domains used by all applications
pDomainIDs sys1,sys2
Example (XC):
> apmgr pdomain -c foo > apstat -P PDomainID Type Uid Cookie Cookie2 LDMS system 0 0x86b80000 0 foo user 20596 0x86bb0000 0x86bc0000
To remove a userspace ptag (XE/XK/XC):
apmgr pdomain -r <somenamehere>
Note: The userid of the ptag being removed must match that of the user running the command or root
Set the following environment variables for either user or system ptags (example shows user ptag values) (XE/XK):
export LDMS_UGNI_PTAG 233 export LDMS_UGNI_COOKIE 0xa1230000
Set the following environment variables for either user or system ptags (example shows user ptag values) (XC):
export LDMS_UGNI_PTAG 0 export LDMS_UGNI_COOKIE 0x86bb0000
When running with user space ptags you must specify the ptag name when using aprun (XE/XK/XC):
aprun <<usual aprun args here>> -p foo ldmsd <<usual ldmsd flags here>> or aprun <<usual aprun args here>> -p foo ldms_ls <<usual ldms_ls flags here>> Note: On some systems you will run aprun after a qsub -I or within a script specified in qsub or similar.
Modify your build environment to use the gnu compiler
module unload PrgEnv-XXX (if XXX is not gnu) module load PrgEnv-gnu
Building LDMS and support libraries and headers
- Download ovis and gpcd from github.com/ovis-hpc
- Download libevent-2.0.x-stable from libevent.org
- In this example source files are assumed to be in ~/Source and builds in ~/Build
Building gpcd libraries and header files
- cp <path>/gpcd-master.zip ~/Source
- cd ~/Source
- unzip gpcd-master.zip
- cd gpcd-master
- Run autogen.sh
./autogen.sh configure.ac:6: warning: AM_INIT_AUTOMAKE: two- and three-arguments forms are deprecated. For more info, see: configure.ac:6: http://www.gnu.org/software/automake/manual/automake.html#Modernize-AM_005fINIT_005fAUTOMAKE-invocation
NOTE: This warning is ok. Makefile.am will soon be updated to address this warning.
- Create config file with the following: — in this example configure.sh
../configure --prefix=<absolute path>/Build/gpcd_build
- make configure.sh executable
chmod +x configure.sh
- Create a build directory
mkdir build
- cd to build directory and run configure.sh
cd build ../configure.sh
- Build and install gpcd libs and includes
make && make install
- Create link from libgpcdlocal.a to libgpcd.a in the build dir
- cd ~Build/gpcd_build/lib
- ln -s libgpcd.a libgpcdlocal.a
Building libevent-2.0
- cp <path>/libevent-2.0.x-stable.tar.gz ~/Source
- cd ~/Source
- untar libevent-2.0.x-stable.tar.gz
- cd libevent-2.0.x-stable
- Run autogen.sh
./autogen.sh
- Create config file with the following: — in this example configure.sh
../configure --prefix=<absolute path>/Build/libevent-2.0_build
- make configure.sh executable
chmod +x configure.sh
- Create a build directory
mkdir build
- cd to build directory and run configure.sh
cd build ../configure.sh
- Build and install libevent-2.0 libs and includes
make && make install
Building LDMS
- cp <path>/ovis-master.zip ~/Source
- cd ~/Source
- unzip ovis-master.zip
- cd ovis-master
- Run autogen.sh
./autogen.sh
- Create config file with the following: — in this example configure.sh
#!/bin/bash
#
# SYNOPSIS: Remove existing build directories, do the automake routine, rebuild,
# and install everything.
#
# REMARK: This script doesn't do uninstall. If you wish to uninstall (e.g. make
# uninstall), please go into each build directories ( */build-$HOSTNAME ) and
# call make uninstall there, or just simply do the following
# for D in */build-$HOSTNAME; do pushd $D; make uninstall; popd; done;
#
#
BUILD_PATH=<absolute path to builds>
PREFIX=$BUILD_PATH/OVIS-3.3
#
# add --enable-FEATURE here
ENABLE="--enable-ugni \
--enable-ldms-python \
--enable-kgnilnd \
--enable-lustre \
--enable-tsampler \
--enable-cray_power_sampler \
--enable-cray_system_sampler \
--enable-aries-gpcdr \
--enable-aries_mmr"
#
# add --disable-FEATURE here
DISABLE="--disable-rpath \
--disable-readline \
--disable-baler \
--disable-sos \
--disable-mmap "
#
# libevent2 prefix
LIBEVENT_PREFIX=$BUILD_PATH/libevent-2.0_build
#
#
WITH="--with-rca=/opt/cray/rca/default --with-krca=/opt/cray/krca/default --with-cray-hss-devel=/opt/cray-hss-devel/default --with-aries-libgpcd=$BUILD_PATH/gpcd_build/lib/,$BUILD_PATH/gpcd_build/include/"
#
if [ -n "$LIBEVENT_PREFIX" ]; then
WITH="$WITH --with-libevent=$LIBEVENT_PREFIX"
fi
#
#
CFLAGS='-g -O0 -Wl,-z,defs -I/opt/cray/rca/default/include -DLDMSD_UPDATE_TIME'
#
# Exit immediately if a command failed
set -e
set -x
#
../configure --prefix=$PREFIX $ENABLE $DISABLE $WITH CFLAGS="$CFLAGS" LDFLAGS=$LDFLAGS CPPFLAGS=$CPPFLAGS
- Nake configure.sh executable
chmod +x configure.sh
- Create a build directory
mkdir build
- Cd to build directory and run configure.sh
cd build ../configure.sh
- Build and install ldms libs, executables, and includes
make && make install
Running LDMS
README files and sample scripts for the XC40 running Rhine Redwood reside in the ovis-master/util/sample_init_scripts/XC40_RR directory.
- Note that the scripts must be modified, as described in the README file, to fit a particular deployment configuration.