Theta-ALCF (XC40)¶

Theta is a Cray XC40 located at Argonne National Laboratory.

$ ssh -XY <username>@theta.alcf.anl.gov

Building WEST¶

WEST executables can be compiled using the following script:

$ cat build_theta.sh
#!/bin/bash

module load miniconda-3.6/conda-4.5.12

export BLAS_LIBS="-L$MKLROOT/intel64/lib -Wl,--start-group -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -Wl,--end-group"
export SCALAPACK_LIBS="-L$MKLROOT/intel64/lib -Wl,--start-group -lmkl_scalapack_lp64 -lmkl_blacs_intelmpi_lp64 -lmkl_intel_lp64 -lmkl_core -lmkl_intel_thread -Wl,--end-group"
export FFT_LIBS=""
export MPIF90="ftn -g -mkl -dynamic"
export CC="cc -g -mkl -dynamic"
export F77="ftn -g -mkl -dynamic"
export FFLAGS="-xMIC-AVX512 -align array64byte -fp-model fast=2 -no-prec-div -assume byterecl -dynamic"

export CRAYPE_LINK_TYPE=dynamic

./install/configure --host=x86_64-build-linux-gnu --build=x86_64-target-linux-gnu --enable-parallel --with-scalapack --enable-openmp LD_LIBS="`python3-config --ldflags`"

make pw -j 16

cd West
make

Running WEST Jobs¶

The following is an example executable script run_west.sh to run the wstat.x WEST executable on two nodes of Theta with 64 MPI ranks per node. The job can be submitted with command qsub run_west.sh where <project_name> is replaced with an active project allocation.

$ cat run_west.sh
#!/bin/bash
#COBALT -n 2 -t 10 -q debug-cache-quad -A <project_name> -O WEST
module load miniconda-3.6/conda-4.5.12
aprun -n 128 -N 64 -d 1 --cc depth -e OMP_NUM_THREADS=1 -j 1 ./wstat.x -i wstat.in > wstat.out

Job submission is done with the following:

$ qsub run_west.sh