IntelMKL

Intel MKL

Next, we are going to use the math library Intel MKL.

salloc -p c5n -N1 --exclusive

Once the job is Running, we are going to log in.

ssh $SLURM_NODELIST
spack env list
spack env activate aws
spack env list

We are using spack to install GROMACS using intel-mkl.

time spack install --no-check-signature --no-checksum gromacs@2021.1 ^intel-mkl

It is able to reuse a lot of the packages we installed earlier.

The installation took about 5min. Again, we are capturing the hash.

read -p "Please paste the hash: " GROMACS_INTELMKL_HASH

We commit the variable to our environment in case we need to log in again.

echo "export GROMACS_INTELMKL_HASH=${GROMACS_INTELMKL_HASH}" |tee -a ~/.bashrc

Now we release the allocation:

exit

36 Ranks

source ~/.bashrc
cat > gromacs-single-node-c5n-intelmkl-36x2.sbatch << \EOF
#!/bin/bash
#SBATCH --job-name=gromacs-single-node-c5n-intelmkl-36x2
#SBATCH --exclusive
#SBATCH --output=/fsx/logs/%x_%j.out
#SBATCH --partition=c5n
#SBATCH --ntasks-per-node=36
NTOMP=2

mkdir -p /fsx/jobs/${SLURM_JOBID}
cd /fsx/jobs/${SLURM_JOBID}

spack env activate aws
echo ">>> spack load /GROMACS_INTELMKL_HASH"
spack load /GROMACS_INTELMKL_HASH

set -x
time mpirun gmx_mpi mdrun -ntomp ${NTOMP} -s /fsx/input/gromacs/benchRIB.tpr -resethway
EOF
sed -i -e 's/GROMACS_INTELMKL_HASH/'${GROMACS_INTELMKL_HASH}'/' gromacs-single-node-c5n-intelmkl-36x2.sbatch
sbatch gromacs-single-node-c5n-intelmkl-36x2.sbatch
sbatch gromacs-single-node-c5n-intelmkl-36x2.sbatch

72 Ranks

source ~/.bashrc
cat > gromacs-single-node-c5n-intelmkl-72x1.sbatch << \EOF
#!/bin/bash
#SBATCH --job-name=gromacs-single-node-c5n-intelmkl-72x1
#SBATCH --exclusive
#SBATCH --output=/fsx/logs/%x_%j.out
#SBATCH --partition=c5n
#SBATCH --ntasks-per-node=72
NTOMP=1

mkdir -p /fsx/jobs/${SLURM_JOBID}
cd /fsx/jobs/${SLURM_JOBID}

spack env activate aws
echo ">>> spack load /GROMACS_INTELMKL_HASH"
spack load /GROMACS_INTELMKL_HASH

set -x
time mpirun gmx_mpi mdrun -ntomp ${NTOMP} -s /fsx/input/gromacs/benchRIB.tpr -resethway
EOF
sed -i -e 's/GROMACS_INTELMKL_HASH/'${GROMACS_INTELMKL_HASH}'/' gromacs-single-node-c5n-intelmkl-72x1.sbatch
sbatch gromacs-single-node-c5n-intelmkl-72x1.sbatch
sbatch gromacs-single-node-c5n-intelmkl-72x1.sbatch

Results

After those runs are done, we grep the performance results.

grep -B2 Performance /fsx/logs/gromacs-single-node-c5n-intelmkl-*

This extends the table started with decomposition.

# execution spec instance Ranks x Threads ns/day
1 native gromacs@2021.1 c5n.18xl 18 x 4 4.7
2 native gromacs@2021.1 c5n.18xl 36 x 2 5.3
3 native gromacs@2021.1 c5n.18xl 72 x 1 5.5
4 native gromacs@2021.1 ^intel-mkl c5n.18xl 36 x 2 5.4
5 native gromacs@2021.1 ^intel-mkl c5n.18xl 72 x 1 5.5

Slightly better.