Build w/ Spack

ECR

TO get everything started, let us login to ECR

export AWS_REGION="$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .region)"
export AWS_ACCT_ID="$(curl -s http://169.254.169.254/latest/dynamic/instance-identity/document | jq -r .accountId)"
aws ecr get-login-password --region ${AWS_REGION} | docker login --username AWS --password-stdin ${AWS_ACCT_ID}.dkr.ecr.${AWS_REGION}.amazonaws.com
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws 

Installing software optimized for the underlaying architecture is essential for HPC to get the performance the underlying hardware can deliver.

Package it up

With spack containerize we are able to bottle up the application into a single container that can be moved around farily easily using the Elastic Container Registry (ECR).

Use AWS env

Firstly, we log into one of the compute nodes. Afterwards we use the already configured aws environment file to get going.

cp /usr/local/spack/var/spack/environments/aws/spack.yaml spack.yaml
yq d -i spack.yaml spack.config
yq d -i spack.yaml spack.specs
yq w -i spack.yaml "spack.specs[+]" gromacs
yq w -i spack.yaml spack.concretization together
yq w -i spack.yaml spack.container.images.build public.ecr.aws/a4y4t9s0/spack/amazonlinux:2.0.20201111.0
yq w -i spack.yaml spack.container.images.final public.ecr.aws/amazonlinux/amazonlinux:2.0.20201111.0
yq w -i spack.yaml spack.container.os_packages.command yum
yq d -i spack.yaml spack.container.os_packages.final
yq w -i spack.yaml "spack.container.os_packages.final[+]" libgomp
yq w -i spack.yaml "spack.container.os_packages.final[+]" libgfortran

To build the container we first create a Dockerfile.

spack containerize > Dockerfile
sed -i -e 's/spack install --fail-fast/spack install --no-check-signature --fail-fast/' Dockerfile
sed -i -e 's/RUN yum update -y && yum install -y epel-release &&/RUN/' Dockerfile

The Dockerfile looks like this:

# Build stage with Spack pre-installed and ready to be used
FROM public.ecr.aws/a4y4t9s0/spack/amazonlinux:2.0.20201111.0 as builder


# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir /opt/spack-environment \
&&  (echo "spack:" \
&&   echo "  view: /opt/view" \
&&   echo "  concretization: together" \
&&   echo "  packages:" \
&&   echo "    all:" \
&&   echo "      providers:" \
&&   echo "        blas:" \
&&   echo "        - openblas" \
&&   echo "        mpi:" \
&&   echo "        - mpich" \
&&   echo "      variants: +mpi" \
&&   echo "      version: []" \
&&   echo "      target: []" \
&&   echo "      compiler: []" \
&&   echo "      buildable: true" \
&&   echo "    mpich:" \
&&   echo "      variants: ~wrapperrpath netmod=ofi" \
&&   echo "      version:" \
&&   echo "      - 3.2.1" \
&&   echo "      target: []" \
&&   echo "      compiler: []" \
&&   echo "      buildable: true" \
&&   echo "      providers: {}" \
&&   echo "    binutils:" \
&&   echo "      variants: +gold+headers+libiberty~nls" \
&&   echo "      version:" \
&&   echo "      - 2.33.1" \
&&   echo "      target: []" \
&&   echo "      compiler: []" \
&&   echo "      buildable: true" \
&&   echo "      providers: {}" \
&&   echo "    cmake:" \
&&   echo "      version:" \
&&   echo "      - 3.17.3" \
&&   echo "      target: []" \
&&   echo "      compiler: []" \
&&   echo "      buildable: true" \
&&   echo "      providers: {}" \
&&   echo "    openfoam:" \
&&   echo "      variants: +paraview" \
&&   echo "      version:" \
&&   echo "      - develop" \
&&   echo "      target: []" \
&&   echo "      compiler: []" \
&&   echo "      buildable: true" \
&&   echo "      providers: {}" \
&&   echo "    paraview:" \
&&   echo "      variants: +osmesa+python3" \
&&   echo "      version:" \
&&   echo "      - 5.8.1" \
&&   echo "      target: []" \
&&   echo "      compiler: []" \
&&   echo "      buildable: true" \
&&   echo "      providers: {}" \
&&   echo "    mesa:" \
&&   echo "      variants: swr=avx,avx2" \
&&   echo "      version:" \
&&   echo "      - 18.3.6" \
&&   echo "      target: []" \
&&   echo "      compiler: []" \
&&   echo "      buildable: true" \
&&   echo "      providers: {}" \
&&   echo "    llvm:" \
&&   echo "      version:" \
&&   echo "      - 6.0.1" \
&&   echo "      target: []" \
&&   echo "      compiler: []" \
&&   echo "      buildable: true" \
&&   echo "      providers: {}" \
&&   echo "    hwloc:" \
&&   echo "      version:" \
&&   echo "      - 1.11.11" \
&&   echo "      target: []" \
&&   echo "      compiler: []" \
&&   echo "      buildable: true" \
&&   echo "      providers: {}" \
&&   echo "  definitions:" \
&&   echo "  - hackathon:" \
&&   echo "    - openfoam" \
&&   echo "    - gromacs" \
&&   echo "  - arch:" \
&&   echo "    - '%gcc@7.3.1 arch=linux-amzn2-skylake_avx512'" \
&&   echo "  mirrors:" \
&&   echo "    mirror: s3://spack-mirrors/amzn2-e4s" \
&&   echo "  compilers:" \
&&   echo "  - compiler:" \
&&   echo "      spec: gcc@7.3.1" \
&&   echo "      paths:" \
&&   echo "        cc: /usr/bin/gcc" \
&&   echo "        cxx: /usr/bin/g++" \
&&   echo "        f77: /usr/bin/gfortran" \
&&   echo "        fc: /usr/bin/gfortran" \
&&   echo "      flags: {}" \
&&   echo "      operating_system: amzn2" \
&&   echo "      target: x86_64" \
&&   echo "      modules: []" \
&&   echo "      environment:" \
&&   echo "        unset: []" \
&&   echo "      extra_rpaths: []" \
&&   echo "  specs:" \
&&   echo "  - gromacs" \
&&   echo "  config:" \
&&   echo "    install_tree: /opt/software") > /opt/spack-environment/spack.yaml

# Install the software, remove unnecessary deps
RUN cd /opt/spack-environment && spack env activate . && spack install --no-check-signature --fail-fast && spack gc -y

# Strip all the binaries
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
    xargs file -i | \
    grep 'charset=binary' | \
    grep 'x-executable\|x-archive\|x-sharedlib' | \
    awk -F: '{print $1}' | xargs strip -s

# Modifications to the environment that are necessary to run
RUN cd /opt/spack-environment && \
    spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh


# Bare OS image to run the installed executables
FROM public.ecr.aws/amazonlinux/amazonlinux:2.0.20201111.0

COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software
COPY --from=builder /opt/view /opt/view
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh


ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]
docker build -t gromacs:$(spack arch) .

This will run spack install within a container and build all its dependencies:

Unfortunately, this fails on skylake: