Compile from sources for GPU

Setup the environment

Note

The current version of CFDEM®coupling for the GPU only compiles with the SPUMA derivation of OpenFOAM®-v2412. Moreover, CFDEM®coupling has only been tested with the NVIDIA backend.

Warning

These instructions assume that there is no OpenFOAM environment loaded when you begin. If you have already installed the CPU-version of CFDEMcoupling and automatically load the corresponding environment, please remove the automatic environment loading before you start installing the GPU-version. You can re-enable the automatic environment loading after you have finished this installation process.

SPUMA (Simulation Processing in Unified Memory Accelerators) is a CFD software released by Cineca based on OpenFOAM®. It implements a full GPU porting of OpenFOAM targeting NVIDIA and AMD GPUs. [Publication]

Prebuilt binaries are unfortunately not available so you will need to download and build, both, SPUMA and OpenFOAM yourself.

  • In addition to the packages required by OpenFOAM, you will need to download the Nvidia HPC SDK. This package provides the necessary CUDA compilers and libraries to compile SPUMA with the the Nvidia backend. Download and install Nvidia SDK according to instructions

  • Make sure that the Nvidia SDK is in your system PATH:

    export PATH="/opt/nvidia/hpc_sdk/Linux_x86_64/<sdk_version>/compilers/bin:$PATH"

    If you installed the HPC SDK to a different location, update above command accordingly.

  • Check if the correct Nvidia nvc++ compiler is functional by running:

    which nvc++     # full path to the compiler -- should match what you exported
nvc++ --version # the version number -- indicates if the compiler actually runs

    Note

    SPUMA requires the Nvidia HPC SDK – A normal CUDA installation is not sufficient!

  • Instead of native OpenFOAM, download and compile SPUMA. The following steps create the required folder structure and clone the repositories to the correct locations:

    cd $HOME
mkdir OpenFOAM
cd OpenFOAM
git clone https://gitlab-hpc.cineca.it/exafoam/spuma --branch develop
cd spuma
git checkout 9c06274b1cc8cc758656b5c55b31a5c6bee33421
git clone https://develop.openfoam.com/Development/ThirdParty-common.git --branch v2412 ThirdParty

    Note

    SPUMA expects the ThirdParty folder inside its root directory. This structure differs from conventional OpenFOAM which expects it one directory up.

    Refer to the complete set of instructions for compiling SPUMA in case you have specific needs or problems.

  • Unzip the CFDEMcoupling archive you received into the folder $HOME/DCS-Computing.

  • Source the bashrc file for CFDEMcoupling. It takes care of setting up the environment variables for SPUMA and CFDEMcoupling:

    export CFDEM_OF_DIR="$HOME/OpenFOAM/spuma"
export NVARCH=70
source $HOME/DCS-Computing/CFDEMcoupling-|package|-|version|/etc/bashrc

    In addition to the default setup for CPU, you need to add settings to account for the SPUMA path and set the architecture of your GPU by setting the CFDEM_OF_DIR variable. SPUMA will set the default compiler to the Nvidia compiler, i.e. export WM_COMPILER=Nvidia.

    Note

    If you are running on a zsh, source zshrc from the same location.

    Note

    The Nvidia HPC SDK must be on your PATH to compile SPUMA and CFDEMcoupling with GPU support. If you are on a new terminal, you need to add the SDK to your PATH variable as discussed above.

    Warning

    Setting NVARCH incorrectly will not cause any compilation errors, but you will have a runtime crash as the necessary GPU kernels have not been compiled.

Compile SPUMA (OpenFOAM® on GPU)

  • In the same terminal as you used to set the environment, compile OpenFOAM by executing the following steps:

    cd $WM_PROJECT_DIR
./Allwmake

    Depending on your hardware and the amount of cores used, compilation of SPUMA can take an hour or more.

  • To test your installation, check that the simpleFoam application is found:

    simpleFoam -help

    This command should print a “Usage” message to screen.

    Note

    Due to how the SPUMA build process for the GPU works, a failed or incorrectly configured compilation run may leave artifacts that prevent a proper re-compilation of the source code. If this issue occurs you may need to delete the build and platforms directories in the SPUMA root directory.

Compile CFDEMcoupling

  • Still in the same terminal, compile CFDEMcoupling by executing

    cfdemCompCFDEM

    from any location. This step behaves in the same manner as for the CPU compilation.

    Note

    Since compiling with SPUMA uses a different compiler toolchain than a normal OpenFOAM/CFDEM®coupling build it is possible to have both versions compiled and installed on your system. It is however not possible to have both versions loaded at the same time and you must make sure that you are not mixing the CPU and GPU environments. Have a look at the next section on approaches to persistently or conditionally set your environment.

  • In case questions concerning the installation arise, please feel free to contact DCS Computing.

  • Proceed to next step.