Getting Started

This section summarizes points that are of importance to new users.

Installation

Have a look at the Installation section for detailed instructions on the separate tools. If you are only interested installation instructions for CFDEM®coupling see Compile CFDEMcoupling.

Case structure

We recommend to keep the CFD and DEM setups in separate sub folders. This approach is taken in all tutorials provided, where these folders are:

  • a directory called CFD covering the CFD relevant settings and data

  • a directory called DEM covering the DEM relevant settings and data

  • the root folder contains setup scripts

This division allows to easily expand a pure CFD or DEM simulation case to a coupled case.

The tutorials can be found in the directory $CFDEM_PROJECT_DIR/tutorials, which can be reached by typing cfdemTut.

Set up a coupled simulation

You can either

  • start with stand-alone CFD and DEM cases

  • start from an existing tutorial case

Virtually all settings for CFDEM®coupling are made in the file constant/couplingProperties the contents of which are documented in couplingProperties.

Also have a look at available tutorial cases.

For settings related to the solver such as discretization schemes, solution settings for the equations and solver control (e.g., corrector or non-orthogonal corrector steps for non-orthogonal meshes) you can find more information in OpenFOAM® documentations (*).

Like in OpenFOAM® collated IO is possible when adding the following settings to controlDict:

OptimisationSwitches
{
    fileHandler collated;
}

Run a simulation

Coupled simulations are most easily run using the cfdemSimulate command or the provided Autorun.sh script in the case root directory.

Note

Make sure the correct environment is loaded before running cfdemSimulate. If you set the environment variables listed in Compile CFDEMcoupling in your .bashrc, then the correct environment is loaded by default.

On a HPC system you most probably need to load the correct module.

Provided that the installation is correct, the tutorials can be run via the Autorun.sh shell scripts. The successful run of the script might need some third party software (e.g., Python, Octave, Evince, etc.).

A typical Autorun.sh script executes the following steps in the two folders:

  • DEM: Aspherix® initialization run (e.g., to produce an initial particle packing)

  • DEM: parallel Aspherix® run with socket communication (corresponding command: mpirun -np X aspherix -in coupled.asx)

  • CFD: mesh generation (e.g., using blockMesh)

  • CFD: mesh decomposition (decomposePar)

  • CFD: parallel CFDEM run with socket communication (corresponding command: mpirun -np X cfdemSolverXXX -parallel &)

  • Post-processing (e.g., using Python or Octave)

In addition to these steps, the Autorun.sh script will prompt the user to select a variant of the test case. In the Allrun.sh script, the user has to set the selected variant manually.

An example of a manual run is described in the Ergun test example page.

Note

Some cases do not contain an Autorun.sh script but the older variant which is typically called Allrun.sh. These scripts features less automation and the user has to select the case variant manually from within the script.

Run a simulation on GPU

CFDEM®coupling can be compiled for GPU with the SPUMA derivation of OpenFOAM®-v2412 (see instructions).

The fluidized bed use case has specifically been engineered and verified for both CFDEM®coupling with SPUMA on GPU and CFDEM®coupling with OpenFOAM-10 on CPU. Smaller case setups are in Settling of a single particle vs. analytic solution or Ergun test.

When running a case from a SPUMA environment using the Autorun.sh script, both the DEM and the CFD are executed in serial on the GPU and decomposition settings are ignored. Moreover, the decomposition step is skipped for the CFD.

Note

While other tutorials, validation cases or use cases may function with the GPU version, they are currently untested and unvalidated. Due to underlying architectural differences between the OpenFOAM packages and the underlying matrix solvers, we can neither guarantee that the provided tutorials / cases will run out of the box nor that the results are consistent.

Run a simulation (advanced)

When running a coupled simulation, the DEM and CFD softwares are executed separately and only exchange necessary trivial data (such as particle position, velocity and dragforce) via socket communication. CFDEM settings for socket communication are explained in the twoWaySocket model.

In general, a coupled simulation can easily be run using the cfdemSimulate command, which launches the CFD and DEM simulations separately. Assuming the folder structure discussed above, cfdemSimulate executes the following operations:

# start DEM
cd ../DEM
mpirun -np 4 aspherix -in run.asx &
# start CFD alongside
cd ../CFD
mpirun -np 4 cfdemSolverPiso -parallel

All CFDEM®coupling tutorials contain an Autorun.sh script which performs these same operations with more convenience functionality.

On clusters or for special use cases, the user may need to start both simulations manually using a sdcript similar to the one indicated above in order to pass the required arguments.

For running simulations using the slurm workload manager see here

Cluster, MPI and multi-node Information

On large scale multi-node clusters, the default behaviour of process binding (core) and placement is fine. Equally ranked processes of the DEM executable must be on the same node as the cfdemSolver processes. E.g., if you have 2 nodes with 4 cores each, the ranks 0-3 of both executables need to be on one node and 4-7 on the other. In queuing system scripts two mpirun commands are now necessary, simply putting the first in the background with & works fine.

Some communication libraries, e.g. Intel PSM2 launch MPI jobs with exclusive resource allocation, such as Infiniband communication lanes, and will not allow 2 MPI applications running at the same time on the same hardware. This can be circumvented with MPI and environment options at launch-time and depend on the specific library that is used. Please contact your system administrator for information on the exact options to be used.

Visualization

Data can be visualized with any tool that can read OpenFOAM® and VTK data, for instance ParaView.

To load OpenFOAM® data, load a *.foam file which is usually located in the CFD folder or in CFD/system. Loading this file will import all CFD meshes and results at different times. By default data for a non-decomposed case is loaded. To load decomposed data, choose Decomposed Case as Case Type.

To load Aspherix particle data, load the aspherix_simulation.pvd file created by the output_settings command which is usually located in DEM/post. If you defined a different name or location for this file, choose the according file. If running a coupled case, the default settings will ensure particle data is available for each writing step in the CFD.

Graphical User Interface

The whole workflow of case setup, execution and post-processing of a coupled CFD-DEM simulation can be done either based on graphical user interface (GUI). The “pure fluid part” of the setup can be done by existing GUIs for OpenFOAM® (one possible option is the open source software Helyx-OS provided by Engys). The “pure particle part” of the setup can be done by the Aspherix-GUI, which also allows for setting fluid-particle interaction such as the drag model.

If you are interested in a GUI based workflow, please feel free to get in touch with DCS Computing.

(*) This offering is not approved or endorsed by OpenCFD Limited, the producer of the OpenFOAM software and owner of the OPENFOAM® and OpenCFD® trade marks.