Purpose

Script to comfortably launch CFDEMcoupling simulations with Aspherix in serial or parallel

Warning

GPU support for this model or solver has not been tested and may not work as expected.

cfdemSimulate

Syntax

cfdemSimulate
  required arguments:
    -s -solver SOLVER     use the cfdemSolver or pure cfdSolver named SOLVER
  optional arguments:
    -i -in FILE           path to Aspherix input script FILE, setting a FILE will start Aspherix simultaneously to SOL
    -c -case FOLDER       path to the CFDEM case folder containing 0, system, constant ...
    -p -np NP             use NP number of processors in parallel
    -n -init FILE         path to Aspherix input script file to run a DEM initialization first with the same MPI parameters
    -l -location          instead of executing Aspherix in the folder the input script resides in, launch it in the CFDEM case folder
    -d -decompose         decompose CFD case before execution, runs a decomposePar -force (does not decompose by default)
    -k -keepPortOffset    keep portOffsetFiles, which are otherwise deleted by default
    -m -mpicmd CMD        change the mpi launch command to CMD, default is $mpicmd
    -o -mpiopts "OPTS"    adds custom string "OPTS" to the mpi launch command as arguments
    -b -useSlurm          use slurm as scheduler, sets mpicmd to use 'srun' and other options accordingly,
                            also set by the environment variable $CFDEM_SIM_USE_SLURM
    -r -useSlurmHead      use resources as specified by SBATCH pragmas, ignores np setting,
                            also set by the environment variable $CFDEM_SIM_SLURM_SBATCH
    -u -uniqueLogFiles    make log file names unique by appending a timestamp, i.e. keep do not override old logs,
                            this option will also change custom log file names
    -a -asxopts "OPTS"    add additional string "OPTS" to Aspherix run command, the full "OPTS" string should be quoted
    -v -var 'VAR VALUE'   set a variable in ASX script via command line
                            variable name and value must be given as space separated key value pair, enclosed in quotes
                            this option may be called several times
    -x -cfdemLog FILE     store cfdemSolver output in FILE,
                            default is $cfdemLog
    -y -asxLog FILE       store Aspherix output in FILE,
                            default is $asxLog
    -dryrun               assemble and pring commands, but do not execute
    -h -help              help, display this text

Examples

cfdemSimulate -in ../DEM/run.asx -solver cfdemSolverPiso -np 4
cfdemsimulate -solver cfdemSolverPiso -in DEM/run.asx -var "myVar 1"
cfdemSimulate -i DEM/run.asx -n DEM/init.asx -c CFD/ -s cfdemSolverPiso -np 16 -o "--oversubscribe"
cfdemSimulate -s cfdemPostproc

Description

cfdemSimulate is a short hand bash-command to start a coupled run. The command internally starts Aspherix with the specified input script and the desired CFDEM solver in parallel. Both, the CFD and the DEM, will be started on the stated number of cores. By default, the command will start the DEM first and wait for the DEM to open the socket communication before starting the CFD side. If no -np flag is stated, a serial run with 1 process each is started.

cfdemSimulate is copied to the executable directory $CFDEM_APP_DIR during the compilation of the CFDEM utilities and is, thus, available as any other solver in CFDEMcoupling.

To decrease the start-up time during socket initialization you can set portFilePath "./"; in your twoWaySocketProps, see twoWaySocket.

On clusters using slurm this command can be configured to use srun instead of mpirun using the -useSlurm flag and will then also use the correct options. With the flag -useSlurmHead resource requirements are handled by slurm and, consequently, the -np argument is ignored. Moreover, the --overlap option will always be used for coupled runs.

Alternatively, the same behavior can be set globally by the environment variables:

• CFDEM_SIM_USE_SLURM: export CFDEM_SIM_USE_SLURM="" will have the same effect as -useSlurm

• CFDEM_SIM_SLURM_SBATCH: export CFDEM_SIM_SLURM_SBATCH="" will have the same effect as -useSlurmHead

Note, that the actual value assigned to these variables does not matter, only the existence of the variable is considered.

In case, you are using SPUMA as OpenFOAM core and do not manually set this option, cfdemSimulate will automatically execute Aspherix with the -gpu command line argument and restrict execution to serial mode.

Restrictions

Note

This command must be run from the CFD case folder containing (e.g.) the constant and system folders.