Running Abaqus

Running Abaqus on ARCHIE-WeSt

.

This is a simple example of Abaqus which should run on ARCHIE-WeSt and take around 15 minutes. The input files and job script are downloadable below. The example used is the standard Abaqus S4b  benchmark that simulates the bolting of a cylinder head onto an engine block, which has 5,000,000 degrees of freedom (DOF) and uses the direct solver (Abaqus Standard).

Known Issues

Nov 2015

It is currently not possible to run Abaqus from the lustre filesystem. This is due to the fact that Abaqus requires to perform a file lock (flock) on temporary data files that it creates. File locking is not enabled on lustre for performance reasons.

Therefore, Abaqus should be run from the /users file system.

This issue is apparently being rectified in Abaqus 6.14

Licensing

Abaqus is commercial software with licenses being shared at Strathclyde between MAE, Naval Architecture, Ocean and Marine Engineering and Bioengineering. There is a license pool of 144 token available the more cores you use, the more tokens are consumed. The following table lists the tokens used by typical HPC jobs:

12 cores 14 tokens
12 cores + 1 GPU 15 tokens
24 cores 19 tokens
24 cores + 2 GPU’s 20 tokens

 
 
 
 

1. Load the appropriate environment modules

.

To run Abaqus load the appropriate module e.g. :

 module load apps/bin/abaqus/6.13.2

2. Prepare input files

.

The following files are required to be present in the working directory (input files):

    • s4b.inp
    • job script: abaqus-standard-parallel-12core.sh  (or alternative)

These can be downloaded as a tar file from here.

The job script (abaqus-standard-parallel.sh) will use 1 node (12 cores):

#
# ************* SGE qsub options ****************
#Export env variables and keep current working directory
#$ -V -cwd
#Select parallel environment and number of parallel queue slots (nodes)
#$ -pe multiway 12
# Add Project Identifier
#$ -P testing.prj
#Combine STDOUT/STDERR
#$ -j y
#Specify output file
#$ -o out.$JOB_ID
#Request resource reservation (reserve slots on each scheduler run until enough have been gathered to run the job
#$ -R y
# Indicate runtime
#$ -l h_rt=12:00:00
# ************** END SGE qsub options ************
abaqus job=s4b input=s4b.inp cpus=$NSLOTS mp_mode=mpi interactive
.

3. Job submission

.

To submit the job, type:

qsub abaqus-standard-parallel.sh
.

As a result, the following files will be produced (output files):

  • s4b.com
  • s4b.sim
  • s4b.odb
  • s4b.dat
  • s4b.msg
  • s4b.prt
  • s4b.sta
  • out.JOBID

4. Output

The output files can be downloaded as a tar file from here for comparison.

5. Using GPU assisted nodes

Abaqus can take advantage of GPU assisted nodes by adding GPUS=N to the abaqus command line:

 abaqus job=s4b input=s4b.inp cpus=12 gpus=1  interactive

for a single node (12 cores) with an additional GPU. On ARCHIE a single GPU boosts performance by almost 25%.

6. Performance

As the benchmark figures below indicate, the addition of a single GPU (NVidia Tesla M2075) has the greatest performance impact of almost 25%, and performs almost as well as the 24-core non-gpu case. Given that there is a finite pool of licenses available, this is probably the most optimal method of running Abaqus on ARCHIE.
abaqus-benchmark