Running Abaqus on ARCHIE-WeSt
This is a simple example of Abaqus which should run on ARCHIE-WeSt and take around 15 minutes. The input files and job script are downloadable below. The example used is the standard Abaqus S4b benchmark that simulates the bolting of a cylinder head onto an engine block, which has 5,000,000 degrees of freedom (DOF) and uses the direct solver (Abaqus Standard).
Known Issues
Nov 2015
It is currently not possible to run Abaqus from the lustre filesystem. This is due to the fact that Abaqus requires to perform a file lock (flock) on temporary data files that it creates. File locking is not enabled on lustre for performance reasons.
Therefore, Abaqus should be run from the /users file system.
This issue is apparently being rectified in Abaqus 6.14
Licensing
Abaqus is commercial software with licenses being shared at Strathclyde between MAE, Naval Architecture, Ocean and Marine Engineering and Bioengineering. There is a license pool of 144 token available the more cores you use, the more tokens are consumed. The following table lists the tokens used by typical HPC jobs:
12 cores | 14 tokens |
12 cores + 1 GPU | 15 tokens |
24 cores | 19 tokens |
24 cores + 2 GPU’s | 20 tokens |
1. Load the appropriate environment modules
To run Abaqus load the appropriate module e.g. :
module load apps/bin/abaqus/6.13.2
2. Prepare input files
The following files are required to be present in the working directory (input files):
- s4b.inp
- job script: abaqus-standard-parallel-12core.sh (or alternative)
These can be downloaded as a tar file from here.
The job script (abaqus-standard-parallel.sh) will use 1 node (12 cores):
# # ************* SGE qsub options **************** #Export env variables and keep current working directory #$ -V -cwd #Select parallel environment and number of parallel queue slots (nodes) #$ -pe multiway 12 # Add Project Identifier #$ -P testing.prj #Combine STDOUT/STDERR #$ -j y #Specify output file #$ -o out.$JOB_ID #Request resource reservation (reserve slots on each scheduler run until enough have been gathered to run the job #$ -R y # Indicate runtime #$ -l h_rt=12:00:00 # ************** END SGE qsub options ************ abaqus job=s4b input=s4b.inp cpus=$NSLOTS mp_mode=mpi interactive |
3. Job submission
To submit the job, type:
qsub abaqus-standard-parallel.sh
As a result, the following files will be produced (output files):
- s4b.com
- s4b.sim
- s4b.odb
- s4b.dat
- s4b.msg
- s4b.prt
- s4b.sta
- out.JOBID
4. Output
The output files can be downloaded as a tar file from here for comparison.
5. Using GPU assisted nodes
Abaqus can take advantage of GPU assisted nodes by adding GPUS=N to the abaqus command line:
abaqus job=s4b input=s4b.inp cpus=12 gpus=1 interactive
for a single node (12 cores) with an additional GPU. On ARCHIE a single GPU boosts performance by almost 25%.
6. Performance
As the benchmark figures below indicate, the addition of a single GPU (NVidia Tesla M2075) has the greatest performance impact of almost 25%, and performs almost as well as the 24-core non-gpu case. Given that there is a finite pool of licenses available, this is probably the most optimal method of running Abaqus on ARCHIE.