% sbatch -t 0-00:30 -n 1 --mem 2G job.sh
sbatch: INFO: Account: ccin2p3
sbatch: INFO: Submission node: cca001
sbatch: INFO: Partition set to: htc
sbatch: INFO: Partition limited to one node per job.
sbatch: INFO: Time limit set to: 0-00:30:00 (30 minutes)
Submitted batch job 936607
-t <j-hh:mm>
specifies the evaluated time limit. All acceptable formats are explained in Essential sbatch options paragraph.
-n <number>
specifies the number of tasks requested (in this syntax equivalent to the number of cores). For a multi-core job, <number> will be larger than 1.
--mem <number>
specifies the amount of memory requested.
job.sh
your executable task script.
The three parameters indicated above must always be expressed upon submission. The -c parameter (number of CPUs per task) can override -n (see Required parameters limits).
Upon submission, information is returned to you in the standard output: the computing group (Account, here ccin2p3), the submission node (here cca001), the partition htc (default) in which the job will be executed, or the job identifier 936607.
If we want to access a resource requiring declaration, we will use the option -L:
% sbatch -t 0-00:30 -n 4 --mem 2G -L sps,matlab job.sh
Submission script
It is possible to define submission options directly in a batch script using the following syntax:
#!/bin/sh
# SLURM options:
#SBATCH --job-name=serial_job_test    # Job name
#SBATCH --output=serial_test_%j.log   # Standard output and error log
#SBATCH --partition=htc               # Partition choice (htc by default)
#SBATCH --ntasks=1                    # Run a single task
#SBATCH --mem=2000                    # Memory in MB per default
#SBATCH --time=1-00:00:00             # Max time limit = 7 days
#SBATCH --mail-user=<email address>   # Where to send mail
#SBATCH --mail-type=END,FAIL          # Mail events (NONE, BEGIN, END, FAIL, ALL)
# Commands to be submitted:
module load python
python my_python_script.py
In this example, we set up a Python environment using modules in order to run the my_python_script.py. All the required Slurm options have been given using #SBATCH instructions, and thus this batch script could be submitted as easily as:
% sbatch my_batch_script.sh
Multiple tasks script
Multiple srun tasks may be launched from the script, provided that the resources declared in the #SBATCH rows are not exceeded. For example:
#!/bin/sh
#SBATCH --job-name=mulitple_jobs
#SBATCH --ntasks=2
#SBATCH --mem=2000
#SBATCH --time=1-00:00:00
#SBATCH --output=mulitple_jobs_%j.log
#SBATCH --licenses=sps
####################################
srun -n 1 --exclusive -L sps script_sps.sh &
srun -n 1 --exclusive script_software.sh &
srun -n 1 --cpus-per-task 2 script_software.sh
The --exclusive option allows the task not to share the resource. The bash syntax allows to:
&: run tasks in parallel; if a task does not find an available resource, it stays pending and a warning message is written in the output,
wait: wait for all tasks to complete before exiting the script.
When the submitted task is a script running multiple parallel processes, one must specifiy the number of cpus and the memory used per task:
#! /bin/bash
#SBATCH --time=1-00:00:00
#SBATCH --mem-per-cpu=1000 # Allocate 1G of memeory per core
#SBATCH --ntasks=2 # Run max 2 parallel tasks
#SBATCH --cpus-per-task=4 # Allocate 4 cores per task, i.e 8 cores in total in this example (ntasks x cpus-per-task)
my_parallel_job.sh
otherwise, all sub-processes launched by my_parallel_job.sh will run on a single CPU.
Interactive job
An interactive job submission is done with the srun command. In the case of a GPU interactive job, the appropriate partition must be specified.
% srun -t 0-08:00 -n 4 --mem 2G --pty bash -i    # HTC interactive session
% srun -t 0-08:00 -n 4 --mem 2G job.sh           # running interactively an HTC executable
% srun -p gpu_interactive -t 0-08:00 --mem 2G --gres=gpu:[type]:1 --pty bash -i   # GPU interactive session
Use the -L option for resources requiring declaration.
GPU [type] specification is optional: please refer to the GPU jobs syntax.
-p
to select the partition
--gres=
allows you to declare the use of a GPU resource, and to define its parameters.
--pty
allows interactivity with the open session (see previous note).
To quit the interactive session:
% exit
Parallel job (MPI)
These are jobs executing parallel operations, possibly on different computing servers, using an MPI type interface through an InfiniBand connection. The hpc partition must be specified:
% sbatch -p hpc -t 0-02:00 -n 8 --mem 2G -N 2 job.sh
-N <number>
specifies the number of required computing servers (nodes)
If the number of computing servers (-N option) is 1, it is not necessary to indicate it, nor to specify the partition.
GPU job
These are jobs that run on computing servers equipped with GPUs. Two syntaxes are allowed:
% sbatch -t 0-01:00 -n <N> --mem 2G --gpus 1 job.sh
% sbatch -t 0-01:00 -n <N> --mem 2G --gres=gpu:v100:1 job.sh # specific request of a GPU type
Here, we request the allocation of a single GPU, but it is possible to use several, up to the limit of GPUs in the corresponding node. The limit on the number of <N> tasks is explained in Required parameters limits.
A single type of GPUs is available at CC-IN2P3: Nvidia V100, labelled by the keyword v100.
Using CUDA
The CC-IN2P3 provide a complete Nvidia environment (drivers, CUDA, CUDnn and NCCL libraries), and upgrade it on a regular basis.
Important
The current version of the Nvidia drivers is 530.30.02-1; the associated CUDA version is 12.1.1-1, CUDnn version is 8.9.1.23 and NCCL version is 2.18.1.
If you need to use a previous Nvidia environment, the CC-IN2P3 is provided Apptainer images. These container images are available from the container repositery:
% ls -lsah /cvmfs/singularity.in2p3.fr/images/HPC/GPU
centos7_cuda10-0_cudnn7-4-2_nccl2-4-2.simg
centos7_cuda10-0_cudnn7-6-5_nccl2-5-6.sif
centos7_cuda10-1_cudnn7-5_nccl2-4-2.simg
centos7_cuda10-1_cudnn7-6_nccl2-7-6.sif
centos7_cuda11-3_cudnn8-2-0_nccl2-9-8.sif
centos7_cuda12-1_cudnn8-9-1_nccl2-18-1.sif
A more detailed documentation about how to use such containers is available in the CC-IN2P3 GitLab.
To compile your CUDA code, you should connect to an interactive GPU server and then use the nvcc compiler:
% /usr/local/cuda-12/bin/nvcc
Once the code is compiled, we recommend you to exit the interactive server and submit your jobs with sbatch.
To profile GPU jobs, CUPTI (CUDA Profiling Tools Interface) is installed on our computing nodes. Since CUPTI is directly linked to CUDA the installed version is the same.
Daemon job and recursivity
For long running jobs with low resource consumption (to motor or orchestrate other jobs) choose the htc_daemon partition (see Required parameters limits):
% sbatch -p htc_daemon -t 90-00:00 -n 1 --mem 1G job.sh
More generally, the computing time limit can be circumvented with a recursive job script which re-submits itself and remains queued until the first job disappears. The command line below should be written inside the script itself, preferably at the beginning of the script.
sbatch --dependency=afterany:$SLURM_JOBID my_batch_script.sh
$SLURM_JOBID being the identifier of the current job from which the job is launched.
Building a job pipeline
It is possible to build a job pipeline using the Slurm dependency option:
% sbatch --dependency=<type:job_id[:job_id][,type:job_id[:job_id]]> -t 0-01:00 -n 2 --mem 2G job.sh
Using the following dependencies types :
after:jobid[:jobid...]
job can begin after the specified jobs have started
afterany:jobid[:jobid...]
job can begin after the specified jobs have terminated
afternotok:jobid[:jobid...]
job can begin after the specified jobs have failed
afterok:jobid[:jobid...]
job can begin after the specified jobs have run to completion with an exit code of zero
singleton
jobs can begin execution after all previously launched jobs with the same name and user have ended. This is useful to collate results of a swarm or to send a notification at the end of a swarm.
A more detailed example of such pipeline is given on this page.