.. _ch:slurm:

Slurm
=====

Slurm, which stands for "Simple Linux Utility for Resource Management," is an open-source cluster management and job scheduling system. Its primary purpose is to efficiently manage and schedule the allocation of computing resources in high-performance computing (HPC) and cluster computing environments. 

It is widely used in HPC and cluster computing environments to optimize resource utilization and manage the complex scheduling requirements of large-scale computing clusters. It provides a flexible and extensible framework for cluster management and job scheduling, making it an essential tool for researchers, engineers, and system administrators in such environments.


Slurm Commands
---------------------

**Check SLURM Version**

To verify that SLURM is installed and running, you can check the version with:

.. code-block:: bash

  sacct --version


**Viewing Cluster Information**

This command provides an overview of the cluster's current state, including node availability, partitions, and node status.

.. code-block:: bash

  sinfo

**Node States**

The ``--states=<flag>`` option in Slurm's ``sinfo`` command allows you to filter nodes based on their state.

- ``all``: Displays nodes in all states (default if ``--states`` is not specified).

- ``idle``: Shows nodes that are currently available for running jobs.

- ``alloc``: Displays nodes that are currently allocated to jobs.

- ``drain``: Lists nodes that are marked for maintenance or have a drain state due to issues.

- ``fail``: Shows nodes in a failed state.

- ``completing``: Lists nodes that are finishing the execution of a job.

- ``mix``: Displays nodes in a combination of states, useful for complex filtering.

- ``down``: Lists nodes that are marked as down, which may be due to hardware or network issues.

- ``unkn``: Shows nodes in an unknown state, typically because Slurm cannot determine their status.

**Allocating Resources**

Request o allocate resources and create an interactive job session on a compute node.

.. code-block:: bash

  salloc [OPTIONS]

**Options**

- ``-n, --ntasks=<number>``: Specifies the number of tasks (processes or threads) you want to run. This option is particularly useful for parallel or multi-threaded applications.

- ``--cpus-per-task=<number>``: Defines the number of CPU cores or threads per task.


- ``-p, --partition=<partition_name>``: Specifies the cluster partition or queue where you want to allocate resources. Different partitions may have varying resource configurations.

- ``--time=<time>``: Sets the maximum time for which the allocated resources will be available. You can specify the time in various formats, such as minutes, hours, or days. For example ``--time=1:00:00`` allocates resources for 1 hour

- ``-N, --nodes=<number>``: Defines the number of compute nodes you want to allocate. This option is useful when you need to distribute your tasks across multiple nodes.

- ``--mem=<memory>``: Specifies the amount of memory required for each task. You can specify memory in various units (e.g., MB, GB). For example, ``--mem=4G`` allocates 4 GB of memory per task

- ``--output=<output_file>``: Redirects the standard output of the interactive session to the specified file.

- ``--error=<error_file>``: Redirects the standard error output of the interactive session to the specified file.

- ``--mail-user=<your_email@example.com>``: Redirects the standard error output of the interactive session to the specified file.

- ``--mail-type=<option1,option2,...>``: Redirects the standard error output of the interactive session to the specified file.

  - ``NONE``: No email notifications will be sent. This is the default if you don't specify --mail-type.

  - ``BEGIN``: An email will be sent when the job starts (begins execution).

  - ``END``: An email will be sent when the job completes successfully (reaches its natural end).

  - ``FAIL``: An email will be sent if the job fails to complete (exits with an error).

  - ``ALL``: This option will trigger email notifications for all of the above events (BEGIN, END, FAIL, REQUEUE).


These options can be combined to tailor the resource allocation according to your specific needs when requesting an interactive session with salloc. Remember that the availability of some options may depend on your cluster's configuration and policies. 


**Monitoring Job Status**

To check the status of all submitted jobs. It displays information about running, pending, and completed jobs. When you want to check just your submitted jobs you can use ``-u <username>`` or in some cluster configurations ``--me`` flags:

.. code-block:: bash

  squeue


To view your submitted job accounting information, such as start and end times, CPU usage, and more:

.. code-block:: bash

  sacct

In order to obtain detailed information about a specific job:

.. code-block:: bash

  scontrol show job <job_id>

To check tha active node:

.. code-block:: bash

  scontrol show hostname $SLURM_NODELIST


**Canceling Jobs**

Cancel a job that you've submitted by specifying its job ID:

.. code-block:: bash

  scancel <job_id>


.. admonition:: Task

  1. Login to the Draco cluster.
  2. Create a `.log` file, then append the outcome and the command used for each of the following tasks.
  3. Check the SLURM version.
  4. List the idle nodes.
  5. List all jobs in the queue.
  6. Allocate a node from either the `short` or `standard` partition with the following specifications: 10 minutes runtime, 2MB Memory, 2 nodes, 3 tasks, 2 CPUs per task, and specify a desired job name.(do not include the the output of the alloc in the log-File)
  7. View your job accounting information.



Module Loading
---------------

Modules are tools for modifying your shell environment. They enable you to load and unload software packages, libraries, and set environment variables as needed.

**View Available Modules**

- ``module avail``: List all available modules that can be loaded.
- ``module avail <keyword>``: Filter modules based on a keyword.

**Load Modules**

- ``module load <module_name>``: Load a specific module.
- ``module load <module_name>/<version>``: Load a module with a specific version.

**Unload Modules**

- ``module unload <module_name>``: Unload a loaded module.

**List Loaded Modules**

- ``module list``: Display the list of currently loaded modules.

SLURM Script
------------

Here's a basic example of a SLURM script that you would submit using sbatch:

.. code-block:: bash

  #!/bin/bash
  #SBATCH --job-name=myjob
  #SBATCH --output=myjob.out
  #SBATCH --error=myjob.err
  #SBATCH --partition=
  #SBATCH --nodes=
  #SBATCH --ntasks=
  #SBATCH --time=
  #SBATCH --cpus-per-task=

  # Load any necessary modules (if needed)
  # module load module_name

  # Enter your executable commands here
  # Execute the compiled program
  echo "Hello, SLURM job!"

This file should be store with ``.sh`` extension. Assuming your job script is saved in a file named ``myjob.sh``, open a terminal and run the following command: 

.. code-block:: bash
  
  sbatch myjob.sh

This command instructs SLURM to submit the job script ``myjob.sh`` for execution. The script will be processed by SLURM, and the job will be placed in the queue, waiting for available resources. When the job reaches the front of the queue, it will be allocated the resources specified in your job script, and the job will begin execution and you can monitor the job's status using commands like ``squeue`` or ``sacct``.

The options you defined in your job script (#SBATCH lines) will be used to configure the job's properties, such as the job name, output file, error file, partition, number of nodes, number of tasks, and the time limit. The actual job commands that perform the work you want to execute should be included at the end of your job script, following the ``# Your job commands here comment`` like ``echo "Hello, SLURM job!"`` in the previous example.


.. admonition:: Task

  #. Log in to the Draco cluster.
  #. Create a `.log` file, then append the outcome and the command used for each of the following tasks.
  #. Write a C++ code which calculates the Fibonacci sequence using recursion until 100.
  #. Load the necessary modules and compile your C++ code.
  #. Create a SLURM job script to submit the job to the SLURM job scheduler. Submit the job to the ``short`` or ``standard`` partition, request 1 node with whole number of CPUs and 1 task, and set a time limit of 10 minutes. Choose a specific job, error file and output file name.
  #. Print out the consumed time for running this job in output file.
  #. Monitor the status of your submitted job.
  #. Once the job is completed, review the output and error files to ensure that your C++ program ran successfully.