Parallelization

Your Code Multitasks

You can make code run faster by executing independent tasks in parallel. This is the core idea of parallel computing.

Suppose your Python code already supports parallelism (for example via joblib or pytorch) and you are connected to a remote cluster. The next step is to choose the hardware used for parallel execution:

• CPU (Central Processing Unit)
• GPU (Graphics Processing Unit, often much faster for large numerical workloads)

Check usage with:

htop     # CPU usage
gpustat  # GPU usage

1) GPU Parallelization

Select GPUs using:

export CUDA_VISIBLE_DEVICES=0,1  # use the first two GPUs

2) CPU Parallelization

Even when your code dispatches jobs, performance depends on how CPU threads are configured.

For large-scale workloads, it is often better to avoid heavy multithreading per process and run more single-threaded jobs in parallel. A common setup is:

# stop per-process multithreading
export MKL_NUM_THREADS=1
export NUMEXPR_NUM_THREADS=1
export OMP_NUM_THREADS=1

You Multitask

Sometimes you want to launch several experiments and leave them running overnight. Three common options are &, screen, and tmux.

1) & (Background Job)

Simple and quick, but the process usually stops when the terminal/session is interrupted.

python3 main.py &
jobs  # check running background jobs

2) screen

screen is a complete and easy session manager.

3) tmux

tmux is a powerful alternative. Useful reference: https://tmuxcheatsheet.com

A tmux session contains windows, and windows can contain multiple panes.