> ## Documentation Index
> Fetch the complete documentation index at: https://wb-21fd5541-update-regex-mention.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

> Parallelize sweep agents on multi-core or multi-GPU machines.

# Parallelize agents

Parallelize your sweep agents on a multi-core or multi-GPU machine to run multiple sweep runs at the same time. Parallelization reduces the time needed to explore your hyperparameter space. This page shows you how to launch parallel agents on multi-CPU and multi-GPU machines so you can use all available compute on a single host.

Before you start, you must initialize your sweep. For more information, see [Initialize sweeps](/models/sweeps/initialize-sweeps).

## Parallelize on a multi-CPU machine

The following tabs describe how to run multiple sweep agents in parallel on the same machine. Parallel agents are most useful on a machine with multiple CPU cores, where each agent can run on its own core. Choose the tab for the CLI or a Jupyter Notebook based on your workflow.

<Tabs>
  <Tab title="CLI">
    Use the [`wandb agent`](/models/ref/cli/wandb-agent) command to parallelize your sweep agent across multiple CPUs with the terminal. Provide the sweep ID that W\&B returned when you [initialized the sweep](/models/sweeps/initialize-sweeps).

    1. Open more than one terminal window on your local machine.
    2. Copy and paste the following code snippet and replace `[SWEEP-ID]` with your sweep ID:

       ```bash theme={null}
       wandb agent [SWEEP-ID]
       ```
  </Tab>

  <Tab title="Notebook">
    Use the W\&B Python SDK library to parallelize your sweep agent across multiple CPUs within Jupyter Notebooks. Provide the sweep ID that W\&B returned when you [initialized the sweep](/models/sweeps/initialize-sweeps) and set the `function` parameter to the name of the function the sweep executes:

    1. Open more than one notebook.
    2. Copy and paste the sweep ID on multiple notebooks to parallelize a sweep. For example, if you have the sweep ID stored in a variable called `sweep_id` and the name of the function is `function_name`, paste the following code snippet on multiple notebooks to parallelize your sweep:

       ```python theme={null}
       wandb.agent(sweep_id=sweep_id, function=function_name)
       ```
  </Tab>
</Tabs>

## Parallelize on a multi-GPU machine

Run sweep agents in parallel across multiple GPUs on the same machine by using `CUDA_VISIBLE_DEVICES` to assign each [`wandb agent`](/models/ref/cli/wandb-agent) to a different GPU, so the agents run in parallel without competing for the same device.

To follow this procedure, you need:

* A machine with more than one GPU.
* The NVIDIA CUDA Toolkit installed.
* A terminal.

For example, suppose you have two NVIDIA GPUs on your local machine. Replace `[SWEEP-ID]` in each command with the sweep ID that W\&B returns when you initialize a sweep:

1. In one terminal, set `CUDA_VISIBLE_DEVICES` to `0` and start an agent:

   ```bash theme={null}
   CUDA_VISIBLE_DEVICES=0 wandb agent [SWEEP-ID]
   ```

2. In a separate terminal, set `CUDA_VISIBLE_DEVICES` to `1` and start another agent with the same sweep ID:

   ```bash theme={null}
   CUDA_VISIBLE_DEVICES=1 wandb agent [SWEEP-ID]
   ```