Using Hopper

How to get onto Hopper, work on it comfortably, and run jobs. For what Hopper is and when it is the right tool, see the overview.

The One Rule

Do not run computations on the head node. CAC states the consequence plainly: “Users should not run codes on the head node. Users who do so will be notified and have privileges revoked.”

The head node is shared by everyone on the cluster. Editing files, moving data, and submitting jobs there is fine. Anything that actually computes belongs in a batch job or an interactive session. This is easy to violate by accident once you are working through a remote IDE — see that section for what to watch for.

Getting an Account

Hopper access runs through a CAC account, which is separate from your Cornell NetID. You get one by being added to the group’s CAC project — message Vivek, and this should be part of onboarding.

Once your account exists:

ssh <your_cac_username>@hopper.cac.cornell.edu

You will be prompted to set a new password on first login. CAC passwords need at least eight characters drawn from at least three of {uppercase, lowercase, digits, symbols}, and expire every six months; reset them at passwordreset.cac.cornell.edu. Repeated failed logins lock the account for about thirty minutes.

Two Things CAC’s Docs Don’t Say

Their documentation does not state whether a VPN is required to reach Hopper from off campus, nor whether two-factor authentication applies. If you are off campus and SSH hangs, try the Cornell VPN before assuming your account is broken, and ask CAC if it persists. We would rather this page say “unknown” than guess.

SSH Keys

Optional, but set them up — it removes the password prompt, and remote IDEs depend on non-interactive login.

On your own machine:

ssh-keygen -t ed25519 -C "hopper"          # accept the defaults
ssh-copy-id <your_cac_username>@hopper.cac.cornell.edu

Then add a host block to ~/.ssh/config so you can type ssh hopper:

~/.ssh/config

Host hopper
    HostName hopper.cac.cornell.edu
    User <your_cac_username>
    IdentityFile ~/.ssh/id_ed25519
    ServerAliveInterval 60
    ServerAliveCountMax 10

ServerAliveInterval stops idle connections from being dropped, which matters for long interactive sessions.

Note that CAC also creates a key pair for you on the cluster itself. That one is for node-to-node communication inside jobs (MPI and similar) and is unrelated to the key you use to log in.

Remote Editing

Working through an editor on your laptop while the files live on Hopper is much more pleasant than editing in a terminal. All three options below rest on ordinary SSH.

Your IDE Runs Code on the Head Node

VS Code and JetBrains both install a background server on the machine you connect to, and it runs as you, on the head node. Normal editing is fine. What is not fine is hitting Run or Debug in the IDE and executing your analysis there, which is exactly what the interface invites you to do.

Use the IDE to edit and to submit jobs. To actually run something interactively, start an interactive session in a terminal and work inside it.

Be aware too that heavyweight language servers and indexing extensions are themselves compute. If you point a Python or Julia extension at a large repository, it will index it on the head node. Keep your extension set lean, and if you are told you are loading the head node, this is the usual reason.

CAC does not document remote IDEs at all. What follows works because Hopper supports standard SSH, not because CAC has endorsed it.

VS Code

Install the Remote - SSH extension, then Remote-SSH: Connect to Host and pick hopper (it reads the ~/.ssh/config block above). VS Code installs its server into ~/.vscode-server on first connect, which takes a minute or two and consumes some home-directory space.

JetBrains

JetBrains Gateway connects with the same SSH configuration and installs a backend IDE under ~/.cache/JetBrains. This is heavier than the VS Code server; if home-directory space is tight, prefer VS Code.

Plain SSH

Always works, never breaks:

ssh hopper
tmux new -s work       # or: tmux attach -t work

tmux means a dropped connection does not kill what you were doing. Even if you use an IDE, learn this — it is the fallback when the IDE’s server will not start. See the shell page.

Moving Files

For a few files, rsync over your SSH config:

rsync -avP ./localdir/ hopper:~/remotedir/      # up
rsync -avP hopper:~/remotedir/ ./localdir/      # down

For large transfers, use Globus, which resumes and verifies. Hopper’s collection is named Hopper Cluster.

Software and Environments

Modules

Hopper uses Lmod:

module avail            # what is installed
module spider <name>    # search, including modules not currently loadable
module load R/4.1.0     # case-sensitive
module list
module purge            # start clean

Load modules inside your job script, not just in your login shell, so the job does not depend on your interactive environment.

Python

conda Advice Does Not Apply Here

The conda page is about your own machine. On Hopper, CAC documents venv and pip only, and there is no python3 module to load — Hopper carries a system Python 3.6, which is old.

If your work needs a current Python, talk to Vivek before fighting with this. Options include building a newer Python yourself, using a Singularity container (module load singularity), or installing Miniforge into your home directory. Do not assume the workflow from your laptop transfers.

python3 -m venv ~/envs/myproject
source ~/envs/myproject/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

R

module load R
mkdir -p $R_LIB

Then inside R, install.packages('<name>') installs into your home directory.

Julia

Julia is available as a module, though the packaged version may lag. Installing juliaup into your home directory gives you control of the version, which is usually what you want for a project with a committed Manifest.toml.

Running Jobs

Hopper uses Slurm. Two partitions:

Partition	Nodes	Time limit
`normal`	all	none
`guest`	all	48 hours

Slurm Counts Hyperthreads

Hyperthreading is on, so Slurm treats each physical core as two CPUs. -n 1 gives you two logical CPUs — one physical core. Add --ntasks-per-core=1 if you want to ignore hyperthreading and count physical cores.

Batch Jobs

job.sh

#!/bin/bash
#SBATCH -J myjob             # job name
#SBATCH -p normal            # partition
#SBATCH -t 04:00:00          # walltime, hh:mm:ss
#SBATCH -n 1                 # tasks
#SBATCH -c 8                 # CPUs per task
#SBATCH -o logs/%x-%j.out    # stdout (%x = name, %j = job id)
#SBATCH -e logs/%x-%j.err    # stderr
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=you@cornell.edu

module purge
module load gnu9

source ~/envs/myproject/bin/activate
python analysis.py

Submit and watch:

mkdir -p logs          # Slurm will not create it, and the job fails if it is missing
sbatch job.sh
squeue -u $USER
scancel <job_id>
scontrol show job <job_id>

Always set -t to something realistic. normal has no limit, but a job that hangs forever holds resources the rest of the cluster wants.

Interactive Sessions

For debugging, exploratory work, or anything you would otherwise be tempted to run on the head node:

srun -p normal -n 1 -c 8 --pty /bin/bash -l

You land on a compute node. Work normally, then exit when finished — CAC asks explicitly that you release the allocation rather than leaving it idle.

Combine this with tmux on the head node so a dropped connection does not kill the session.

CAC documents srun --pty and does not document salloc. salloc may work, but srun --pty is the supported path.

Job Arrays

The workhorse for this group, since most of what we run is an ensemble of independent tasks — Monte Carlo replicates, parameter sweeps, one task per scenario:

array.sh

#!/bin/bash
#SBATCH -J ensemble
#SBATCH -p normal
#SBATCH -t 02:00:00
#SBATCH -n 1
#SBATCH -c 4
#SBATCH --array=1-500%50     # 500 members, at most 50 running at once
#SBATCH -o logs/%x-%A_%a.out

module purge
source ~/envs/myproject/bin/activate
python run_member.py --member $SLURM_ARRAY_TASK_ID

The %50 throttle matters: it leaves capacity for the other three groups sharing the cluster. $SLURM_ARRAY_TASK_ID is how each task knows which member it is — use it to seed, to index into a parameter table, and to name the output file.

Chained and Dependent Jobs

When a workflow has stages — calibrate, then simulate, then post-process — submit them with dependencies rather than babysitting:

jid1=$(sbatch --parsable calibrate.sh)
jid2=$(sbatch --parsable --dependency=afterok:$jid1 simulate.sh)
sbatch --dependency=afterok:$jid2 postprocess.sh

afterok runs only if the previous job succeeded. To wait on an entire array, --dependency=afterok:$jid on the array’s job id works — the dependent job starts when every task has finished.

Staging Data Through `/tmp`

Home directories are on NFS shared across all nodes, and hammering them from many tasks at once slows the filesystem for everyone. CAC is emphatic about this: “We cannot stress enough how important this is to avoid delays on the file systems.”

If your job does heavy file I/O, copy inputs to the node’s local /tmp first, work there, and copy results back at the end:

MYTMP=/tmp/$USER/$SLURM_JOB_ID
mkdir -p $MYTMP || exit 1
cp -rp $SLURM_SUBMIT_DIR/inputs $MYTMP/ || exit 1

cd $MYTMP && python analysis.py

cp -rp $MYTMP/output $SLURM_SUBMIT_DIR/ || exit 1
rm -rf $MYTMP

This matters most for job arrays, where the multiplier is the number of concurrent tasks.

FAQ

My job is stuck in PENDING. squeue -u $USER shows a reason. Resources means the cluster is busy — normal, wait. Priority means others are queued ahead. PartitionTimeLimit means your -t exceeds the partition’s limit. If it is pending for days, ask before resubmitting.

My job is much slower than on my laptop. Two usual causes. Hopper’s Intel cores run at 2.1 GHz, well below a workstation, so serial code genuinely is slower — see when to use Hopper. Or you are I/O-bound against NFS, in which case stage through /tmp.

I need more memory. The AMD nodes (c0023-c0026) have 567 GB against the Intel nodes’ 192 GB. Request memory with --mem= or --mem-per-cpu=, or target the large nodes with -w.

How do I see what a finished job actually used? Try sacct -j <job_id> or seff <job_id>. These are standard Slurm tools but are not in CAC’s documentation, so they may or may not be configured here — worth trying, and worth telling the rest of us if they work.

I ran out of space in my home directory. Check with du -sh ~/*. IDE servers, conda environments, and old job output are the usual culprits. Remember that nothing on Hopper is backed up — see Data Management.

Is there a web interface? No. CAC’s Open OnDemand portal serves the Seneca cluster, not Hopper. Hopper is SSH only.

Something is broken and it isn’t my code. Check portal.cac.cornell.edu/status, then email help@cac.cornell.edu with “Hopper” in the subject. Tell Vivek too, since it probably affects the rest of the group.

More Information

CAC TechDocs: Hopper — the authoritative reference
Slurm and Slurm Quick Start
slurm_tutorial — the group’s example submission scripts
cluster-training — older HPC training materials
Water Programming on HPC and Slurm

--- title: Using Hopper --- How to get onto [Hopper](hopper.qmd), work on it comfortably, and run jobs. For what Hopper is and when it is the right tool, see the [overview](hopper.qmd). ::: {.callout-important} ## The One Rule **Do not run computations on the head node.** CAC states the consequence plainly: *"Users should not run codes on the head node. Users who do so will be notified and have privileges revoked."* The head node is shared by everyone on the cluster. Editing files, moving data, and submitting jobs there is fine. Anything that actually computes belongs in a [batch job](#batch-jobs) or an [interactive session](#interactive-sessions). This is easy to violate by accident once you are working through a remote IDE — see [that section](#remote-editing) for what to watch for. ::: ## Getting an Account Hopper access runs through a **CAC account**, which is separate from your Cornell NetID. You get one by being added to the group's CAC project — message Vivek, and this should be part of [onboarding](../procedures/checklist.qmd). Once your account exists: ```bash ssh <your_cac_username>@hopper.cac.cornell.edu ``` You will be prompted to set a new password on first login. CAC passwords need at least eight characters drawn from at least three of {uppercase, lowercase, digits, symbols}, and **expire every six months**; reset them at [passwordreset.cac.cornell.edu](https://passwordreset.cac.cornell.edu/passwordreset/Passwordreset). Repeated failed logins lock the account for about thirty minutes. ::: {.callout-note} ## Two Things CAC's Docs Don't Say Their documentation does not state whether a **VPN** is required to reach Hopper from off campus, nor whether **two-factor authentication** applies. If you are off campus and SSH hangs, try the Cornell VPN before assuming your account is broken, and ask CAC if it persists. We would rather this page say "unknown" than guess. ::: ## SSH Keys Optional, but set them up — it removes the password prompt, and remote IDEs depend on non-interactive login. On your own machine: ```bash ssh-keygen -t ed25519 -C "hopper" # accept the defaults ssh-copy-id <your_cac_username>@hopper.cac.cornell.edu ``` Then add a host block to `~/.ssh/config` so you can type `ssh hopper`: ```{.default filename="~/.ssh/config"} Host hopper HostName hopper.cac.cornell.edu User <your_cac_username> IdentityFile ~/.ssh/id_ed25519 ServerAliveInterval 60 ServerAliveCountMax 10 ``` `ServerAliveInterval` stops idle connections from being dropped, which matters for long interactive sessions. Note that CAC also creates a key pair for you on the cluster itself. That one is for node-to-node communication inside jobs (MPI and similar) and is unrelated to the key you use to log in. ## Remote Editing {#remote-editing} Working through an editor on your laptop while the files live on Hopper is much more pleasant than editing in a terminal. All three options below rest on ordinary SSH. ::: {.callout-warning} ## Your IDE Runs Code on the Head Node VS Code and JetBrains both install a background server on the machine you connect to, and it runs as you, on the **head node**. Normal editing is fine. What is not fine is hitting **Run** or **Debug** in the IDE and executing your analysis there, which is exactly what the interface invites you to do. Use the IDE to edit and to submit jobs. To actually run something interactively, start an [interactive session](#interactive-sessions) in a terminal and work inside it. Be aware too that heavyweight language servers and indexing extensions are themselves compute. If you point a Python or Julia extension at a large repository, it will index it on the head node. Keep your extension set lean, and if you are told you are loading the head node, this is the usual reason. CAC does not document remote IDEs at all. What follows works because Hopper supports standard SSH, not because CAC has endorsed it. ::: ### VS Code Install the [Remote - SSH](https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh) extension, then **Remote-SSH: Connect to Host** and pick `hopper` (it reads the `~/.ssh/config` block above). VS Code installs its server into `~/.vscode-server` on first connect, which takes a minute or two and consumes some home-directory space. ### JetBrains [JetBrains Gateway](https://www.jetbrains.com/remote-development/gateway/) connects with the same SSH configuration and installs a backend IDE under `~/.cache/JetBrains`. This is heavier than the VS Code server; if home-directory space is tight, prefer VS Code. ### Plain SSH Always works, never breaks: ```bash ssh hopper tmux new -s work # or: tmux attach -t work ``` `tmux` means a dropped connection does not kill what you were doing. Even if you use an IDE, learn this — it is the fallback when the IDE's server will not start. See the [shell page](../kb/programming/shell.qmd). ## Moving Files For a few files, `rsync` over your SSH config: ```bash rsync -avP ./localdir/ hopper:~/remotedir/ # up rsync -avP hopper:~/remotedir/ ./localdir/ # down ``` For large transfers, use [Globus](https://portal.cac.cornell.edu/techdocs/files/FileTransferGlobus/), which resumes and verifies. Hopper's collection is named **`Hopper Cluster`**. ## Software and Environments ### Modules Hopper uses [Lmod](https://portal.cac.cornell.edu/techdocs/clusterinfo/modules/): ```bash module avail # what is installed module spider <name> # search, including modules not currently loadable module load R/4.1.0 # case-sensitive module list module purge # start clean ``` Load modules **inside your job script**, not just in your login shell, so the job does not depend on your interactive environment. ### Python ::: {.callout-warning} ## conda Advice Does Not Apply Here The [conda page](../kb/programming/conda.qmd) is about your own machine. On Hopper, CAC documents `venv` and `pip` only, and there is **no `python3` module** to load — Hopper carries a system Python 3.6, which is old. If your work needs a current Python, talk to Vivek before fighting with this. Options include building a newer Python yourself, using a [Singularity](https://portal.cac.cornell.edu/techdocs/clusters/Hopper/) container (`module load singularity`), or installing Miniforge into your home directory. Do not assume the workflow from your laptop transfers. ::: ```bash python3 -m venv ~/envs/myproject source ~/envs/myproject/bin/activate pip install --upgrade pip pip install -r requirements.txt ``` ### R ```bash module load R mkdir -p $R_LIB ``` Then inside R, `install.packages('<name>')` installs into your home directory. ### Julia Julia is available as a module, though the packaged version may lag. Installing [`juliaup`](https://github.com/JuliaLang/juliaup) into your home directory gives you control of the version, which is usually what you want for a project with a committed `Manifest.toml`. ## Running Jobs Hopper uses [Slurm](https://portal.cac.cornell.edu/techdocs/clusterinfo/slurm/). Two partitions: | Partition | Nodes | Time limit | |---|---|---| | `normal` | all | none | | `guest` | all | 48 hours | ::: {.callout-note} ## Slurm Counts Hyperthreads Hyperthreading is on, so Slurm treats each physical core as two CPUs. `-n 1` gives you **two** logical CPUs — one physical core. Add `--ntasks-per-core=1` if you want to ignore hyperthreading and count physical cores. ::: ### Batch Jobs {#batch-jobs} ```{.bash filename="job.sh"} #!/bin/bash #SBATCH -J myjob # job name #SBATCH -p normal # partition #SBATCH -t 04:00:00 # walltime, hh:mm:ss #SBATCH -n 1 # tasks #SBATCH -c 8 # CPUs per task #SBATCH -o logs/%x-%j.out # stdout (%x = name, %j = job id) #SBATCH -e logs/%x-%j.err # stderr #SBATCH --mail-type=END,FAIL #SBATCH --mail-user=you@cornell.edu module purge module load gnu9 source ~/envs/myproject/bin/activate python analysis.py ``` Submit and watch: ```bash mkdir -p logs # Slurm will not create it, and the job fails if it is missing sbatch job.sh squeue -u $USER scancel <job_id> scontrol show job <job_id> ``` Always set `-t` to something realistic. `normal` has no limit, but a job that hangs forever holds resources the rest of the cluster wants. ### Interactive Sessions {#interactive-sessions} For debugging, exploratory work, or anything you would otherwise be tempted to run on the head node: ```bash srun -p normal -n 1 -c 8 --pty /bin/bash -l ``` You land on a compute node. Work normally, then `exit` when finished — CAC asks explicitly that you release the allocation rather than leaving it idle. Combine this with `tmux` on the head node so a dropped connection does not kill the session. CAC documents `srun --pty` and does not document `salloc`. `salloc` may work, but `srun --pty` is the supported path. ### Job Arrays The workhorse for this group, since most of what we run is an ensemble of independent tasks — Monte Carlo replicates, parameter sweeps, one task per scenario: ```{.bash filename="array.sh"} #!/bin/bash #SBATCH -J ensemble #SBATCH -p normal #SBATCH -t 02:00:00 #SBATCH -n 1 #SBATCH -c 4 #SBATCH --array=1-500%50 # 500 members, at most 50 running at once #SBATCH -o logs/%x-%A_%a.out module purge source ~/envs/myproject/bin/activate python run_member.py --member $SLURM_ARRAY_TASK_ID ``` The `%50` throttle matters: it leaves capacity for the other three groups sharing the cluster. `$SLURM_ARRAY_TASK_ID` is how each task knows which member it is — use it to seed, to index into a parameter table, and to name the output file. ### Chained and Dependent Jobs When a workflow has stages — calibrate, then simulate, then post-process — submit them with dependencies rather than babysitting: ```bash jid1=$(sbatch --parsable calibrate.sh) jid2=$(sbatch --parsable --dependency=afterok:$jid1 simulate.sh) sbatch --dependency=afterok:$jid2 postprocess.sh ``` `afterok` runs only if the previous job succeeded. To wait on an entire array, `--dependency=afterok:$jid` on the array's job id works — the dependent job starts when every task has finished. ### Staging Data Through `/tmp` Home directories are on NFS shared across all nodes, and hammering them from many tasks at once slows the filesystem for everyone. CAC is emphatic about this: *"We cannot stress enough how important this is to avoid delays on the file systems."* If your job does heavy file I/O, copy inputs to the node's local `/tmp` first, work there, and copy results back at the end: ```bash MYTMP=/tmp/$USER/$SLURM_JOB_ID mkdir -p $MYTMP || exit 1 cp -rp $SLURM_SUBMIT_DIR/inputs $MYTMP/ || exit 1 cd $MYTMP && python analysis.py cp -rp $MYTMP/output $SLURM_SUBMIT_DIR/ || exit 1 rm -rf $MYTMP ``` This matters most for job arrays, where the multiplier is the number of concurrent tasks. ## FAQ **My job is stuck in `PENDING`.** `squeue -u $USER` shows a reason. `Resources` means the cluster is busy — normal, wait. `Priority` means others are queued ahead. `PartitionTimeLimit` means your `-t` exceeds the partition's limit. If it is pending for days, ask before resubmitting. **My job is much slower than on my laptop.** Two usual causes. Hopper's Intel cores run at 2.1 GHz, well below a workstation, so serial code genuinely is slower — see [when to use Hopper](hopper.qmd). Or you are I/O-bound against NFS, in which case stage through `/tmp`. **I need more memory.** The AMD nodes (`c0023`-`c0026`) have 567 GB against the Intel nodes' 192 GB. Request memory with `--mem=` or `--mem-per-cpu=`, or target the large nodes with `-w`. **How do I see what a finished job actually used?** Try `sacct -j <job_id>` or `seff <job_id>`. These are standard Slurm tools but are not in CAC's documentation, so they may or may not be configured here — worth trying, and worth telling the rest of us if they work. **I ran out of space in my home directory.** Check with `du -sh ~/*`. IDE servers, conda environments, and old job output are the usual culprits. Remember that [nothing on Hopper is backed up](hopper.qmd) — see [Data Management](data-management.qmd). **Is there a web interface?** No. CAC's Open OnDemand portal serves the Seneca cluster, not Hopper. Hopper is SSH only. **Something is broken and it isn't my code.** Check [portal.cac.cornell.edu/status](https://portal.cac.cornell.edu/status), then email `help@cac.cornell.edu` with "Hopper" in the subject. Tell Vivek too, since it probably affects the rest of the group. ## More Information - [CAC TechDocs: Hopper](https://portal.cac.cornell.edu/techdocs/clusters/Hopper/) — the authoritative reference - [Slurm](https://portal.cac.cornell.edu/techdocs/clusterinfo/slurm/) and [Slurm Quick Start](https://portal.cac.cornell.edu/techdocs/clusterinfo/slurmQS/) - [`slurm_tutorial`](https://github.com/srikrishnan-lab/slurm_tutorial) — the group's example submission scripts - [`cluster-training`](https://github.com/srikrishnan-lab/cluster-training) — older HPC training materials - [Water Programming on HPC](https://waterprogramming.wordpress.com/category/high-performance-computing/) and [Slurm](https://waterprogramming.wordpress.com/category/slurm/)