UNIX and the Shell

Whether you’re using a Linux laptop or workstation (or working on the command line in macOS!) or working on a cluster, you’ll need to be familiar with shell commands and scripting, which can save you a lot of time and typing.

You do not need to be an expert. But once your work moves to Hopper there is no graphical interface to fall back on, so a working command-line vocabulary stops being optional.

What You Actually Need

Enough to move around a filesystem, inspect files without opening them, and chain a few commands together:

Task Commands
Move around cd, pwd, ls
Inspect files cat, head, tail, less, wc
Move and copy cp, mv, mkdir, rm
Find things find, grep
Chain things pipes (\|), redirection (>, >>)
Remote work ssh, scp, rsync
Long-running jobs tmux or screen
Disk usage du -sh, df -h

Two that repay learning early:

  • rsync rather than scp for anything large or repeated. It resumes, it skips unchanged files, and rsync -avn (with -n for a dry run) lets you see what would transfer before committing to it.
  • tmux on any remote machine. Without it, a dropped SSH connection kills whatever you were running. With it, you reattach and your session is still there.
Warningrm Does Not Ask Twice

There is no undo and no trash can. rm -rf on the wrong path is unrecoverable, and on a cluster with no backups (which includes Hopper) so is deleting the wrong directory with any command. Check pwd before destructive operations, and prefer rm -i while you are still getting comfortable.

Shell Scripts

A shell script is just a file of commands you would otherwise type. Any analysis step you have run more than twice belongs in one, both to save typing and because the script is a record of what you actually did. This matters for reproducibility: “I ran some commands” is not a method, and a .sh file in the repository is.

Start scripts with #!/usr/bin/env bash and set -euo pipefail, which makes the script stop on the first error instead of continuing with bad state. Without that line a failed step is silently ignored and the script reports success.

Your Slurm submission scripts on Hopper are shell scripts with extra #SBATCH directives at the top, so this is the same skill.

Learn More

  • The Unix Shell from Software Carpentry, the standard introduction. Work through it once; it takes an afternoon.
  • The Missing Semester of Your CS Education from MIT, covering the shell, tmux, and the wider command-line toolchain.
  • ShellCheck will catch bugs in a script you have written, including several classes of quoting error that fail silently.
  • Explainshell breaks down what an unfamiliar command and its flags actually do, which is useful when you inherit a script or copy something from a tutorial.

When You’re Stuck

The shell’s error messages are terse and its manual pages assume you already know what you are looking for. A rough order of attack:

  1. man <command> or <command> --help. Man pages are reference, not tutorial — skim for the flag you need rather than reading top to bottom.
  2. Explainshell when you have a command and do not know what it does. Paste it in and it annotates each flag. Better than guessing at a line you copied from a tutorial.
  3. ShellCheck when a script misbehaves. It catches quoting bugs that fail silently, which is most of them.
  4. The Unix Shell if the gap is conceptual rather than a specific command.
  5. Ask. Shell problems are unusually well suited to a two-minute conversation, because the person helping can usually see the answer in your command.