lab-hpc

Files (6)

• SKILL.md
• arc-hpc.md · references/
• arc-slurm.md · references/
• ircbc-hpc.md · references/
• ircbc-slurm.md · references/
• check-hpc-config.sh · scripts/

SKILL.md (first lines)

# Lab HPC clusters & remote-compute workflow

Per-cluster detail is in `references/`: `*-hpc.md` for hosts, network and storage; `*-slurm.md` for partitions and submitting.

Before designing a large-data pipeline or starting a long data job, read and apply
[`cluster-run`](../hq-mode/references/playbooks/cluster-run.md), including on a held node.

## Step 0 — before the first ssh

1. Run `bash scripts/check-hpc-config.sh` (`--live` also tests ssh). If the cluster the task needs
   is `NOT CONFIGURED`, **refuse**: name the missing aliases, point at the README's "Per-user
   config", never improvise with raw addresses or keys, and resolve hosts from `~/.ssh/config` only.
   `config dir: OLD LOCATION` → **refuse** too, and relay the printed fix for the user to run.
2. Read `~/.claude/lab/personal.md` — usernames, code dirs, the standing job, sbatch
   defaults. It **overrides** this skill. Say so if it is absent.
3. Ask which cluster unless context pins it down. Then develop locally, push to GitHub,
   and `git pull` on the remote — code reaches a cluster through git, not scp.

## Shell and environment

- **The local shell on lab Macs is zsh.** Quote every URL (`?` and `*` are globs). An unquoted
  `$var` is not split into words, so use an array. End a probe that may find nothing with
  `|| true`, so an empty result doesn't hide real errors.
- **Run project code on a cluster through `pixi run`,** not the system `python3`.

## Work on a compute node, not the login node

Get onto a compute node and stay there. A login node is one shared machine per cluster, and a
fan-out of "small" agent commands takes it down. `ssh arc '<cmd>'` is the wrong reflex even for
a one-liner: the whole Slurm client (`squeue`, `sbatch`, `scancel`) runs on the compute node.
Only **two reasons** justify a login node:

1. **You hold no job**, so no compute node is yours yet. Only a login node can submit that
   first one.
2. **The compute node lacks what you need:** on ircbc, internet and `git`. On **arc, nothing**:
   its compute nodes have internet and a full toolchain, so once you hold one, never leave.

## SOP — take a foothold, then keep it

1. **Find and reuse the node you already hold.** The lab parks a long-lived idle GPU job on arc for this.
   Check `personal.md`, then probe aliases with `ssh -o BatchMode=yes -o ConnectTimeout=10
   <alias> true`. Judge by exit status, not stderr.
2. **Land and stay.** On arc, wrap commands in a login shell — `ssh <node> 'bash -lc "…"'` — or
   `pixi` and other user-installed tools are invisible.
   Put each probe's commands into one ssh call: every call is a connection through the login
   node. If ssh hangs after a network drop, clear the stale shared connection with
   `ssh -O exit <alias>`, then retry once.
3. **No node anywhere?** Only then use the login node, and only to fix that: show the exact
   sbatch/salloc, get confirmation, submit, poll to RUNNING, ssh in, and leave. If a foothold
   is preempted, repeat this step.

## Watching a job

Poll a job only with `bash scripts/watch-job.sh <alias> <jobid> [file]` in the background, never
a `sleep` loop; `-h` lists its exit codes. Every job-watching subagent's brief names it.

## Hard rules

- **Work on a compute node; the login node has exactly the two uses above.**
- **No sbatch/srun outside an approved run plan.** Approved before the pilot, sized for the
  full run, scripts shown: work folder, partitions, most CPU, memory and wall time per job,
  most jobs at once, target measure and range. Inside it, submit and cancel its own jobs
  unasked. A job or pilot estimate past a limit needs a revised plan. Cancelling a job the
  plan did not start, and writes to shared data or the shared package store, need a yes.
  Post it on the run's issue (or work folder README); quote it verbatim in subagent briefs.
  It ends with the run. No plan: a yes per script. Probes like `squeue` need none.
- **Reachable never means permitted.** ircbc lets you ssh to a node you hold no job on; working
  there steals from the scheduler. Hold an allocation.
- If ssh to `ircbc` hangs, **stop and tell the user to check the VPN** (atrust, managed by
  hand). arc is not behind it. Never retry in a loop.

From liuhlab/liulab-skills, MIT.

More from liuhlab/liulab-skills