Where Your SUs Actually Go: Right-Sizing PBS Jobs on Gadi

Where Your SUs Actually Go: Right-Sizing PBS Jobs on Gadi#

Most people learn the Gadi cost model the same way: someone asks why the project burned a quarter’s allocation in three weeks, and you go digging. The answer is almost never “we did too much science.” It’s usually a handful of jobs requesting resources they never used, and one specific mistake that can cost you 12× more SU than you expect without a single warning.

Here’s how charging actually works, and how to stop leaking allocation.

The formula#

Every job on Gadi is charged by the same equation:

Job Cost (SU) = Queue Charge Rate × Resource Requested × Walltime Used (hours)

where the middle term is the part that surprises people:

Resource Requested = Max(NCPUs Requested,
                         (Memory Requested / Max Memory Per Node) × Max NCPUs Per Node)

Three things follow immediately, and they’re worth internalising before anything else:

  1. You’re charged on what you request, not what you use. Ask for 48 cores and use 4, you pay for 48. PBS reserves the resources; the scheduler doesn’t care that your code was single-threaded.

  2. Walltime is the exception — you’re charged for walltime used, not requested. A job that asks for 10 hours and finishes in 2 is charged for 2. (The full 10 is reserved against your grant while it runs, so it still affects what else you can submit.)

  3. CPUs and memory don’t add up — they compete. You pay for whichever is larger. This is the whole game.

The memory trap#

That Max() is where allocations quietly die.

On the normal queue, a node has 48 cores and 192 GiB. So memory is charged as the fraction of the node’s RAM you took, expressed in core-equivalents. Request a lot of memory with few cores, and you’ve made the rest of the node unusable for everyone else — so you’re billed as if you’d taken it all.

Watch the same 8-core job get more expensive purely through the mem flag, over 5 hours on normal:

ncpus

mem

Charged on

Cost

8

16 GB

CPUs — Max(8, 4)

80 SU

8

128 GB

Memory — Max(8, 32)

320 SU

8

190 GB

Memory — Max(8, 47.5)

475 SU

Same cores. Same walltime. Nearly 6× the cost, entirely from memory you may not have needed. The 190 GB case is the pathological one: you asked for 8 cores and got billed for 47.5, because 190 of 192 GiB is effectively the whole node.

The breakeven is your node’s memory-per-core ratio. On normal that’s 4 GB/core (192 ÷ 48). Stay at or under 4 × ncpus GB and CPUs dominate your bill; go above it and memory takes over. On normalsr (Sapphire Rapids) the ratio is ~4.8 GB/core (500 ÷ 104); on normalbw it’s ~4.6 (128 ÷ 28).

So: mem=190GB is not a safe default. It’s the single most expensive thing you can casually type into a PBS script. If your job needs 30 GB, ask for 30 GB.

Know your queue’s charge rate#

Rates vary by more than 4× across CPU queues. Current rates (per resource-hour):

Queue

Hardware

Charge rate

Cores/node

Mem/node

Mem/core

normalbw

Broadwell

1.25 SU

28

128 / 256 GB

~4.6 GB

normalsl

Skylake

1.5 SU

32

192 GB

6 GB

normal

Cascade Lake

2 SU

48

190 GB

~4 GB

normalsr

Sapphire Rapids

2 SU

104

500 GB

~4.8 GB

hugemem

Cascade Lake

3 SU

48

1470 GB

~30 GB

hugemembw

Broadwell

1.25 SU

28

1020 GB

~36 GB

megamem

Cascade Lake

5 SU

48

2990 GB

~62 GB

express

Cascade Lake

6 SU

48

190 GB

~4 GB

gpuvolta

V100

3 SU

12/GPU

382 GB

~8 GB

dgxa100

A100

4.5 SU

16/GPU

2000 GB

~15 GB

gpuhopper

H200

7.5 SU

12/GPU

1024 GB

~21 GB

A few consequences worth acting on:

express costs 3× normal. It buys queue priority, not speed — identical hardware. If you’re not genuinely blocked waiting, you’re paying triple for nothing. Debug on express with tiny jobs; run production on normal.

normalbw is 1.25 SU vs normal at 2 SU. Broadwell is older and slower per core, but at 62% of the price. If your job isn’t wall-clock-critical and doesn’t scale beautifully, benchmark it — for a lot of embarrassingly parallel analysis work, the older nodes are the cheaper answer.

hugemembw (1.25 SU) vs hugemem (3 SU) is the sleeper. Need 400 GB for a modest core count? hugemembw gives you more memory per core (~36 vs ~30 GB) at less than half the charge rate. Slower silicon, but if you’re memory-bound rather than compute-bound — which describes a lot of large-array climate analysis — that’s the trade you want. The catch: cores come in multiples of 7 on hugemembw.

Don’t reach for hugemem reflexively. A memory-hungry job on normal charged at Max(ncpus, mem_proportion) may still beat a hugemem job at 3 SU. Do the arithmetic both ways.

Whole nodes: the rounding tax#

Any job larger than one node must request CPUs in multiples of the full node. On normal that’s multiples of 48; on normalsr, 104; on normalbw, 28.

So a job asking for 20,000 cores on normal actually requests 20,016 — you round up to 417 nodes and pay for all of it. If your decomposition lands at 50 cores, you’re paying for 96 and wasting 46. Choose core counts that land on node boundaries, or shrink to fit one node.

This is also why ncpus=1, mem=190GB is such a trap: you didn’t request a node, but you priced one.

Walltime: request generously, but not absurdly#

Since you’re billed on walltime used, over-requesting walltime doesn’t directly cost SU. But it isn’t free either:

  • The full requested walltime is reserved against your grant while the job runs, cutting what else you can submit. nci_account shows this under Reserved.

  • Shorter walltime requests schedule sooner. PBS backfills — a 2-hour job slots into gaps a 48-hour job can’t.

  • Walltime limits tighten as core count rises: on normal, 48 hours up to 672 cores, but only 5 hours at 3024+ cores. Scaling up shortens your ceiling.

  • If the job doesn’t have enough SU available for the requested walltime at submission, it goes straight to H (held). That’s a nci_account problem, not a code problem.

Rule of thumb: request ~1.3× your benchmarked runtime. Enough headroom to survive a slow node, not so much that you never get scheduled.

Measure what you actually used#

The .o file at job end tells you everything. Read it every time:

   Service Units: 96.00
   NCPUs Requested: 48          NCPUs Used: 48
   CPU Time Used: 09:12:33
   Memory Requested: 190.0GB    Memory Used: 31.4GB      <-- !!
   Walltime requested: 02:00:00 Walltime Used: 01:00:12
   JobFS requested: 100.0MB     JobFS used: 0B

That job paid 190 GB rates and touched 31 GB. Requesting mem=40GB would have dropped it from ~96 SU to ~48 SU — half, for a one-line change.

The diagnostic to compute is CPU efficiency:

efficiency = CPU Time Used / (NCPUs Used × Walltime Used)

Here: 9:12:33 / (48 × 1:00:12) ≈ 19%. You’re paying for 48 cores and using the equivalent of about 9. Something is serial, I/O-blocked, or badly threaded. NCI’s guidance is to target the sweet spot where you’re using at least 80% of what you reserved.

Check your balance before submitting a campaign:

nci_account -P gb02          # grant, used, reserved, available
nqstat -P gb02               # what's running and queued

The five most expensive habits#

  1. mem=190GB by default. Costs up to 6× on identical work. Set memory to your measured high-water mark plus ~20%.

  2. express for production runs. 3× the rate for priority you don’t need.

  3. Requesting cores a serial script can’t use. 48 cores at 19% efficiency is 39 cores of pure waste. Profile first.

  4. Ignoring normalbw / hugemembw. Often 40–60% cheaper for work that isn’t clock-critical.

  5. Never reading the .o file. Every job hands you a free audit and most people delete it unread.

A worked right-sizing#

Before — a Dask analysis job someone wrote in a hurry:

#PBS -q express
#PBS -l ncpus=48
#PBS -l mem=190GB
#PBS -l walltime=10:00:00
#PBS -l storage=gdata/gb02+scratch/gb02

Ran in 3 hours, peaked at 45 GB, and the .o file showed 12 of 48 cores meaningfully busy (Lustre metadata was the bottleneck, not CPU).

Cost: 6 SU × Max(48, 47.5) × 3h = 864 SU.

After:

#PBS -q normal
#PBS -l ncpus=12
#PBS -l mem=48GB
#PBS -l jobfs=100GB
#PBS -l walltime=4:00:00
#PBS -l storage=gdata/gb02+scratch/gb02

Cost: 2 SU × Max(12, 12) × 3h = 72 SU.

12× cheaper, same science, same wall-clock. Nothing exotic — just matching the request to the measurement. (The jobfs line is separate from cost: it’s not charged, but the default is only 100 MB, so any real temp I/O needs it declared. Keeping spill on node-local NVMe is what let the core count drop honestly.)

Checklist before you submit a campaign#

  • Ran one job first and read the .o file — cores, memory, walltime used.

  • mem ≤ 4 × ncpus GB on normal? If not, is the memory genuinely needed?

  • CPU efficiency ≥ 80%? If not, cut ncpus rather than hoping.

  • Checked whether normalbw / hugemembw does the job at 1.25 SU.

  • On express only for genuine debugging.

  • Multi-node core count lands on a node boundary.

  • Walltime ≈ 1.3× benchmark, not a round 48.

  • jobfs requested if you use more than 100 MB of temp space.

  • nci_account -P <proj> has room before you queue 500 jobs.