← Back to skills

CURATED FROM PUBLIC GIT

hf-mem

Estimate the GPU memory needed to run a Hugging Face model (weights, KV cache and inference overhead) and whether it fits on given GPUs, without downloading it. Use when asked how much VRAM a model needs, whether it fits on a GPU, the longest context that fits, or which GGUF quantization to pick.

↓ 0★ 0by greatsage_sh
5291c2b9365e
skillmarket install greatsage_sh/hf-mem
Public Git sourcehttps://github.com/huggingface/skills.gitcommit 5291c2b9365e06b175dc04817610c778623f4836
Part of skillsetskills →

About this skill

---
name: hf-mem
description: Estimate the GPU memory needed to run a Hugging Face model (weights, KV cache and inference overhead) and whether it fits on given GPUs, without downloading it. Use when asked how much VRAM a model needs, whether it fits on a GPU, the longest context that fits, or which GGUF quantization to pick.
license: mit
---

# hf-mem

```bash
uvx hf-mem <model> -o json [options]
```

`<model>` is a Hub ID (`org/name`, `org/name@revision`), a Hub URL, a local path, or `org/name-GGUF:Q4_K_M` for one GGUF quantization.

Options:

- `--gpu-memory 24` (GiB per GPU) or `8x80` (8 GPUs): adds `fit`.
- `--max-model-len N`: context length, defaults to the model's maximum, which can dominate the total.
- `--batch-size N`: concurrent sequences, defaults to 1.
- `--kv-cache-dtype fp8` (vLLM names for Safetensors, llama.cpp cache types such as `q8_0` for GGUF).
- `--file <file or variant>` and `--format safetensors|gguf|onnx` to pick the weights.
- `--max-num-batched-tokens 2048` on A100s.
- Gated or private models need `HF_TOKEN` or `--hf-token`.

JSON (sizes in bytes, divide by 2^30 for GiB):

- `total` is the GPU memory for one copy of the model: `weights` + `kv_cache` + `overhead` (vLLM for Safetensors, llama.cpp for GGUF, none for ONNX).
- `offloaded` are weights the engine keeps in CPU memory: excluded from `weights` and `total`, but needed as free system RAM.
- `fit` (with `--gpu-memory`): `fits`, `needed` GPUs, `max_model_len` (the longest context that fits), and the `required` vs `usable` memory across the given GPUs. Use `required` / GPUs for the memory per GPU, not `total` / GPUs, as each GPU also holds its own overhead and replicated KV cache.
- GGUF and ONNX repositories with several files return a `files` list to compare quantizations; pick one with `:TAG` or `--file` to get `fit`.
- Report `warnings` to the user, and errors explain how to fix them.