← Back to skills

CURATED FROM PUBLIC GIT

fastgen-dataset

>-

↓ 0★ 0by greatsage_sh
4fbbff39da1a
skillmarket install greatsage_sh/fastgen-dataset
Public Git sourcehttps://github.com/NVIDIA/skills.gitcommit 4fbbff39da1a5780d8bfd796f1674618c6349ef7
Part of skillsetskills →

About this skill

---
name: fastgen-dataset
description: >-
  Prepare, register or repair datasets for NVIDIA FastGen loaders: class-conditional ZIPs, WebDataset shards of images, prompts, latents, videos, audio or camera data, and KD pairs or trajectories; then verify decoded batches. For experiment configuration over already prepared data, use fastgen-configure.
license: Apache-2.0
metadata:
  author: "Julius Berner <[email protected]>"
  domain: ml
---

# Prepare a FastGen dataset

Apply [fastgen-runtime](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-runtime/SKILL.md). The
[dataset README](https://github.com/NVlabs/FastGen/blob/main/fastgen/datasets/README.md) owns formats, preparation commands,
loader presets and key parameters; the loaders in
[wds_dataloaders.py](https://github.com/NVlabs/FastGen/blob/main/fastgen/datasets/wds_dataloaders.py) and
[wds_utils.py](https://github.com/NVlabs/FastGen/blob/main/fastgen/datasets/wds_utils.py) document decoding, partitioning and
resume in their docstrings. Read [loader contracts](references/loader-contract.md) for
shard acceptance, coverage and resume accounting.

1. **Scope.** Resolve source, permitted use, destination, task, the method's supervision
   needs ([fastgen-configure](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-configure/SKILL.md) if unsettled), the
   network's preprocessing and the available sample count. Inspect existing inputs before
   downloading; keep provenance and counts. Follow the pretrained model's conditioning
   requirements; recaption prompts to match its reference style when needed, preserving meaning.
2. **Loader.** Choose the existing class-conditional or WebDataset loader. Shards need
   unique sample keys with the extensions named in `key_map`; constants come from
   `files_map` and `presets_map`. Raw media and prompts use the `*_raw` keys and cached
   latents and embeddings the plain keys; never label placeholders as model-space data,
   and record the model revision and preprocessing of every cache. Rebuild affected caches
   after edits.
3. **Pairing and geometry.** Keep task pairing (source and target for editing, reference
   frames for image-conditioned video, camera data, aligned audio). Derive frame counts,
   FPS, aspect ratio and temporal compression from the network; the representable frame
   count is not the nominal clip length.
4. **Splits and size.** Separate training from held-out evaluation inputs before deriving
   labels or trajectories. Estimate the unique examples needed from throughput, effective
   batch, updates and acceptable repetition; a tiny dataset validates plumbing only.
5. **Register.** Import and copy a preset from [data.py](https://github.com/NVlabs/FastGen/blob/main/fastgen/configs/data.py)
   rather than reproducing its loader fields; add a shared preset there when needed.
   Supply paths through the config or environment roots; keep data and preprocessing
   artifacts outside the source tree. For data-dependent fine-tuning or distillation, consider
   teacher-generated targets via [fastgen-inference](https://github.com/NVlabs/FastGen/blob/main/skills/fastgen-inference/SKILL.md).
   Save the method's required supervision, including
   [KD pairs or trajectories](https://github.com/NVlabs/FastGen/blob/main/fastgen/datasets/README.md#knowledge-distillation-loaders).
6. **Verify.** Iterate the real loader and a prepared batch (`prepare_batch`) in the
   proper runtime; check keys, shapes, dtypes, ranges, captions, pairing, negative
   conditions and modality lengths. Use enough batches and ranks to expose empty worker
   or shard splits, and keep a manifest of usable counts. Before prompt-only training,
   sample the teacher on a few real training prompts to confirm they are usable.

Report provenance, the split/count manifest, dataset location, data-config path and the
observed batch contract. For empty or short batches inspect tar members, `key_map`,
filters, shard/worker partitioning and `files_map` before touching the network; report
corrupt or excluded samples instead of counting them.