← Back to skills

CURATED FROM PUBLIC GIT

compileiq-run-search

>

↓ 0★ 0by greatsage_sh
4fbbff39da1a
skillmarket install greatsage_sh/compileiq-run-search
Public Git sourcehttps://github.com/NVIDIA/skills.gitcommit 4fbbff39da1a5780d8bfd796f1674618c6349ef7
Part of skillsetskills →

About this skill

---
name: compileiq-run-search
description: >
  Use when composing the Search(...) call and calling .start(). Covers the
  four worker classes (MultiProcessWorker / IsoMultiProcessWorker / RayWorker
  / AsyncWorker) and when to pick each, SearchConfiguration sizing rules,
  dump_results checkpointing, tracker_config choice (Disabled / Loguru /
  MLflow), num_workers/task_timeout semantics, and GPU clock locking for
  stable measurements. Triggers on "Search()", "tuner.start()", "pool_size",
  "num_workers", "task_timeout", "IsoMultiProcessWorker", "RayWorker",
  "dump_results", "MLflow", "GPU clocks".
when_to_use: |
  - About to instantiate Search() and call .start().
  - Search converges too slow / too fast and the user is unsure how to size.
  - Search is hanging on a few configs (need IsoMultiProcessWorker + timeout).
  - Migrating from local to a Ray cluster.
license: CC-BY-4.0 AND Apache-2.0
metadata:
  version: "1.2.0"
  author: CompileIQ Team <[email protected]>
  domain: compiler-optimization
  tags: [compileiq, cuda, ptxas, compiler-optimization, autotuning]
allowed-tools: Bash Read
paths: ["**/*.py"]
---

# compileiq-run-search

After you have an objective function (from `compileiq-author-objective`) and
a search space (from `compileiq-search-space`), this skill helps you choose
the worker, size the configuration, and run the search safely.

## When

- About to instantiate `Search(...)` and call `.start()`.
- Search is converging too fast or too slow and the user is unsure how to
  re-size pool/generations.
- Search hangs on individual configs and the worker doesn't kill them.
- Scaling out from one GPU to a Ray cluster.

## Worker selection

Pass either a built-in `WorkerTypes` enum value or the worker class itself to
`Search(worker_type=...)`:

```python
from compileiq.types import WorkerTypes
from compileiq.worker import (
    MultiProcessWorker,    # default
    IsoMultiProcessWorker, # spawns fresh process per task; kill-safe
    RayWorker,             # distributed
    AsyncWorker,           # asyncio for async def objectives
)
```

| Situation | Worker class | Why |
|---|---|---|
| GPU kernel that may hang, OOM, or leak CUDA context | **`IsoMultiProcessWorker`** | One fresh process per task; parent kills on `task_timeout`. Defaults to `fork`. ([`docs/workers.md`](https://github.com/NVIDIA/CompileIQ/blob/main/docs/workers.md)) |
| Triton mixed example on Blackwell-class GPUs | `WorkerTypes.ISOLATED` + `CIQ_PROCESS_MODE=spawn` | Isolates each evaluation and avoids leaking illegal memory access state across runs. |
| Fast (<100ms), stateless objective | `MultiProcessWorker` (default) | Reuses a pool; lower overhead. Defaults to `forkserver`. |
| Multi-node / multi-GPU cluster | `RayWorker` | User must set up Ray cluster + install compileiq on every worker. Both `num_workers` and `task_timeout` are ignored. ([`docs/workers.md`](https://github.com/NVIDIA/CompileIQ/blob/main/docs/workers.md)) |
| I/O-bound `async def` objective | `AsyncWorker` | Concurrency, not parallelism. Rare for GPU work. |

**Default recommendation for compiler tuning of GPU kernels:**
`IsoMultiProcessWorker` with `task_timeout` between **30s** (small kernels) and
**180s** (large attention / XLA HLO).

## SearchConfiguration sizing

Reference: `compileiq/types.py:473-615`. Defaults auto-derive; only set what
you must.

```python
from compileiq.types import SearchConfiguration, ProblemType

config = SearchConfiguration(
    problem_type=ProblemType.MIN,   # MIN for latency; MAX for throughput
    generations=10,                  # required, > 0
    pool_size=15,                    # > 5; auto-derives if omitted
    # num_objectives defaults to 1
    # normalize defaults to False (set True for cross-GPU runs)
)
```

| Knob | Default | When to override |
|---|---|---|
| `generations` | required | 10 for initial exploration; 20-40 for a deep run. |
| `pool_size` | auto (≥32) | 15 for tiny spaces; 32 for ≥1k design points; 64-128 for ≥10k. |
| `num_objectives` | 1 | Must equal `len(return_tuple)` from the objective. |
| `normalize` | False | True when running across heterogeneous nodes or GPUs. |

The default engine, `amber`, owns its optimizer settings. Leave the
`SearchConfiguration` fields `cull_size`, `mutate_rate`, `objective_weights`,
and `enable_large_fail_pool` unset: they have no effect since CompileIQ 1.1, and
`Search` ignores them with a `FutureWarning`. Pass amber options
(`mutation_probability`, `survivor_count`, `reference_direction_weights`,
`seed`) through `engine=EngineConfig(name="amber", options={...})` instead.
`amber` never repeats a candidate, so keep `pool_size * generations` well below
the number of distinct points in a small discrete space; otherwise `start()` ends
early with a `SearchSpaceExhaustedWarning` and returns the results so far.

**Sanity rule of thumb:** if `pool_size * generations < 50`, you are exploring,
not optimizing. If `> 2000`, you are probably overfitting to measurement noise
— `compileiq-validate-result` will earn its keep there.

## Search(...) constructor — every relevant kwarg

```python
from pathlib import Path
from compileiq.ciq import Search
from compileiq.search_spaces.compilers import PtxasSearchSpace
from compileiq.tracker import LoguruTrackerConfig

tuner = Search(
    objective_function=objective,
    search_space=PtxasSearchSpace(version="13.3", variant="att"),
    search_config=config,
    worker_type=IsoMultiProcessWorker,                 # or WorkerTypes.ISOLATED
    tracker_config=LoguruTrackerConfig(sink="optimization.log"),
    dump_results=Path("results.csv"),                  # ALWAYS set this
    cache_folder=None,                                  # default ~/.cache/compileiq
    disable_progress_bar=False,
    exit_on_failure=True,
    debug=False,
)
```

Always set `dump_results=Path(...)`. CSV is flushed every batch, so a crashed
or killed run leaves recoverable state.

## start(...) semantics

```python
results = tuner.start(num_workers=4, task_timeout=120)
```

- `num_workers`: ignored by workers where `respects_num_workers=False`
  (`RayWorker`, `AsyncWorker`); CompileIQ emits the warning
  `"num_workers is not supported by <WorkerName>"` (`compileiq/ciq.py:449-451`)
  so users recognize it.
- `task_timeout`: ignored where `supports_timeout=False` (`RayWorker`).
  Critical for `IsoMultiProcessWorker` — without it a hung config wedges that
  branch.
- Returns a `SearchResult`. Don't process inline; hand off to
  `compileiq-validate-result`.

## Tracker choice (one-line each)

```python
from compileiq.tracker import DisabledTrackerConfig, LoguruTrackerConfig, MLflowTrackerConfig
```

- `DisabledTrackerConfig()` — default, no overhead. Fine for one-off runs.
- `LoguruTrackerConfig(sink="optimization.log", level="INFO")` —
  recommended for serious campaigns. Negligible overhead.
- `MLflowTrackerConfig(experiment_name="...", tracking_uri="...", run_name="...")`
  — when integrating with ML Ops; creates a nested MLflow run per evaluation.

## Sample before you search

`Search.sample(n)` returns `n` randomly sampled parameter dicts from the
search space without running the search. Use it to:

1. Confirm the search space resolves at all (cheaper than the bootstrap
   round-trip; uses the in-memory state of `Search`).
2. Eyeball that the dicts have the keys your objective expects.
3. Feed a single sample into the objective by hand to verify it runs.

```python
sample = tuner.sample(1)[0]
print(sample)
print(objective(sample))   # should return a real float, not raise
```

## GPU clock locking (operator-level)

Stable measurements need locked clocks, locked **before** `tuner.start()` and
reset after the search. Locking needs root privileges, so do not run these
commands yourself: show them to the user and ask them (or their cluster
admin) to run them as root, then confirm before starting the search.

```bash
# Run as root, before the search
nvidia-smi -pm 1
MAX_GPU=$(nvidia-smi --query-gpu=clocks.max.graphics --format=csv,noheader,nounits | head -1)
MAX_MEM=$(nvidia-smi --query-gpu=clocks.max.memory --format=csv,noheader,nounits | head -1)
nvidia-smi --lock-gpu-clocks=$MAX_GPU,$MAX_GPU --lock-memory-clocks=$MAX_MEM,$MAX_MEM

# Run as root, after the search
nvidia-smi --reset-gpu-clocks --reset-memory-clocks
```

Check the current state without privileges:
`nvidia-smi --query-gpu=clocks.gr,clocks.max.graphics --format=csv`.

If the user can't lock clocks (CI container, shared cluster), skip this and
report the higher CV% to the validation skill so it knows to compensate.

## Self-test

```bash
python scripts/smoke_search.py
```

Runs a 2-generation search on `x**2 + y` with `MultiProcessWorker` and
verifies `results.get_best_result()` returns a dict with `score_1` and `params`.

## Gotchas

- **Forgetting `task_timeout`** with `IsoMultiProcessWorker` is the most
  common reason a search hangs for hours. The worker will *kill* a stuck
  process but only after `task_timeout` elapses.
- **`forkserver` issues** on some hosts manifest as `EOFError` or "Broken pipe"
  on the first eval. Set `CIQ_PROCESS_MODE=spawn`.
- **Local workers don't assign GPUs.** `MultiProcessWorker` and
  `IsoMultiProcessWorker` give every worker process the same visible GPUs, so
  `num_workers > 1` on a GPU-kernel objective makes evaluations share a GPU
  and skews the timings. On one GPU, keep `num_workers=1` for timing-sensitive
  kernels. For several GPUs (one machine or many), use `RayWorker`, the only
  worker that schedules a GPU per task: call `tuner.start(num_gpus=1)` and read
  the assigned device from `CUDA_VISIBLE_DEVICES` (or `ray.get_gpu_ids()`)
  inside the objective, or write a custom worker
  ([`docs/workers.md`](https://github.com/NVIDIA/CompileIQ/blob/main/docs/workers.md)).
- **Don't put GPU-clock lock calls inside the objective.** They need root
  and are per-host operator setup, not per-eval.

## Next

- After `.start()` returns: `compileiq-validate-result`.
- If something's wrong: `compileiq-debug`.