About this skill
---
license: Apache-2.0
name: doca-dpa-hl-tracer
description: >
Use this skill when the user runs doca_dpa_hl_tracer to
capture/decode DPA-side traces at the programming-events
layer (kernel entry/exit, sync points, comm primitive
calls, RDMA WR submission, completion drain) — picking
TRACE vs CRIT, tuning the JSON config (file-size limits
+ file_size_limit_policy, thread priorities/cores),
decoding against the matching DPA-side ELF, or
diagnosing empty/noisy captures. Trigger even when the
user does not explicitly mention "DOCA DPA tracer" or
"high-level tracer" — typical implicit phrasings include
"DPA kernel returns wrong result but host completions
look clean", "kernel-entry to first-comm latency is
huge", "RDMA WR to drain gap on the DPA", "trace file
truncated mid-run", "TRACE doubled my DPA latency", or
"tracer wrote a file but parser shows zero events".
Refuse and route elsewhere for writing DPA kernels,
DPA-Comms/DPA-Verbs programming, raw per-cycle DPA
profiling, host-side doca-dpa debugging, or production
DPA telemetry — those belong to other skills.
metadata:
kind: tool
compatibility: >
Requires DOCA SDK installed at /opt/mellanox/doca on
Linux (Ubuntu 22.04/24.04 or RHEL/SLES) with a BlueField
device whose DPA processor is exposed to the host, plus
the DOCA DPA Tools optional component (binary at
/opt/mellanox/doca/tools/doca_dpa_hl_tracer). Requires a
DPACC-built DPA-side ELF and a live doca-dpa-launched
workload for events to fire.
---
# DOCA DPA High-Level Tracer
**Where to start:** This is a tool skill for invoking
`doca_dpa_hl_tracer` — the documented host-side CLI that
captures DPA-side execution traces in higher-level terms
(DPA programming events: kernel entry / exit, sync points,
comm primitive calls, RDMA WR submission, completions) rather
than raw cycle counts. Open [`TASKS.md`](TASKS.md) and start at
[`## configure`](TASKS.md#configure) for the
mode-vs-overhead decision and the JSON config layout, then
[`## run`](TASKS.md#run) for the
capture → decode → render pipeline. Open
[`CAPABILITIES.md`](CAPABILITIES.md) when the question is
*what does this tool actually trace*, *which DPA programming
events does it expose*, *what is the trace-overhead vs
fidelity tradeoff*, or *how does it slot into a DPA debug
loop alongside [`doca-dpa`](../../libs/doca-dpa/SKILL.md)
and [`doca-debug`](../../doca-debug/SKILL.md)*. If DPA is not
the right surface for the user's question (e.g. the bug is
host-side, the bug is in the DPACC-produced image, the user
wants raw cycle counts), the path-selection rule in
[`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
routes the agent before any capture is attempted.
## Example questions this skill answers well
The CLASSES of `doca_dpa_hl_tracer` questions this skill is
built to answer, each with one worked example. The class is
the load-bearing piece; the worked example is one instance.
- **"My DPA kernel is doing the wrong thing — where do I
look?"** — worked example: *"my host-side
`doca_dpa_kernel_launch_update_*` completes, but the
kernel's reported result is wrong; no host-side
`DOCA_ERROR_*`"*. Answered by the *when DPA-side
high-level tracing is the right surface* gate in
[`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
+ the
capture → decode → render flow in
[`TASKS.md ## run`](TASKS.md#run) + the
*which DPA programming events to focus on* rule in
[`TASKS.md ## debug`](TASKS.md#debug).
- **"My DPA kernel is slow at a granularity that doesn't
show up in cycle profiles — how do I see kernel-entry to
first-comm-call latency?"** — worked example: *"my DPA
kernel runs but the time between launch and the first
RDMA WR submission is bigger than I expected"*. Answered
by the event-taxonomy table in
[`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
+ the iterative loop in
[`TASKS.md ## test`](TASKS.md#test) which treats trace
overhead, mode (`TRACE` vs `CRIT`), and capture window
as axes to tune.
- **"How do I capture a trace without burying the DPA in
observation overhead?"** — worked example: *"`TRACE` mode
is producing too much data and my measured DPA latency
went up by 2x compared to without the tracer"*. Answered
by the mode-vs-overhead tradeoff in
[`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
+ the `CRIT`-first guidance in
[`TASKS.md ## configure`](TASKS.md#configure) (start
with critical-events-only; widen to `TRACE` only when the
bug demands per-event detail).
- **"My trace file got truncated mid-run — how should I
configure the file-size limits?"** — worked example:
*"binary trace file hit 5 GB and the capture stopped"*.
Answered by the `log_file_max_size_in_bytes` /
`bin_file_max_size_in_bytes` / `file_size_limit_policy`
triple in
[`CAPABILITIES.md ## Capabilities and modes`](CAPABILITIES.md#capabilities-and-modes)
+ the JSON config layout in
[`TASKS.md ## configure`](TASKS.md#configure).
- **"Is the tracer on my install, and is it paired with the
matching `doca-dpa` library and DPACC compiler
version?"** — worked example: *"is the tracer ABI on my
install compatible with the DPA image my DPACC just
produced?"*. Answered by the version-overlay in
[`CAPABILITIES.md ## Version compatibility`](CAPABILITIES.md#version-compatibility),
which redirects to the canonical
[`doca-version`](../../doca-version/SKILL.md) chain and
adds the *tracer ↔ `doca-dpa` library ↔ DPACC compiler*
match rule.
- **"The capture file looks empty / decode failed — is the
install broken, no events fired, or am I tracing the
wrong thing?"** — worked example: *"`doca_dpa_hl_tracer`
ran, wrote a file, but the parser shows zero events"*.
Answered by the layered error taxonomy in
[`CAPABILITIES.md ## Error taxonomy`](CAPABILITIES.md#error-taxonomy)
(install / device-binding / DPA-image-instrumented /
capture-window / decode-vs-elf / overhead-saturated /
version / cross-cutting) + the layered walk in
[`TASKS.md ## debug`](TASKS.md#debug).
## Audience
This skill serves **external developers, platform operators,
and AI agents who have already brought up a DPA-side workload
through [`doca-dpa`](../../libs/doca-dpa/SKILL.md) and now
need higher-level visibility into what the DPA kernel is
actually doing on the wire** — DPA programming events
ordering, sync gaps, comm-call latencies, RDMA-WR / completion
timing — without dropping all the way down to raw cycle
counters. Concretely:
- A DPA developer who can launch their kernel cleanly from
the host side but whose kernel's *result* is wrong or
whose *DPA-side performance* is below expectation, and
who needs a DPA-side ground truth before triaging.
- A platform operator running a DPA-using workload (RDMA
offload from accelerator, custom CC algorithm via
`doca-pcc`) and needs to localize a regression to the
DPA side without instrumenting the application.
- An AI agent producing a *DPA-side trace report* as
evidence for the host-side
[`doca-dpa TASKS.md ## debug`](../../libs/doca-dpa/TASKS.md#debug)
ladder when the host side reports clean completions but
the DPA-side behaviour is wrong.
It is **not** for users debugging the tracer binary itself,
**not** a substitute for the live public DOCA DPA Tools
guide, **not** the right place for users learning how to
write a DPA kernel (that audience belongs in
[`doca-dpa`](../../libs/doca-dpa/SKILL.md) plus the public
DOCA DPA / DPACC / DPA-Comms / DPA-Verbs guides), and **not**
the right place for raw per-instruction cycle profiling
(different surface, different tool — route via
[`doca-public-knowledge-map ## DOCA tools`](../../doca-public-knowledge-map/SKILL.md#doca-tools)).
The tracer is shipped as a **CLI binary** under
`/opt/mellanox/doca/tools/`, not a library you link against.
The skill uses the same `kind: tool` three-file shape as
the rest of the bundle so the agent's task-verb contract is
uniform across libraries, services, and tools.
## Language scope
`doca_dpa_hl_tracer` is a C++ host-side CLI. Its inputs are
its JSON config file, the DPA-side ELF (the
`doca_dpa_app`-class image produced by DPACC), and a running
DPA-side workload that the host-side `doca-dpa` lifecycle
already started. Its outputs are a binary trace file
(`bin_file`) and a human-readable log file (`log_file`).
The skill keeps the workflow guidance language-neutral —
the DPA-side workload it traces can be C compiled by DPACC
or any other DPA translation unit DPACC accepts — and
routes per-language questions to the public DPA / DPACC
guides via
[`doca-public-knowledge-map`](../../doca-public-knowledge-map/SKILL.md).
## When to load this skill
Load this skill when the user is — or the agent needs to —
invoke `doca_dpa_hl_tracer` on a real host with DOCA
installed against a BlueField with a DPA processor visible to
the host, and the host-side
[`doca-dpa`](../../libs/doca-dpa/SKILL.md) lifecycle has
already brought a DPA workload up at least once. Concretely:
- Capturing a DPA-side trace to localize a DPA kernel's
wrong-result or wrong-ordering behaviour when the
host-side `doca-dpa` lifecycle reports clean completions.
- Capturing a DPA-side trace to localize a DPA-side
performance gap (kernel-entry to first-comm latency,
RDMA-WR-issue to completion gap, sync-point dwell time)
at a granularity above raw cycle counts.
- Choosing between `TRACE` and `CRIT` capture modes based
on the bug-vs-overhead tradeoff and the available
capture window.
- Tuning the JSON config (thread priorities, core
affinities, file size limits, file-size-limit policy) so
the capture itself does not perturb the workload more
than the bug it is investigating.
- Decoding a captured `bin_file` against the matching
DPA-side ELF to render the human-readable event stream.
- Capturing a side-effect-bounded trace as prerequisite
evidence for a host-side
[`doca-dpa TASKS.md ## debug`](../../libs/doca-dpa/TASKS.md#debug)
ladder step.
Do **not** load this skill for general DOCA orientation,
DPA-side programming model questions, raw cycle profiling,
or DOCA / DPACC install. For those, route to
[`doca-public-knowledge-map`](../../doca-public-knowledge-map/SKILL.md),
[`doca-dpa`](../../libs/doca-dpa/SKILL.md), or
[`doca-setup`](../../doca-setup/SKILL.md).
## What this skill provides
This is a **thin loader**. Substantive material lives in two
companion files:
- `CAPABILITIES.md` — what `doca_dpa_hl_tracer` captures: the
DPA programming event taxonomy (kernel entry / exit, sync
points, comm primitive calls, RDMA WR submission and
completion drain), the two documented capture modes
(`TRACE` for full per-event, `CRIT` for critical-events
only), the trace-overhead-vs-fidelity tradeoff, the
config-file shape (receiver / binary-writer / file-writer
/ printer threads with priority + core affinity, file
size limits, `file_size_limit_policy`), the
capture-window + workload-must-be-running invariant, the
ELF-must-match-image rule for decode, the
version-availability overlay (tracer ↔
[`doca-dpa`](../../libs/doca-dpa/SKILL.md) library ↔
DPACC compiler), the layered error taxonomy
(install / device-binding / image-instrumented /
capture-window / decode / overhead-saturated / version /
cross-cutting), the observability surface (binary trace
file + log file + tool's own stderr), and the safety
policy (capture is bounded; tracing is not a production
observability surface).
- `TASKS.md` — step-by-step workflows for the in-scope
task verbs: `install` (route to host-side DOCA install +
DPA prerequisites), `configure` (mode + JSON config
layout + capture window), `build` (route to install —
the binary is shipped, the DPA-side application is
user-built by DPACC), `modify` (refuse — do not patch
the binary; modify the JSON config and the invocation
instead), `run` (the capture flow with `--mode`,
`--config-file`, `--output-file`), `test` (iterative
loop tuning mode, window, and overhead), `debug` (walk
the error taxonomy), `use` (consume the decoded trace
in a `doca-dpa` debug session), plus a `Deferred task
verbs` block.
The skill assumes a host where DOCA is already installed at
the standard location, a BlueField with a DPA processor is
present and visible to the host, the DPACC compiler is
installed at a version matched to the host-side DOCA, the
DPA-side application image (the ELF the tracer decodes
against) is on disk and matches what the
[`doca-dpa`](../../libs/doca-dpa/SKILL.md) lifecycle loaded,
and the operator has the privileges the public DOCA DPA
Tools guide requires.
## What this skill deliberately does not ship
This skill is **agent guidance**, not a samples or scripts
bundle. To keep the boundary clean, it deliberately does not
contain — and pull requests should not add:
- **Specific flag strings, event names, or mode tokens
beyond what the public DOCA DPA Tools page and `--help`
document.** The DPA programming events surface evolves
release to release; `--help` on the installed binary is
the authoritative inventory.
- **Pre-baked example traces or expected event timings.**
Trace output is workload-, DPA-image-, BlueField-, and
firmware-specific; a captured example pinned to one
setup misleads operators elsewhere.
- **Wrappers, parsers, or rendering scripts** in any
language that consume the binary trace format. The
format is documented; users who want to script against
it should read the live guide and write the parser
against their installed version.
- **A specific tuning recommendation derived from a single
trace.** A DPA-side perf decision (move a sync, batch a
comm call, change a launch argument) is a workload
question and the skill prescribes how to *capture and
read* traces — it refuses to translate a captured gap
into a kernel-rewrite recommendation without the user's
own analysis.
- **A `samples/` or `reference/` subtree.** This is a thin
loader for a shipped CLI; substantive material lives on
the public page, in `--help`, and in
[`doca-dpa`](../../libs/doca-dpa/SKILL.md).
## Loading order
1. Read this `SKILL.md` first to confirm the user's question
is in scope (DPA-side high-level tracing, not DPA-side
programming and not raw cycle profiling).
2. **For the event taxonomy, capture modes, overhead
tradeoff, JSON config layout, version overlay, error
taxonomy, observability, and safety policy, see
[CAPABILITIES.md](CAPABILITIES.md).**
3. **For the documented invocations and the
capture → decode → render workflow — `install`,
`configure`, `build`, `modify`, `run`, `test`, `debug`,
`use` — see [TASKS.md](TASKS.md).**
## Related skills
- [`doca-dpa`](../../libs/doca-dpa/SKILL.md) — the host-side
DPA control library whose loaded application image the
tracer captures. Pair them in every DPA debug session:
`doca-dpa` brings the workload up; the tracer captures
what the workload does at the DPA programming event
layer. Conflating the library with the tracer is the
most common DPA-debug first-touch error.
- [`doca-debug`](../../doca-debug/SKILL.md) — the
cross-cutting debug ladder. The tracer slots in at the
*runtime* layer as the DPA-side ground truth before any
DPA-side perf or correctness conclusion is made.
- [`doca-public-knowledge-map`](../../doca-public-knowledge-map/SKILL.md)
— routing to the public DOCA DPA Tools page on
`docs.nvidia.com` and the rest of the public DOCA
documentation set.
- [`doca-version`](../../doca-version/SKILL.md) — canonical
DOCA version-handling rules. The `## Version
compatibility` section in [`CAPABILITIES.md`](CAPABILITIES.md)
is a concise overlay that redirects here for the body
and adds the *tracer ↔ `doca-dpa` library ↔ DPACC
compiler* matching rule.
- [`doca-setup`](../../doca-setup/SKILL.md) — env
preparation, install verification, DPACC compiler
install / verification, BlueField mode (the DPA
processor must be exposed before any tracing is
meaningful), and the *I have no install yet* path with
the public NGC DOCA container.
- [`doca-structured-tools-contract`](../../doca-structured-tools-contract/SKILL.md)
— the bundle's detect → prefer → fall back → report
contract for structured helper tools. The command
appendix in [`TASKS.md`](TASKS.md) honors this contract.
- [`doca-programming-guide`](../../doca-programming-guide/SKILL.md)
— general DOCA programming patterns shared by every
library / tool surface, including the cross-library
`DOCA_ERROR_*` taxonomy this tool's host-side error
layer overlays on top of when host-side `doca-dpa`
calls fail in tandem.
The DPA-side companion libraries `doca-dpa-comms` (comm
primitives the DPA kernel itself calls) and
`doca-dpa-verbs` (RDMA verbs the DPA kernel itself calls)
are **different artifacts** that the tracer's *DPA
programming events* surface visibly names; for the
DPA-side programming model itself, route through
[`doca-public-knowledge-map`](../../doca-public-knowledge-map/SKILL.md)
to the public DOCA DPA-Comms and DPA-Verbs guides and to
the shipped `/opt/mellanox/doca/samples/doca_dpa/` samples.
This tool *traces* their use; it does not redefine them.