performance-investigation

Files (2)

• SKILL.md
• openai.yaml · agents/

SKILL.md (first lines)

# Performance Investigation

First establish the unexpected delay, observed contention, quota constraint, or requested measurement.
Missing, late, or out-of-order UI updates alone call for ordinary correctness debugging: check event delivery, parsing, and rendering first.
Use this workflow if those checks reveal slow execution or blocked rendering, or if profiling or benchmarking is explicitly requested.
An unexpectedly stalled build triggers this skill. A quiet build that is progressing within its expected duration does not.

Before editing, record the baseline command, environment, metric, narrowest representative workload, elapsed time, and whether the run is cold or warm. Reproduce the user's actual command resolution; the agent's inherited shell may differ. Prior baselines are useful when environment differences are named, but precise comparisons require same-machine before/after measurements.

## Preserve evidence

Before measuring, choose an evidence directory that survives session or environment resets and benchmark cleanup. Save the exact runner, query, and analysis sources there with the raw results for each run. Preserve the source version used for every run. Temporary copies, terminal output, or hashes alone cannot recover lost evidence.

Prefer the runner's discovery roots or exclusions to keep captured test sources outside the measured workload.
Do not assume Git-ignored directories are excluded; check the runner's discovery behavior.
Use archives or non-test suffixes only as a fallback, preserving original paths and file bytes for reconstruction.

Keep disposable build outputs separate from evidence. Before cleanup, verify that retained sources and raw results suffice to recompute reported measurements.

## Choose the simplest measurement

Before choosing tools, name the question, the decision it informs, and the least attribution detail sufficient to decide.
Separate user limits and preservation requirements from diagnostic choices, such as sampling windows and observation points.
Do not reject an alternative because it cannot satisfy conditions needed only by the current diagnostic.

### Whole-command costs

For whole-command latency, begin with a phase-time budget. Account for overlap; do not add concurrent phases as if they ran sequentially.
Estimate the maximum end-to-end benefit before pursuing a local optimization.
After a meaningful improvement, refresh the budget and rank the remaining avoidable costs. Do not keep following the previous bottleneck without rechecking.

### Profiler selection

Distinguish a progressing, CPU-heavy computation from a hang.
For a progressing computation, prefer an existing profiler that can answer the question over sampling hooks or custom telemetry.
Use existing evidence or the cheapest viability check; do not require an expensive profiling run just to rule out a tool.
Completion-only profiling cannot diagnose a hang, but may capture a computation that completes or safely unwinds on timeout.
Check the output path under relevant stop conditions and user limits; name conditions that can lose results.

### Timing and attribution scope

Use [measurement scope and isolation](references/measurement-scope.md) to choose call-site, counting, timing, and attribution boundaries and decide which measurements need isolation.

### Custom instrumentation

Before adding custom instrumentation, identify the required answer an existing tool cannot provide and the evidence of that limitation.

## Diagnose boundedly

When a command exceeds its expected duration or stays unexpectedly silent for 30 seconds, work through these steps:

1. Record elapsed time, output so far, and the process tree. Use shell timestamps if no portable timing tool exists. Save diagnostic logs and stack samples in the evidence directory.
2. Capture descendants recursively. Repeat the capture if children are short-lived. Identify the active child operation instead of attributing its time to the outer command.
3. Classify the active phase: CPU-bound, I/O-bound, blocked, sleeping, or waiting on a child.
4. For a hang, take a JVM thread dump or a bounded stack or syscall sample for other runtimes.
5. Separate launcher setup, dependency resolution, discovery/loading, execution, reporting, and shutdown.
6. Stop when the sample answers the current question. If you terminate a wrapper, clean up only descendants created by that invocation; preserve shared daemons and pre-existing workers.

For a hang, use bounded samples to locate the active phase before attempting a representative completion run.

## Improve the measured boundary

Look first for repeated process startup, duplicate runtime loading, overly broad dependency or source scope, unnecessary freshness checks, uncached generated work, and metadata-heavy filesystem traffic. Use temporary local storage for disposable intermediates only when measurement shows significant filesystem overhead.

Before optimizing subprocess calls or adding a daemon, check whether an existing runtime and supported API can remove the boundary.
If consolidation needs new quoting, framing, or parsing, compare moving the work into that runtime before adding another representation.
For runtime-configuration regressions, inspect effective state, reload idempotence, hidden callbacks, and subprocesses.

## External and quota-bound benchmarks

Use this procedure when the measured system is an API, cloud query, remote archive, registry, or other service where points, requests, bytes, reset windows, throttling, or server-side execution constrain throughput.

1. Define the measured contract before running: exact query or request shape, production fields and pagination paths included, sample authority, cold/warm state, credential class, quota window, concurrency, retries, and required finalization.
2. Make the runner write the final evidence artifact directly. Record the script or query hash, configuration hash, sample identifiers, start/end timestamps, provider request IDs when available, environment, raw per-request measurements, incomplete work, and projection formula. Do not manually transcribe totals into a second report.
3. Keep provider limits separate: primary quota consumed, reset waits, secondary throttles, concurrency caps, server CPU/time, response bytes, retries, and wall time. Concurrency may reduce latency without reducing quota cost.
4. Compare batch sizes or concurrency sequentially against the same workload. Randomize or alternate order when provider load or cache state could bias later runs. Stop increasing concurrency after throttling, timeout growth, or a worse completed-throughput result.
5. Exercise every overflow or continuation path required by the claimed contract. A first-page or query-shape probe cannot establish end-to-end cost when production requires additional fields, pagination, storage writes, normalization, or artifact finalization.
6. Classify the output precisely:
   - *Query-shape probe:* one request design's fields, cost, size, and failure behavior.
   - *Batching probe:* batch/concurrency behavior for the declared sample.
   - *Cold end-to-end benchmark:* all required phases from named empty caches through finalization.
   - *Population projection:* an explicit extrapolation from measurements to a larger workload.
7. Label projections as a *representative estimate*, *conservative bound*, *lower bound*, *upper bound*, or *planning datum only*. Bounds require a sampling model and calculation proving their direction. Convenience, adversarial, or deliberately skewed samples are planning data unless weighted against the population.
8. Treat quota resets and provider variability as part of literal elapsed time. A quota-capacity calculation is not an elapsed-time minimum unless starting balance, reset boundaries, retry behavior, and secondary limits support that claim.

Stop when the cheapest measurement settles the decision. Do not spend quota to improve precision that cannot change the selected design.

## Verify

Before comparing timings, verify that the optimization is active using runtime evidence and a negative control. Do not infer activation from configuration alone: in Node, even `NODE_DISABLE_COMPILE_CACHE=0` disables the compile cache.

When comparing toolchains, alternate sequential trials on the real workload, include artifact finalization, and exclude setup runs with unequal cache state.

Measure the changed phase and the full user command. Preserve correctness checks, compare cold and warm results on the same machine, and repeat runs when variance could change the conclusion. Report missing tools and other environmental failures separately from performance results. Confirm the optimization did not merely move work into an unmeasured phase.

When controllable, instrument opaque subprocesses with phase timing; an enclosing test or CI-job duration is insufficient attribution. Completion requires the consumer-visible evidence artifact, not terminal output or a handwritten summary.

From jyn514/dotfiles, Unlicense.

More from jyn514/dotfiles