observability-instrument

Files (3)

• SKILL.md
• SKILL.md.bundle
• SKILL.md.sig

SKILL.md (first lines)

# Observability Instrument

> OpenTelemetry, OpenInference semantic conventions, baggage for per-tenant
> attribution, PII redaction inline. Production-grade from day one.

**Tradeoff:** Every new code path needs instrumentation. Worth it: full tracing
catches more agent quality regressions before users see them.

---

## Why baggage matters

Baggage propagates `user_id` and `tenant_id` through the trace context. Every
span automatically inherits them through the call tree. Teams that try to
propagate identity through ad-hoc argument passing always lose it at the first
tool boundary.

Decision: add baggage in week one. Costs two days. Buys per-tenant cost
attribution forever.

---

## Setup (TypeScript / Node)

```typescript
// otel-setup.ts
import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { propagation, ROOT_CONTEXT } from '@opentelemetry/api';
import { W3CBaggagePropagator, W3CTraceContextPropagator, CompositePropagator } from '@opentelemetry/core';
import { resourceFromAttributes } from '@opentelemetry/resources';
import { SemanticResourceAttributes } from '@opentelemetry/semantic-conventions';
import { piiRedactingExporter } from './pii-redaction.js';

propagation.setGlobalPropagator(new CompositePropagator({
  propagators: [new W3CTraceContextPropagator(), new W3CBaggagePropagator()],
}));

const sdk = new NodeSDK({
  resource: resourceFromAttributes({
    [SemanticResourceAttributes.SERVICE_NAME]: 'auth-service',
    [SemanticResourceAttributes.SERVICE_VERSION]: process.env.GIT_SHA,
    [SemanticResourceAttributes.DEPLOYMENT_ENVIRONMENT]: process.env.ENV,
  }),
  traceExporter: piiRedactingExporter(new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,
  })),
  instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();
```

---

## Adding baggage

At the request entry point:

```typescript
import { context, propagation } from '@opentelemetry/api';

app.use(async (req, res, next) => {
  const baggage = propagation.createBaggage({
    'user.id': { value: req.user?.id ?? 'anonymous' },
    'tenant.id': { value: req.tenant?.id ?? 'unknown' },
    'session.id': { value: req.session?.id ?? 'none' },
  });
  const ctx = propagation.setBaggage(context.active(), baggage);
  context.with(ctx, () => next());
});
```

From this point on, every child span (HTTP call, DB query, agent invocation,
external API) inherits user.id and tenant.id automatically.

---

## OpenInference for LLM calls

Use the OpenInference semantic conventions so you can swap observability
backends (Langfuse, Laminar, Arize Phoenix) without re-instrumenting:

```typescript
import { OpenAI } from 'openai';
import { OpenAIInstrumentation } from '@arizeai/openinference-instrumentation-openai';
// or @arizeai/openinference-instrumentation-anthropic for Anthropic

registerInstrumentations({ instrumentations: [new OpenAIInstrumentation()] });
```

This adds spans like:
- `llm.provider`: anthropic
- `llm.model`: claude-sonnet-4-6
- `llm.input_messages.0.role`: user
- `llm.input_messages.0.content`: (redacted)
- `llm.token_count.prompt`: 1234
- `llm.token_count.completion`: 567
- `llm.cost_usd`: 0.0122

---

## PII redaction (mandatory)

PII never reaches the observability backend. Implement an inline-redacting
exporter at `observability/pii-redaction.ts`:

```typescript
export function piiRedactingExporter(baseExporter) {
  return {
    export(spans, cb) {
      const redacted = spans.map(span => ({
        ...span,
        attributes: redactObject(span.attributes),
        events: span.events?.map(e => ({ ...e, attributes: redactObject(e.attributes) })),
      }));
      return baseExporter.export(redacted, cb);
    },
    shutdown() { return baseExporter.shutdown(); },

From MILTONADINA/DevOPs, MIT.

More from MILTONADINA/DevOPs