---
title: Monitoring
description: The same agent, watched in production — observe, create_monitor, why the explicit client draws no waterfall until you give it one, and what the platform does with a session on its own.
---

Monitoring is the other half of the product: the agent doing real work for
real users, reported as it happens and read by the same rules as an
experiment. Sessions, Violations and Intents in the console fill from
production instead of from a lab. One integration, two doorways — this page is
the second one.

## Two records, and which one you are looking at

This is the thing that confuses people first, so it goes first.

| record | opened by | filled by | where |
| --- | --- | --- | --- |
| **the run** | `redline dev` claiming an experiment | `ctx.thinking` / `ctx.tool` / `ctx.tool_result` / `ctx.log` in your doorway, plus spans | Experiments → the run |
| **the session** | `@observe` in production, or the worker during a dev conversation | conversation turns, plus **spans only** | Monitor → Sessions |
| **the session, hand-reported** | `create_monitor` / `createMonitor` | **only the rows you write** — no spans are collected for it | Monitor → Sessions |

`ctx.*` writes to the **run**. It never reaches a session. So a doorway that
maps every tool call onto `ctx.tool` and emits no spans produces a run with a
full transcript and a session with none — both correct, and easy to read as a
bug. What fills a session's waterfall is OpenTelemetry spans, from your
framework's instrumentation or from your own loop.

## Under `redline dev` — nothing to write

While the worker is up, monitoring is on (`REDLINE_MONITOR=1`). Every
conversation the agent works — its user message, model calls with token
counts, both halves of every tool call, reasoning, the final answer — also
streams to Monitor → Sessions as a live session. The worker records the turns;
the spans supply everything between.

**What decides whether it sees anything is the instrumentation.** The SDK only
listens; the framework has to emit. Most emit nothing until their package is in
the venv, and `redline dev` activates whatever is installed **at start**:

| framework | install | notes |
| --- | --- | --- |
| Pydantic AI | nothing | the SDK calls `Agent.instrument_all()` |
| LangChain / LangGraph | `openinference-instrumentation-langchain` | |
| OpenAI Agents SDK | `openinference-instrumentation-openai-agents` | |
| CrewAI | `openinference-instrumentation-crewai` **and** `openinference-instrumentation-openai` | CrewAI 1.x calls the model through the OpenAI client |
| Agno | `openinference-instrumentation-agno` | |
| Vercel AI SDK v7 | nothing | |
| Vercel AI SDK ≤6 | nothing to install | `experimental_telemetry: { isEnabled: true }` on the agent or each call |
| **your own loop** | nothing to install | **emit your own** — see below |

Verify after the first run: the session must show model calls and a token
count. One line per turn and nothing between means the package is missing or
was installed after `redline dev` started. Install and restart.

## In production — `observe`

For the deployed agent — real users, no `redline dev` — one decorator on the
function that handles a message:

**Python**

```python
from redline import observe

@observe(agent="my-agent", session_arg="conversation_id",
         user_arg="user_id", input_arg="text")
async def handle_message(conversation_id: str, user_id: str, text: str) -> str:
    return await your_agent(text)
```

- `session_arg` — the argument carrying the conversation id. Calls sharing it
  land in one transcript; without it every call is its own session.
- `user_arg` — the argument naming the end user, shown on the session.
- `input_arg` — the argument carrying the user's message; default is the
  call's single string argument.

**TypeScript**

```ts
import { observe } from "@redlineai/sdk";

export const handleMessage = observe(
  async (conversationId: string, userId: string, text: string) => yourAgent(text),
  { agent: "my-agent", sessionArg: 0, userArg: 1, inputArg: 2 },
);
```

Same options, by argument **index**.

The decorator records the input and the returned answer itself; every span the
call emits lands in the same session, keyed to the call's trace. Config is the
environment by default — `REDLINE_API_KEY`, `REDLINE_URL`, `REDLINE_AGENT` —
so a deploy needs no code. **Without a key it is inert** and the handler runs
untouched. Errors are reported and re-raised; nothing in it can take the
handler down.

### When the key cannot be in the environment

Some apps strip their secrets out of the environment on purpose. The usual
reason is a child process: anything the app spawns inherits `os.environ` /
`process.env`, so a Redline key left sitting there is a key that subprocess can
report sessions with. Pass it as an argument instead:

**Python**

```python
@observe(agent="my-agent", session_arg="conversation_id",
         api_key=settings.redline_key)      # never read from os.environ
async def handle_message(conversation_id: str, text: str) -> str:
    return await your_agent(text)
```

**TypeScript**

```ts
export const handleMessage = observe(
  async (conversationId: string, text: string) => yourAgent(text),
  { agent: "my-agent", sessionArg: 0, apiKey: settings.redlineKey },
);
```

`baseUrl` / `base_url` and `projectId` / `project_id` take arguments the same
way. Anything you leave out still falls back to the environment, so the
ordinary deploy is unchanged. This is the supported answer for an app in that
position — it keeps you on the wrapper, with spans collected for you, instead
of hand-building the waterfall below.

> **It wraps a function that returns the answer**
>
> `observe` records what the wrapped function **returns**, so a streaming entry
> point needs one layer of care — in both languages.
>
> **Python** — `observe` checks `inspect.iscoroutinefunction`, which is `False`
> for an async generator. Decorate one directly and it records the generator
> object as the answer and never sees the turn.
>
> **TypeScript** — a handler that returns an SSE `Response`, a `ReadableStream`
> or an async iterator records that **object** as the answer, and every span the
> stream emits after the handler returned lands outside the session.
>
> Either way, wrap the innermost layer that can hand back the whole turn — the
> chat or engine function the route calls, not the route itself — and let it
> accumulate the deltas it is already sending, so the browser still streams and
> `observe` still gets the answer. The
> [own tool loop](/integrations/own-tool-loop#3-the-doorway--production) page
> shows the shape.

### Wiring it into an existing app

A decorator only records calls to the function it decorates. If the repository
already handles inbound messages — a webhook, a queue worker, a websocket loop
— that pipeline has to **call** `handle_message`. It will not find it on its
own.

That is one call-site change in existing code, and it is the only one the
integration asks for. Keep the pipeline's own signals intact: if it read
something off the stream besides the reply (an "ended" flag, a step count),
pass a `state` dict the handler fills, so nothing downstream changes.

Without this, experiments work, `redline dev` sessions work, and production
sessions **never appear**. An integration reviewed as "done" with this missing
is the commonest shape we see.

## `create_monitor` — the client underneath

For a handler shape the decorator cannot wrap, the explicit client:

**Python**

```python
from redline import create_monitor
monitor = create_monitor(agent="my-agent")          # env for key and url

s = monitor.session(id=conversation_id, user=user_id)
s.input(text)
s.thinking("…")
s.tool("lookup", args={"q": "x"}, result="…", ok=True, latency_ms=120)
s.output(reply, tokens=439)
s.end("closed")
```

**TypeScript**

```ts
import { createMonitor } from "@redlineai/sdk";
const monitor = createMonitor({ apiKey: process.env.REDLINE_API_KEY! });

const s = monitor.session({ id: conversationId, agent: "my-agent", user: userId });
s.input(text);
s.thinking("…");
s.tool("lookup", { q: "x" }, "…", { latencyMs: 120 });
s.output(reply, { tokens: 439 });
await s.end("closed");
```

Same span types the ingest API accepts — `input`, `output`, `thinking`,
`tool`, `retrieval`, `error`, `event`. Batched and flushed on a timer; `end()`
closes the trace.

### It collects nothing on its own

`observe` does two things this client does not. It installs the span processor,
and it binds the call's trace to the session. `create_monitor` does neither, by
design: it is a client for rows you write, and it writes exactly those.

**So a session opened this way has no waterfall** — not a thin one, none — even
in a process that is emitting perfectly good OpenTelemetry, because nothing is
listening to it. The transcript reads fine, which is what makes this hard to
spot: the session is there, the turns are there, and the chart beside them
simply never appears.

Two ways out, in order of preference:

1. **Use `observe`.** It wires both halves for you, and since it takes the key
   as an argument there is no longer an environment it cannot run in.
2. **Draw the bars yourself**, below — for a loop whose shape `observe` cannot
   wrap at all.

### Drawing the waterfall by hand

Every row takes an optional span ref. Four fields, all yours to mint:

| field | what it is |
| --- | --- |
| `id` | this row's span id. Rows sharing one fold into a single bar |
| `parent` | the enclosing span's `id` — the indentation of the chart. `None`/`null` at the root |
| `startMs` | when the work started, ms epoch |
| `durMs` | how long it ran, ms |

**Python**

```python
import time, uuid
from redline import create_monitor

monitor = create_monitor(agent="my-agent", api_key=settings.redline_key)
s = monitor.session(id=conversation_id, user=user_id)

span_id = lambda: uuid.uuid4().hex[:16]
turn = span_id()
t0 = int(time.time() * 1000)

s.input(text, span={"id": turn, "parent": None, "startMs": t0, "durMs": 0})

started = int(time.time() * 1000)
result = await lookup(q)
s.tool("lookup", args={"q": q}, result=result,
       span={"id": span_id(), "parent": turn, "startMs": started,
             "durMs": int(time.time() * 1000) - started})

s.output(reply, tokens=439,
         span={"id": span_id(), "parent": turn, "startMs": started, "durMs": 12})
s.end("closed")
```

**TypeScript**

```ts
import { createMonitor, type SpanRef } from "@redlineai/sdk";
import { randomUUID } from "node:crypto";

const monitor = createMonitor({ apiKey: settings.redlineKey });
const s = monitor.session({ id: conversationId, agent: "my-agent" });

const spanId = () => randomUUID().replace(/-/g, "").slice(0, 16);
const turn = spanId();
const t0 = Date.now();

s.input(text, { span: { id: turn, parent: null, startMs: t0, durMs: 0 } });

const started = Date.now();
const result = await lookup(q);
s.tool("lookup", { q }, result, {
  span: { id: spanId(), parent: turn, startMs: started, durMs: Date.now() - started },
});

s.output(reply, { tokens: 439, span: { id: spanId(), parent: turn, startMs: started, durMs: 12 } });
await s.end("closed");
```

The ref rides inside the row's `data` under a reserved key, so your own payload
is untouched. `durMs` also becomes the row's latency — do not pass both. A row
with no ref is still a perfectly good row; it just is not work, so it stays out
of the chart. That is deliberate: the user's words and the agent's answer are
conversation, and a chart that drew them as bars would be measuring typing.

> **Prefer spans to hand-minted refs**
>
> If anything in your process can emit OpenTelemetry, emit it and let the SDK
> derive these refs — they are what the capture writes for itself. Minting them
> by hand means your waterfall is only as honest as your bookkeeping, and a
> parent id that points at nothing silently flattens the tree.

> **Close the session, or you get no intents**
>
> Intent mining runs when a session **closes** — `s.end("closed")`, or the
> decorator returning. A multi-turn conversation is one session; close it when
> the conversation is over, not after every turn, or the miner sees a hundred
> one-line sessions instead of one conversation.

## Emitting your own spans

For a hand-rolled loop there is no package to install; the loop emits its own,
in the OTel GenAI convention the SDK reads:

| span | attributes |
| --- | --- |
| one per **user turn** (the parent) | none — a container carries no `gen_ai.*` |
| one per **model call** | `gen_ai.operation.name="chat"`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens` |
| one per **tool call** | `gen_ai.operation.name="execute_tool"`, `gen_ai.tool.name`, `gen_ai.tool.call.arguments`, `gen_ai.tool.call.result` |

Parent the model and tool spans to the turn span explicitly
(`start_span(..., context=set_span_in_context(turn))`), not by making the turn
current — a loop that yields leaks a current span into whatever runs next.
Without the parent, every span is read as its own turn.

The full, working version is on the
[own tool loop](/integrations/own-tool-loop#4-the-spans--what-makes-the-waterfall) page.

## What the platform does with a session

With no further code:

- It appears in **Monitor → Sessions** as a readable conversation, with the
  waterfall beside it.
- Rule detectors match **every span as it is written** — a violation on a tool
  call is raised the moment the call lands. Within minutes the monitor agent
  re-examines the whole session against every rule the project defines
  (**Monitor → Violations**). Rules are written in the console, in plain
  language, not in the repository.
- When the session closes, the intent miner reads it for recurring user
  behaviour (**Monitor → Intents**). *Custom instructions* on that page steers
  it, also in plain language.

Arrival is the event. Post the session and the rest follows.