What is a trace?

Before traces, finding a slow request meant reproducing it locally and hoping your laptop behaved like production. A trace removes the hoping. It records what one real request actually did, step by step, with a number next to each step — so "the checkout page is slow" becomes "this one query took 820 of the 980 milliseconds." The waterfall is the whole point: you stop guessing and start reading.

The shape of a trace

A trace is a tree. The root span is the whole request. Child spans are the operations that ran inside it. Each span has a start time, end time, and optional parent pointer.

A simplified Laravel trace

POST /api/checkout                        [root, 980ms]
├── middleware.auth                       [3ms]
├── middleware.throttle                   [1ms]
├── Controller@store                      [962ms]
│   ├── db.query SELECT user              [4ms]
│   ├── db.query SELECT cart WITH items   [820ms]   ← the bottleneck
│   ├── http.out stripe.com/v1/charges    [140ms]
│   └── view.render checkout.success      [12ms]
└── response.send                         [2ms]

At a glance you can see db.query SELECT cart WITH items owns 84% of the request. No guessing — you drill in and inspect the SQL, see it's loading every cart item for every user ever (an N+1 disguised as an inefficient join), and fix it.

Trace IDs and span IDs

Every span carries a trace ID (shared across the whole trace) and its own span ID. Parent-child pointers rebuild the tree. In a multi-service architecture, trace IDs propagate between services via HTTP headers (traceparent) so spans from different services stitch into one trace.

Traces within a Laravel monolith

Every Laravel request is already a trace once instrumented. The Nightwatch package auto-instruments the framework's common operations — DB queries, cache, mail, queue dispatches, outgoing HTTP — and ties them back to the originating request. NightOwl and Laravel Nightwatch Cloud both consume this data and render the waterfall.

What traces don't solve

• Aggregation — one trace tells you about one request. For patterns you still need aggregated metrics (p95 per route).
• Long-running work — queue jobs after the HTTP response need their own trace or a linked span.
• Cost — storing every trace at high traffic is expensive. Sampling is usually required.

Reading a waterfall fast

The skill is less about the tooling and more about where to look. Two heuristics carry most of the weight.

Look for the widest bar, not the deepest tree. A trace with forty spans is intimidating until you notice one of them owns 80% of the width. That bar is your bottleneck; the other thirty-nine are noise for this particular investigation.

Then look for things that run in sequence but could run in parallel. Three external API calls stacked end to end, 100ms each, is 300ms you might collapse to 100ms by firing them concurrently. The waterfall makes that obvious in a way the code doesn't — the code reads fine, it's the timing that gives it away.