> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ewake.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluations

> Replay alerts that already fired, read how ewake answered, and rate the answers to teach it what good looks like.

<Info>
  **What you'll get:** a way to ask "how good is ewake on *our* alerts?" using alerts that
  already fired in your own channels, rather than waiting for the next incident.
</Info>

An evaluation takes real alerts from a Slack channel, replays them through ewake as if
they had just fired, and collects the answers side by side. You rate each one, and a
rating with a comment feeds back into ewake's [skills](/working-with-ewake/skills).

***

## Prerequisites

* **Slack connected**, and ewake in the channel you want to import from
* **A channel with past alerts** from a bot — Datadog, Grafana, PagerDuty, whichever posts yours
* **An admin account** — Evaluation is an admin page

***

## Run an evaluation

<Steps>
  <Step title="Open Evaluation and click New">
    The page lists your evaluations, newest first, with how far each one's runs have got.
  </Step>

  <Step title="Choose the channel, the bot and a date range">
    Pick the channel the alerts landed in, the bot that posted them, and the window to
    look back over.

    Ewake reads that channel's history, keeps the chosen bot's **top-level firing
    alerts**, and skips anything it has already imported. Thread replies, resolutions and
    chatter are ignored.
  </Step>

  <Step title="Review what was imported">
    You see how many alerts were found before anything runs. Nothing has been spent yet.
  </Step>

  <Step title="Start the replay">
    Each alert is scheduled as its own run, **five minutes apart**. They arrive exactly as
    a live alert would — same investigation lifecycle, same tools, same models.

    <Warning>
      **A replay is a full agent run and costs the same as a real investigation.** Forty
      replays is forty investigations. The limits below exist to make that hard to trigger
      by accident.
    </Warning>
  </Step>

  <Step title="Come back and audit">
    Runs land over the following minutes. Open the evaluation to see each alert, ewake's
    answer, latency, cost and the rating.
  </Step>
</Steps>

***

## Limits

Both halves are bounded, deliberately.

|                             | Limit      |
| --------------------------- | ---------- |
| Import range                | 90 days    |
| Messages scanned per import | 500        |
| Alerts kept per import      | 50         |
| Replays per batch           | 40         |
| Spacing between replays     | 5 minutes  |
| Wait between batches        | 60 minutes |

If a channel is busier than the message limit, the import says so rather than silently
taking a slice. The wait between batches is a spend guard — you cannot start a second
evaluation until the previous one has had time to finish.

***

## What each status means

| Status      | Meaning                                          |
| ----------- | ------------------------------------------------ |
| **Queued**  | Scheduled, not started yet                       |
| **Running** | The investigation is in progress                 |
| **Done**    | Finished — the answer is there to read           |
| **Failed**  | The run errored                                  |
| **Missed**  | Scheduled over an hour ago and never started     |
| **Stalled** | Started over an hour ago and never reported back |

<Note>
  **Queued** and **Missed** are the same underlying state — no run has been recorded yet.
  A run that never starts shows as **Queued** for an hour before it becomes **Missed**, so
  a recently queued run and a dead one look alike for that first hour.
</Note>

***

## Rating an answer

Each run takes a thumb up or down, and optionally a comment.

* **A bare thumb** records the judgement.
* **A thumb with a comment** goes further: ewake reads the comment against the run and
  turns it into a skill — a new one, or an edit to an existing one. That is how an
  evaluation changes future behaviour rather than just scoring past behaviour.

<Tip>
  Comments are worth the extra seconds. "Wrong service — the 500s were upstream in
  `auth-api`, not `checkout`" teaches something. A bare thumbs-down does not.
</Tip>

***

## Deleting an evaluation

Deleting removes the evaluation and its runs. The imported alerts stay, so you can replay
them again without re-importing.

Investigations that already ran are unaffected — they remain in your history like any
other.

***

## Good candidates for a first evaluation

* **A noisy channel you already distrust.** If ewake handles those well, that is the
  strongest signal you will get.
* **A week that contained a real incident**, so the set includes something that mattered
  rather than only routine noise.
* **A single bot at a time.** Mixing sources makes the results harder to read.

<Note>
  Evaluations replay alerts against **today's** integrations and knowledge graph, not the
  state at the time the alert fired. An alert from three months ago is answered with what
  ewake knows now.
</Note>