Skip to main content
What you’ll get: a way to ask “how good is ewake on our alerts?” using alerts that already fired in your own channels, rather than waiting for the next incident.
An evaluation takes real alerts from a Slack channel, replays them through ewake as if they had just fired, and collects the answers side by side. You rate each one, and a rating with a comment feeds back into ewake’s skills.

​
Prerequisites

  • Slack connected, and ewake in the channel you want to import from
  • A channel with past alerts from a bot — Datadog, Grafana, PagerDuty, whichever posts yours
  • An admin account — Evaluation is an admin page

​
Run an evaluation

1

Open Evaluation and click New

The page lists your evaluations, newest first, with how far each one’s runs have got.
2

Choose the channel, the bot and a date range

Pick the channel the alerts landed in, the bot that posted them, and the window to look back over.Ewake reads that channel’s history, keeps the chosen bot’s top-level firing alerts, and skips anything it has already imported. Thread replies, resolutions and chatter are ignored.
3

Review what was imported

You see how many alerts were found before anything runs. Nothing has been spent yet.
4

Start the replay

Each alert is scheduled as its own run, five minutes apart. They arrive exactly as a live alert would — same investigation lifecycle, same tools, same models.
A replay is a full agent run and costs the same as a real investigation. Forty replays is forty investigations. The limits below exist to make that hard to trigger by accident.
5

Come back and audit

Runs land over the following minutes. Open the evaluation to see each alert, ewake’s answer, latency, cost and the rating.

​
Limits

Both halves are bounded, deliberately. If a channel is busier than the message limit, the import says so rather than silently taking a slice. The wait between batches is a spend guard — you cannot start a second evaluation until the previous one has had time to finish.

​
What each status means

Queued and Missed are the same underlying state — no run has been recorded yet. A run that never starts shows as Queued for an hour before it becomes Missed, so a recently queued run and a dead one look alike for that first hour.

​
Rating an answer

Each run takes a thumb up or down, and optionally a comment.
  • A bare thumb records the judgement.
  • A thumb with a comment goes further: ewake reads the comment against the run and turns it into a skill — a new one, or an edit to an existing one. That is how an evaluation changes future behaviour rather than just scoring past behaviour.
Comments are worth the extra seconds. “Wrong service — the 500s were upstream in auth-api, not checkout” teaches something. A bare thumbs-down does not.

​
Deleting an evaluation

Deleting removes the evaluation and its runs. The imported alerts stay, so you can replay them again without re-importing. Investigations that already ran are unaffected — they remain in your history like any other.

​
Good candidates for a first evaluation

  • A noisy channel you already distrust. If ewake handles those well, that is the strongest signal you will get.
  • A week that contained a real incident, so the set includes something that mattered rather than only routine noise.
  • A single bot at a time. Mixing sources makes the results harder to read.
Evaluations replay alerts against today’s integrations and knowledge graph, not the state at the time the alert fired. An alert from three months ago is answered with what ewake knows now.