What you’ll get: a way to ask “how good is ewake on our alerts?” using alerts that
already fired in your own channels, rather than waiting for the next incident.
Prerequisites
- Slack connected, and ewake in the channel you want to import from
- A channel with past alerts from a bot — Datadog, Grafana, PagerDuty, whichever posts yours
- An admin account — Evaluation is an admin page
Run an evaluation
1
Open Evaluation and click New
The page lists your evaluations, newest first, with how far each one’s runs have got.
2
Choose the channel, the bot and a date range
Pick the channel the alerts landed in, the bot that posted them, and the window to
look back over.Ewake reads that channel’s history, keeps the chosen bot’s top-level firing
alerts, and skips anything it has already imported. Thread replies, resolutions and
chatter are ignored.
3
Review what was imported
You see how many alerts were found before anything runs. Nothing has been spent yet.
4
Start the replay
Each alert is scheduled as its own run, five minutes apart. They arrive exactly as
a live alert would — same investigation lifecycle, same tools, same models.
5
Come back and audit
Runs land over the following minutes. Open the evaluation to see each alert, ewake’s
answer, latency, cost and the rating.
Limits
Both halves are bounded, deliberately.
If a channel is busier than the message limit, the import says so rather than silently
taking a slice. The wait between batches is a spend guard — you cannot start a second
evaluation until the previous one has had time to finish.
What each status means
Queued and Missed are the same underlying state — no run has been recorded yet.
A run that never starts shows as Queued for an hour before it becomes Missed, so
a recently queued run and a dead one look alike for that first hour.
Rating an answer
Each run takes a thumb up or down, and optionally a comment. Both are recorded against the run and shown next to it, so you can read the batch as a score and go back to the reasoning behind any single verdict.Ratings on this page score the batch; they do not teach ewake. Only the 👍 / 👎
buttons under an answer in Slack feed the loop that writes and edits
skills.
Deleting an evaluation
Deleting removes the evaluation and its runs. The imported alerts stay, so you can replay them again without re-importing. Investigations that already ran are unaffected — they remain in your history like any other.Evaluate on live alerts in a separate channel
Replaying past alerts is one way to judge ewake. The other is to let it answer your real alerts as they fire, but send every answer to a channel only you are watching. Your on-call channel is untouched — no threads appear under the alerts your team is working — and you still get ewake’s answer on each one, at the moment it mattered. This is a routing rule on a Slack alert source: ewake watches the channel your alerts already land in, and a rule sends the answers somewhere else.Prerequisites
- Slack connected, and ewake invited to both channels — the one it watches and the one it answers in
- A channel to evaluate in, for example
#ewake-trial. A private channel is fine - Alerts arriving from a bot in your existing channel
1
Open On-call and create an alert source
Click On-call in the sidebar, then New source. Alert sources are how alerts
reach ewake.
2
Choose Slack
Pick Slack as the source kind. Ewake watches a channel and answers the alerts a bot
posts there, so there is nothing to configure on the sending side.
3
Fill in the Configure step
Name the source, choose the Slack channel your alerts already land in, and choose
the Slack bot that posts them — Datadog, Grafana, whichever is yours. Ewake responds
to that bot only.Leave the prompt as it is for a first evaluation, so you are judging ewake’s default
behaviour rather than your own wording.
4
Add a routing rule that catches every alert
Still on the Configure step, under Routing rules, click to add a rule and set:
Every alert from a Slack source carries
vendor: slack, so this rule matches all of
them. That is what makes it a catch-all rather than a filter.With this rule saved, ewake answers in your evaluation channel instead of threading
under the alert. Without it, answers go to a thread under the alert itself.
5
Invite ewake to the evaluation channel
The Connect step lists the channels you need ewake in, including any your rules
route to. Invite
@ewake to each one.6
Wait for an alert
The next alert that bot posts gets investigated, and the answer lands in your evaluation
channel. To try it without waiting, fire a test alert from your monitoring tool.
Narrowing it further
vendor is the catch-all. To evaluate on a slice instead — one monitor, one severity —
use a more specific field and leave the rest threading as normal:
The Alert field box suggests common paths, but you can type any top-level alert field
or
attributes.<name> — attribute names come from your vendor, so the list is a suggestion,
not the permitted set. Matching is exact.
You can add up to 20 rules per source. The first rule that matches wins, so order is
precedence — put your narrow rules above the catch-all, or the catch-all will swallow them.
When you are done
- Delete the rule to send answers back to a thread under each alert. This is the switch from evaluating to running for real — nothing else changes.
- Delete the source to stop ewake answering these alerts at all.
Answers go to the evaluation channel instead of the alert thread, not in addition to it.
While the catch-all rule is in place, your on-call channel sees no ewake replies.
Good candidates for a first evaluation
- A noisy channel you already distrust. If ewake handles those well, that is the strongest signal you will get.
- A week that contained a real incident, so the set includes something that mattered rather than only routine noise.
- A single bot at a time. Mixing sources makes the results harder to read.
Evaluations replay alerts against today’s integrations and knowledge graph, not the
state at the time the alert fired. An alert from three months ago is answered with what
ewake knows now.