Skip to main content
This page lists failures of a Bring Your Own Cloud deployment, how to find their cause, and how to fix them. In the commands below, replace these values with your own:

Agent runs fail with ENOTFOUND reactive.<company>.internal

Symptom

Investigations and other agent runs do not start. The dashboard still works. The log group /ewake/<tenant>/<company>/reactive-lambda in CloudWatch Logs shows Error processing message in Lambda, with an error like this one:
The stack trace contains claimAction or another call in reactiveClient.ts.

Cause

The agent Lambda calls the dashboard service over a private address in your VPC, not over the load balancer. The Terraform sets this up:
  1. It creates an AWS Cloud Map private DNS namespace, <company>.internal. Cloud Map stores it as a Route 53 private hosted zone that is attached to your VPC.
  2. It registers each running task of the ECS service <company> under the name reactive.<company>.internal.
  3. It gives the Lambda the address http://reactive.<company>.internal:3000.
ENOTFOUND means that the DNS server of the VPC returned no address for that name. Only the Amazon-provided DNS resolver of the VPC reads private hosted zones. The lookup fails when:
  • A. A Route 53 Resolver forwarding rule matches the name and sends the query to another DNS server.
  • B. The DHCP options set of the VPC gives your own DNS servers to the Lambda instead of the Amazon-provided resolver.
  • C. DNS support or DNS hostnames is off on the VPC.
  • D. No running task is registered in Cloud Map, so the name has no record.
A, B and C are most frequent when you deploy into a VPC that you already run (existing_network).

Find the cause

1

Check that a task is registered

If runningCount is 0 or the instance list is empty, go to Fix D. If an instance has an AWS_INSTANCE_IPV4 address, the record exists. Continue with the next step.
If you have more than one Ewake deployment in the account, list-services returns one reactive service for each. Pick the one in the namespace <company>.internal.
2

Resolve the name from inside the VPC

The dashboard task runs in the same subnets as the Lambda, so it sees the same DNS. Open a shell in it with ECS Exec. You need the Session Manager plugin for the AWS CLI.
In the shell, run:
  • If the lookup prints an IP address, DNS works from the task. Compare the subnets and security groups of the Lambda <tenant>-<company>-reactive with those of the task, then contact Ewake support.
  • If nameserver is not the Amazon-provided resolver, go to Fix B. The Amazon-provided resolver is the base of the VPC CIDR plus two (10.10.0.2 for 10.10.0.0/16), or 169.254.169.253.
  • If nameserver is the Amazon-provided resolver and the lookup prints ENOTFOUND, check Fix C, then Fix A.

A. A forwarding rule captures the name

List the forwarding rules that are associated with the VPC:
A forwarding rule for <company>.internal, or for a domain that contains it, sends the query to the target of the rule. That DNS server does not know the private hosted zone. To fix it, add a system rule for the exact name. A system rule tells the Resolver to answer the name itself, from the private hosted zone, and it takes precedence over the forwarding rule:
The rule names only reactive.<company>.internal, so it does not change how the VPC resolves the other names in your domain.

B. The VPC uses your own DNS servers

Find the DNS servers of the VPC:
If domain-name-servers does not contain AmazonProvidedDNS, your DNS servers answer the query. Use one of these fixes:
  • Forward the name to Route 53 (recommended in a shared VPC). On your DNS servers, add a conditional forwarder for reactive.<company>.internal to a Route 53 Resolver inbound endpoint. The endpoint must be in a VPC that is attached to the private hosted zone. If your central DNS uses a different VPC, attach the zone to that VPC too:
    If that VPC is in another AWS account, first run aws route53 create-vpc-association-authorization in the account of the deployment.
  • Use the Amazon-provided resolver. Create a DHCP options set with domain-name-servers=AmazonProvidedDNS and associate it with the VPC. Do this only if no other workload in the VPC needs your DNS servers.

C. DNS support is off on the VPC

A private hosted zone needs both VPC attributes:
If either is false, turn it on:

D. No task is registered

The ECS service registers a task in Cloud Map only while the task runs. Find out why the service has no running task:
Then read the logs of the stopped task in CloudWatch Logs. When the service runs a task again, ECS registers it and the name resolves.

Confirm the fix

  1. Run the lookup from Find the cause again. It must print an IP address.
  2. Start a new investigation from Slack or from the dashboard.
The record has a TTL of 10 seconds, so the change takes effect without a redeploy. SQS delivers a failed run again, up to five times in total. After the fifth failure, SQS moves the run to the dead-letter queue <tenant>-<company>-lambda-dlq.

If the run still fails

Send these to Ewake support:
  • the full error from /ewake/<tenant>/<company>/reactive-lambda
  • the output of the commands in Find the cause
  • whether the deployment uses existing_network