Agent runs fail with ENOTFOUND reactive.<company>.internal
Symptom
Investigations and other agent runs do not start. The dashboard still works. The log group/ewake/<tenant>/<company>/reactive-lambda in CloudWatch Logs shows Error processing message in Lambda, with an error like this one:
claimAction or another call in reactiveClient.ts.
Cause
The agent Lambda calls the dashboard service over a private address in your VPC, not over the load balancer. The Terraform sets this up:- It creates an AWS Cloud Map private DNS namespace,
<company>.internal. Cloud Map stores it as a Route 53 private hosted zone that is attached to your VPC. - It registers each running task of the ECS service
<company>under the namereactive.<company>.internal. - It gives the Lambda the address
http://reactive.<company>.internal:3000.
ENOTFOUND means that the DNS server of the VPC returned no address for that name. Only the Amazon-provided DNS resolver of the VPC reads private hosted zones. The lookup fails when:
- A. A Route 53 Resolver forwarding rule matches the name and sends the query to another DNS server.
- B. The DHCP options set of the VPC gives your own DNS servers to the Lambda instead of the Amazon-provided resolver.
- C. DNS support or DNS hostnames is off on the VPC.
- D. No running task is registered in Cloud Map, so the name has no record.
existing_network).
Find the cause
1
Check that a task is registered
runningCount is 0 or the instance list is empty, go to Fix D. If an instance has an AWS_INSTANCE_IPV4 address, the record exists. Continue with the next step.If you have more than one Ewake deployment in the account,
list-services returns one reactive service for each. Pick the one in the namespace <company>.internal.2
Resolve the name from inside the VPC
The dashboard task runs in the same subnets as the Lambda, so it sees the same DNS. Open a shell in it with ECS Exec. You need the Session Manager plugin for the AWS CLI.In the shell, run:
- If the lookup prints an IP address, DNS works from the task. Compare the subnets and security groups of the Lambda
<tenant>-<company>-reactivewith those of the task, then contact Ewake support. - If
nameserveris not the Amazon-provided resolver, go to Fix B. The Amazon-provided resolver is the base of the VPC CIDR plus two (10.10.0.2for10.10.0.0/16), or169.254.169.253. - If
nameserveris the Amazon-provided resolver and the lookup printsENOTFOUND, check Fix C, then Fix A.
A. A forwarding rule captures the name
List the forwarding rules that are associated with the VPC:<company>.internal, or for a domain that contains it, sends the query to the target of the rule. That DNS server does not know the private hosted zone.
To fix it, add a system rule for the exact name. A system rule tells the Resolver to answer the name itself, from the private hosted zone, and it takes precedence over the forwarding rule:
reactive.<company>.internal, so it does not change how the VPC resolves the other names in your domain.
B. The VPC uses your own DNS servers
Find the DNS servers of the VPC:domain-name-servers does not contain AmazonProvidedDNS, your DNS servers answer the query. Use one of these fixes:
-
Forward the name to Route 53 (recommended in a shared VPC). On your DNS servers, add a conditional forwarder for
reactive.<company>.internalto a Route 53 Resolver inbound endpoint. The endpoint must be in a VPC that is attached to the private hosted zone. If your central DNS uses a different VPC, attach the zone to that VPC too:If that VPC is in another AWS account, first runaws route53 create-vpc-association-authorizationin the account of the deployment. -
Use the Amazon-provided resolver. Create a DHCP options set with
domain-name-servers=AmazonProvidedDNSand associate it with the VPC. Do this only if no other workload in the VPC needs your DNS servers.
C. DNS support is off on the VPC
A private hosted zone needs both VPC attributes:false, turn it on:
D. No task is registered
The ECS service registers a task in Cloud Map only while the task runs. Find out why the service has no running task:Confirm the fix
- Run the lookup from Find the cause again. It must print an IP address.
- Start a new investigation from Slack or from the dashboard.
<tenant>-<company>-lambda-dlq.
If the run still fails
Send these to Ewake support:- the full error from
/ewake/<tenant>/<company>/reactive-lambda - the output of the commands in Find the cause
- whether the deployment uses
existing_network