> ## Documentation Index
> Fetch the complete documentation index at: https://docs.ewake.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting Bring Your Own Cloud

> Find and fix common failures of a self-hosted Ewake deployment, such as an agent Lambda that cannot resolve the internal name of the reactive service.

This page lists failures of a [Bring Your Own Cloud](https://docs.ewake.ai/architecture/byoc.md) deployment, how to find their cause, and how to fix them.

In the commands below, replace these values with your own:

| Placeholder | Value |
| - | - |
| `<tenant>` | `tenant_name` in your `terraform.tfvars` |
| `<company>` | `company.name` in your `terraform.tfvars` |
| `<vpc-id>` | `terraform output vpc_id` |

***

## Agent runs fail with `ENOTFOUND reactive.<company>.internal`

### Symptom

Investigations and other agent runs do not start. The dashboard still works.

The log group `/ewake/<tenant>/<company>/reactive-lambda` in CloudWatch Logs shows `Error processing message in Lambda`, with an error like this one:

```json theme={null}
{
  "code": "ENOTFOUND",
  "message": "getaddrinfo ENOTFOUND reactive.<company>.internal"
}
```

The stack trace contains `claimAction` or another call in `reactiveClient.ts`.

### Cause

The agent Lambda calls the dashboard service over a private address in your VPC, not over the load balancer. The Terraform sets this up:

1. It creates an AWS Cloud Map private DNS namespace, `<company>.internal`. Cloud Map stores it as a Route 53 private hosted zone that is attached to your VPC.
2. It registers each running task of the ECS service `<company>` under the name `reactive.<company>.internal`.
3. It gives the Lambda the address `http://reactive.<company>.internal:3000`.

`ENOTFOUND` means that the DNS server of the VPC returned no address for that name. Only the Amazon-provided DNS resolver of the VPC reads private hosted zones. The lookup fails when:

* **A.** A Route 53 Resolver forwarding rule matches the name and sends the query to another DNS server.
* **B.** The DHCP options set of the VPC gives your own DNS servers to the Lambda instead of the Amazon-provided resolver.
* **C.** DNS support or DNS hostnames is off on the VPC.
* **D.** No running task is registered in Cloud Map, so the name has no record.

A, B and C are most frequent when you deploy into a VPC that you already run (`existing_network`).

### Find the cause

**1. Check that a task is registered**

```bash theme={null}
aws ecs describe-services --cluster <tenant> --services <company> \
  --query 'services[0].[runningCount,desiredCount]'

SERVICE_ID=$(aws servicediscovery list-services \
  --query 'Services[?Name==`reactive`].Id' --output text)
aws servicediscovery list-instances --service-id $SERVICE_ID
```

If `runningCount` is `0` or the instance list is empty, go to [Fix D](#d-no-task-is-registered). If an instance has an `AWS_INSTANCE_IPV4` address, the record exists. Continue with the next step.

> [!NOTE]
>
> If you have more than one Ewake deployment in the account, `list-services` returns one `reactive` service for each. Pick the one in the namespace `<company>.internal`.

**2. Resolve the name from inside the VPC**

The dashboard task runs in the same subnets as the Lambda, so it sees the same DNS. Open a shell in it with ECS Exec. You need the [Session Manager plugin](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html) for the AWS CLI.

```bash theme={null}
TASK=$(aws ecs list-tasks --cluster <tenant> --service-name <company> \
  --query 'taskArns[0]' --output text)
aws ecs execute-command --cluster <tenant> --task $TASK \
  --container reactive --interactive --command "/bin/sh"
```

In the shell, run:

```bash theme={null}
cat /etc/resolv.conf
node -e "require('dns').lookup('reactive.<company>.internal', (e, a) => console.log(e ? e.code : a))"
```

* If the lookup prints an IP address, DNS works from the task. Compare the subnets and security groups of the Lambda `<tenant>-<company>-reactive` with those of the task, then contact Ewake support.
* If `nameserver` is not the Amazon-provided resolver, go to [Fix B](#b-the-vpc-uses-your-own-dns-servers). The Amazon-provided resolver is the base of the VPC CIDR plus two (`10.10.0.2` for `10.10.0.0/16`), or `169.254.169.253`.
* If `nameserver` is the Amazon-provided resolver and the lookup prints `ENOTFOUND`, check [Fix C](#c-dns-support-is-off-on-the-vpc), then [Fix A](#a-a-forwarding-rule-captures-the-name).

### A. A forwarding rule captures the name

List the forwarding rules that are associated with the VPC:

```bash theme={null}
aws route53resolver list-resolver-rule-associations \
  --filters Name=VPCId,Values=<vpc-id> \
  --query 'ResolverRuleAssociations[].ResolverRuleId'

aws route53resolver list-resolver-rules \
  --query 'ResolverRules[?RuleType==`FORWARD`].[Id,DomainName]' --output table
```

A forwarding rule for `<company>.internal`, or for a domain that contains it, sends the query to the target of the rule. That DNS server does not know the private hosted zone.

To fix it, add a system rule for the exact name. A system rule tells the Resolver to answer the name itself, from the private hosted zone, and it takes precedence over the forwarding rule:

```bash theme={null}
RULE=$(aws route53resolver create-resolver-rule \
  --creator-request-id ewake-reactive-$(date +%s) \
  --rule-type SYSTEM \
  --domain-name reactive.<company>.internal \
  --name ewake-reactive-internal \
  --query ResolverRule.Id --output text)

aws route53resolver associate-resolver-rule --resolver-rule-id $RULE --vpc-id <vpc-id>
```

The rule names only `reactive.<company>.internal`, so it does not change how the VPC resolves the other names in your domain.

### B. The VPC uses your own DNS servers

Find the DNS servers of the VPC:

```bash theme={null}
DHCP=$(aws ec2 describe-vpcs --vpc-ids <vpc-id> \
  --query 'Vpcs[0].DhcpOptionsId' --output text)
aws ec2 describe-dhcp-options --dhcp-options-ids $DHCP
```

If `domain-name-servers` does not contain `AmazonProvidedDNS`, your DNS servers answer the query. Use one of these fixes:

* **Forward the name to Route 53 (recommended in a shared VPC).** On your DNS servers, add a conditional forwarder for `reactive.<company>.internal` to a Route 53 Resolver inbound endpoint. The endpoint must be in a VPC that is attached to the private hosted zone. If your central DNS uses a different VPC, attach the zone to that VPC too:

  ```bash theme={null}
  ZONE=$(aws route53 list-hosted-zones-by-name --dns-name <company>.internal \
    --query 'HostedZones[0].Id' --output text)
  aws route53 associate-vpc-with-hosted-zone --hosted-zone-id $ZONE \
    --vpc VPCRegion=eu-west-3,VPCId=<dns-vpc-id>
  ```

  If that VPC is in another AWS account, first run `aws route53 create-vpc-association-authorization` in the account of the deployment.

* **Use the Amazon-provided resolver.** Create a DHCP options set with `domain-name-servers=AmazonProvidedDNS` and associate it with the VPC. Do this only if no other workload in the VPC needs your DNS servers.

### C. DNS support is off on the VPC

A private hosted zone needs both VPC attributes:

```bash theme={null}
aws ec2 describe-vpc-attribute --vpc-id <vpc-id> --attribute enableDnsSupport
aws ec2 describe-vpc-attribute --vpc-id <vpc-id> --attribute enableDnsHostnames
```

If either is `false`, turn it on:

```bash theme={null}
aws ec2 modify-vpc-attribute --vpc-id <vpc-id> --enable-dns-support '{"Value":true}'
aws ec2 modify-vpc-attribute --vpc-id <vpc-id> --enable-dns-hostnames '{"Value":true}'
```

### D. No task is registered

The ECS service registers a task in Cloud Map only while the task runs. Find out why the service has no running task:

```bash theme={null}
aws ecs describe-services --cluster <tenant> --services <company> \
  --query 'services[0].events[:10].[createdAt,message]' --output table
```

Then read the logs of the stopped task in CloudWatch Logs. When the service runs a task again, ECS registers it and the name resolves.

### Confirm the fix

1. Run the lookup from [Find the cause](#find-the-cause) again. It must print an IP address.
2. Start a new investigation from Slack or from the dashboard.

The record has a TTL of 10 seconds, so the change takes effect without a redeploy. SQS delivers a failed run again, up to five times in total. After the fifth failure, SQS moves the run to the dead-letter queue `<tenant>-<company>-lambda-dlq`.

### If the run still fails

Send these to Ewake support:

* the full error from `/ewake/<tenant>/<company>/reactive-lambda`
* the output of the commands in [Find the cause](#find-the-cause)
* whether the deployment uses `existing_network`


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.