Troubleshooting API Gateway Private API

~ Rahul Baisla
Table Of Contents
  1. Connectivity Issues
  2. Invoke URL
  3. Permission Issues
  4. References

This article focuses on Troubleshooting API Gateway Private API. Following are the common causes of restricted access to Private API

  1. Connectivity issues due to Amazon VPC security groups, NACL rules incorrectly configured.
  2. The invoke URL for the private API
  3. Permission related issues due to API Gateway Resource and VPC Endpoint policy

Before you start troubleshooting API Gateway Private APIs, enable API Gateway CloudWatch Access and Execution logs. Refer: Enable API Gateway CloudWatch Logs

Logging Settings
Log level, choose INFO
Log full requests/responses data Set to True

Note: If there is no log entry populated in access logs It would mean that request did not reach API Gateway endpoint.

Connectivity Issues

When you try to access your private API and the request times out without a response this would indicate a connection problem.

Make sure the DNS is resolving and IP addresses returned are for the correct VPC Endpoint.

DNS resolution 

# Private DNS disabled on VPC Endpoint
dig {VPCEndpoint_Public_DNS}
nslookup {VPCEndpoint_Public_DNS}
# Route53 Alias configured
dig {REST_API_ID}-{VPCEndpoint_ID}.execute-api.{region}.amazonaws.com
nslookup {REST_API_ID}-{VPCEndpoint_ID}.execute-api.{region}.amazonaws.com
# Private DNS enabled on VPC Endpoint
dig {REST_API_ID}.execute-api.{region}.amazonaws.com
nslookup {REST_API_ID}.execute-api.{region}.amazonaws.com
Note: You don't need to enable private DNS if you're invoking the private API using its public DNS name or an Amazon Route 53 alias.

Connectivity test

If the DNS is resolving, Test If the connection can be made on port 443 

#using VPCEndpoint_Public_DNS to connect
telnet {VPCEndpoint_Public_DNS} 443
#using Private DNS to connect:
telnet {REST_API_ID}.execute-api.{region}.amazonaws.com 443
#using Route53 Alias configured:
telnet {REST_API_ID}-{VPCEndpoint_ID}.execute-api.{region}.amazonaws.com 443

If during the telnet test, the connection times out, check your VPC configuration to make sure that:

  • The security group of AWS resource from where the request is made have a security group rule that allows TCP Port 443 outbound traffic to the interface VPC endpoint’s IP address range or security group.
  • The interface VPC endpoint’s security group is correctly configured. It must have a rule that allows TCP Port 443 inbound traffic from the IP address range or security group of the AWS resource that’s making requests.
  • From on-prem, make sure the request is reaching the endpoint. If the request is blocked from on-prem, a good test would be to spin up an EC2 instance in the same vpc and run telnet connectivity test from within vpc to eliminate issues with VPCEndpoint iteslf

SSL Test

If there is connectivity on port 443, Next test would be to see If the SSL handshake passes. 

#Private DNS disabled on VPC Endpoint
openssl s_client -connect {VPCEndpoint_Public_DNS}:443 -msg
#Private DNS enabled on VPC Endpoint
openssl s_client -connect {REST_API_ID}.execute-api.{region}.amazonaws.com:443 -msg
#Route53 Alias configured
openssl s_client -connect {REST_API_ID}-{VPCEndpoint_ID}.execute-api.{region}.amazonaws.com:443 -msg