DevOps Quick Fix are solutions to common DevOps problems, usually encountered during client engagements.

Problem

From Google Cloud Platform (GCP), your outgoing HTTPS connections to third-party REST APIs are slow or dropping.

Solution

Preamble

The cause of the issue is most likely GCP Cloud NAT port exhaustion.

If you are using private network Compute Engine, GKE, or any of the GCP serverless services like Cloud Run, your outgoing connections are going through Cloud NAT.

Cloud NAT needs a unique 5-tuple [source IP, source port, destination IP/port/protocol] to make an outgoing connection. If all the 5-tuples are used, the connections are slowed or dropped. This is known as NAT port exhaustion.

Turn off Endpoint-Independent Mapping

You can read the gnarly technical details about Endpoint-Independent Mapping conflicts, but bottom-line, EIM is more trouble that it is worth. Turn EIM off and increase the "Minimum ports per VM instances" setting (2048 is a good starting option).

Steps:
1. In GCP console, go to Cloud NAT
2. Click your Cloud NAT Gateway link
3. Click the "Edit" button on top, then click "Advanced Configuration"
4. Uncheck "Enable Endpoint-Independent Mapping" and increase the "Minimum port per VM instance" value
5. Click the "Save" button at the bottom

Optional: Enable Dynamic Port Allocation

Dynamic Port Allocation will scale up the number of VM ports when it senses port exhaustion. The main issue is connections can drop during scaling. I usually prefer to turn it off and set the "Minimum ports per VM instance" to a high number.

If you prefer to turn on "Dynamic Port Allocation", set the minimum port high to reduce the chance of dropped connections.

Optional: Assign more IPs to Cloud NAT

If you are using static reserved IPs for Cloud NAT because of third-party API firewall allow-lists, you can manually add more reserved IPs to reduce port exhaustion.

Steps:
1. In GCP console, go to Cloud NAT
2. Click your Cloud NAT Gateway link
3. Click the "Edit" button on top, then go to the "Cloud NAT mapping" section.
4. Click the "+ ADD IP ADDRESS" button to add more IPs to Cloud NAT
5. Click the "Save" button at the bottom