HTTP 502, 503, and 504 Errors: A Developer's Diagnostic Guide

There is nothing quite like the specific panic of your pager screaming at 2 AM, only to open Sentry or Datadog and find dead silence. While a classic 500 error is polite enough to leave a stack trace pointing right to an unhandled exception in your code, 502s, 503s, and 504s are whole different beasts.

They aren't code bugs - they are infrastructure black boxes living somewhere between your reverse proxy, container overlays, and app instances. Let's stop blindly changing configs and look straight at what the proxy error logs are actually trying to tell you.

Gateway Errors at the Protocol Layer

Before we jump into specific platforms, it helps to remember how the reverse proxy sees the world. In this setup, your proxy (whether it's Nginx, an ALB, or an Ingress controller) is acting as an HTTP client making a request to your backend service.

When the proxy tries to establish a TCP handshake with your Node, Python, or Go app and gets turned away at the door, that’s an immediate 502. If the door opens and the request goes through, but your backend leaves the proxy hanging without headers for too long, the proxy gives up and hands the user a 504.

A 503 is a bit different: it’s intentional. The infrastructure already knows the upstream isn't fit to take traffic - usually because a health check flunked - so it doesn't even bother attempting a full connection.

The Diagnostic Ladder: Reading Nginx Error Logs

Whenever an Nginx proxy spits out a 500-series error, its error.log almost always leaves an explicit OS-level error code explaining what went sideways. Stop guessing and grep your logs for these exact signatures.

110: Connection timed out (The 504 Trigger)

upstream timed out (110: Connection timed out) while reading response header from upstream

This log line means the network pipe was fine: Nginx successfully connected to your backend, tossed the request over the fence, and waited. The issue is that Nginx's internal proxy_read_timeout hit its threshold before your app sent back so much as a single HTTP header byte. Nine times out of ten, your backend is pinned waiting on a lock in Postgres, an external third-party API that fell over, or an unindexed table scan.

111: Connection refused (The 502 Trigger)

connect() failed (111: Connection refused) while connecting to upstream

Here, Nginx tried to initiate a TCP handshake, and the kernel on the destination machine immediately replied with an explicit RST packet. That means nothing is listening on that socket.

Either your app process crashed completely, you typoed the port in your upstream block, or the app is bound strictly to 127.0.0.1 inside a container while Nginx is trying to reach it over a shared bridge network. If you aren't sure what's bound where, you'll want to inspect your open sockets and find and kill the process by port number before trying to bring the backend back up cleanly.

104: Connection reset by peer (The Mid-Flight 502)

recv() failed (104: Connection reset by peer) while reading response header from upstream

This one is notoriously sneaky. Nginx opened the socket without an issue, handed off the request, and sat waiting for data - only for the connection to violently snap shut from the other side.

This almost never means a clean application error; it means the host OS pulled the plug on your process. Look directly at your system logs for an Out-Of-Memory (OOM) killer strike, or check if your native runtime threw a sudden segmentation fault mid-request.

Fixing 502 and 504 in Nginx Reverse Proxies

When tuning timeouts in Nginx, keep in mind there are three separate knobs, and twisting the wrong one won't do anything for your 504s:

• proxy_connect_timeout: How long Nginx waits just to finish the initial TCP handshake. This should rarely be more than a few seconds.

• proxy_send_timeout: The timeout between two successive write operations when pushing the request body over to the backend.

• proxy_read_timeout: How long Nginx waits between two successive read operations for the backend's response headers. This is the dial you actually want to nudge if you have legitimate, long-running batch endpoints or heavy export jobs.

If you're deploying PHP-FPM on enterprise Linux distros like RHEL, Rocky, or AlmaLinux, you'll sometimes run into instantaneous 502s right after setup, even though both Nginx and PHP are running without errors. You’re almost certainly running face-first into SELinux policies.

By default, SELinux prevents the web server daemon from opening outbound network sockets. Flipping the boolean with setsebool -P httpd_can_network_connect 1 will instantly clear the logjam.

Resolving 502s in Docker Compose Networks

The single most common headache when migrating a project from bare metal into Docker Compose is the classic localhost misunderstanding. When Nginx was running natively on your laptop, pointing proxy_pass http://localhost:3000; worked like a charm. Once you drop that into a container, localhost stops pointing to your host machine - it points straight to the Nginx container’s own isolated network namespace.

Because there’s obviously no backend server running inside that specific proxy container on port 3000, you’re greeted with a swift 111: Connection refused. To fix it, you need to use the service name declared in your compose file as the target domain. Swapping your directive to proxy_pass http://backend-api:3000; lets Docker's internal DNS daemon step in and resolve the request straight to the right container IP.

If you're taking a closer look at whether Docker Compose is still the best fit for your stack, modern runtime tooling like Podman and nerdctl provide rootless container setups that handle daemonless networking a bit differently, cutting out a lot of these proxy headaches entirely.

Kubernetes Ingress 502s During Rolling Deploys

Ever rolled out a routine release in Kubernetes, watched all your pods go green, and still noticed a sudden spike of 502s in your metrics? You're running into the fact that the Kubernetes control plane is eventually consistent. An Ingress controller or kube-proxy might still be routing incoming requests to an old pod that just received its termination notice, or forwarding connections to a shiny new pod that technically marked itself "Running" before its application runtime was fully booted.

To get honest zero-downtime rollouts, you need to coordinate two fundamental pieces:

• Readiness Probes: Never trust a pod just because the process started. Make sure Kubernetes holds back traffic until your framework returns a clean 200 on an explicit /health or /ready endpoint.

• PreStop Hooks: Give your ingress a head start by dropping a preStop hook with a simple sleep 5 into your pod spec. That brief delay pauses the pod before it gets hit with the SIGTERM, giving the Ingress controller and iptables rules enough time to pull that pod’s IP out of active rotation before the app actually stops listening.

AWS Specifics: API Gateway, CloudFront, and ALB

Once you move into managed AWS territory, you have to play by their hard architectural boundaries. Things that pass muster on a plain VPS can fail without a trace behind AWS load-balancing layers.

The Lambda 29-Second Hard Limit

AWS lets you configure a Lambda function to crunch numbers for up to 15 full minutes. But if that Lambda is wired up behind an AWS API Gateway, you run directly into an immutable, non-negotiable 29-second integration ceiling.

If your function takes 30 seconds to return, API Gateway cuts the cord and hands the user a 504, leaving your Lambda happily burning compute cycles in the background. If a process takes that long, you have to pull it out of the HTTP lifecycle and route it through an SQS queue or Step Function instead.

API Gateway Malformed Integration 502

If your Lambda code runs without an error, finishes in 200ms, and CloudWatch logs look clean, but API Gateway still throws a 502 Bad Gateway to the user, the issue is almost always your payload format. When you're using Lambda Proxy Integration, API Gateway demands a strict, top-level JSON structure. If your handler returns an unformatted string, or an object that forgets the explicit statusCode and stringified body keys, API Gateway marks the response as corrupted and kills it with a 502.

CloudFront SSL Certificate Mismatches

Getting a 502 Bad Gateway directly from CloudFront usually means the edge location hit a brick wall during the upstream TLS handshake with your origin server. CloudFront is exceptionally strict: the SSL certificate presented by your origin must cleanly cover the exact domain name configured in your distribution's Origin Domain setting. If your origin is an ALB relying on a self-signed cert, or if you pointed CloudFront directly to an IP address without a matching Subject Alternative Name (SAN), CloudFront will refuse the connection every single time.

Strategic Fixes: Raising Timeouts vs. Fixing the Backend

When your dashboards start lighting up with 504 Gateway Timeouts, the natural human urge is to just bump proxy_read_timeout from 60 seconds to 120 seconds so people stop seeing errors. Most of the time, this is throwing fuel on the fire.

Bumping that timeout rarely solves the underlying problem - it just leaves backend worker threads and database connections hanging wide open for twice as long. If the root cause is a dropped index or a table lock in your database, dragging out the timeout simply causes requests to pile up, exhausting your connection pools and triggering a massive cascading 503 outage as the whole host runs out of memory. Keep timeouts tight, profile your slow queries first, and only increase timeouts on specific route locations you know run genuine, long-haul background processing.

How to Return 503 Correctly During Planned Maintenance

If you need to pull your backend down to run a big database migration and you don't prep your proxy, your users - and search engine scrapers - are going to run into a messy wall of 502s. That's dangerous: when Googlebot crawls your site and gets slammed with 502s, it assumes your application is broken, and persistent gateway errors can actively damage your search index standing.

The right way to handle this, straight out of the RFC 7231 spec, is to deliberately serve a 503 Service Unavailable paired with a well-configured Retry-After header.

You can configure Nginx to catch all incoming traffic during the maintenance window, serve a friendly static maintenance HTML page, and return a clean 503 status code. Adding a header like Retry-After: 3600 tells clients to try again in an hour. Real users get an intentional maintenance message, and search bots understand this is planned downtime, pausing their crawl budget without penalizing your rankings.

Reproducing Gateway Errors Locally

The best way to demystify these errors is to break things on purpose on your local machine so you recognize the symptoms next time.

To see a pure 502 Bad Gateway, point your local Nginx proxy block to a port where you know nothing is listening (like localhost:9999). That instantaneous connection rejection will trigger an immediate 502 in your browser and drop a clear 111: Connection refused in your local error log.

To manufacture a clean 504 Gateway Timeout, drop a simple artificial pause into an endpoint. In a Node.js route, you can just toss in await new Promise(r => setTimeout(r, 10000));, and then set your local Nginx proxy_read_timeout down to 2s. Send a request to that URL: Nginx will neatly sever the connection after two seconds with a 504, while your Node server quietly keeps waiting in the background.