How to Fix 403 and 502 Errors on Azure Application Gateway v2

What the Azure Application Gateway error pages mean

Azure Application Gateway is a load balancer and reverse proxy that sits between visitors and your web servers. When it cannot or will not pass a request on, it answers itself with a plain error page that ends in the server name Microsoft-Azure-Application-Gateway/v2. That footer tells you something useful: the error came from the gateway, not from your application, so your application logs will often show nothing at all.

The two errors people search for most have quite different causes:

A 403 from the gateway is the WAF doing its job, sometimes wrongly. A form post containing something that looks like SQL, a long cookie, or a content management system admin screen sending unusual parameters can all trigger a managed rule. In Prevention mode, a matched rule set that scores high enough blocks the request with a 403. In Detection mode, the same request is logged but allowed through.

A 502 almost always means that, from the gateway's point of view, the backend pool is unhealthy. The gateway sends health probes to every backend and only routes traffic to those that answer correctly. If every member fails the probe, every request gets a 502 even if the site works perfectly when you visit the server directly.

Common causes of 403 and 502 on Application Gateway v2

403 Forbidden causes

502 Bad Gateway causes

The timeout errors guide covers the slow backend case in more depth, because the fix is usually in the application rather than in Azure.

How to diagnose a 403 from the WAF

You need the WAF logs to see which rule fired. Without diagnostic settings the gateway keeps no record you can search, so turn them on first: open the Application Gateway in the Azure portal, choose Diagnostic settings, add a setting, tick the firewall log and the access log, and send them to a Log Analytics workspace. Reproduce the error after that, because earlier requests were not recorded.

Then open the workspace, choose Logs, and run a query. If your diagnostic setting uses the Azure diagnostics table, this lists recent firewall matches for your test IP address:

AzureDiagnostics
| where Category == "ApplicationGatewayFirewallLog"
| where clientIp_s == "203.0.113.25"
| where TimeGenerated > ago(1h)
| project TimeGenerated, requestUri_s, ruleId_s, action_s, Message, details_message_s, details_data_s, transactionId_g
| order by TimeGenerated desc

With the resource-specific table option, the same data sits in AGWFirewallLogs with slightly different column names. Look at the rows that share one transaction ID. With anomaly scoring, several rules show as Matched, and a final row shows the request was Blocked once the total score passed the threshold. The Matched rows tell you which rules to deal with and details_data_s tells you what part of the request set them off, such as a cookie name or a form field.

How to diagnose a 502 Bad Gateway

Start with the Backend health page of the gateway in the portal. It lists every backend pool member, its status, and for unhealthy members a message that names the problem. The same information is available from the Azure CLI:

az network application-gateway show-backend-health \
  --resource-group rg-example-web \
  --name agw-example \
  --query "backendAddressPools[].backendHttpSettingsCollection[].servers[].{address:address, health:health, log:healthProbeLog}" \
  --output table

Next, check from inside the virtual network that the backend answers the probe the way the gateway expects. From a virtual machine in the same network, request the probe path with the host name the probe uses:

# Plain HTTP backend on a private IP, with the host header the probe sends
curl -sI -H "Host: www.example.com" http://10.0.2.4/health

# HTTPS backend: check the certificate name and chain the gateway will see
curl -svI --resolve www.example.com:443:10.0.2.4 https://www.example.com/health 2>&1 | grep -E "subject:|issuer:|HTTP/"

The probe counts any status from 200 to 399 as healthy by default. A probe path that redirects to a login page off the site, or returns 401 or 404, marks the member as down. If the backend looks fine but users still get intermittent 502s, check the access log for the time taken and compare it with the request timeout in the backend setting. A response slower than that limit is cut off and returned as a 502.

AzureDiagnostics
| where Category == "ApplicationGatewayAccessLog"
| where httpStatus_d == 502
| where TimeGenerated > ago(1h)
| project TimeGenerated, requestUri_s, serverRouted_s, serverStatus_s, timeTaken_d
| order by timeTaken_d desc

An empty serverStatus_s on 502 rows usually means the backend never answered at all, which points at network rules or a dead service rather than the application.

The key fixes

Fixing a WAF false positive with an exclusion

An exclusion tells the WAF to skip inspecting one named part of the request, for example a cookie or form field, while still checking everything else. In the WAF policy, open Managed rules, choose Exclusions, and add one. Where your rule set version supports it, scope the exclusion to the specific rule IDs from the log rather than to the whole rule set. For a rich text field called body on a blog editor that trips SQL injection rules, the exclusion would be: match variable Request arg names, operator Equals, selector body, applied to the rule IDs that fired.

If one rule fires on legitimate traffic across many fields, you can disable that single rule in the managed rule set instead. Prefer exclusions, because a disabled rule protects nothing anywhere on the site.

Allowing gateway traffic through the network security group

Application Gateway v2 needs inbound traffic from the GatewayManager service tag on ports 65200 to 65535 on its own subnet, plus traffic from the Azure load balancer. The backend subnet must accept the probe and request traffic from the gateway subnet.

az network nsg rule create \
  --resource-group rg-example-web --nsg-name nsg-agw-subnet \
  --name Allow-GatewayManager --priority 100 --direction Inbound \
  --access Allow --protocol Tcp \
  --source-address-prefixes GatewayManager \
  --destination-port-ranges 65200-65535

Fixing certificate and host name problems

For HTTPS backends, the gateway checks that the backend certificate is trusted and that its name matches the host name in the backend setting. If the backend uses a certificate from a public certificate authority, make sure the backend setting sends the right host: either enter the host name, such as www.example.com, or turn on Pick host name from backend target for App Service and other multi-tenant backends. If the backend uses an internal or self-signed certificate, upload its root certificate as a trusted root certificate in the backend setting. See the SSL host name mismatch guide for how the name check works.

Giving the probe a page that tells the truth

Create a custom probe pointing at a light page such as /health that returns 200 when the application can serve pages and an error when it cannot. Set the host name to match your site, and use an interval of 30 seconds with an unhealthy threshold of 3 as a sensible start.

A worked example

Example Bakery moves its WordPress site behind Application Gateway v2 with the WAF in Prevention mode. Two problems appear on launch day.

First, staff get 403 Forbidden, Microsoft-Azure-Application-Gateway/v2 every time they save a post that contains a recipe table. The firewall log query shows three Matched rows from SQL injection and cross-site scripting rules, all with details_data_s pointing at the post argument content on /wp-admin/post.php, followed by a Blocked row. Rather than disable the rules, the administrator adds a per-rule exclusion for the argument name content on those three rule IDs, and adds a custom rule that only allows /wp-admin from the office IP range. Saving works, and the admin area is better protected than before.

Second, the public site returns 502 Bad Gateway for all visitors. Backend health shows both servers as unhealthy with the message Received invalid status code from the backend. Visiting the servers directly works, so the administrator tests with curl from a VM in the same network, sending the host name the backend setting uses. The cause is that the backend setting overrides the host to the old staging domain, and the server answers 404 for that name. Correcting the host name in the backend setting and adding a custom probe on /health.txt turns both members healthy within a minute, and the 502s stop.

The fix, step by step

Edge cases worth knowing

• Search engine crawlers being blocked. Bot protection and geo rules can block legitimate crawlers. If Search Console reports 403 for pages that load for you, check the firewall log for the crawler's requests. Our website 403 Forbidden guide covers how to confirm a crawler is being refused.
• A 403 that is really from the app. If the error page looks like your application's design rather than the plain gateway page, the WAF passed the request and the backend refused it.
• Intermittent 502s during deploys. If servers restart while still in the pool, the gateway routes to them until the probe marks them down. Use connection draining in the backend setting and deploy one member at a time.
• User-defined routes. A route table on the gateway subnet that sends internet traffic through a firewall appliance can break the gateway's own management traffic and probe results.
• Long uploads and websockets. Long-running requests can hit the request timeout. Raise it only for the backend setting that serves those paths.
• Outages and SEO. Long runs of 5xx errors slow Google's crawling. See Server error (5xx) in Search Console for what to do afterwards.

Keeping the gateway healthy

• Keep diagnostic logs flowing to Log Analytics at all times, so the next 403 or 502 can be traced in minutes rather than reproduced.
• Set an Azure Monitor alert on the Unhealthy Host Count metric above zero, and another on a rise in failed requests.
• Before turning on Prevention mode, run the WAF in Detection mode for a week of normal traffic, review the Matched rules, and add exclusions for genuine false positives first.
• Track backend certificate expiry dates and renew before they lapse, because an expired backend certificate makes every member unhealthy at once.
• Review exclusions and disabled rules every few months, and remove any that no longer apply.
• After any change, crawl the site from outside and look for 403 and 5xx responses. The HTTP error audit guide explains how to read the results.