Certificate Transparency Logs Leak Origin IP

▶ Watch (5:40)

Every certificate from Let’s Encrypt, DigiCert, or any CA gets logged into a public immutable database. When your load balancer uses that certificate, the subject alternative name points to your domain and subdomain. Attackers query CT logs, find the subdomain’s IP, and bypass the CDN. A direct curl to the load balancer returns a 200 while the main domain returns a 403. Defense: network isolation that allows inbound only from the load balancer, plus origin authentication via X-Origin-Verify and mutual TLS. A five-minute fix: audit security group rules, add a rule to allow ingress only from the load balancer, then test and delete any any-any rule.

Exposed Services Behind the Load Balancer

▶ Watch (10:31)

With the load balancer IP known, attackers scan for open ports. Port 6379 exposes Redis without a password. Port 8080 exposes the HAProxy stats page showing the entire infrastructure map. Attackers extract API keys, database passwords, and session tokens from Redis in seconds. The DAX: all services accept connections from anywhere because they bind to 0.0.0.0. Defense: bind every service to localhost and enforce authentication. A 10-minute fix: run ss -tnlp, identify wildcard mappings, rebind Redis to localhost with a single-line config change, enable auth with an openssl-generated hex key.

Error-Based Endpoint Enumeration

▶ Watch (15:00)

Most applications return 403 for restricted paths and 404 for non-existent paths. Attackers fuzz common paths, separate 403 from 404, and build the full API surface in 30 seconds. Each 403 is a confirmed endpoint. This enumeration works even through the CDN because error codes come from the application. Defense: return a generic 404 for every unauthenticated request, whether the path exists or not. A quick win: set a default deny policy at the root location returning 404 for everything. Only explicitly allowed public paths remain. Validate by running the same fuzzing script; the 403/404 difference disappears.

Gray Cloud DNS Bypass

▶ Watch (18:52)

Cloudflare’s Orange Cloud proxies traffic and applies WAF, rate limiting, DDoS protection. Gray Cloud records are DNS-only and expose the origin IP directly. A leaked subdomain with a Gray Cloud record pointing to the same production application bypasses all edge security. Curling the main domain gets a 403; curling the Gray Cloud subdomain gets a 200. Attackers can flood the origin with no rate limit, causing a total outage while the security dashboard shows green. Defense: use VPC with private DNS so the origin has no public IP. If public IPs are required, restrict inbound to Cloudflare’s published IP ranges, enforce mTLS, and validate the origin-pull header. A 10-minute fix: scan for Gray Cloud records, toggle them to Orange Cloud, and delete residual staging configurations.

Q&A

Did you consider attackers bypassing filtering by spoofing Cloudflare source IPs at layer 3? Enforce mutual TLS between Cloudflare and the load balancer so that even with a spoofed IP, the connection requires a client certificate validated against a common CA. ▶ 31:01

Notable Quotes

Your edge is just a theatre just like this.

Your edge is just a theater just like this. If your application allows traffic from everywhere, you can spend half a million dollars on enterprisegrade security. Arjun Sharma · ▶ 1:38

The gold standard is VPC with private DNS. Your origin has no IP at all. Arjun Sharma · ▶ 21:50

Your 100k investment got wasted right there. Arjun Sharma · ▶ 8:09

Each 403 is a confirmed endpoint. Arjun Sharma · ▶ 15:14

One graycloud record undoes everything else. Arjun Sharma · ▶ 24:24

Key Takeaways

  • Audit certificate transparency logs to detect leaked subdomains pointing directly to your load balancer.
  • Bind every backend service to localhost and enforce authentication even for internal services.
  • Normalize error codes to generic 404 to prevent endpoint enumeration.
  • Toggle all DNS records to proxied (Orange Cloud) and delete residual Gray Cloud records.
  • Combine network isolation, mTLS, and header validation to protect origin IP from bypass.

About the Speaker(s)

Arjun is Team Lead for IBM Cloud’s Global Load Balancer Service, managing infrastructure that processes billions of monthly requests for Fortune 500 customers across global regions. With over a decade of experience in cloud infrastructure and security, Arjun leads engineering teams in architectural reviews, threat modeling, and preventing infrastructure bypasses.