Merge pull request #1 from acaloiaro/sergiy-analysis

TCP stream analysis

content/post/2023/2023-04-16-att-traffic-shaping-makes-websites-unusable.md

@@ -4,15 +4,15 @@ image: "/img/att_traffic_shaping.png"
 published: true
 title: AT&T Wireless traffic shaping apparently making some websites unusable
 tags:
-- wireless
-- at&t
+    - wireless
+    - at&t
 ---
 
 When I'm living in my RV, wireless service providers are my primary source of connectivity. So when either AT&T or Verizon make major changes, I take notice.
 
 I recently noticed that multiple websites are quite slow when browsing with my AT&T business plan, listed in AT&T Premier (business account management UI) as "Wireless Broadband Ultra for Router or Hotspt (sic)". This is an "unlimited" **100Mbit** plan with 50GB for Business Fast Track (prioritized) data. Being that I was far below the 50GB of monthly Fast Track data, my data should have had top priority, so I became suspicious. To be honest, I've never noticed a discernible difference between Fast Track and non-Fast Track data rates. This is all to say that I have no reason to believe that I'm being deprioritized due to usage.
 
-Naturally, the first hing I did was conduct a speed test. I already knew from previous experience that for some reason, AT&T traffic to [fast.com](https://fast.com)  is throttled. Why AT&T wants bandwidth to appear lower than reality is a mystery to me, but I digress. Linode.com has speed tests that AT&T has no special treatment for, and the [nearest one to me](http://speedtest.fremont.linode.com/) was in Fremont, CA.
+Naturally, the first hing I did was conduct a speed test. I already knew from previous experience that for some reason, AT&T traffic to [fast.com](https://fast.com) is throttled. Why AT&T wants bandwidth to appear lower than reality is a mystery to me, but I digress. Linode.com has speed tests that AT&T has no special treatment for, and the [nearest one to me](http://speedtest.fremont.linode.com/) was in Fremont, CA.
 
 The speedtest revealed **21Mbps** down and **4.5Mbps** up -- pretty reasonable in a relatively rural area like Durango, CO. Latency was ~**130ms**. That speed certainly wouldn't explain why it took anywhere between 15 seconds and 2 minutes to load `strava.com`.
 
@@ -45,9 +45,11 @@ As of the time of writing, I'm unsure what is causing such a significant slowdow
 Do you have any ideas how to diagnose this problem further? Do you know the best way engage AT&T's technical folks to take this seriously? Write me at `att-traffic-shaping @ this domain`. I'll add updates here if anything changes, or I get a response on https://bizcommunity.att.com.
 
 # Updates
+
 ---
 
 ## Update #1 (19:03:09 UTC)
+
 Here is a pcap file captured with `sudo tcpdump host 'dgpcy4fyk1eox.cloudfront.net' -w capture.pcap`: [capture.pcap](/files/capture.pcap)
 
 I've also attempted to disable/enable IPv6 on my local machine and my VPN connection to check for any differences. No differences were observed.
@@ -57,6 +59,7 @@ I've also attempted to disable/enable IPv6 on my local machine and my VPN connec
 Traceroutes
 
 Connected to AT&T iPhone via Wifi hotspot
+
 ```
 traceroute web-assets.strava.com                                                                                                                                                    130 ↵
 traceroute to web-assets.strava.com (99.84.208.10), 30 hops max, 60 byte packets
@@ -84,6 +87,7 @@ traceroute to web-assets.strava.com (99.84.208.10), 30 hops max, 60 byte packets
 ```
 
 Connected to router on AT&T business plan
+
 ```
 traceroute web-assets.strava.com
 traceroute to web-assets.strava.com (99.84.208.115), 30 hops max, 60 byte packets
@@ -113,8 +117,39 @@ traceroute to web-assets.strava.com (99.84.208.115), 30 hops max, 60 byte packet
 
 Here is a pcap file captured with `sudo tcpdump host 'dgpcy4fyk1eox.cloudfront.net' -w iphone-capture.pcap` while tethered to my AT&T iPhone and fetching `827.js` with `wget -O /dev/null -q --show-progress https://web-assets.strava.com/assets/federated/find-and-invite-friends/827.js` : [iphone-capture.pcap](/files/iphone-capture.pcap)
 
-# Update #4 (23:39:00 UTC)
+## Update #4 (23:39:00 UTC)
 
 As my initial post mentioned, I had hoped to get some sort of resolution by posting to http://bizcommunity.att.com. As of right now, that post (https://bizcommunity.att.com/conversations/the-break-room/im-apparently-experiencing-extremely-aggressive-att-traffic-shaping/643c3d81aba4c269accb1a2b) has been marked as "Private". This URL also 404s, unless I'm logged in under the account with which I posted it.
 
 I can't say whether this is normal practice on the bizcommunity forum, but I posted a followup post with a much more benign title, "Are all new posts here marked as private?" (https://bizcommunity.att.com/conversations/the-break-room/are-all-new-posts-here-marked-as-private/643c866bb67b2d067eb0985c), which as of right now is not private and still accessible to the public.
+
+## Update #5
+
+Hello reader, I'm another HN member trying to help get to the bottom of this! I've analyzed all the data provided and feel confident there is some kind of manipulation / throttling happening with Adriano's AT&T business plan, but if anyone wants to correct my interpretation or provide other insights feel free to reach out to me on my [blog](https://rasbora.dev/)!
+
+We start by analyzing the packet capture from the throttled connection. There are a few takeaways from this:
+
+-   TCP segment & window sizes have normal values.
+-   There are almost no retransmissions or lost packets.
+-   There appears to be ample room for the TCP stream to scale up to a much faster speed.
+-   The TCP stream starts scaling up to 800Kbps before dramatically going down.
+-   The TCP stream regularly blocks for 50-200ms intervals waiting for new data to arrive from cloudfront.
+-   These blocks coincide with throughput scaling up to ~350Kbps.
+
+![ATT Throughput](/img/att_throughput.png "ATT Throughput")
+![ATT RTT](/img/att_rtt.png "ATT RTT")
+
+All of the spikes in round trip time correspond to a segment of the stream where the client acknowladges receipt of new data and is forced to wait 50-200ms for any new data to arrive. This happens at a regular cadence that appears artificial in nature (i.e not regular network congestion) because the throughput chart stays within a well defined boundary.
+
+If this were a problem with AT&T's network or a side effect of LTE networks being inherently high latency we would expect to observe similair patterns over the iphone connection:
+
+![iphone Throughput](/img/iphone_throughput.png "iphone Throughput")
+![iphone RTT](/img/iphone_rtt.png "iphone RTT")
+
+We can draw a few conclusions from these charts:
+
+-   The iphone AT&T connection doesn't exhibit the same RTT or throughput pattern despite having similair latency & network routing.
+-   The throughput chart is indicative of a healthy TCP stream that slowly scales up the total bandwidth to make the most out of available network resources.
+-   We are never waiting very long for fresh data to arrive.
+
+In my opinion these patterns rule out any extraneous factors affecting the network and are indicative of intentional manipulation of the TCP stream. If AT&T were to occasionaly drop an incoming packet than the client would never send an acknowledgement to cloudfront and after a little while cloudfront would automatically retransmit that packet, on our client side packet capture we would observe this as a packet coming in with a big delay, if we were being rate limited we would expect to see these delayed packets coming in regularly around the time when we were starting to hit a threshold, it seems likely AT&T is dropping certain inbound packets once the incoming rate reaches around ~350Kbps.