me/adriano.fyi
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions 1

Reorg traffic shaping post

Browse source
This commit is contained in:
Adriano Caloiaro 2023-07-27 12:17:48 -06:00
parent c44669ae91
commit 1065441e1c
No known key found for this signature in database
GPG key ID: C2BC56DE73CE3F75
1 changed files with 22 additions and 18 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

40
content/posts/2023/2023-04-16-att-traffic-shaping-makes-websites-unusable.md
View file

@ -8,15 +8,19 @@ tags:
- at&t
---
## Post-incident Sumary
[Hacker News Discussion](https://news.ycombinator.com/item?id=35592607)
To summarize this post, I was seemingly throttled by AT&T for days. While gathering data for this post and attempting solutions, I changed my LTE router's APN, which caused it to re-authenticate to AT&T, and after I was re-authenticated I was no longer being throttled. One Hacker News user [suggested](https://news.ycombinator.com/item?id=35596351) that re-authenticating may have caused me to route through a different PGW (Packet Data Network Gateway) with a different traffic shaping policy.
To summarize this post, I was seemingly throttled by AT&T for days. See the [Original Post](#original-post) for the full write up.
While gathering data for this post and attempting solutions, I changed my LTE router's APN, which caused it to re-authenticate to AT&T, and after I was re-authenticated I was no longer being throttled. One Hacker News user [suggested](https://news.ycombinator.com/item?id=35596351) that re-authenticating may have caused me to route through a different PGW (Packet Data Network Gateway) with a different traffic shaping policy.
I won't know for certain unless AT&T responds directly to my request for more information.
The bottom of this post contains multiple updates from throughout the incident and beyond.
# Original Post
## Original Post
When I'm living in my RV, wireless service providers are my primary source of connectivity. So when either AT&T or Verizon make major changes, I take notice.
I recently noticed that multiple websites are quite slow when browsing with my AT&T business plan, listed in AT&T Premier (business account management UI) as "Wireless Broadband Ultra for Router or Hotspt (sic)". This is an "unlimited" **100Mbit** plan with 50GB for Business Fast Track (prioritized) data. Being that I was far below the 50GB of monthly Fast Track data, my data should have had top priority, so I became suspicious. To be honest, I've never noticed a discernible difference between Fast Track and non-Fast Track data rates. This is all to say that I have no reason to believe that I'm being deprioritized due to usage.
@ -53,17 +57,17 @@ As of the time of writing, I'm unsure what is causing such a significant slowdow
Do you have any ideas how to diagnose this problem further? Do you know the best way engage AT&T's technical folks to take this seriously? Write me at `att-traffic-shaping @ this domain`. I'll add updates here if anything changes, or I get a response on https://bizcommunity.att.com.
# Updates
## Updates
---
## Update #1 (19:03:09 UTC)
### Update #1 (19:03:09 UTC)
Here is a pcap file captured with `sudo tcpdump host 'dgpcy4fyk1eox.cloudfront.net' -w capture.pcap`: [capture.pcap](/files/capture.pcap)
I've also attempted to disable/enable IPv6 on my local machine and my VPN connection to check for any differences. No differences were observed.
## Update #2 (19:43:00 UTC)
### Update #2 (19:43:00 UTC)
Traceroutes
@ -122,17 +126,17 @@ traceroute to web-assets.strava.com (99.84.208.115), 30 hops max, 60 byte packet
20 server-99-84-208-115.iad79.r.cloudfront.net (99.84.208.115) 152.577 ms 140.190 ms 141.460 ms
```
## Update #3 (21:54:03 UTC)
### Update #3 (21:54:03 UTC)
Here is a pcap file captured with `sudo tcpdump host 'dgpcy4fyk1eox.cloudfront.net' -w iphone-capture.pcap` while tethered to my AT&T iPhone and fetching `827.js` with `wget -O /dev/null -q --show-progress https://web-assets.strava.com/assets/federated/find-and-invite-friends/827.js` : [iphone-capture.pcap](/files/iphone-capture.pcap)
## Update #4 (23:39:00 UTC)
### Update #4 (23:39:00 UTC)
As my initial post mentioned, I had hoped to get some sort of resolution by posting to http://bizcommunity.att.com. As of right now, that post [https://bizcommunity.att.com/conversations/the-break-room/im-apparently-experiencing-extremely-aggressive-att-traffic-shaping/643c3d81aba4c269accb1a2b](https://bizcommunity.att.com/conversations/the-break-room/im-apparently-experiencing-extremely-aggressive-att-traffic-shaping/643c3d81aba4c269accb1a2b) has been marked as "Private". This URL also 404s, unless I'm logged in under the account with which I posted it.
I can't say whether this is normal practice on the bizcommunity forum, but I posted a followup post with a much more benign title, "Are all new posts here marked as private?" [https://bizcommunity.att.com/conversations/the-break-room/are-all-new-posts-here-marked-as-private/643c866bb67b2d067eb0985c](https://bizcommunity.att.com/conversations/the-break-room/are-all-new-posts-here-marked-as-private/643c866bb67b2d067eb0985c), which as of right now is not private and still accessible to the public.
## Update #5 (23:54:00 UTC)
### Update #5 (23:54:00 UTC)
Hello reader, I'm another HN member trying to help get to the bottom of this! I've analyzed all the data provided and feel confident there is some kind of manipulation / throttling happening with Adriano's AT&T business plan, but if anyone wants to correct my interpretation or provide other insights feel free to reach out to me on my [blog](https://rasbora.dev/)!
@ -165,7 +169,7 @@ We can draw a few conclusions from these charts:
In my opinion these patterns rule out any extraneous factors affecting the network and are indicative of intentional manipulation of the TCP stream. If AT&T were to occasionaly drop an incoming packet than the client would never send an acknowledgement to cloudfront and after a little while cloudfront would automatically retransmit that packet, on our client side packet capture we would observe this as a packet coming in with a big delay, if we were being rate limited we would expect to see these delayed packets coming in regularly around the time when we were starting to hit a threshold, it seems likely AT&T is dropping certain inbound packets once the incoming rate reaches around ~350Kbps.
## Update #6 (00:43:00 UTC)
### Update #6 (00:43:00 UTC)
Refer to Update #4 above: I'm getting shut down on the AT&T bizcommunity site. My posts are being marked private, so I'll save them here for posterity.
@ -208,7 +212,7 @@ Notice the `Private` label on my response. So it looks to others like I impolite
![response and followup to forum post 2](/img/att_bizcommunity_marked_private.png "getting shut down in bizcommunity")
## Update #6 (2023-04-17 04:12:00 UTC)
### Update #6 (2023-04-17 04:12:00 UTC)
This ended with the least satisfying end one could expect. Over on Hacker News `kevin_nisbet` [recommended changing the APN](https://news.ycombinator.com/item?id=35595891) on the device to see if there were any routing differences. I changed the APN to `NXTGENPHONE`, which I figured would work as a "generic LTE device", per https://www.att.com/support/article/wireless/KM1062162/. However, the router was not able to authenticate with the `NXTGENPHONE` APN, so I switched back to `broadband`.
@ -218,17 +222,17 @@ I'll keep monitoring for future throttling and add any responses from the `bizco
For now, we're done here.
## Update #7 (2023-04-17 13:43:00)
### Update #7 (2023-04-17 13:43:00)
My original `Forum Post 1` on the `bizcommunity` is still marked as private and not deleted. In a [followup to Forum Post 2](https://bizcommunity.att.com/conversations/the-break-room/are-all-new-posts-here-marked-as-private/643c866bb67b2d067eb0985c?commentId=643d4ccb61a580258da175f9), I asked that my original post be made public so I can have a discussion with other business community members to see if they're having similar problems.
My `Response and followup to Forum Post 2` has been deleted by a forum moderator. I presume if they provide any explanation, it will be that I linked to an external site. In any case, despite claiming to have direct messaged me, I never received a direct message from anyone, and they didn't seem to care that I didn't receive it.
## Update #8 (2023-04-17 14:30:00)
### Update #8 (2023-04-17 14:30:00)
I figured it would be helpful to post some "post-incident" (I doubt it's actually over) data for completeness.
### Traceroute
#### Traceroute
```
traceroute web-assets.strava.com 130 ↵
@ -258,7 +262,7 @@ traceroute to web-assets.strava.com (13.225.103.44), 30 hops max, 60 byte packet
We can see it's taking a distinctly different route from `Connected to router on AT&T business plan` from `Update #2`. Current resolver is `1.1.1.1`. I imagine some of you will spot that this was routed to Hong Kong...from Durango, CO. The internet is a mysterious place.
### Current DNS Resolution
#### Current DNS Resolution
```
host web-assets.strava.com
@ -279,7 +283,7 @@ dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2363:3800:17:4613:2840:9
Note this is a completely different set of hosts from "during the incident". I'm not sure that this new host list is in any way related to this being "post-incident". Cloudflare runs a lot of POPs, and I'm not about to try to understand the interplay between my resolver `1.1.1.1` and Cloudflare's global DNS. It's notable, in any case.
### During incident DNS resolution
#### During incident DNS resolution
For reference, these are the host from "during the incident". I can't say with 100% certainty, but I believe I was using `1.1.1.1` as a resolver at the time, and whether I was connected via iphone tether or LTE router, I received the same host list:
@ -300,7 +304,7 @@ dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:f000:17:4613:2840:9
dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:400:17:4613:2840:93a1
```
### Pcap data & analysis
#### Pcap data & analysis
Here is a pcap file captured with `sudo tcpdump host 'dgpcy4fyk1eox.cloudfront.net' -w post-incident-capture.pcap` while my router is connected to my AT&T business plan and fetching `827.js` with `wget -O /dev/null -q --show-progress https://web-assets.strava.com/assets/federated/find-and-invite-friends/827.js` : [post-incident-capture.pcap](/files/post-incident-capture.pcap)
@ -310,13 +314,13 @@ Throughput and RTT graphs from Wireshark of the above pcap file
![post incident RTT](/img/post_incident_rtt.png "post incident RTT")
## Update #9 (2023-04-17 15:39:00)
### Update #9 (2023-04-17 15:39:00)
For posterity, since `bizcommunity` has deleted at least one post. I want to record my [current comment awaiting response](https://bizcommunity.att.com/conversations/the-break-room/are-all-new-posts-here-marked-as-private/643c866bb67b2d067eb0985c).
![bizcommunity followup](/img/bizcommunity_post_2_followup.png "bizcommunity followup")
## Update #10 (2023-04-20 17:17:00)
### Update #10 (2023-04-20 17:17:00)
My connection continues not to be throttled, but I still want an answer from AT&T. On that front, they've been dragging their feet.