me/adriano.fyi
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Reorg traffic shaping post

Browse source
This commit is contained in:
Adriano Caloiaro 2023-07-27 12:17:48 -06:00
parent c44669ae91
commit 1065441e1c
No known key found for this signature in database
GPG key ID: C2BC56DE73CE3F75
1 changed files with 22 additions and 18 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

40
content/posts/2023/2023-04-16-att-traffic-shaping-makes-websites-unusable.md
View file

@ -8,15 +8,19 @@ tags:
- at&t - at&t
--- ---
## Post-incident Sumary
[Hacker News Discussion](https://news.ycombinator.com/item?id=35592607) [Hacker News Discussion](https://news.ycombinator.com/item?id=35592607)
To summarize this post, I was seemingly throttled by AT&T for days. While gathering data for this post and attempting solutions, I changed my LTE router's APN, which caused it to re-authenticate to AT&T, and after I was re-authenticated I was no longer being throttled. One Hacker News user [suggested](https://news.ycombinator.com/item?id=35596351) that re-authenticating may have caused me to route through a different PGW (Packet Data Network Gateway) with a different traffic shaping policy. To summarize this post, I was seemingly throttled by AT&T for days. See the [Original Post](#original-post) for the full write up.
While gathering data for this post and attempting solutions, I changed my LTE router's APN, which caused it to re-authenticate to AT&T, and after I was re-authenticated I was no longer being throttled. One Hacker News user [suggested](https://news.ycombinator.com/item?id=35596351) that re-authenticating may have caused me to route through a different PGW (Packet Data Network Gateway) with a different traffic shaping policy.
I won't know for certain unless AT&T responds directly to my request for more information. I won't know for certain unless AT&T responds directly to my request for more information.
The bottom of this post contains multiple updates from throughout the incident and beyond. The bottom of this post contains multiple updates from throughout the incident and beyond.
# Original Post ## Original Post
When I'm living in my RV, wireless service providers are my primary source of connectivity. So when either AT&T or Verizon make major changes, I take notice. When I'm living in my RV, wireless service providers are my primary source of connectivity. So when either AT&T or Verizon make major changes, I take notice.
I recently noticed that multiple websites are quite slow when browsing with my AT&T business plan, listed in AT&T Premier (business account management UI) as "Wireless Broadband Ultra for Router or Hotspt (sic)". This is an "unlimited" **100Mbit** plan with 50GB for Business Fast Track (prioritized) data. Being that I was far below the 50GB of monthly Fast Track data, my data should have had top priority, so I became suspicious. To be honest, I've never noticed a discernible difference between Fast Track and non-Fast Track data rates. This is all to say that I have no reason to believe that I'm being deprioritized due to usage. I recently noticed that multiple websites are quite slow when browsing with my AT&T business plan, listed in AT&T Premier (business account management UI) as "Wireless Broadband Ultra for Router or Hotspt (sic)". This is an "unlimited" **100Mbit** plan with 50GB for Business Fast Track (prioritized) data. Being that I was far below the 50GB of monthly Fast Track data, my data should have had top priority, so I became suspicious. To be honest, I've never noticed a discernible difference between Fast Track and non-Fast Track data rates. This is all to say that I have no reason to believe that I'm being deprioritized due to usage.
@ -53,17 +57,17 @@ As of the time of writing, I'm unsure what is causing such a significant slowdow
Do you have any ideas how to diagnose this problem further? Do you know the best way engage AT&T's technical folks to take this seriously? Write me at `att-traffic-shaping @ this domain`. I'll add updates here if anything changes, or I get a response on https://bizcommunity.att.com. Do you have any ideas how to diagnose this problem further? Do you know the best way engage AT&T's technical folks to take this seriously? Write me at `att-traffic-shaping @ this domain`. I'll add updates here if anything changes, or I get a response on https://bizcommunity.att.com.
# Updates ## Updates
--- ---
## Update #1 (19:03:09 UTC) ### Update #1 (19:03:09 UTC)
Here is a pcap file captured with `sudo tcpdump host 'dgpcy4fyk1eox.cloudfront.net' -w capture.pcap`: [capture.pcap](/files/capture.pcap) Here is a pcap file captured with `sudo tcpdump host 'dgpcy4fyk1eox.cloudfront.net' -w capture.pcap`: [capture.pcap](/files/capture.pcap)
I've also attempted to disable/enable IPv6 on my local machine and my VPN connection to check for any differences. No differences were observed. I've also attempted to disable/enable IPv6 on my local machine and my VPN connection to check for any differences. No differences were observed.
## Update #2 (19:43:00 UTC) ### Update #2 (19:43:00 UTC)
Traceroutes Traceroutes
@ -122,17 +126,17 @@ traceroute to web-assets.strava.com (99.84.208.115), 30 hops max, 60 byte packet
20 server-99-84-208-115.iad79.r.cloudfront.net (99.84.208.115) 152.577 ms 140.190 ms 141.460 ms 20 server-99-84-208-115.iad79.r.cloudfront.net (99.84.208.115) 152.577 ms 140.190 ms 141.460 ms
``` ```
## Update #3 (21:54:03 UTC) ### Update #3 (21:54:03 UTC)
Here is a pcap file captured with `sudo tcpdump host 'dgpcy4fyk1eox.cloudfront.net' -w iphone-capture.pcap` while tethered to my AT&T iPhone and fetching `827.js` with `wget -O /dev/null -q --show-progress https://web-assets.strava.com/assets/federated/find-and-invite-friends/827.js` : [iphone-capture.pcap](/files/iphone-capture.pcap) Here is a pcap file captured with `sudo tcpdump host 'dgpcy4fyk1eox.cloudfront.net' -w iphone-capture.pcap` while tethered to my AT&T iPhone and fetching `827.js` with `wget -O /dev/null -q --show-progress https://web-assets.strava.com/assets/federated/find-and-invite-friends/827.js` : [iphone-capture.pcap](/files/iphone-capture.pcap)
## Update #4 (23:39:00 UTC) ### Update #4 (23:39:00 UTC)
As my initial post mentioned, I had hoped to get some sort of resolution by posting to http://bizcommunity.att.com. As of right now, that post [https://bizcommunity.att.com/conversations/the-break-room/im-apparently-experiencing-extremely-aggressive-att-traffic-shaping/643c3d81aba4c269accb1a2b](https://bizcommunity.att.com/conversations/the-break-room/im-apparently-experiencing-extremely-aggressive-att-traffic-shaping/643c3d81aba4c269accb1a2b) has been marked as "Private". This URL also 404s, unless I'm logged in under the account with which I posted it. As my initial post mentioned, I had hoped to get some sort of resolution by posting to http://bizcommunity.att.com. As of right now, that post [https://bizcommunity.att.com/conversations/the-break-room/im-apparently-experiencing-extremely-aggressive-att-traffic-shaping/643c3d81aba4c269accb1a2b](https://bizcommunity.att.com/conversations/the-break-room/im-apparently-experiencing-extremely-aggressive-att-traffic-shaping/643c3d81aba4c269accb1a2b) has been marked as "Private". This URL also 404s, unless I'm logged in under the account with which I posted it.
I can't say whether this is normal practice on the bizcommunity forum, but I posted a followup post with a much more benign title, "Are all new posts here marked as private?" [https://bizcommunity.att.com/conversations/the-break-room/are-all-new-posts-here-marked-as-private/643c866bb67b2d067eb0985c](https://bizcommunity.att.com/conversations/the-break-room/are-all-new-posts-here-marked-as-private/643c866bb67b2d067eb0985c), which as of right now is not private and still accessible to the public. I can't say whether this is normal practice on the bizcommunity forum, but I posted a followup post with a much more benign title, "Are all new posts here marked as private?" [https://bizcommunity.att.com/conversations/the-break-room/are-all-new-posts-here-marked-as-private/643c866bb67b2d067eb0985c](https://bizcommunity.att.com/conversations/the-break-room/are-all-new-posts-here-marked-as-private/643c866bb67b2d067eb0985c), which as of right now is not private and still accessible to the public.
## Update #5 (23:54:00 UTC) ### Update #5 (23:54:00 UTC)
Hello reader, I'm another HN member trying to help get to the bottom of this! I've analyzed all the data provided and feel confident there is some kind of manipulation / throttling happening with Adriano's AT&T business plan, but if anyone wants to correct my interpretation or provide other insights feel free to reach out to me on my [blog](https://rasbora.dev/)! Hello reader, I'm another HN member trying to help get to the bottom of this! I've analyzed all the data provided and feel confident there is some kind of manipulation / throttling happening with Adriano's AT&T business plan, but if anyone wants to correct my interpretation or provide other insights feel free to reach out to me on my [blog](https://rasbora.dev/)!
@ -165,7 +169,7 @@ We can draw a few conclusions from these charts:
In my opinion these patterns rule out any extraneous factors affecting the network and are indicative of intentional manipulation of the TCP stream. If AT&T were to occasionaly drop an incoming packet than the client would never send an acknowledgement to cloudfront and after a little while cloudfront would automatically retransmit that packet, on our client side packet capture we would observe this as a packet coming in with a big delay, if we were being rate limited we would expect to see these delayed packets coming in regularly around the time when we were starting to hit a threshold, it seems likely AT&T is dropping certain inbound packets once the incoming rate reaches around ~350Kbps. In my opinion these patterns rule out any extraneous factors affecting the network and are indicative of intentional manipulation of the TCP stream. If AT&T were to occasionaly drop an incoming packet than the client would never send an acknowledgement to cloudfront and after a little while cloudfront would automatically retransmit that packet, on our client side packet capture we would observe this as a packet coming in with a big delay, if we were being rate limited we would expect to see these delayed packets coming in regularly around the time when we were starting to hit a threshold, it seems likely AT&T is dropping certain inbound packets once the incoming rate reaches around ~350Kbps.
## Update #6 (00:43:00 UTC) ### Update #6 (00:43:00 UTC)
Refer to Update #4 above: I'm getting shut down on the AT&T bizcommunity site. My posts are being marked private, so I'll save them here for posterity. Refer to Update #4 above: I'm getting shut down on the AT&T bizcommunity site. My posts are being marked private, so I'll save them here for posterity.
@ -208,7 +212,7 @@ Notice the `Private` label on my response. So it looks to others like I impolite
![response and followup to forum post 2](/img/att_bizcommunity_marked_private.png "getting shut down in bizcommunity") ![response and followup to forum post 2](/img/att_bizcommunity_marked_private.png "getting shut down in bizcommunity")
## Update #6 (2023-04-17 04:12:00 UTC) ### Update #6 (2023-04-17 04:12:00 UTC)
This ended with the least satisfying end one could expect. Over on Hacker News `kevin_nisbet` [recommended changing the APN](https://news.ycombinator.com/item?id=35595891) on the device to see if there were any routing differences. I changed the APN to `NXTGENPHONE`, which I figured would work as a "generic LTE device", per https://www.att.com/support/article/wireless/KM1062162/. However, the router was not able to authenticate with the `NXTGENPHONE` APN, so I switched back to `broadband`. This ended with the least satisfying end one could expect. Over on Hacker News `kevin_nisbet` [recommended changing the APN](https://news.ycombinator.com/item?id=35595891) on the device to see if there were any routing differences. I changed the APN to `NXTGENPHONE`, which I figured would work as a "generic LTE device", per https://www.att.com/support/article/wireless/KM1062162/. However, the router was not able to authenticate with the `NXTGENPHONE` APN, so I switched back to `broadband`.
@ -218,17 +222,17 @@ I'll keep monitoring for future throttling and add any responses from the `bizco
For now, we're done here. For now, we're done here.
## Update #7 (2023-04-17 13:43:00) ### Update #7 (2023-04-17 13:43:00)
My original `Forum Post 1` on the `bizcommunity` is still marked as private and not deleted. In a [followup to Forum Post 2](https://bizcommunity.att.com/conversations/the-break-room/are-all-new-posts-here-marked-as-private/643c866bb67b2d067eb0985c?commentId=643d4ccb61a580258da175f9), I asked that my original post be made public so I can have a discussion with other business community members to see if they're having similar problems. My original `Forum Post 1` on the `bizcommunity` is still marked as private and not deleted. In a [followup to Forum Post 2](https://bizcommunity.att.com/conversations/the-break-room/are-all-new-posts-here-marked-as-private/643c866bb67b2d067eb0985c?commentId=643d4ccb61a580258da175f9), I asked that my original post be made public so I can have a discussion with other business community members to see if they're having similar problems.
My `Response and followup to Forum Post 2` has been deleted by a forum moderator. I presume if they provide any explanation, it will be that I linked to an external site. In any case, despite claiming to have direct messaged me, I never received a direct message from anyone, and they didn't seem to care that I didn't receive it. My `Response and followup to Forum Post 2` has been deleted by a forum moderator. I presume if they provide any explanation, it will be that I linked to an external site. In any case, despite claiming to have direct messaged me, I never received a direct message from anyone, and they didn't seem to care that I didn't receive it.
## Update #8 (2023-04-17 14:30:00) ### Update #8 (2023-04-17 14:30:00)
I figured it would be helpful to post some "post-incident" (I doubt it's actually over) data for completeness. I figured it would be helpful to post some "post-incident" (I doubt it's actually over) data for completeness.
### Traceroute #### Traceroute
``` ```
traceroute web-assets.strava.com 130 ↵ traceroute web-assets.strava.com 130 ↵
@ -258,7 +262,7 @@ traceroute to web-assets.strava.com (13.225.103.44), 30 hops max, 60 byte packet
We can see it's taking a distinctly different route from `Connected to router on AT&T business plan` from `Update #2`. Current resolver is `1.1.1.1`. I imagine some of you will spot that this was routed to Hong Kong...from Durango, CO. The internet is a mysterious place. We can see it's taking a distinctly different route from `Connected to router on AT&T business plan` from `Update #2`. Current resolver is `1.1.1.1`. I imagine some of you will spot that this was routed to Hong Kong...from Durango, CO. The internet is a mysterious place.
### Current DNS Resolution #### Current DNS Resolution
``` ```
host web-assets.strava.com host web-assets.strava.com
@ -279,7 +283,7 @@ dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2363:3800:17:4613:2840:9
Note this is a completely different set of hosts from "during the incident". I'm not sure that this new host list is in any way related to this being "post-incident". Cloudflare runs a lot of POPs, and I'm not about to try to understand the interplay between my resolver `1.1.1.1` and Cloudflare's global DNS. It's notable, in any case. Note this is a completely different set of hosts from "during the incident". I'm not sure that this new host list is in any way related to this being "post-incident". Cloudflare runs a lot of POPs, and I'm not about to try to understand the interplay between my resolver `1.1.1.1` and Cloudflare's global DNS. It's notable, in any case.
### During incident DNS resolution #### During incident DNS resolution
For reference, these are the host from "during the incident". I can't say with 100% certainty, but I believe I was using `1.1.1.1` as a resolver at the time, and whether I was connected via iphone tether or LTE router, I received the same host list: For reference, these are the host from "during the incident". I can't say with 100% certainty, but I believe I was using `1.1.1.1` as a resolver at the time, and whether I was connected via iphone tether or LTE router, I received the same host list:
@ -300,7 +304,7 @@ dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:f000:17:4613:2840:9
dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:400:17:4613:2840:93a1 dgpcy4fyk1eox.cloudfront.net has IPv6 address 2600:9000:2199:400:17:4613:2840:93a1
``` ```
### Pcap data & analysis #### Pcap data & analysis
Here is a pcap file captured with `sudo tcpdump host 'dgpcy4fyk1eox.cloudfront.net' -w post-incident-capture.pcap` while my router is connected to my AT&T business plan and fetching `827.js` with `wget -O /dev/null -q --show-progress https://web-assets.strava.com/assets/federated/find-and-invite-friends/827.js` : [post-incident-capture.pcap](/files/post-incident-capture.pcap) Here is a pcap file captured with `sudo tcpdump host 'dgpcy4fyk1eox.cloudfront.net' -w post-incident-capture.pcap` while my router is connected to my AT&T business plan and fetching `827.js` with `wget -O /dev/null -q --show-progress https://web-assets.strava.com/assets/federated/find-and-invite-friends/827.js` : [post-incident-capture.pcap](/files/post-incident-capture.pcap)
@ -310,13 +314,13 @@ Throughput and RTT graphs from Wireshark of the above pcap file
![post incident RTT](/img/post_incident_rtt.png "post incident RTT") ![post incident RTT](/img/post_incident_rtt.png "post incident RTT")
## Update #9 (2023-04-17 15:39:00) ### Update #9 (2023-04-17 15:39:00)
For posterity, since `bizcommunity` has deleted at least one post. I want to record my [current comment awaiting response](https://bizcommunity.att.com/conversations/the-break-room/are-all-new-posts-here-marked-as-private/643c866bb67b2d067eb0985c). For posterity, since `bizcommunity` has deleted at least one post. I want to record my [current comment awaiting response](https://bizcommunity.att.com/conversations/the-break-room/are-all-new-posts-here-marked-as-private/643c866bb67b2d067eb0985c).
![bizcommunity followup](/img/bizcommunity_post_2_followup.png "bizcommunity followup") ![bizcommunity followup](/img/bizcommunity_post_2_followup.png "bizcommunity followup")
## Update #10 (2023-04-20 17:17:00) ### Update #10 (2023-04-20 17:17:00)
My connection continues not to be throttled, but I still want an answer from AT&T. On that front, they've been dragging their feet. My connection continues not to be throttled, but I still want an answer from AT&T. On that front, they've been dragging their feet.