TECH TALK Dealing With Data Caps and Proxy Caches
THIS PAST MONTH, I did something I’ve never managed before: I went over the 1TB data allowance that comes with my Internet service. There was no fee—this time—but my data use has been on an upward track for decades, and it’s only a matter of time before 1
My Internet speeds increased from 20Mb/s to 300Mb/s over the past 15 years. In that same period, PC games moved from CDs and DVDs to digital downloads, and 2–6GB installs to 50GB and more. Video drivers can update several times a month, at 400MB a pop. Cut the cable TV and move to streaming video, and that can be hundreds of GB every month, with 4K streams using even more. Don’t forget to back up all your data in the cloud, which in my case happens over a paltry 12Mb/s upstream connection.
I’ve never liked the idea of data caps, and looking at smartphones only reinforces how bad it can get. 4G/LTE can download at speeds of several hundred Mb/s, but most “unlimited” plans throttle your speed after 22GB per month of data. You could exceed that in just over 30 minutes. It’s not that you’d normally chew through that much data on a smartphone, but the telecoms love this sort of pricing model, and would be happy to apply it to high-speed Internet connections. And in the post-net neutrality world, paying more for my Internet seems inevitable.
With multiple PCs for testing purposes, each grabbing the same game patches and Windows 10 updates, it can add up to hundreds of GB per month. Not content to sit back and watch the data usage continue to climb, I took matters into my own hands, and set up a reverse proxy cache. I put together a PC with a 1TB SSD cache, and looked for solutions.
There are a few problems, like the fact that most web traffic is now HTTPS (SSL encrypted), so a proxy can’t do anything to cache frequently accessed web pages. Even for non-encrypted files, content delivery networks (CDNs) mean you could get the same file from multiple sources. Thankfully, smart people have done a lot a lot to address such problems, and Steamcache, Steam-Squid, Lancache, and other open-source projects reduce the amount of work.
I ended up running Debian Linux with Lancache installed. It almost worked. Or rather, it almost worked without a bunch of additional futzing about, and now several days later it mostly works. My main problem is my ISP, which runs its own Steam proxy cache. That wouldn’t be so bad, except it uses SSL encryption (unlike Steam’s normal CDNs), so my own proxy would get directed to the ISP proxy, open an SSL connection, and not cache any of the resulting files. The solution was to block my system from seeing the ISP proxies, but that’s not a perfect solution—Steam downloads happen at about 50–100Mb/s, for example. At least Origin, UPlay, Blizzard, and several other gaming platforms work fine, along with Microsoft’s Windows Update servers.
Still, I find it grating that my ISP wants to limit its external data use by running a Steam proxy cache, yet in doing so it interferes with my own attempts to do the same thing. Even more irritating is the data cap in the first place. Yes, I’m in the top 1 percent of data users, I’m sure of that. Maybe I should be charged extra… except I’m already paying $30 per month for a faster Internet connection. I can choose from Internet speeds of 15Mb/s to 400Mb/s, but all of those include the same 1TB data cap. You could theoretically download 1TB in about a week with a 15Mb/s connection, while the 60Mb/s connection would only take a day and a half. My 300Mb/s connection? I could blow through my entire data allowance in one working day of eight hours.
There’s no truly unlimited 1Gb/s connection in my area. There’s only one other ISP as well, except it’s a DSL provider with speeds that top out at 40Mb/s. Ending net neutrality is not going to help people like me who are stuck with a single provider.
I could blow through my entire data allowance in one working day of eight hours.
Are routine data cap overcharges the future of our post-net neutrality Internet?