Arman's stuff
Bandwidth Savings and Useful Information

(Tue Jan 18 17:42:26 2011)

I love learning new things!

Before I get into all this, I'd like to give a few basic definitions, just in case someone who doesn't know about the depths of Linux is trying to read this.

The main program I talk about today is apt - this program will go out to servers on the Internet (called repositories) and check for updates, not just to the OS, but to every single game and application you have installed. If there is an update, it will download it (and anything it depends on) and install it, all automatically. If you want to get rid of something, it can do that, too! There are lots of ways for apt to download/upgrade/install/uninstall things, and lots of tools that use apt, improve apt, extend apt, and otherwise leverage one of the best parts of Ubuntu.

And now, on to the meat of the post!

In getting stuff ready for the next upcoming LAN party, I'm installing and updating quite a few computers - mine, two desktops as a "free seat" and a game server, and my laptop, probably another free seat. It's not hard work, but it is time consuming - and time's not something I have a lot of. The part that takes the most time is moving all that data from the Ubuntu (and other) servers down to my computers; it's not bad for a basic upgrade, but when you're installing games on four computer, it's a lot of bytes. As an example, upgrading to the current debs after a fresh install was nearly an 800 megabyte download - and that was just one PC!

To help lift that burden, I've mirrored the entire Ubuntu repository on my own home server, using a program called "apt-cache". It acts as a go-between; every night at 4AM, it goes out to the repositories and sees what's new. If anything has changed, it happily downloads the changes, and updates its own repository. Then, when one of my many computers checks for an upgrade, instead of going out to the big, slow Internet, it can just grab all the files it needs from my own, internal server. A gigabit connection is rather a lot faster than a 6 Mbps connection...

The downside, of course, is that I am undoubtedly downloading programs I'll never need or even want, updating them every time a new release comes out. The repository is only around 40 GBs, all told, so it doesn't take up a terribly large amount of space; the repository that updates most frequently (security updates) is itself fairly small.

There are sites that I don't mirror, however; while the main, restricted, universe, and multiverse sections of the lucid, lucid-updates, lucid-backports, and lucid-security repositories, plus the Opera stable and webmin 'sarge' repositories, are only about 40 gigs, some repositories are much, much bigger - specifically the getdeb repository. A single game (Alien Arena) is almost 400 MB; multiply that by hundreds of games, and there are many, many gigabytes of information that I would have to mirror. And chances are, I'll never want even a fraction of all those games; I've only installed two or three total.

Thus, the next step: apt-proxy. While apt-cache is a direct, complete copy of the original repository, apt-proxy is more of a "collect and store" function. When you first request a file, apt-proxy will go straight to the original repository and download it, simultaneously sending it to you and to its own storage. If a second computer asks for the same file, apt-proxy uses the already-downloaded file.

Like apt-cache, this means that if five computers want an update, it still only gets downloaded from the original repository; however, unlike apt-cache, it waits to download any file until it is first requested. At the expense of the wait for the first download, you save quite a bit of disk space. When I install Alien Arena the first time, it'll take an hour or two to download that 380 MB file from the overburdened getdeb server. The second time, though, it'll only take seconds, because it won't need to download it again - it'll use its stored copy.

Slowly, I'm adding tools like this to the vast information database in my head, and eventually, they come in handy. As I prep for my LAN party, I've had to not only install KUbuntu from scratch (leading to the almost 800 MB download from apt-cache), I've also had to update several programs (and thus the installation of apt-proxy). If the LAN party gets as big as I'd like it to, eventually I'll have entire rows of built-up computers, all begging for installations of Alien Arena, Wolfenstein: Enemy Territory, Smokin' Guns, and WarZone 2100. Assuming we play those, at any rate.

The next step, I think, is actually starting my own repository; instead of copying the files from elsewhere, I'd actually build it myself. It would have files I've built myself from source code, files you can't get elsewhere (like pre-packaged Star Craft or Kohan Wine builds), and lots of configuration files. Beyond that, I can envision creating win-get (apt for Windows) or Fink (apt for Mac) repositories; every person that comes to the LAN can automatically update all their programs, patching everything as needed, without that nasty pass-the-USB-drive-to-everyone bit.

Like I said - I love learning new things! :-)

<< I've worked myself into a corner.Game Servers Are Hard >>

This blag is tagged: Apt, Linux, Mac, Server, Ubuntu, Windows, All