Setting up an off grid server, part 5. Apt-mirror.

One of the great things with Linux systems is their great package managers. Bring up a friendly graphical application, and you’re presented with a list of every application you could ever want to install. Check off the new ones, click on “Go” and the app is downloaded and installed automatically. At my home, however, I don’t have any internet access. Yes, there is digital life away from Facebook, but this makes it difficult to install anything from the ‘net. To get around this, I use an application called apt-mirror. I’ll often set up a mirror on my laptop, go to work, and update it. When I get home, I’ll move the mirror to the server, and use it to keep all my machines updated.

To setup a mirror, first, we must install it from the package manager, with “sudo apt-get install apt-mirror”. Second, we’ll setup the config file for all of our sources. The file is at /etc/apt/mirror.list. Here is mine from the Panda.

############# config ##################
#
set base_path    /var/samba/root/apt-mirror
#
set mirror_path  $base_path/mirror
set skel_path    $base_path/skel 
set var_path     $base_path/var
set cleanscript $var_path/clean.sh
# set defaultarch  
set postmirror_script $var_path/postmirror.sh
# set run_postmirror 0
set nthreads     20
set _tilde 0
############## end config ##############
#ubuntu official
deb-i386 http://10.0.23.99/us.archive.ubuntu.com/ubuntu/ lucid main restricted
deb-i386 http://10.0.23.99/us.archive.ubuntu.com/ubuntu/ lucid-updates main restricted
deb-i386 http://10.0.23.99/us.archive.ubuntu.com/ubuntu/ lucid universe
deb-i386 http://10.0.23.99/us.archive.ubuntu.com/ubuntu/ lucid-updates universe
deb-i386 http://10.0.23.99/us.archive.ubuntu.com/ubuntu/ lucid multiverse
deb-i386 http://10.0.23.99/us.archive.ubuntu.com/ubuntu/ lucid-updates multiverse
deb-i386 http://10.0.23.99/us.archive.ubuntu.com/ubuntu/ lucid-backports main restricted universe multiverse
deb-i386 http://10.0.23.99/archive.canonical.com/ubuntu lucid partner
deb-i386 http://10.0.23.99/security.ubuntu.com/ubuntu lucid-security main restricted
deb-i386 http://10.0.23.99/security.ubuntu.com/ubuntu lucid-security universe
deb-i386 http://10.0.23.99/security.ubuntu.com/ubuntu lucid-security multiverse

# google chrome and others
deb-i386 http://10.0.23.99/dl.google.com/linux/deb/ stable main

#mediabuntu
deb-i386 http://10.0.23.99/packages.medibuntu.org/ lucid free non-free 

#arduino packages
deb-i386 http://10.0.23.99/ppa.launchpad.net/arduino-ubuntu-team/ppa/ubuntu lucid main 

#audio drivers update
deb-i386 http://10.0.23.99/ppa.launchpad.net/ubuntu-audio-dev/ppa/ubuntu lucid main

#Virtual Box Free-Closed Source
deb-i386 http://10.0.23.99/download.virtualbox.org/virtualbox/debian lucid non-free

#xbmc
deb-i386 http://10.0.23.99/ppa.launchpad.net/team-xbmc/ppa/ubuntu lucid main

#rabbitsvn
deb-i386 http://10.0.23.99/ppa.launchpad.net/rabbitvcs/ppa/ubuntu lucid main

#playdeb
deb-i386 http://10.0.23.99/archive.getdeb.net/ubuntu lucid-getdeb games

#handbrake
deb-i386 http://10.0.23.99/ppa.launchpad.net/stebbins/handbrake-snapshots/ubuntu lucid main 

#official arm for panda
deb-armel http://10.0.23.99/ports.ubuntu.com/ubuntu-ports maverick main restricted universe multiverse
deb-armel http://10.0.23.99/ports.ubuntu.com/ubuntu-ports maverick-security main restricted universe multiverse
deb-armel http://10.0.23.99/ports.ubuntu.com/ubuntu-ports maverick-updates main restricted universe multiverse

#panda specific arm repository 
deb-armel http://10.0.23.99/ppa.launchpad.net/tiomap-dev/release/ubuntu maverick main 
deb-armel http://10.0.23.99/ppa.launchpad.net/tiomap-dev/omap-trunk/ubuntu maverick main 

#cleaners
clean http://10.0.23.99/us.archive.ubuntu.com/ubuntu/
clean http://10.0.23.99/archive.canonical.com/ubuntu
clean http://10.0.23.99/security.ubuntu.com/ubuntu
clean http://10.0.23.99/dl.google.com/linux/deb/
clean http://10.0.23.99/packages.medibuntu.org/
clean http://10.0.23.99/ppa.launchpad.net/arduino-ubuntu-team/ppa/ubuntu 
clean http://10.0.23.99/ppa.launchpad.net/aegirxx-googlemail/dftd-latest/ubuntu
clean http://10.0.23.99/ppa.launchpad.net/ubuntu-audio-dev/ppa/ubuntu 
clean http://10.0.23.99/download.virtualbox.org/virtualbox/debian 
clean http://10.0.23.99/ppa.launchpad.net/team-xbmc/ppa/ubuntu
clean http://10.0.23.99/ppa.launchpad.net/rabbitvcs/ppa/ubuntu 
clean http://10.0.23.99/archive.getdeb.net/ubuntu /va
clean http://10.0.23.99/linux.toribash.com/apt/
clean http://10.0.23.99/ppa.launchpad.net/stebbins/handbrake-snapshots/ubuntu 
clean http://10.0.23.99/ppa.launchpad.net/phobie/ubuntu 

#clean arm
clean http://10.0.23.99/ports.ubuntu.com/ubuntu-ports
clean http://10.0.23.99/ppa.launchpad.net/tiomap-dev/release/ubuntu
clean http://10.0.23.99/ppa.launchpad.net/tiomap-dev/omap-trunk/ubuntu
#end file /etc/apt/mirror.list

There are a number of things here which I want to point out. Some are specific to my system, and some are just a little odd. At first glance, this may seem to be a copy of the sources list, and for the most part it is. Only the initial settings at the front and the “clean” lines at the bottom are added. In my case, I’m actually mirroring for 2 architectures (soon to be 3), so I explicitly stated the architecture with each line, using deb-i386 or deb-armel instead of simply deb. Second, the clean section is just a copy of the deb sections, with deb changed to clean, and all but the url removed. Finally, the initial settings were changed to point to my usb drive.

To actually update the mirror, which in the first run will download EVERYTHING (82 GB in my case), run the following commands.

sudo apt-mirror
cp /var/samba/root/apt-mirror/mirror/clean.sh ~
chmod +x ~/clean.sh
sudo ~/clean/sh

Again, my case is a little unique. The first time I ran this, I had quite a bit of difficulty with the apt-mirror script attempting to write to files or directories which didn’t exist. It seems that the script could use a bit of an update to make these first runs more clean. However, in each case I created the directory or touched the file requested and it continued on a second attempt. Also, because I left my usb drive formated as FAT32 (just in case the server dies and it needs to be plugged into something else), it’s not possible to change the permissions of anything to executable, so the clean script needed to be copied locally and marked as executable. This can take two days on a good internet connection, and in my case, I pulled from my local laptop via the 802.11G connection and it still took about 10 hours.

Finally, we change our local /etc/apt/sources.list file to point to our local drive. After that, we can update ourself in the usual way. Again, here is my file. You can see I left the original lines, but just commented them out.

#deb http://ports.ubuntu.com/ubuntu-ports maverick main restricted universe multiverse
#deb http://ports.ubuntu.com/ubuntu-ports maverick-security main restricted universe multiverse
#deb http://ports.ubuntu.com/ubuntu-ports maverick-updates main restricted universe multiverse
#deb http://ppa.launchpad.net/tiomap-dev/release/ubuntu maverick main

deb file:///var/samba/root/apt-mirror/mirror/10.0.23.99/ports.ubuntu.com/ubuntu-ports maverick main restricted universe multiverse
deb file:///var/samba/root/apt-mirror/mirror/10.0.23.99/ports.ubuntu.com/ubuntu-ports maverick-security main restricted universe multiverse
deb file:///var/samba/root/apt-mirror/mirror/10.0.23.99/ports.ubuntu.com/ubuntu-ports maverick-updates main restricted universe multiverse
deb file:///var/samba/root/apt-mirror/mirror/10.0.23.99/ppa.launchpad.net/tiomap-dev/release/ubuntu maverick main
#end file /etc/apt/sources.list

That’s it for this entry. I still have to set up the printer and we’ll want to share this mirror to all on our network, but that’ll come later.

Advertisements

About Grant Wagner

General class ham: KC9SJQ https://grantwagner.wordpress.com
This entry was posted in Uncategorized. Bookmark the permalink.

3 Responses to Setting up an off grid server, part 5. Apt-mirror.

  1. Mark says:

    Hello Grant – I operate a public apt-mirror implementation in Olympia, WA at http://nwlinux.us/sources.list. Do you know of any other public, non-official mirrors in the U.S. or Canada that could be added to http://apt-mirror.com?

  2. Grant Wagner says:

    Howdy Mark!

    Hey, the author of the app himself! Thanks for such a great tool. As you may get from this, I have a pretty limited access to the net, and putting mirrored copy of the repos gives me a great deal of flexibility. It may be hard on the bandwidth at my office, but even a full copy is just a speck on a modern HD.

    As far as I know, the only non-official public mirror I can think of is yours. Even that I didn’t know about until you brought it to my attention. Thanks for the comment, and I wish you the best of luck with your projects.

  3. Mark says:

    Hello Grant – The apt-mirror application is not mine. However, I find it strange that the package developer did not snatch up the domain earlier. Either way, this package has really made things easier for me – one of my top 3 favorites!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s