Skip to content

Dual ISPs or How To Survive Out In The Sticks

So you are living in a remote area that has poor Internet connectivity options and you are a nerd that can’t survive off of a single slow DSL connection. What can one do?! Well with an older computer, three NICs, and a little help from our favorite Linux distro we make a buffet of low quality Internet connections seem like one semi-decent connection. What we want to get out of this is the download bandwidth of a satellite connection but the latency (ping) of a DSL connection. That way we can watch Netflix and YouTube on the satellite and play games, SSH, and do other latency sensitive things on the DSL.

What I used specifically was a DSL connection from the local phone company that is sold as 1.5 Mbit/s but typically shows speeds of around 800 Kbit/s on a good day and a satellite connection from Exede. An old dual CPU AMD Opteron with 3 gigabit NICs and an install of Gentoo or your favorite Linux distro will be our router. In this setup eth0 will be the LAN-connected NIC, eth1 will be the DSL and eth2 the satellite connection. Each of the connections were tested individually with my laptop to ensure they were functional and to get some speed tests for comparison. eth1-2 are dynamically (dhcp) assigned IPs from the ISP provided modems and eth0 has a static IP for our LAN.

Once you have everything ready, connect and then after unconnect, each ISP modem to its designated NIC and then let’s test each of the connections real quick:

$ ping -c10 http://www.google.com


Now we will connect all of the ISP modems and verify everything is connected with ‘ifconfig’ or ‘ip addr’ and check that each NIC has an appropriate IP address. You will want to note each IP address and which modem/ISP it is from/for – maybe even write it down on a piece of paper for quick reference.

If each one works, then let’s continue with creating a router script at /usr/local/bin/router.sh:

#/bin/bash
# Set what interface is which
LAN=eth0
WAN0=eth2
WAN1=eth1
LAN_IP=192.168.1.1
WAN0_IP=192.168.0.3
WAN1_IP=162.72.156.86
LAN_GW=192.168.1.1
WAN0_GW=192.168.0.1
WAN1_GW=162.72.152.1

# SNAT packets going out WAN0 to DSL ISP
iptables -t nat -A POSTROUTING -o ${WAN0} -j SNAT –to-source ${WAN0_IP}

# SNAT packets going out WAN1 to SAT ISP
iptables -t nat -A POSTROUTING -o ${WAN1} -j SNAT –to-source ${WAN1_IP}

# chain which marks a packet (MARK) and its connection (CONNMARK) with MARK 1 for DSL ISP
iptables -t mangle -N MARK-DSL-ISP
iptables -t mangle -A MARK-DSL-ISP -j MARK –set-mark 1
iptables -t mangle -A MARK-DSL-ISP -j CONNMARK –save-mark
# icmp echo requests (ping)
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p icmp -j MARK-DSL-ISP
# ssh
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p tcp –dport 22 -j MARK-DSL-ISP
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p udp –dport 22 -j MARK-DSL-ISP
# time
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p tcp –dport 37 -j MARK-DSL-ISP
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p udp –dport 37 -j MARK-DSL-ISP
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p tcp –dport 123 -j MARK-DSL-ISP
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p udp –dport 123 -j MARK-DSL-ISP
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p tcp –dport 23 -j MARK-DSL-ISP
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p udp –dport 23 -j MARK-DSL-ISP
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p tcp –dport 992 -j MARK-DSL-ISP
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p udp –dport 992 -j MARK-DSL-ISP
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p tcp –dport 107 -j MARK-DSL-ISP
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p udp –dport 107 -j MARK-DSL-ISP
# dns
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p tcp –dport 53 -j MARK-DSL-ISP
# Star Trek Online
for STO_IP in 208.95.184.{0..255} 208.95.185.{0..255} 208.95.186.{0..255} 208.95.187.{0..255}; do
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p tcp -d $STO_IP -j MARK-DSL-ISP
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p udp -d $STO_IP -j MARK-DSL-ISP
done
# chain which marks a packet (MARK) and its connection (CONNMARK) with MARK 2 for SAT ISP
iptables -t mangle -N MARK-SAT-ISP
iptables -t mangle -A MARK-SAT-ISP -j MARK –set-mark 2
iptables -t mangle -A MARK-SAT-ISP -j CONNMARK –save-mark
# http
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p tcp –dport 80 -j MARK-SAT-ISP
# https
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p tcp –dport 443 -j MARK-SAT-ISP
# smtp
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p tcp –dport 25 -j MARK-SAT-ISP
# imap2
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p tcp –dport 143 -j MARK-SAT-ISP
# pop2
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p tcp –dport 109 -j MARK-SAT-ISP
# pop3
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p tcp –dport 110 -j MARK-SAT-ISP
# imaps
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p tcp –dport 993 -j MARK-SAT-ISP
# pop3s
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p tcp –dport 995 -j MARK-SAT-ISP
# ftp
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p tcp –dport 21 -j MARK-SAT-ISP
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p udp –dport 21 -j MARK-SAT-ISP
# ftp-data
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p tcp –dport 20 -j MARK-SAT-ISP
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p udp –dport 20 -j MARK-SAT-ISP
# sftp
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p tcp –dport 115 -j MARK-SAT-ISP
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p udp –dport 115 -j MARK-SAT-ISP
# rsync
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p tcp –dport 873 -j MARK-SAT-ISP
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p udp –dport 873 -j MARK-SAT-ISP
#
# Special rules for certain hosts (they have static IPs)
#
# We want the Wii to use DSL all the time for playing online games
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p tcp -s 192.168.1.95 -j MARK-DSL-ISP
#
# For Samsung Bluray Player using Netflix we want it to use the SAT
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate NEW -p tcp -s 192.168.1.90 -j MARK-SAT-ISP
# If a packet is not NEW, then there must be a connection for it somewhere, so go find the connection mark and apply it to the packet
# Packets from Internal network
iptables -t mangle -A PREROUTING -i ${LAN} -m conntrack –ctstate ESTABLISHED,RELATED -j CONNMARK –restore-mark

# add local routes too
ip route flush table dsl
ip route add table dsl default dev ${WAN0} via ${WAN0_GW}
ip route add table dsl 192.168.0.0/24 dev ${WAN0} src 192.168.0.3
ip route add table dsl 162.72.156.0/24 dev ${WAN1} src 162.72.156.86
ip route add table dsl 192.168.1.0/24 dev ${LAN} src 192.168.1.1/24

# ditto
ip route flush table sat
ip route add table sat default dev ${WAN1} via ${WAN1_GW}
ip route add table sat 192.168.0.0/24 dev ${WAN0} src 192.168.0.3
ip route add table sat 162.72.156.0/24 dev ${WAN1} src 162.72.156.86
ip route add table sat 192.168.1.0/24 dev ${LAN} src 192.168.1.1/24

# Now add rules to actually use them…
ip rule del from all fwmark 2 2>/dev/null
ip rule del from all fwmark 1 2>/dev/null
ip rule add fwmark 1 table dsl
ip rule add fwmark 2 table sat
ip route flush cache

# We need to allow packet forwarding
echo 1 > /proc/sys/net/ipv4/ip_forward

# Finally, make sure that the rp_filter option is disabled on the router, otherwise it could drop packets!
for i in /proc/sys/net/ipv4/conf/*/rp_filter; do echo 0 > “$i”; done

# That’s it!


In this firewall I route traffic based on the port number and I also included one example of how to route traffic for a game (Star Trek Online). That should be enough to get you up and running and some examples to make your own customizations.

The next step is to make the firewall script executable and run it:

# chmod +x /usr/local/bin/firewall

# /usr/local/bin/firewall


The last part is to add the script so it gets autoloaded by your system on boot, but that depends on the distribution you are using so consult with Google.

Now you have a semi-decent Internet connection despite being in a remote location and traffic should get routed to the most appropriate connection.

WAL-E on PostgreSQL with AWS S3

I run PostgreSQL on two AWS EC2 instances and have binary streaming replication between them for high availability. Since both VMs are in AWS, it only makes sense to use S3 for archiving and backups. I will assume you already have two working EC2 instances running PostgreSQL with replication and only wish to add WAL-E with S3 into the mix.

Before we get started you will need to make sure you have python pip and python virtualenv installed. I also needed to install zlib but that may have been because I started from a very minimal install.

I will go over the setup once as I have my master and slave configured 100% identical with the exception that the master has recovery.conf named recovery.conf.use to keep it from becoming a slave.

We will need the AWS CLI tools before we get into WAL-E.

# pip install awscli


You will need an AWS account/IAM user that has permissions on the S3 bucket we will be using. If you haven't set an account or an S3 bucket do that now. Both my EC2 instance and S3 bucket are in the US-West region so be sure to adjust that if needed. Now we need to create config and credential files for user postgres:

$ sudo -u postgres -i

postgres ~ $ mkdir ~/.aws

postgres ~ $ echo -e "[default]\nregion = us-west-2\noutput = json" > ~/.aws/config

postgres ~ $ chmod 600 ~/.aws/config

postgres ~ $ echo -e "[default]\naws_secret_access_key = \naws_access_key_id = " > ~/.aws/credentials

postgres ~ $ chmod 600 ~/.aws/config


Now would be a good time to test your awscli install and your credentials:

postgres ~ $ aws s3 ls


If you get an error go back and check your credentials as well as your IAM permissions for the user and bucket.

Next we will install WAL-E:

# pip install wal-e


We need to make a directory to hold the configuration files:

# mkdir -p /etc/wal-e.d/env/

# chown root:postgres /etc/wal-e.d

# chmod 750 /etc/wal-e.d

# chown root:postgres /etc/wal-e.d/env

# chmod 750 /etc/wal-e.d/env


Now we need to create the configuration/credential files:

# echo "" > /etc/wal-e.d/env/AWS_ACCESS_KEY_ID

# echo "us-west-2" > /etc/wal-e.d/env/AWS_REGION

# echo "" > /etc/wal-e.d/env/AWS_SECRET_ACCESS_KEY

# echo "s3://" > /etc/wal-e.d/env/WALE_S3_PREFIX

# chown postgres:postgres /etc/wal-e.d/env/

# chmod 640 /etc/wal-e.d/env/*


Note that the S3 URL prefix must be a lowercase "s3://" or it will fail!

Now is a good place to stop and manually do a basebackup to test the WAL-E install, configuration, and credentials (be sure to adjust the path to match where your database resides on the filesystem):

postgres ~ $ envdir /etc/wal-e.d/env wal-e backup-push /var/lib/postgresql/9.5/main


If everything goes properly it should automatically start a backup, copy the files, and finish. If you encounter any errors be sure to double check your credentials and paths.

Next we will edit our postgresql.conf to use WAL-E for archiving:

archive_command = 'envdir /etc/wal-e.d/env /usr/bin/wal-e wal-push %p'


Then we will edit our recovery.conf to also use WAL-E:

restore_command = 'envdir /etc/wal-e.d/env /usr/bin/wal-e wal-fetch "%f" "%p"'


At this point we can restart postgresql to make use of the new configuration (be sure to adjust this commend if you are using systemd or another init system):

service postgresql-9.5 restart


We also want our basebackups to go to S3 so we'll create a cronjob for that:

postgres ~ $ crontab -e


Then add a line similar to this (be sure to adjust the path to match where your database resides on the filesystem):

0 2  postgres envdir /etc/wal-e.d/env wal-e backup-push /var/lib/postgresql/9.5/main


We should be good to go at this point and you can watch your postgresql log file (postmaster.log for me) to make sure everything is running smoothly on both the master and slave respectively.

If you plan on keeping a lot of archive files but do not need quick access to them you may want to consider setting up an S3 bucket policy that moves things to glacier after a certain number of days. For me I have it configured to move files older than one week into a cheaper storage medium and then into glacier after two weeks.

That's about all there is to it and if you need further reading check out the WAL-E github page and of course Google is your friend when it comes to error messages!