Lunarhotel.co.uk

This article deals with routing over two (or more) internet connections. There are many different ways of setting this up, including load balancing, auto fail over, and selective routing (routing depending on traffic type) configurations. The one I've detailed here features selective routing, and inbound routing. The inbound routing is actually slightly more difficult to do than the outbound routing, but once its set up would allow the ipsec vpn service to be available on both public ip addresses, or have dual access points to an internal web server.

This configuration also allows mapping types of traffic to specific internet connections. For example, have all http traffic going out via internet connection 1 and all ftp traffic going out via internet connection 2.

Note: While I havent checked (and would love to be proved wrong) I dont think this is possible to do on a windows based pc. If you know any different, and care to share that information. Leave a comment at the end of the article

So, lets have some background:

Lets say you have two internet connections (this could be two xdsl lines, or an adsl and a standard modem). If you want to utilise both lines, you have a problem, which takes the form of the gateway setting. You can only define one default gateway, and ergo it will only ever use that one (and ergo one internet connection) to service any internet requests. The routing table is obviously central to this process. You can list the routing table in linux by issuing the following command

 

ip route show table main

 

You'll note at the bottom of this table is an entry that look something like

 

default via <your gateway address> dev <internet device>

 

Fortunatley, with Linux, its possible to set more than one routing table, which is key to routing over multiple internet connections. First, lets look at the pc we need to setup to act as our router. The one I used was a pentium 3 500mhz with 256MB RAM. into this I put 4 network cards. "Why 4?" I hear you ask! The work network has two subnets for reasons of security, The "Repro" network gets exposed to a lot of removable media, usb sticks and various other potentiall sources of viruses. It also carries alot more traffic than the other, and requires a faster and better quality switch. So, we have:

 

 

Of course, the two internal networks just add to the complication of things, but if you can get your head around all this, then you should be able to do the same with only one internal network range.

The two internet connections are maintained by two Netgear DG834G, which considering they're aimed at the home market are brilliant little boxes and excellent value for money. Ive never had to reboot one (because it crashed or whatever) and they just keep going. Netgear even issued firmware which allowed the creation of up to five IPSEC vpns! and all for just shy of £60! How good is that?

The router is always addressed .2 so on the admin network its 192.168.2.2, on the repro its 192.168.1.2 and so on. for the internet access the routers are 192.168.4.1 (connection A) and 192.168.5.1 (connection B)

So on the router I installed Fedora core 5. Now I know there are better Linux distro's out there for doing work like routing, but Im still learning linux and like the comfort factor having a pretty look GUI i can fall back on should I get annoyed using vi all the time! Also, it comes with most of the stuff needed pre-installed. Anything that isnt can normally be installed with

yum -y install <application name>

Which (IMHO) makes thing infinitely easier.

 

In order to get OpenSwan working with KLIPS I had to recompile to kernel, but I wont cover that here. Its more part of setting up Openswan which is (or will be) covered in another article.

So broadly, how does it all work? Well, basically, using a combination of iptables (not only to act as the firewall, but also mark certain packets), multiple routing tables, and ip routing rules, we can achieve a wide variety of "effects". For this configuration, the following occurrs when an outbound request is made:

1. request is recieved by the router
2. iptables either lets it through or doesnt, if it does it checks to see what type of traffic it is
3. if the traffic type is something we want going out via internet connection b we tag it
4. the ip rules are processed, which includes a rule to say that any tagged traffic should use routing table b
5. routing table b sends the packet out through internet connection b

Keep in mind that we must still conform to the "one default gateway" rule, and any traffic that needs to be routed over the other internet connection needs to explicitly tagged to do so. Or put another way, unless its specified otherwise, the traffic will always use the default connection. With this in mind, we setup the router to have a default gateway (in my case I set it to internet connection A)

So, lets get started. Set up the router with a default gateway of INTERNET CONNECTION A, and the relavent dns servers. Id try getting a web page up (you see that GUI does come in usefull) or wget-ing a page to verify you have a connection.

Next, we need to configure IPTABLES to act as a simple router. For that I set up a simple sh script :

IPTABLES="/sbin/iptables"
INTERNETIFA="eth2" # internet connection a
INTERNETIFB="eth1" # internet connection b
ADMINIF="eth3" # internal network
REPROIF="eth0"
$IPTABLES --flush
$IPTABLES --table nat --flush
$IPTABLES --delete-chain
$IPTABLES --table nat --delete-chain
$IPTABLES --table nat --append POSTROUTING --out-interface $INTERNETIFA -d ! $ADMINRANGE -j MASQUERADE
$IPTABLES -A INPUT -i lo -p all -j ACCEPT
$IPTABLES -A OUTPUT -o lo -p all -j ACCEPT
$IPTABLES -A INPUT -i $INTERNETIFA -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A INPUT -i $ INTERNETIFB -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A INPUT -i $ADMINIF -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A INPUT -i $REPROIF -m state --state ESTABLISHED,RELATED -j ACCEPT

Obviously if you're just dealing with one internal subnet range, you dont need the last line. Also, if you copy this script, make sure you change the value of the ADMINIF and/or REPROIF variables accordingly

Id suggest putting that into a file called router.sh. (dont forget you need to chmod 755 router.sh, and run it by typing ./router.sh) If we keep the various aspects of the routing separate, we can tweak bits of it without interfering with whats working already. In the end I combined all of commands needed into one script. Which we can look at later.

Setup a client pc so that it points at your new linux router for it default gateway, and dont forget to set the dns servers. Run the above script on your router, and your client should be able to access the internet. Which will always use INTERNET CONNECTION A. To satify any requests for external resources. So before we can usitlise the other internet connection, we need to set up a routing table for it. Lets have a look at the current main routing table:

ip route show table main

It should show something like this:

192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.2
192.168.2.0/24 dev eth3 proto kernel scope link src 192.168.2.2
192.168.4.0/24 dev eth2 proto kernel scope link src 192.168.4.2
192.168.5.0/24 dev eth1 proto kernel scope link src 192.168.5.2
169.254.0.0/16 dev eth0 scope link
default via 192.168.4.1 dev eth2

So we can see above the four address ranges and IP addresses of the router, of the repro, admin, internet A and internet B connections. We can also see at the bottom that currently the default route is via internet connection A. So we need to define another routing table but that has INTERNET CONNECTION B as its default route. We can do this with the ip command which takes the form

ip route add <route> table <tablename>

route is (for example) 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.2 and tablename could be tbr

 

So, to transpose the main table to another routing table called tbr we can issue the following commands:

 

ip route add 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.2 table tbr
ip route add 192.168.2.0/24 dev eth3 proto kernel scope link src 192.168.2.2 table tbr
ip route add 192.168.4.0/24 dev eth2 proto kernel scope link src 192.168.4.2 table tbr
ip route add 192.168.5.0/24 dev eth1 proto kernel scope link src 192.168.5.2 table tbr
ip route add 169.254.0.0/16 dev eth0 scope link table tbr

 

Now we can issue the command for the default gateway for table tbr, but this time of course we change it so it is using INTERNET CONNECTION B :

 

ip route add default via 192.168.5.1 dev eth2

ip route show table tbr

we should get:

192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.2
192.168.2.0/24 dev eth3 proto kernel scope link src 192.168.2.2
192.168.4.0/24 dev eth2 proto kernel scope link src 192.168.4.2
192.168.5.0/24 dev eth1 proto kernel scope link src 192.168.5.2
169.254.0.0/16 dev eth1 scope link
default via 192.168.5.1 dev eth1

Notice that the default route is different to that of the main table (its using INTERNET CONNECTION B) So we have our secondary routing table which is all well and good. But at the moment, it isnt going to do anything with it, because the ip rules are dictating that the main table always be used. So lets have a look at the ip rules:

 

ip rule show

 

should product something like this:

 

0: from all lookup local
32766: from all lookup main
32767: from all lookup default

we can see that the rule table says "from where ever the packet comes from, lookup the main table". We need to add a rule in there that says if the packetis marked, send it via the tbr table. We can do this using the following command

 

ip rule add from all fwmark 0x4 table tbr

 

So this adds the rule that if the packet is marked in a specific way, we should use the tbr table instead of the main routing table

 

0: from all lookup local
32765: from all fwmark 0x4 lookup tbr
32766: from all lookup main
32767: from all lookup default

We can see the new rule in just above the rule 32766 that says process everything with the main routing table. So allwe have to do now, is to mark the packets we want to route via INTERNET CONNECTION B. This can be done with iptables. The following line can be added to the bottom of your router.sh file:

$IPTABLES -t mangle -A PREROUTING -p tcp --dport 21 -s 192.168.1.0/24 -j MARK --set-mark 4

So we're essentially saying "if the traffic is of type tcp 21 mark it with a 4" . Note as well we're being specific about where the traffic is coming from with -s 192.168.1.0/24 in other words, only mark it if the traffic is coming from the repro network. Im pretty sure you'll need to specify this even if you only have one network range.

So run your updated router.sh and you should now have a router which routes ftp traffic over INTERNET CONNECTION B and all other traffic over INTERNET CONNECTION A. its difficult to get any visual confirmation that its actually working. To test mine, I simply visited www.whatismyip.com and noted my public ip address. I then changed the above rule so that tcp port 80 traffic was routed over the B connection, and revisted www.whatismyip.com and made sure that they changed. Oh! and one other thing: Dont try run the above test on the router itself. If you goto www.whatismyip.com on there, it will always report the public ip address of the default connection in the MAIN table. It only seems to work from a client pc. If anyone can offer any ideas as to why this might be, Im all ears

And there we have it! Just like we did with the firewall rules, you might want to put all of the commands which set up the secondary routing table, and define the ip rules into a sh script. Again, it makes it easier for debugging as we go along, and if you reboot your router, the secondary routing table (tbr) and the ip rule will be lost.

Conclusion

So theres still a lot to do on this. What Ive detailed here is just one possible application of multiple internet connections. As mentioned in the intro, theres lots of other configs we could use including load balancing over the two connections, automatic fail over (so when one fails the other kicks in) Of course we can expand on what rules we've defined so that can have all port 80 traffic (http) going over the quicker 2mb line, and all email (not so time critical) going out via the 512k line. Or even Email comes in on the 512k line, but gets sent on the 2mb line. Ive read (although never attempted) that you can even create rules based on the user thats logged in, giving rise to situations like "IT Department gets to use fast line, everyone else can slum it on the slow line" Which while cruel, is something I think alot of us would like to do!

Check out Part2 where we deal with incoming connections like a webserver, or a mailserver .

UPDATE: at the end of part 3 there is a small discussion on changing which connection a given traffic type gets router over on the fly (e.g. not having to reload the router script everytime you want to make a change) Its best to read part two to get a full idea of how it all works. But if you really cant wait, part 3 is here.