A few years back I worked for an organization that had a single 100Mb Internet connection. Not bad for just typical corporate traffic, but we also hosted our production web site out of that location as well. An incident occurred where our website was down due to Internet issues during an extremely inconvenient time. So we decided to procure a second Internet uplink through a different provider. At the time, I had no practical experience doing something like this – yet I was put in charge of the project. Let’s go over some of what I learned…
The easy part of the whole process is the first step – ordering a second Internet connection. Our CIO at the time placed a few calls and had a quote back pretty quickly. A local carrier was willing to run new fiber cables to our building in less than a month. Depending on how important uptime is to your organization, this is the point where you might want to ask about a diverse path into the building. If both connections run though the same physical paths, then a single incident could still cause an outage. For example – I once worked somewhere where the redundant Internet connections shared the same telephone poll across the street. So even though the connections were redundant, a single accident involving that telephone poll and both connections were severed.
Next – Ask about IP space. In terms of IPv4, the general rule for external BGP peering is that ISP’s don’t like to accept any prefixes smaller than a /24. In our case, we had a single /25 block already allocated by our current provider – which wasn’t going to work. Luckily, the new service provider offered to give up a free /24 block along with the installation costs. Unfortunately, this meant that we had to re-address all of our public-facing services, which is almost always a pain to do. I have a few tips for this, which helped us to minimize downtime – but that’s a story for another time.
Next, we need to obtain a globally unique Autonomous System (AS) number, which will be used to advertise our network to the world. Since we were located in North America, we went though ARIN for this process – which was fairly painless. Sign up for an account, prove that you’re associated with the business, fill out a few forms to justify your need, and then just wait for the approval. One thing to watch out for is 2-byte vs 4-byte AS numbers. 2-byte is the standard and has been around forever, but only allows for up to 65,535 unique IDs. A 4-byte ASN allows for significantly more unique IDs, but I have actually run into instances where an ISP doesn’t support these. I would hope that in most cases a 4-byte ASN will be just fine, but it might be worth asking your ISP just in case.
At this point, you should be ready to hit the ground running as soon as that second Internet uplink is installed. This is also assuming you already run a router or multilayer switch on the edge of your network, which also has BGP capabilities. So let’s get down to the fun stuff – an extremely basic configuration to peer between two ISPs. I’ll dedicate another post to additional recommended settings and configurations – but for now let’s focus on getting this running. The configuration sample below is aimed at Cisco devices, but the same concepts apply to most vendors:
EdgeRouter(config)# router bgp <YOUR AS NUMBER> ! The AS number provided by ARIN
EdgeRouter(config-router)# network <YOUR LOCAL SUBNET> ! The subnet we need to advertise out both ISPs
EdgeRouter(config-router)# neighbor <ISP1 PEER IP> remote-as <ISP1 ASN> ! Provided by the first ISP - Their remote peer IP and ASN
EdgeRouter(config-router)# neighbor <ISP2 PEER IP> remote-as <ISP2 ASN> ! Provided by the second ISPAs I mentioned, this config is very basic and will just accomplish what we need to get going. Follow up with a quick ‘show ip bgp neighbors’ and hopefully you’ll see two peers in the established state. Any other state indicates a problem bringing up the peer connection. I won’t get into too much detail here – but check the physical connection, ping the peer, and make sure there are no firewalls blocking TCP port 179 between the peer addresses.
Hope this was helpful! Comment below and let me know how your experiences have gone with this type of setup – and look forward to a few more posts regarding BGP peering setup with multiple ISPs.