So in my last post, I picked up a Qotom Mini-PC to run OPNsense on. After a few months, the device has been running well & I’m very happy with it.

One of the new things I got to try out with OPNsense was Wireguard VPN. I had previously been using something else for VPN connectivity back to my home network - but I heard good things about Wireguard & wanted to give it a try.

So far my experience has been good! I’ve been pleasantly suprised with how easy it is to configure & get running. In addition, the performance & overall experience has been very positive. The VPN connects quicker than anything I’ve used in the past, and just simply works without issue.

All that being said - I wanted to put together a quick guide on how to configure Wireguard on OPNsense. Specifically, this configuraion will be for remote-access VPN - where clients will connect to a VPN headend. We’ll walk through the OPNsense configuration & a few clients as well. So let’s dig in!

Topology

For the purpose of this blog post, we’ll be using the lab topology below:

I will be using the reserved IP range 203.0.113.0/24 for the WAN-side addressing. I’ll have one Windows & one Android client that we’ll walk through & connect to the VPN.

Installing the Wireguard Plugin

To get started, first thing we will want to do is install the Wireguard plugin for OPNsense. By default, OPNsense will have standard IPSec & OpenVPN already available - but other VPN options can be enabled easily.

So in OPNsense, we’ll navigate down to System > Firmware > Plugins, then search for wireguard and click the plus icon.

This should pull down the package & install pretty quickly. No reboot required here!

Once installed, you may have to refresh the page or navigate to a new page so that the menu bar has a chance to reload. Then we’ll have a new option under VPN:

Wireguard Tunnel Configuration

Next we’ll begin configuring Wireguard on the OPNsense side.

There is a little bit of a chicken & egg scenario here since everything is based on cryptographic keys. We’ll need to generate keys on the firewall, which we need to enter on the client - but we also need the client keys to enter on the firewall. A bit of bouncing between the two - but for now we’ll try to complete as much as we can on the firewall side.

We’ll enable Wireguard by dropping down to VPN > WireGuard then clicking Enable and Apply

Next we’ll set up the Wireguard tunnel interface on OPNsense. This will be a virtual tunnel interface that will be created as interface wg<instance number>.

To do this, we’ll navigate to the Local tab, and click the plus icon to add a new tunnel.

In the above screenshot, I’ve filled in just a few details.

For Name, I’ve entered our virtual interface name wg1. Since OPNsense shows the Instance as 1 - it will create a wg interface with that instance number.

We’ll leave Public Key & Private Key blank for now. OPNsense will auto-generate these keys once we save this config.

For Listen Port, I’ve set this to 51820 which is the default for Wireguard. It’s not stated here, but this is a UDP tunnel.

For Tunnel Address - this is where we add the IP address of the virtual tunnel interface. This will be the gateway for our remote clients. In my lab I’ll be using 10.50.50.1/24 here.

We have no peers configured yet - so we can’t select any here. We’ll leave this blank for now, but come back later.

We will leave Disable Routes unchecked. By default, OPNsense will add static/connected routes for any client via the tunnel interface. You might not want this behavior if you wanted to do custom routing - for example, in a site-to-site VPN connection - but we’ll leave this enabled.

Once we’re done, we’ll click Save.

Like I mentioned before, OPNsense will now auto-generate our crypto keys for the tunnels. So if we edit our tunnel, we’ll now see those fields populated:

We’ll want to copy the Public Key & save it for later. This will need to be imported onto our clients, so that they can communicate securely with our firewall.

Wireguard Interface Assignment

Now that we have our headend tunnel interface defined, we can map our wg1 interface to an OPNsense interface. The OPNsense documentation suggests this is optional, but I would recommend it since it will allow us to create firewall rules to permit/deny access to clients.

We’ll navigate to Interfaces > Assignments, and we should see a New interface available: our wg1 tunnel.

We can assign this a name, then click the plus icon & Save.

Next we’ll enable the interface by navigating to Interfaces > WG1. Here we’ll only need to click Enable & save the change - nothing else is necessary.

Of course, we’ll be prompted to apply the changes - which we will do:

By default, all traffic through our WG1 firewall interface will be blocked - so please make sure to configure a firewall rule to permit traffic from the Wireguard clients.

Firewall Rules: Allow Inbound Wireguard Traffic

Next we need to permit the Wireguard traffic into our firewall. By default the WAN interface will block all traffic that isn’t explicitly allowed - including our Wireguard traffic.

For this, we’ll navigate to Firewall > Rules > WAN. Then click the plus icon to add a new rule.

The screenshot above shows what our firewall rule will look like.

Here’s the summary:

• Action: Pass
• Interface: WAN
• Direction: In
• TCP/IP Version: IPv4
  • You can enable IPv6 as well, if you have IPv6 connectivity (this is a lab box, which does not)
• Protocol: UDP
• Source: Any
  • This will allow anyone on the internet to reach our VPN. We could restrict source IP addresses, if our clients had permanent, static IPs.
• Destination: WAN Address
  • The firewall itself is the destination for this traffic
• Destination Port Range: (other) / 51820

I also enabled logging & added a quick description. After we have all this configured, we can click Save - then Apply Changes.

Client Setup - Windows

So first we’ll start with an easy configuration on a Windows client. Wireguard client software can be found on the Wireguard site here.

For the sake of the walkthrough, we will manually configure each client. However, this can be a difficult task if there is a large number of clients. Wireguard does support importing configurations, and there are a number of free tools available to help automate generating config files for clients - including some which will generate QR codes for easy import on mobile clients.

So on my lab Windows machine, we’ll open up the Wireguard client & click Add Empty Tunnel:

Then we’ll be given a blank config file, with only the devices public & private key pair generated for us.

We’re going to fill in details similar to the below screenshot:

The configuration under [Interface] is the local, client-side configuration. I’ve added the client’s tunnel address - which will be 10.50.50.15/32. I’ve also configured DNS servers which the client can reach via the VPN.

Then we’ll add the [Peer] section, which contains info about our VPN headend. Here’s where we’ll need the public key from our OPNsense firewall. We’ll also specify the Endpoint address, which is the IP or hostname of our VPN headend & the port (which by default is 51820).

We’ve also configured AllowedIPs as 0.0.0.0/0. This will force all client traffic over the VPN tunnel - including general internet traffic. However, we could limit this to specific subnets. For example, let’s say your network only used 172.16.90.0/24 & 10.1.1.0/16 subnets & we only wanted the user to be able to access those. We would configure the following: AllowedIPs = 172.16.90.0/24, 10.1.1.0/16. In this case, only traffic for those subnets would be routed over the VPN - any other traffic would use the devices default internet connection.

Now, we’ll save this - and again need to copy the device’s Public Key, which we’ll need to enter on the OPNsense firewall.

Client Setup - Adding Clients to OPNsense

In order for the Windows machine to connect to OPNsense, we’ll also need to configure a client profile on the firewall.

In OPNsense, we’ll navigate back to VPN > WireGuard, then click on the Endpoints tab.

Here we’ll configure a name for our client & paste in the client’s Public Key.

We’ll also set AllowedIPs to the client’s IP address, which we have configured as 10.50.50.15/32. This controls what IP addresses are reachable via this endpoint.

We do have some additional fields available, which we will leave blank. For example - Endpoint Address & Endpoint Port would be used to define a public IP that we expect our client to connect from. Since a remote-access client could connect from any IP, we leave those fields blank to allow this.

Once we’re done, we’ll click Save then Apply.

Next we’ll jump back to the Local tab, and edit our headend tunnel configuration.

We should now see our windows client in the Peers dropdown. We’ll select that client, so that Wireguard will permit that client to connect via this tunnel interface.

Then, as always, click Save & Apply

Last but not least - we should restart our Wireguard server on OPNsense. This can be done by either disabling & re-enabling Wireguard - or by navigating back to the OPNsense dashboard & clicking the restart icon next to the wireguard-go service.

Testing The Connection

Okay - now that we have all that completed, it’s finally time to test connectivity from our client.

On my Windows client, I’ll just click the Activate button for the tunnel:

And we’ll see that the Status shows Active, and we’ll start to see updates to the Last Handshake & Transfer fields - indicating we are connected & sending data.

To further validate, we can check the Log tab:

The key things to look for here, are the following messages:

• Receiving handshake response which indicates our firewall responded to our request to connect
• Keypair 1 created which indicates that our connection is healthy to our peer
• Receiving keepalive packet from peer - we should see these periodically to maintain our connection. If keepalives stop flowing, then we may have a break in connectivity between client & peer.

We should also be able to reach out wg1 tunnel address from the Windows client:

On the OPNsense side of things, we can check what client is connected via the List Configuration tab, under VPN > Wireguard:

On this screen, we can see we have 1 peer connected - which matches the IP & public key of our Windows client. We’ll see similar output like the Windows client, where we can see the latest handshake & data transfer.

Additionally, for easier access we can add the Wireguard widget to the OPNsense dashboard.

If we navigate to our dashboard, then click Add widget - we can add the Wireguard widget.

Here’s what that looks like:

Additional Client Setup - Android

Let’s also take a quick look at a mobile client. I’ll get through this pretty quickly, since the configuration will be very similar to our other client - just a different UI.

On my Android device - if I open up the Wireguard app, I have a few options for creating or importing a tunnel:

In this case, we’ll again walk through creating a tunnel manually. Again, it’s worth mentioning that there are 3rd party apps available to auto-generate config imports or QR codes to make this easier.

First we’ll have a handful of fields to fill in about the Android-side configuration. This includes a name for the tunnel, an address, and DNS servers.

We can click on the refresh icon in the Private Key field to auto-generate our key pairs:

One of the interesting parts of the mobile app is the ability to permit/exclude individual apps from traversing the VPN tunnel.

Here’s a quick screenshot, where we can pick either to allow only certain apps to use the VPN - or permit all except a few we choose to exclude:

In my case, I’ll keep this set to allow all applications.

Next we’ll work on the peer config:

After that, all we need to do is save our VPN configuration - then we can toggle the tunnel on or off:

And similar to the Windows client, we can click on the tunnel itself to see current status / data transfer:

So it looks like we should be connected & can try accessing VPN resources!

We can also use the same methods as earlier to check connectivity from the OPNsense side.

Okay - that’s all I wanted to share today. I’ve been quite pleased with how easy to use WireGuard has been - and how well it performs! I hope this blog post was helpful if you’re interested in trying it out.

Thanks!!

^{Note: I may receive commissions for purchases made through links in this post. This is to help support my blog and does not have any impact on my recommendations.}

One day I would love to have symmetric internet speeds, but for the meantime I have to work with what I have - which is dismal upload speeds that don’t come anywhere close to the download speed.

So I’m currently at 200Mb/s down & 10Mb/s up - and I would prefer to have higher upload speeds. However, even just 20Mb/s upload requires me to purchase a 500Mb/s download plan. 🙄

I’ve been hesitant to upgrade, since my current Meraki MX64 tops out at ~250Mb/s throughput. I would have to upgrade my firewall or half of my new download speed would be unusable.

While Meraki does have faster firewalls, they’re a bit expensive & at the moment somewhat difficult to obtain. So instead I searched around for alternatives, and came across the Qotom Q750G5 - which is a barebones mini-PC that I could load pfSense or OPNsense onto.

So in this blog post - I wanted to talk about the device & some of the performance testing I did.

Qotom Q750G5 - Hardware

First let’s take a quick look at the hardware. What got my attention quickly was the five 2.5GbE ports, but this small device offers quite a lot for how inexpensive it was.

A quick look at the specs:

• Intel Celeron J4125 Quad Core 2.0Ghz (Burst 2.7Ghz)
• Five Intel I225-V 2.5Gb Ethernet ports
• Optional WiFi slot
• Optional 3G Cellular & SIM card slots
• 1 RAM slot
• 1 mSATA slot
• 1 2.5in SATA HDD slot
• 2x USB 2.0 & 3x USB 3.0 ports

By default it seems this is a barebones kit, however I did opt to buy a model that included 16GB RAM & a 256GB mSATA SSD. It arrived with a single 16GB TeamGroup DDR4 3200Mhz module & 256GB Hoodisk mSATA SSD.

Are the specs a bit overkill for an embedded firewall? Yeah definitely. But the whole kit was only around $230 USD - which seemed crazy to me.

The big thing that pulled me toward this model (besides price) was the 2.5GbE ports. Since a lot more places have gigabit residential internet these days, and some lucky places are starting to see 2Gb/s - I was hoping that this little box would offer me a bit of future-proofing.

Of course, I was a little curious about what performance this device could realistically achieve - but more on that later!

I wanted to provide a few photos of the unit as well. The casing itself is pretty minimal, but the box is definitely heavier than you would expect!

Front Photo:

On the front there isn’t a whole lot to see. Power button, reset button, status LEDs, and USB ports. Oh - and there is an HDMI port as well, which comes in handy when installing an operating system (or if you’re using this as an actual PC).

Back Photo:

The back shows off the five 2.5Gb Ethernet ports, as well as the power plug. While I didn’t get a WiFi-enabled unit, the back faceplate still has cutouts for wireless antenna. I could opt to add wireless to this later, but I do wish the faceplate came with plugs or something to cover the holes in the meantine.

Inside Photo:

Now here’s where things get interesting! This box surprisingly packs a lot in it’s tiny size.

In the upper right, there is a PCI slot for WiFi. There’s also a WiFi/3G PCI slot in the lower left.

Just above the 3G slot is the SSD mSATA slot. When a drive is installed, it does cover the SIM card slot - meaning that the SSD would need to be removed to change SIM cards.

Also on the left, mostly out of the photo, is a connector for a 2.5in HDD. While I didn’t snap a photo of the bottom of the unit, the drive would screw into the bottom plate.

Interestingly enough, the CPU is on the other side of the motherboard. While you might think that the overall unit kinda looks like a heatsink - I was surprised to see that’s exactly what it is. The CPU is located right under the four screws with black washers and has thermal paste pre-applied. It rests up against the top of the case & uses the top as a heatsink.

As a last note about the CPU - it does get quite toasty at times. During one of my performance tests, the CPU reached ~90°C - and the top of the unit was very hot to the touch. It may not act as the world’s best heatsink, so I would avoid placing anything on top of the unit - or resting your hand there for too long 🙂.

Performance Testing

As a quick note before we get to the good stuff: All of my tests were performed using iPerf3. This may not show us the best real-world throughput tests, but I had quite a difficult time getting dpdk drivers to compile correctly for the Intel 2.5GbE ports (so I could use something like TRex). I may attempt to go back to this later, but the iPerf data is what I’ll provide for now.

The Testbed

For the performance testing, I used the two Intel NUC11 PCs that I purchased recently for my VMware lab. These already come with a 2.5GbE port, which I am currently using for vMotion between the two. I disconnected the vMotion port temporarily, and instead enabled PCI-passthrough to connect the ports directly to a VM.

I built a VM on each NUC with the following specs:

• Debian 11
• 8x vCPU
• 16GB RAM
• PCI-passthrough to 2.5GbE adapter

iPerf 3.9 was used for all tests. The iPerf server was enabled with the iperf3 -s command, and the client tests were run with iperf3 -c <client ip> -P 8 -t 600. Each test was run for 10 minutes.

The devices were connected & configured with the following topology:

Note: In the below paragraphs, I will talk about the configuration I used for each test. If you’re interested in more detail, please check out the video at the top of this blog post. In the video, I walk through the configuration & setup for most of the tests below.

Test 1: No Firewall

Avg: 2.35Gb/s

Okay, so expecting that I likely wouldn’t get the full 2.5Gb speeds once I added the firewall - I wanted to perform a baseline test with the VM’s directly connected via the 2.5GbE passthrough port.

In each of these tests, I was able to reach an average speed of: 2.35Gb/s

Test 2: Routing, NAT, Simple Firewall rules

Avg: 2.35Gb/s

In this test case, OPNsense was configured to match the diagram above. The iPerf server was located on the LAN segment, with an IP of 10.2.2.1 & a default route toward the LAN interface of the OPNsense box (10.2.2.2). The client is connected via the WAN interface, at 203.0.113.50.

I created a proxy-ARP virtual address for the 203.0.113.25 IP address, then created a NAT rule to forward traffic sent to that address to 10.2.2.1.

Next, there was a single firewall rule created to permit TCP port 5201 inbound from the WAN. This is the default port that iPerf will use.

During these tests, I averaged about 2.35Gb/s and relatively low CPU usage around 10-20%.

As an interesting side note to this: Originally I didn’t use the default LAN & WAN ports (ports 1 & 2), but instead used ports 3 & 4. During testing between those ports, I could only reach ~1.7Gb/s. Once I switched to ports 1 & 2, I could easily hit 2.35Gb/s. I’m curious to dig in later & see if this is a hardware issue, or possibly OPNsense is prioritizing the LAN/WAN traffic differently

Test 3: Large Firewall Ruleset

Avg: 2.35Gb/s

While my home network will likely have a somewhat minimal firewall ruleset, I was curious to see how well the device would perform with a large ruleset.

So I wrote a script to auto-generate ~1,200 firewall rules with random IP addresses, ports, and a mix of permit/block actions. I applied the ruleset inbound on both the LAN & WAN ports.

Under this test, I was still able to reach the 2.35Gb/s speeds - but now the CPU was creeping up to 30-40%.

So what next? Well I decided to see if I could stress the box a little - and increased the auto-generated ruleset to just over ~15,000 rules.

The increased ruleset took about 5-10 minutes to load into the system, and CPU was pegged at 100% the whole time. In fact, the CPU never really came back down. Even after the rules had loaded, the CPU was flat at 100% - and it took anywhere from 2-4 minutes to navigate between pages in the web GUI. (This is the part where my CPU temps went from <50°C to over 90°C 🙃)

I ended up having to re-image the box.

Test 4: Suricata IPS

Avg: 2.35Gb/s (But sometimes much, much less)

Next I loaded up the built-in IDS/IPS, which uses suricata. I downloaded all available free rulesets, and enabled all of them. Ideally, you wouldn’t necessarily enable everything - but I wanted to start off with a full load test.

My experience with these tests was highly variable. Most times, I was surprised to see that I could still push 2.35Gb/s through without any issue. During these tests, the CPU usually bounced between 50-80% usage.

Strangely, every so often I would test and get significantly less than that. Most times when this happened, I would instead only see somewhere between 400-700Mb/s - but on one test the box slowed to just over 200Mb/s. Each time this happened, the performance tests would remain degraded until I restarted the suricata service - then I would be able to reach the 2.35Gb/s again.

My best guess is that something is getting stuck. I know some older IPS systems I’ve worked with in the past had issues with CPU-pinning, and you might get garbage performance if your traffic hit the wrong CPU core. I didn’t spend a lot of time digging into this, but it feels a bit similar.

Test 5: Wireguard VPN

Avg: 800Mb/s, 650-700Mb/s with IPS also enabled

I also wanted to try VPN performance. There are a lot of options for VPN services within OPNsense, including standard IPSec and OpenVPN. However, I opted to try out wireguard - since it’s really easy to set up & get running.

In this case, I definitely hit a limitation with CPU-pinning. With my initial tests, I could only reach about ~450-500Mb/s - but the Qotom CPU was only spiking up to 70%.

Seems like the wireguard client (at least for same source/destination/port traffic) does pin everything to a single CPU. So my client CPU, between encrypting the traffic & generating it, was actually maxing out long before the Qotom was.

I ended up spinning up a second VM on the client side (which required disabling PCI passthough & adding both VMs to a shared vSwitch). With both of these running, I was able to max out the Qotom CPU at 100% - and hit a fairly consistent 800Mb/s VPN throughput.

As a final test, I enabled the suricata IPS on top of the VPN as well. Now the traffic would need to be decrypted & inspected before reaching the server. With a single client, I averaged ~400-450Mb/s - and with both clients it was around ~650-700Mb/s

Overall I’m really pleased with how well this thing performs, especially for how relatively inexpensive it was. I also expected OPNsense to have a steep learning curve, but it was surprisingly easy to get up and running very quickly.

Next I’ll be working on getting this firewall up & running at home. I may be tempted to write up some more on OPNsense - so if there are any questions or specific configurations you might like to see, please leave a comment!

I’ve been getting a handful of questions lately on the process of bringing a Cisco Catalyst 8000v or CSR 1000v into an SD-WAN environment. So I figured maybe I should put something together to share.

On a related note - I’ve been debating for a while doing a blog and/or video on building out a Cisco Viptela SD-WAN lab in EVE-NG. This would (tentatively) include everything from building controllers, bringing up remote sites, and template/policy configs. If this is something you might get value from, please let me know!! I’m looking for some motivation :)

Okay - all that being said, let’s go ahead and get started.

Note: This guide should work with CSR 1000v devices as well. But it will NOT be 100% accurate for physical ISR/IOS-XE routers, as there are some additional steps with certificates & the Plug and Play portal to get those running.

Topology

So to start with, figured I would share the topology that I’m working from. If you read my last post you might recognize this, but with an added location. This new location (site id 400) contains our Catalyst 8000v VM, running IOS-XE version 17.04.01a.

Controller or Autonomous Mode?

Back in the earlier days of IOS-XE SD-WAN, there used to be two separate software images to load on your network appliance - one for traditional IOS-XE, and one for SD-WAN code.

With the newer releases of IOS-XE, we’re now getting a unified image that contains both software sets. So our options now are two modes: autonomous (traditional IOS-XE) or controller (SD-WAN).

One way we can check this, is by running a show version and looking for Router operating mode:

Router# show version
<-- Output omitted -->
cisco C8000V (VXE) processor (revision VXE) with 2035355K/3075K bytes of memory.
Processor board ID XXXXXXXXXXX
Router operating mode: Autonomous
4 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
3965744K bytes of physical memory.
5234688K bytes of virtual hard disk at bootflash:.

Configuration register is 0x2102

Okay, and looks like the Catalyst 8000v image I’m using booted up in autonomous mode. No big deal, we can change modes pretty easily!

So in normal exec-mode, we’ll use the command controller-mode enable:

Router# controller-mode enable
Enabling controller mode will erase the nvram filesystem, remove all configuration files, and reload the box!
Ensure the BOOT variable points to a valid image
Continue? [confirm]

As noted in the snippet above - this command will erase the config! So you will want to take a backup snapshot of your existing configuration, if this is already being used. In my case, it’s a brand new VM - so we’ll continue on.

Once the device is back online, we’ll log in with the default login of admin/admin. Note that you will be forced to change this upon first login.

User Access Verification

Username: admin
Password:

Default admin password needs to be changed.

Enter new password:
Confirm password:
Router#

And just for fun, we’ll run a show version again to ensure we’re in controller mode:

Router# show version 
<-- Output omitted -->
cisco C8000V (VXE) processor (revision VXE) with 2035355K/3075K bytes of memory.
Processor board ID XXXXXXXXXXX
Router operating mode: Controller-Managed
4 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
3965756K bytes of physical memory.
5234688K bytes of virtual hard disk at bootflash:.

Configuration register is 0x2102

Catalyst 8000v Initial Config

So since this is a lab environment, and I’m not trying to provide any Day0 provisioning files - I’ll have to complete some manual configuration to get started.

First, we’ll start with some SD-WAN specific config (site id, org name, etc). I’ll be using the values that apply to my lab, so be sure to change these in yours!

Note: If you haven’t used IOS-XE SD-WAN previously, be aware that “conf t” doesn’t work! In SD-WAN/controller mode, you’ll use “config-transaction”. In this mode, all changes will need to be committed before they’re applied to the device.

Router# config-transaction
 ! Set hostname
Router(config)# hostname Cat8k-Site400
 ! Required SD-WAN system configs
Router(config)# system
Router(config-system)# organization-name "SDWAN-LAB"
Router(config-system)# site-id 400
Router(config-system)# vbond 192.168.99.241
Router(config-system)# system-ip 10.10.10.237
Router(config-system)# commit
Commit complete.

Cat8k-Site400# 

Once we have those basics configured, at a minimum we’ll also need to configure our internet-facing tunnel interface. This allows our Catalyst 8000v to communicate with our control plane for bring-up.

Cat8k-Site400#config-transaction
 ! Config physical interface (This is a lab, so I'm using a static IP)
Cat8k-Site400(config)# interface GigabitEthernet1
Cat8k-Site400(config-if)# ip address 192.168.99.237 255.255.255.0
Cat8k-Site400(config-if)# no shut
 ! Config tunnel interface
Cat8k-Site400(config-if)# interface Tunnel1
Cat8k-Site400(config-if)# no shut
Cat8k-Site400(config-if)# ip unnumbered GigabitEthernet1
Cat8k-Site400(config-if)# tunnel source GigabitEthernet1
Cat8k-Site400(config-if)# tunnel mode sdwan
Cat8k-Site400(config-if)# exit
 ! SD-WAN tunnel config
Cat8k-Site400(config)# sdwan
Cat8k-Site400(config-sdwan)# interface GigabitEthernet1
Cat8k-Site400(config-interface-GigabitEthernet1)# tunnel-interface
Cat8k-Site400(config-tunnel-interface)# encapsulation ipsec
Cat8k-Site400(config-tunnel-interface)# color biz-internet
Cat8k-Site400(config-tunnel-interface)# exit
 ! Default route to our internet gateway
Cat8k-Site400(config)# ip route 0.0.0.0 0.0.0.0 192.168.99.1
Cat8k-Site400(config)# commit
Commit complete.

Cat8k-Site400#

A Note on Certificates

Since this is a lab environment, I’m using self-signed local certificate authority to provision all of my certificate infrastructure. Because of this, I’ll need to install my local CA certificate on the Catalyst 8000v. If you’re using the default Cisco-provisioned certificate setup, you won’t need to do this.

Since my SD-WAN lab doesn’t have direct access to my local TFTP server - I do have an out-of-band management interface connected to my Cat 8000v. We’ll start by configuring that interface:

Cat8k-Site400# config-transaction
Cat8k-Site400(config)# vrf definition 512
Cat8k-Site400(config-vrf)# address-family ipv4
Cat8k-Site400(config-vrf)# exit
Cat8k-Site400(config)# interface GigabitEthernet 3
Cat8k-Site400(config-if)# vrf forwarding 512
Cat8k-Site400(config-if)# ip address dhcp
Cat8k-Site400(config-if)# exit
Cat8k-Site400(config)# ip tftp source-interface GigabitEthernet 3
Cat8k-Site400(config)# commit

The standard management VRF/VPN for SD-WAN is 512, so I kept that config to match when I configured this management interface. This will all be over-written anyways once we get connected to vManage & configure/push our template configs.

Once that’s done, we can go ahead and copy our CA certificate to bootflash.

Cat8k-Site400# copy tftp://10.0.0.2/cacert.pem bootflash:
Destination filename [cacert.pem]?
Accessing tftp://10.0.0.2/cacert.pem...
Loading cacert.pem from 10.0.0.2 (via GigabitEthernet3): !
[OK - 1406 bytes]

1406 bytes copied in 0.112 secs (12554 bytes/sec)

Cat8k-Site400# dir bootflash: | inc cacert.pem
16      -rw-             1406  May 14 2021 14:41:27 +00:00  cacert.pem
```text

Then we'll use the command below to install the CA certificate:

```text
Cat8k-Site400# request platform software sdwan root-cert-chain install bootflash:cacert.pem
Uploading root-ca-cert-chain via VPN 0
Copying ... /bootflash/cacert.pem via VPN 0
Updating the root certificate chain..
Successfully installed the root certificate chain

And we can validate by using the show sdwan certificate root-ca-cert command:

Cat8k-Site400 show sdwan certificate root-ca-cert
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            11:3b:0a:12:17:b0:e0:b5:4b:fa:c2:e9:2c:9c:12:84
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: DC = local, DC = 0x2142, O = SDWAN-LAB, CN = 0x2142-0XWIN1-CA-2
        Validity
            Not Before: Nov 29 20:00:11 2018 GMT
            Not After : Nov 29 20:10:11 2043 GMT
        Subject: DC = local, DC = 0x2142, O = SDWAN-LAB, CN = 0x2142-0XWIN1-CA-2
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
        <-- Output omitted -->

Activating the Catalyst 8000v

Almost there! Now that we’ve finished our pre-config & added our root CA certificate - we’re ready to join the Catalyst 8000v to the SD-WAN fabric.

We’ll start over in vManage - by going to Configuration > Devices.

Then we’ll find our target, unused Catalyst 8000v device. Click the ellipsis on the right side, then select Generate Bootstrap Configuration

This will give us a prompt to select which configuration style to generate. We’ll leave this on “Cloud-init”:

Once we hit okay - we’ll be presented with the info we need. We won’t necessarily need all of this information, but we’ll want to take note of our uuid and otp:

We’ll drag this info back over to our Catalyst 8000v, and we can now use it to activate the device & join to our SD-WAN fabric.

For the command below, chassis-number will be our uuid value - and token will be our otp.

Cat8k-Site400# request platform software sdwan vedge_cloud activate chassis-number C8K-178BXXXX-XXXX-XXXX-XXXX-XXXXXXXXBC24 token 421ecxxxxxxxxxxxxxxxxxxxxxbd53b5

Validation

After a few moments, we’ll see some log messages start to appear showing our control connections coming up. You might see these on the terminal if you’re using the console port, or you can use show log:

*May 14 15:39:06.910: %Cisco-SDWAN-Cat8k-Site400-OMPD-3-ERRO-400002: R0/0: OMPD: vSmart peer 10.10.10.242 state changed to Init
*May 14 15:39:07.980: %DMI-5-AUTH_PASSED: R0/0: dmiauthd: User 'vmanage-admin' authenticated successfully from 10.10.10.240:48140 and was authorized for netconf over ssh. External groups:
*May 14 15:39:08.798: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400002: R0/0: OMPD: vSmart peer 10.10.10.242 state changed to Handshake
*May 14 15:39:08.804: %Cisco-SDWAN-Cat8k-Site400-OMPD-5-NTCE-400002: R0/0: OMPD: vSmart peer 10.10.10.242 state changed to Up
*May 14 15:39:08.808: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400005: R0/0: OMPD: Number of vSmarts connected : 1
*May 14 15:39:09.827: %Cisco-SDWAN-Cat8k-Site400-OMPD-3-ERRO-400002: R0/0: OMPD: vSmart peer 10.10.10.243 state changed to Init
*May 14 15:39:10.584: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
*May 14 15:39:11.756: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400002: R0/0: OMPD: vSmart peer 10.10.10.243 state changed to Handshake
*May 14 15:39:11.762: %Cisco-SDWAN-Cat8k-Site400-OMPD-5-NTCE-400002: R0/0: OMPD: vSmart peer 10.10.10.243 state changed to Up
*May 14 15:39:11.762: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400005: R0/0: OMPD: Number of vSmarts connected : 2
*May 14 15:39:12.738: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400007: R0/0: OMPD: Using policy from peer 10.10.10.242
*May 14 15:39:13.570: %Cisco-SDWAN-Cat8k-Site400-FTMD-6-INFO-1000020: R0/0: FTMD: SLA class added : class 'Default' at index '1' loss = 25%, latency = 300ms, jitter = 100ms, app-probe-class = None
*May 14 15:39:14.738: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400007: R0/0: OMPD: Using policy from peer 10.10.10.242

We can also see our control connections using the command show sdwan control connections:

Cat8k-Site400# show sdwan control connections
                                                                                       PEER                                          PEER                                          CONTROLLER
PEER    PEER PEER            SITE       DOMAIN PEER                                    PRIV  PEER                                    PUB                                           GROUP
TYPE    PROT SYSTEM IP       ID         ID     PRIVATE IP                              PORT  PUBLIC IP                               PORT  ORGANIZATION            LOCAL COLOR     PROXY STATE UPTIME      ID
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
vsmart  dtls 10.10.10.242    100        1      192.168.99.242                          12446 192.168.99.242                          12446 SDWAN-LAB  biz-internet    No    up     0:00:00:51  0
vsmart  dtls 10.10.10.243    100        1      192.168.99.243                          12446 192.168.99.243                          12446 SDWAN-LAB  biz-internet    No    up     0:00:00:49  0
vmanage dtls 10.10.10.240    100        0      192.168.99.240                          12646 192.168.99.240                          12646 SDWAN-LAB  biz-internet    No    up     0:00:00:52  0

Of course, we can also check to see our device status in the vManage dashboard as well.

Over on the Monitor > Network page, we can see that our new Catalyst 8000v is now online:

Extra: How do I check the routing table on an IOS-XE SD-WAN device?

So - if you’ve only used the vEdge software devices, you may be used to using the show ip route or show ip route vpn 10 commands.

In the IOS-XE world, most SD-WAN commands are prefixed with the sdwan keyword. For example: show sdwan bfd sessions (where on vEdges, it would just be show bfd sessions)

This might lead you to believe that you can use show sdwan ip route or show sdwan ip route vrf 10 - but these won’t work! In fact, you’ll get the following message:

Cat8k-Site400# show sdwan ip route
% Error: This command is not supported

For the IOS-XE based devices, they actually just use the standard IOS-XE routing table and VRF constructs.

So on a vEdge, you would have VPN 0 as your transport VPN. On IOS-XE, this is just the default global routing table, shown with show ip route.

But what about our LAN-side service VPNs? In this case, our routes are being dumped into a VRF on the IOS-XE device.

So for example, I have VPN 10 in my lab which is used for LAN-side clients. We can use the show vrf command to see that this exists, and then show ip route vrf 10 to see the routes from our other SD-WAN locations:

Cat8k-Site400# show vrf
  Name                             Default RD            Protocols   Interfaces
  10                               <not set>             ipv4        Gi2
  512                              <not set>             ipv4        Gi3
  65528                            <not set>             ipv4        Lo65528

Cat8k-Site400# show ip route vrf 10

Routing Table: 10
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
       n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       H - NHRP, G - NHRP registered, g - NHRP registration summary
       o - ODR, P - periodic downloaded static route, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR
       & - replicated local route overrides by connected

Gateway of last resort is not set

      10.0.0.0/24 is subnetted, 2 subnets
m        10.2.2.0 [251/0] via 10.10.10.235, 00:09:02, Sdwan-system-intf
m        10.3.3.0 [251/0] via 10.10.10.236, 00:09:02, Sdwan-system-intf

Okay, that’s it! Pretty quick process overall - and now we can get into applying our device/feature templates.

Hope this was helpful!!

In this post, we’ll walk through configuring a Cisco Viptela SD-WAN network to integrate with Cisco Umbrella’s Secure Internet Gateway (SIG) & Secure Web Gateway (SWG).

If you’re interested in reading more about what SIG & SWG are, please check out my last post where I walked through this integration with a Meraki MX firewall.

For additional reading, there’s also a good Umbrella blog post on all the components of Cisco’s SASE architecture, which you can find here.

Okay, with that being said - let’s get started!

Step 1: Umbrella API Keys

Okay, so we’ll start on the Umbrella side. We’ll need to generate a set of API keys, which we’ll push out to our WAN edge devices. These API keys will allow our remote devices to reach out to Umbrella & auto-configure their SIG tunnels.

We’ll log into our Umbrella dashboard at https://login.umbrella.com/

Once we log in, we’ll hop over to Admin > API Keys. Then up in the upper-right corner, click Create.

For this integration, we’ll need to select Umbrella Management to make sure our API keys have the access they need. Then click Create. Easy enough, right?

Once we do that, the Umbrella dashboard will display our API keys - which we’ll need to copy over to our Viptela SD-WAN environment.

We’ll also need to collect our Umbrella Organization ID. This is actually just embedded within the Umbrella Dashboard URL, and should be a seven-digit number. For example, if we look at our current URL on the API keys page:

https://dashboard.umbrella.com/o/<ORG ID>/#/admin/apikeys

And that’s all we need for now from the Umbrella side. So let’s hop over to Viptela.

Step 2: Viptela Templates & Config

For this post, I’ll be using my SD-WAN lab in EVE-NG. Currently I just upgraded the lab to run Viptela version 20.4.1, though the SIG integration has been supported since 20.1.

Just for reference, here is the lap topology that I’m working with:

This lab includes a bridged connection to allow direct internet access from the lab network.

To configure our SIG automatic tunnels, we’ll need to create / update a few templates:

• Create a SIG Credentials feature template
• Create a SIG feature template
• Assign SIG templates to device templates
• Edit Service-side VPN Template to inject a service route

2a: Creating a SIG Credentials Template

First we’ll create a template which will store our Umbrella API credentials.

In the vManage dashboard, we’ll go to Configuration > Templates, then drop into the Feature tab, and click on Add Template.

Once we get to the next screen, we’ll select our device type. In my case, I’m using a vEdge Cloud.

A list of templates will pop up - we’ll drop down to SIG Credentials, which will be near the bottom under the Other Templates header.

We’ll then be taken to the page where we create our template.

We’ll fill in our template name & description, then paste in our Umbrella details (Org ID, API Key, API Secret).

Once we’re done - We’ll hit Save.

Note: At time of writing, the “Get Keys” button only functions if you’ve purchased your SD-WAN subscription with DNA Premier licensing. This license level includes Umbrella SIG, and allows this 1-click integration that pulls your Umbrella API keys automatically.

2b: Creating a SIG Tunnel Template

Okay, next we’ll need to create another feature template to specify our IPSec tunnel configuration.

Just like before, we’ll head back over to Configuration > Templates > Feature > Add Template.

This time, after selecting our device type, we’ll choose Secure Internet Gateway (SIG) / WAN - which is located under the VPN header.

In the template configuration, we’ll give the template a name & description as always. Then we’ll jump down to the tunnel config.

We’ll select Umbrella for SIG Provider, then click Add Tunnel.

Within the tunnel config, we’ll specify an interface name - I’ll name mine ipsec1 (and I’ll create an ipsec2 shortly). We’ll also specify a Tunnel Source, which in my lab is ge0/0 for the biz-internet VPN 0 interface.

We’ll keep Data-Center as Primary, then click Add. Then - we’ll add a second tunnel configuration, but using Data-Center as Secondary and the interface name as ipsec2.

When all that is done, we should have the following:

If we scroll down to the bottom of the template config, we’ll have some settings for High Availability. Here, we’ll specify what our active & backup IPSec tunnels are, and their weights. I’ll specify ipsec1 as active & ipsec2 as backup. Then click Save to finish our template.

Note: Weight settings are only available starting with 20.4.1 & allow for ECMP routing to SIG. Not shown above, you can create up to four HA tunnel pairs. Depending on weighting, you can equal-cost or unequal-cost load balance between those pairs.

Why would you want to load balance across multiple tunnels? At time of writing, each IPSec tunnel is limited to a maximum 250Mb/s throughput. By creating multiple tunnels & load balancing, we can overcome this limitation if we need higher bandwidth.

2c: Attaching SIG to Device Templates

After we’ve built out our two SIG templates, we can now attach them to our device templates.

For me, I currently only have a single device template which is applied to all of my remote WAN edge devices.

So we’ll go back to Configuration > Templates and find whichever device template we want to use - then click the ellipsis on the far right, end select Edit.

In the device template, we’ll scroll down and look for our VPN 0 configuration under Transport & Management VPN. We’ll attach our SIG tunnel template to VPN 0, since that’s where those IPSec tunnels are being sourced from.

On the right side, under Additional VPN 0 Templates, we’ll click Secure Internet Gateway to add our template. Then from the drop-down, we can select the feature template we just created:

We’ll also include our SIG credentials template, which we can find at the bottom of the page, under Additional Templates:

Once we’re done, we can click Save at the bottom. This will take us through the process of pushing out these changes to our remote WAN edge devices. When this configuration is deployed, each vEdge will now reach out to Umbrella & auto-configure an IPSec tunnel.

Note: While the IPSec tunnels will be established when this configuration is pushed - no traffic will flow over them yet. We’ll need to add a service route for that, which we’ll do next!

2d: Injecting a Service Route

Now we have our IPSec tunnels deployed - all we need to do is start routing traffic out to Umbrella. We’ll accomplish this using a service route on our LAN-side VPN.

So we’ll go over to Configuration > Templates > Feature - and we’ll scroll down & find whichever template we’re currently using on our LAN side. For me, I want VPN 10 to route over the tunnels, so I’ll be editing my VPN 10 template.

Within the template, we’ll scroll down to the Service Route section, click New Service Route, and we’ll enter our desired prefix. This will be what traffic will be routed over the IPSec tunnel to Umbrella. Since this is intended to be an internet gateway, I’ll enter 0.0.0.0/0 to inject a default route to Umbrella.

Service should be pre-selected as SIG, so we’ll click Add, then Update & deploy our changes to the remote edge devices.

Step 3: Validation & Troubleshooting

Awesome! Now we should have everything configured & working. So let’s jump through some initial testing and validation we can do.

Within vManage, we can check the status of our IPSec tunnels. We’ll go over to Monitor > Network then select one of our WAN edge devices.

We’ll click on the Interfaces tab on the left side - and we should be able to see a list of all interfaces on our device. This should include an ipsec1 & ipsec2 interface.

I’ve also selected the Real Time monitor on mine, and filtered to just show traffic on the ipsec1 interface:

You might recall from my topology above, that I have a linux VM running behind each of the two vEdge Cloud appliances. Using these, I can also test web access & see what my external IP is:

And sure enough, I am getting a 146.112.x.x address - which belongs to the Umbrella datacenter.

If we log into one of our vEdge devices, we can check the routing table with show ip route vpn 10 (or whichever VPN you’re using for the LAN-side). We should see our default 0.0.0.0/0 route via ipsec1, with a next-hop-VPN of VPN0:

vEdge-01# show ip route vpn 10

     ADDRESS               PATH             PROTOCOL          NEXTHOP  NEXTHOP                         NEXTHOP
VPN  FAMILY   PREFIX       ID    PROTOCOL   SUB TYPE  METRIC  IFNAME   ADDR     TLOC IP  COLOR  ENCAP  VPN      STATUS
------------------------------------------------------------------------------------------------------------------------
10   ipv4     0.0.0.0/0    0     std-ipsec  -         0       ipsec1   -        -        -      -      0        F,S
10   ipv4     10.2.2.0/24  0     connected  -         0       ge0/2    -        -        -      -      -        F,S

We can also check specifically our SIG tunnel status using the command show secure-internet-gateway tunnels:

vEdge-01# show secure-internet-gateway tunnels

TUNNEL                                                        API   LAST
IF                                                            HTTP  SUCCESSFUL     TUNNEL
NAME    TUNNEL ID  TUNNEL NAME                     FSM STATE  CODE  REQ            STATE
-------------------------------------------------------------------------------------------
ipsec1  530233543  SITE200SYS10x10x10x235IFipsec1  st-tun-up  200   create-tunnel  -
ipsec2  530233542  SITE200SYS10x10x10x235IFipsec2  st-tun-up  200   create-tunnel  -

In that output above, we’ll see that we have both ipsec1 & ipsec2 tunnels shown. The last API queries to Umbrella have a HTTP 200, which is good. We’ll also see our tunnel names, which we can use to find our device tunnel configuration in the Umbrella dashboard.

The tunnel name might look like a mess at first, but it’s a unique identifier to represent each tunnel that was created. So if we break it down for this vEdge:

• This vEdge is at Site ID 200
  • Shown as “SITE200”
• This vEdge has a system IP of 10.10.10.235
  • Shown as “SYS10x10x10x235”
• This vEdge has two IPSec interfaces, ipsec1 & ipsec2
  • Shown as “IFipec1” and “IFipsec2”

If we jump over to the Umbrella dashboard, we’ll see the same. Within the Umbrella dashboard, we can jump to Deployments > Network Tunnels.

On this page, we should see a total of four tunnels listed (two from each vEdge appliance):

And with everything configured & validated - now we can move onto configuring firewall and web filtering policies within Umbrella!

If you’re interested in seeing how to configure Umbrella’s cloud firewall & web filtering policies - please check out my YouTube Video above!

In today’s blog post - we’ll be checking out how to integrate a Meraki MX firewall to Cisco’s Umbrella Secure Internet Gateway (SIG) & Secure Web Gateway (SWG) services.

Interested in seeing how to do this with Cisco Viptela SD-WAN? Check out this post

Note: This integration is still rather new - so I anticipate some of the screenshots & steps below may differ as this post ages.

What are SIG & SWG?

Good question! A lot of us are probably familiar with Cisco Umbrella (formerly OpenDNS) for the DNS-layer security products. Using Cisco Umbrella, we can configure our end PCs or DNS forwarders to use the Cisco DNS servers. Then, we apply security policies to allow/deny DNS queries based on our policy.

Now applying security policy at the DNS level is great! Typical web filtering only inspects HTTP/HTTPS traffic, but inspecting DNS traffic allows us to stop threats that might not use those typical ports. While there may be some applications or malware that use hard-coded IP addresses, a good majority use a domain name that requires a DNS lookup.

But what if we wanted to still do web filtering too? Or maybe we want a centralized, cloud-hosted firewall service? That’s where Umbrella SIG & SWG come in.

Secure Web Gateway (SWG) is pretty much exactly what it sounds like. It’s a cloud-hosted web filter or web proxy, where we can set URL-based policys to permit or deny access. It supports the usual allow-lists, deny-lists, HTTPS inspection, file inspection, and policies based on user/group identities.

Secure Internet Gateway (SIG) is a large, cloud-hosted firewall service. What we can do with SIG is tunnel all of our traffic to one centrallized location to apply firewall policies (permit/deny/etc).

Why would we want either of these functions to be cloud-hosted? Well we might not have the resources at every remote site to perform firewalling & web filtering - or we don’t want to invest in beefy hardware in our corporate datacenter, and backhaul all of our remote traffic back for inspection. The cloud-hosted piece also means a central management point, so only one place to make change for all of our remote sites - rather than having to touch dozens or hundreds of remote devices…..Did someone say SASE? 🙃

With all that said - Let’s get into making this work!

Note: There are many ways to use Umbrella SIG/SWG (IPSec tunnel, PAC file, Anyconnect, etc). This particular post will only cover IPSec via Meraki MX.

Step 1: Getting our Keys

First thing we’ll need to do is on the Umbrella side. We’ll need to generate tunnel keys for our Meraki MX to use for IPSec negotiation.

We’ll log into our Umbrella dashboard at https://login.umbrella.com/

Once we’re in, we’ll navigate to Deployments > Core Identities > Network Tunnels. Shown below, we currently have no tunnels configured:

In the upper right corner, let’s click the Add Tunnel button.

We’ll now see the following screen to begin creating our IPSec tunnel:

Here, we’ll need to specify a name for our tunnel & which device type. The name can be anything we choose that helps us identify what locations are using this tunnel. I’ve specified MX64 as my tunnel name, since I only have one MX that I’ll be testing this with. We also have Meraki MX as an option in the Device Type drop down - so we’ll select that as well. Then, we’ll click Next.

Then we’ll need to configure our Tunnel ID and Passphrase:

The Tunnel ID we’ll be sending later, as part of our IPSec local ID. The passphrase will be used as our pre-shared key for the tunnel config. Once we’re done with that, click Save.

As long as our ID & passphrase are all good, we’ll be presented with a box to easily copy these parameters into the Meraki config:

Finally, we’ll be dropped back to our Network Tunnels page - where we can see that our tunnel is configured, but not yet established. Let’s fix that & hop over to the Meraki dashboard!

Step 2: Meraki MX IPSec Configuration

Okay, now onto the fun stuff.

We’ll log into our Meraki Dashboard at https://account.meraki.com/secure/login/dashboard_login

Once we’re in, you’ll need to select the network with the MX you want to connect. For me, I currently only have a single network, which contains my MX firewall.

Then we’ll navigate to Security & SD-WAN > Configure > Site-to-site VPN.

Assuming we have no other VPNs configured, you’ll have to change the VPN Type to Hub (Mesh) or Spoke. In my case, Umbrella SIG will be the only VPN I’ll have configured - so I’ll go ahead and use Hub (Mesh). Don’t mind the error regarding exit hubs, as we’ll be configuring a non-Meraki peer below.

Okay, scrolling down the page just a bit - we’ll need to specify which VLANs/internal subnets will be routed across the VPN & pushed through Umbrella for SIG/SWG:

For testing purposes, I have a VM in a subnet labeled DMZ that I’ll be using - So that will be the only VLAN that I’ll set to VPN On.

Next we’ll take a look at configuring Non-Meraki VPN Peers.

As shown in the screenshot below, we’ll need to configure the following settings:

• Name - This is locally significant, I set mine to Umbrella_SIG
• IKE Version - Set this to IKEv2
• IPSec Policies - Custom - See below
• Public IP - This is the peer Umbrella Datacenter
  • Choose the DC closest to you here
  • For this post, I’m using the New York DC
• Local ID - Set this to the Tunnel ID we got from Umbrella
• Remote ID - Leave blank
• Private Subnets - This is what destination IPs will be routed over the tunnel
  • Since this is intended as an Internet gateway, we’ll use 0.0.0.0/0
• Preshared Secret - Set this to the passphrase we configured in Umbrella
• Availability - Set this to which networks this tunnel should apply
  • Since I only have 1 network & 1 MX, I set this to All Networks

Okay - I mentioned above we’ll need to set some custom IPSec parameters. Here’s what we’ll configure as a custom policy:

Note: Turns out in the drop down menu, there is also an option for “Umbrella”, so you don’t have to create a custom policy. That being said, the Umbrella preset uses DH group 5, but Umbrella’s docs ask for DH group 14.

Lastly, we’ll be able to configure whether or not we want firewall logging (I enabled this) and also whether we want to add access-lists to permit/deny any traffic over the VPN. For now I’ll be leaving this as permit any.

Finally - we can click Save to push our configuration to the MX appliance!

Step 3: Validation & Testing

Hopefully once we get all that configuration done, we’ll be able to see our tunnel come up.

We can validate from both sides, but let’s start with Meraki. In the Dashboard menu, we’ll navigate to Security & SD-WAN > Monitor > VPN Status.

Then we’ll need to click the tab for Non-Meraki peer:

With any luck, we should see the little green circle showing that our tunnel is online.

If your tunnel doesn’t come up immediately, you may have to generate some traffic from your VPN subnet to the internet. This will force the MX to try and connect to Umbrella.

It’s also worth noting - sometimes it may take the Meraki dashboard to show a successful connection. A better indicator is by checking the Umbrella dashboard, or checking manually with a device you are expecting to route over the VPN.

If for any reason we have trouble, we can also check our VPN negotiation logs by going to Network-Wide > Monitor > Event Log.

On this screen, make sure you’ve selected for security appliances at the top. If needed, we can also filter events by All Non-Meraki VPN / Client VPN.

Since my tunnel came up with no issues, the output above shows a successful tunnel negotiation.

Let’s jump over to the Umbrella side, and see what the Umbrella dashboard shows.

Back on the Network Tunnels page - we should see that our tunnel now shows as Active.

This screen will also so the public IP that our MX is using to connect, as well as which Umbrella datacenter we’ve connected to.

Okay! That’s it for now. To try and keep this blog post from getting too long, I’ve decided to split this into two parts.

If you’re interested in seeing how to configure Umbrella’s cloud firewall & web filtering policies - please check out my YouTube Video above!

This post has been on my backlog for a while… I’ve been intermittently trying out the IOS-XE guestshell feature for a few different projects, and never feeling like any of them were good enough to write about.

So why not just write about guestshell itself? It’s such a nifty tool.

What is Guestshell?

Now that Cisco’s Catalyst platform has standardized on IOS-XE, we can take advantage of some of the features of the Linux-based platform. One of which being Linux Containers (LXC).

Most of the latest Catalyst routers, switches, and access points support what Cisco calls Application Hosting - which is a fancy way of just saying it can run containers on-box. Another acronym to be aware of is IOx - which is what Cisco calls their IOS-XE/Linux appliction framework.

Guestshell is a specific feature within IOS-XE, where a dedicated pre-built container is enabled for on-box access to a Linux shell.

What Hardware Supports Guestshell?

Okay - so for this example, I’ll be using the new Catalyst 8000V platform (a virtual Catalyst router). I do also have a Catalyst 9200L on hand, but unfortunately it’s one of the few Catalyst platforms that does not support Guestshell (due to lower-end HW specs).

You can also use the Catalyst 8000V to test this in a lab, or you could use most of the Catalyst 8000 / 9000 series platforms if you had them available (or the CSRv & some ISR routers!).

You can find a list of supported platforms & other hardware requirements here.

All that being said - let’s get started!

Enabling IOx

So before we can actually use the guestshell feature, there is some pre-configuration to be done.

First we’ll need to ensure that the IOx application framework is enabled & running. We can check this by using the show iox command:

0x8KV01# show iox

IOx Infrastructure Summary:
---------------------------
IOx service (CAF)              : Not Running
IOx service (HA)               : Not Supported
IOx service (IOxman)           : Not Running
IOx service (Sec storage)      : Not Supported
Libvirtd 5.5.0                 : Running

And as we see above - most components are listed as “Not Running” or “Not Supported”.

So we’ll enter config mode, and use the iox command to enable these services:

0x8KV01# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
0x8KV01(config)# iox 
0x8KV01(config)#
0x8KV01(config)# exit
0x8KV01# show iox

IOx Infrastructure Summary:
---------------------------
IOx service (CAF)              : Running
IOx service (HA)               : Not Supported
IOx service (IOxman)           : Running
IOx service (Sec storage)      : Not Supported
Libvirtd 5.5.0                 : Running

Configuring Guestshell Networking

There’s a good chance we’ll want our container to have access to the outside world, so let’s take a quick look at the network configuration.

Note: The networking config syntax will differ if you are using a Catalyst router or a Catalyst switch. Since I am using a Catalyst 8000V - we will look at the config for a Catalyst router first.

In order to allow external access to our container, we’ll need to configure a gateway interface and NAT translations. Then we can provide a full network configuration to our container:

## Enable NAT on our external interface - in my case, gigabitEthernet1
0x8KV01(config)# interface gigabitEthernet 1
0x8KV01(config-if)# ip nat outside

## Create a virtual interface for the guestshell container
0x8KV01(config-if)# int virtualportgroup0
0x8KV01(config-if)# ip address 192.168.100.1 255.255.255.0
0x8KV01(config-if)# ip nat inside

## Create an ACL to match traffic from guestshell & Enable NAT
0x8KV01(config)# ip access-list standard IOX_NAT
0x8KV01(config-std-nacl)# permit 192.168.100.0 0.0.0.255
0x8KV01(config)# ip nat inside source list IOX_NAT interface gigabitEthernet 1 overload

## Assign network configuration to our guestshell container
0x8KV01(config)#app-hosting appid guestshell
0x8KV01(config-app-hosting)# app-vnic gateway0 virtualportgroup 0 guest-interface 0
0x8KV01(config-app-hosting-gateway0)# guest-ipaddress 192.168.100.5 netmask 255.255.255.0
0x8KV01(config-app-hosting-gateway0)# app-default-gateway 192.168.100.1 guest-interface 0
0x8KV01(config-app-hosting)# name-server0 8.8.8.8

Now, as I mentioned before - this config will look different if you’re using a switch vs a router.

So let’s take a quick look at what this would look like, if we were on a Catalyst Switch instead:

Cat9k(config)# interface AppGigabitEthernet 1/0/1
Cat9k(config-if)# switchport mode trunk

Cat9k(config)# app-hosting appid name
Cat9k(config-app-hosting)# app-vnic AppGigabitEthernet trunk
Cat9k(config-app-hosting-trunk)# vlan 10 guest-interface 0
Cat9k(config-app-hosting-vlan-access-ip))# guest-ipaddress 192.168.100.1 netmask 255.255.255.0

Cat9k(config-app-hosting)# app-default-gateway 192.168.100.1 guest-interface 0
Cat9k(config)# name-server 8.8.8.8 

Once all that config is done - Let’s move on to getting our container started!!

Managing the Guestshell Process

In order to start a container on our Catalyst device, we’ll need to go through a process of deploying the app, activating it, then starting it.

For guestshell, there is a shortcut command that will perform all of these steps for you - and make managing the guestshell process much easier.

So let’s go ahead and spin up our guestshell container with the command guestshell enable:

0x8KV01# guestshell enable
Interface will be selected if configured in app-hosting
Please wait for completion
guestshell installed successfully
Current state is: DEPLOYED
guestshell activated successfully
Current state is: ACTIVATED
guestshell started successfully
Current state is: RUNNING
Guestshell enabled successfully

0x8KV01# show app-hosting list
App id                                   State
---------------------------------------------------------
guestshell                               RUNNING

And as you can see above, everything gets automatically deployed & started. We can verify using the show app-hosting list command to see that our container has been started.

If we need to, we can also stop & deactivate our container using the command guestshell disable.

We could also use the commands below if we wanted to perform these steps manually:

0x8KV01# app-hosting activate appid guestshell
0x8KV01# app-hosting start appid guestshell

0x8KV01# app-hosting stop appid guestshell
0x8KV01# app-hosting deactivate appid guestshell

Before we move onto accessing the guestshell interface, I also wanted to show where you can get more detail about the guestshell container.

Using the show app-hosting detail appid guestshell command, we can see details around the container version, current MAC/IP config, and resource consumption:

0x8KV01# show app-hosting detail appid guestshell
App id                 : guestshell
Owner                  : iox
State                  : RUNNING
Application
  Type                 : lxc
  Name                 : GuestShell
  Version              : 3.2.0
  Description          : Cisco Systems Guest Shell XE for x86_64
  Path                 : /guestshell/:guestshell.tar
  URL Path             :
Activated profile name : custom

Resource reservation
  Memory               : 256 MB
  Disk                 : 1 MB
  CPU                  : 800 units
  CPU-percent          : 3 %
  VCPU                 : 1

Attached devices
  Type              Name               Alias
  ---------------------------------------------
  serial/shell     iox_console_shell   serial0
  serial/aux       iox_console_aux     serial1
  serial/syslog    iox_syslog          serial2
  serial/trace     iox_trace           serial3

Network interfaces
   ---------------------------------------
eth0:
   MAC address         : 52:54:dd:d:bf:da
   IPv4 address        : 192.168.100.5
   IPv6 address        : ::
   Network name        : VPG0

Port forwarding
  Table-entry  Service  Source-port  Destination-port
  ---------------------------------------------------

Accessing the Guestshell CLI

Now for the fun part! Let’s get into our guestshell container.

We’ll just need to use the guestshell command, and we’ll be dropped into the Linux shell:

0x8KV01# guestshell
[guestshell@guestshell ~]$
[guestshell@guestshell ~]$ uname -a
Linux guestshell 5.4.40 #1 SMP Tue Oct 6 17:57:00 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Let’s also make sure our network & NAT configuration worked:

[guestshell@guestshell ~]$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=64 time=3.57 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=64 time=2.64 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=64 time=2.54 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 4ms
rtt min/avg/max/mdev = 2.543/2.916/3.567/0.465 ms

If needed, we can also run commands on our Catalyst device CLI without leaving the guestshell interface by using the dohost utility:

[guestshell@guestshell ~]$ dohost "show app-hosting list"
App id                                   State
---------------------------------------------------------
guestshell                               RUNNING

[guestshell@guestshell ~]$

Just a note, the reverse is also possible. Using the guestshell run command from our Catalyst CLI, we can run commands in our container:

0x8KV01# guestshell run uname -a
Linux guestshell 5.4.40 #1 SMP Tue Oct 6 17:57:00 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

0x8KV01#

It’s also worth reiterating, that the guestshell container is a full Linux container (It runs CentOS). So if we need to install additional Linux packages, we can use our Linux package manager (YUM) to do so.

Anyways - Let’s wrap this up by looking at the built-in Python modules & capabilities

Using Python in Guestshell

One of the more appealing use cases for guestshell is the ability to run native Python scripts & code directly on your Catalyst device.

A couple of possible use cases could be:

• Troubleshoot intermittent issues - Automatically collect device state (routes, MAC changes, etc) on a regular basis to log locally or send via email
• Automated resolution - If a certain type of state change is observed, we could run a pre-set list of commands to try and automatically fix the issue
• Local monitoring / troubleshooting - We could run a series of ping, web page load tests, etc from a python script to help monitor performance from the switch’s perspective
• Change management / notification - A script could be written to monitor the device configuration for any changes made, then send an automated email when something was detected
• Have a risky change that might sever your connection to a remote device? A script could execute the change for you, and if unsuccessful, revert to a working configuration (without doing a reload after)

These are just a handful of ideas that came to mind while writing this blog post… But there are many more possibilities!

Within the guestshell container, there is a special cli python module that comes pre-loaded. This module allows us to run CLI commands on our catalyst device directly from a python script & receive the command output for parsing.

Let’s look at a quick example using the Python interactive interpreter to save the device configuration:

[guestshell@guestshell ~]$ python3
Python 3.6.8 (default, Aug 24 2020, 17:57:11)
[GCC 8.3.1 20191121 (Red Hat 8.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> import cli
>>> status = cli.execute("write memory")
>>> if "OK" in status: print("Device configuration was saved!")
...
Device configuration was saved!

The example above uses the cli.execute Python function, which will run commands in Cisco’s exec mode.

What if we wanted to make configuration changes? In that case we would use the cli.configure function:

>>> config_data = """interface loopback200
... ip address 172.16.99.10 255.255.255.0
... """
>>>
>>> cli.configure(config_data)
'Line 1 SUCCESS:  interface loopback200\nLine 2 SUCCESS:  ip address 172.16.99.10 255.255.255.0\n'
>>>
>>> cli.execute("show run interface loopback200")
'Building configuration...\nCurrent configuration : 68 bytes\n!\ninterface Loopback200\n ip address 172.16.99.10 255.255.255.0\nend\n'

In the above example, we used a multi-line comment to provide the list of commands to enter. The cli.configure function also supports a Python list - so we could provide the commands in that format as well.

You’ll notice that the Python interpreter returns a status for each command that we attempted to run. This should allow us to error-check & ensure that all of our intended configuration was accepted.

Lastly, we used the cli.execute command again to validate that our running configuration looks correct.

There’s a bit more functionality built into the cli Python module, but I just wanted to cover some of the common use cases. More details can be found here

A Few Common Questions

Does this give me access to the underlying IOS-XE Linux shell?

Nope, anything running in guestshell is running in a container - which runs on-top of the IOS-XE image.

What about resource conflicts?

The linux containers running on the IOS-XE platform are given their own dedicated resources. Any resource constraints that affect the containers will not impact normal network functions.

So if you’ve read some of my recent posts - you may have seen that I purchased a NetGear LB 1121 LTE Cellular modem to use for home internet backup.

Well - I decided to upgrade!

After using the NetGear modem for a while, I started having some issues where it would disconnect from the cellular connection intermittently. Since it’s not necessarily intended for the purpose I’m using it for, there wasn’t any good way to set up monitoring for it either.

So I opted to upgrade to a Meraki MG21. This is one of the latest additions to the Meraki family of network devices & is available with internal (MG21) and external (MG21E) antenna.

I was pretty excited to get one, since Meraki tends to have decent analytics & configuration - and they make everything so easy!

We’ll walk through the setup of the MG below - but if you’re interested in seeing what the device looks like as well, definitely check out the video above!

MG Setup - Changing the APN

Okay - so after I got the SIM card inserted into the MG, the first configuration step is making sure we have the correct APN configured. As a reminder, I’m using Google Fi as my cellular provider.

This change will need to be done on the local web management interface - not from Meraki Dashboard.

When the MG powers on, by default it will hand out DHCP addresses to any device connected to port 1. At the time of writing, these addresses were in the 192.168.5.x range.

Connecting a PC directly to the MG port, we should be able to reach the local management web page - either by typing in the IP address into our web browser, or using mg.meraki.com:

In my case, I could see that the MG had auto-detected my Carrier as Google Fi. However, it still had the incorrect APN.

Using the Configure tab, we can change that setting. You’ll be prompted for a username & password. By default, the username will be the serial number of the device (including dashes) with a blank password.

As shown in the screenshot above, we have a handful of options - though we’ll only care about APN.

First - change the Cellular Override option to Override SIM Settings. Then type in your APN. In my case, it’s h2g2 for Google Fi. Lastly, hit save at the bottom (just outside the view in the screenshot above).

With any luck, the modem will connect and you’ll see something like this:

The modem connects, gets an IP from the provider, and is able to validate connectivity to both the internet & Meraki Cloud.

When I originally set up my MG, I did run into some issues with this. My MG connected to the internet successfully, but said it couldn’t reach the Meraki Cloud. Not sure what caused it, but it shortly resolved itself within a few minutes. Just gotta be patient sometimes, I suppose!

Configuring the MG in Meraki Dashboard

Note: I won’t get into how to claim your device in dashboard or how to attach it to a network. If you need help, please check out the video above where I did show how to accomplish these steps

Okay! Now that our MG is configured for the correct cell network, we can log into the Meraki Dashboard and begin configuring it.

After we’ve added the MG to our network, we’ll see a new Cellular Gateway menu:

We’ll start first by going over to Configure > Settings

First section we’ll see is for Addressing & NAT:

As of today, the MG doesn’t support any form of direct internet pass-through. Instead, our only option is routed mode - where the MG will hold the IP provided by our Carrier & NAT any requests from the devices behind it.

We can change the DHCP subnet configuration here, which will affect what IP addresses are handed to clients behind the MG. In my case, I’m connecting this directly to a firewall as a secondary internet uplink - so the addressing & subnet doesn’t matter as much. By default, the MG will always consume the first available address as it’s own.

Next, we have a section for DHCP & subnets:

Here we can change our DHCP lease time, and what DNS servers are provided to our clients. The DNS setting does have pre-defined options for Umbrella DNS, Google DNS, or using whatever the upstream carrier provides. You’re also welcome to manually specify which DNS servers to use.

We can also configure reserved & fixed IP addresses here.

Reserved IP ranges are IP addresses that we don’t want the MG to provide via DHCP. So if we had any statically configured IP addresses, we could reserve them here.

Fixed IP addresses are for any client that needs a DHCP address, but we want that IP assignment to be permanent. We’ll enter the client name & MAC Address here, as well as the IP we want assigned to that device. In my case, I went ahead and inserted my firewall MAC address - and I’ll just allow the firewall to get its IP via DHCP from the MG.

By default, the MG will block all inbound traffic from the cellular network. If we need to allow any traffic inbound, we can change the Port Forwarding settings:

This allows for a light configuration of an inbound NAT. Right now, I probably won’t be using this. However, I may permit VPN access into my network via the MG at a later date.

If I needed to add anything here, the MG allows us to translate an external / public IP & port to any internal IP / port combination. It appears we can even add a IP filter to permit only trusted source addresses.

Lastly - We can configure settings for Traffic Shaping:

In this section, we can throttle our cellular throughput & configure uplink monitoring.

By default, the cell bandwidth is set to unlimited - but we can drop this down if needed. In my case, I am not using an unlimited cell data plan - so I will throttle cellular speeds to preserve data & reduce charges.

In addition, we can configure one or more IP addresses to check uplink connectivity. These addresses will be used to collect loss & latency data via the cellular connection. The MG monitoring dashboard will collect & graph this data for easy insight into the performance metrics.

Note: As a word of warning, these uplink monitors are constantly sending ICMP/ping requests. If you have a limited amount of cellular data, this may consume more data than you would like. In my testing, using only one IP for uplink monitoring consumed about 70-100M per day. More on this below…

Monitoring the MG

Now we get to the good stuff! The primary reason I opted to buy an MG was for monitoring & analytics.

Back on the dashboard, if we use the lefthand menu - we’ll go over to Cellular Gateway > Monitor > Cellular Gatways. Then select our MG out of the list.

The primary summary page isn’t too exciting:

The MG does have two gigabit ethernet ports - and we’ll see the status here.

We’ll also see the connectivity history to the Meraki cloud - which in my case is nearly 100%. Seems like one very minor blip just after 4am.

We can also see the current network utilization on the MG. This is great to have - though my current utilization is pretty low… (I am using this as a backup modem, after all).

On the left side of the page, we’ll see some of the usual info we expect from a Meraki device. Current IP, location, Serial number, and IMEI. Just below the view of the screenshot, there is also an indicator for firmware version.

Onto the Uplink tab! Let’s see what we have:

First we’ll see the Configuration section. This just gives us a quick view into what settings the MG currently has.

We’ll see the current IP info provided by our carrier, and also some statistics on our cellular connection.

Just below that info, we’ll see our cellular graphs:

This is what I wanted! It’s great to see a quick view into what our active uplink traffic is - as well as look back historically at what our LTE signal quality has been.

Not pictured here - but there is also a section of graphs on this page for our uplink monitor. This is where we can see our current & historical loss & latency stats for the cellular connection. After a few days of use - I disabled the uplink monitor due to the amount of data the feature consumes.

Finally, we also have the DHCP tab:

This will show our current DHCP subnet & any clients that have been provided an address. In my case, there isn’t any current leases here - because my firewall has a fixed IP.

Performance && Considerations

I’ve had the MG running for about a week now, and wanted to provide some things to think about.

First - How does the modem perform? Well, the first day I had it set up - I was able to get ~150M download speeds using the MG’s built-in speed test utility:

That being said - I’m lucky if I get 30-50M on an average day. I might have just gotten lucky that day with some light cellular utilization in my area. Overall though, I’m pleased with the speeds I get - they’ll certainly fit my needs.

For the few days that I had the uplink monitoring running, I saw good results. Usually 0% packet loss, with a rare spike of 5-10%. Latency was a little less reliable, but usually bounced between 50-150ms. This was also much less than the NetGear modem I had been using, which averaged 200-250ms.

Speaking of uplink tests! Let’s talk about data usage….

By default, the MG communicates intermittently with the Meraki cloud - which consumes some data. By my measurement, this is usually less than 10Mb/day. No problem here.

The uplink tests, on the other hand, do consume a bit of data. I’m not sure what frequency these run on, but it’s fairly often. Even with one uplink monitor to 8.8.8.8 configured, I was seeing data usage of 70-100Mb a day.

While seeing those metrics is valuable to me, it’s also not worth the data charges. If I was using a SIM card with an unlimited data plan - no doubt I would keep this feature enabled. However, since I am paying for the cell data used - I opted to disable this feature.

The MG does still perform it’s check-ins to the Meraki Cloud - so you’ll have availability statistics & monitoring… But disabling the uplink monitor means you’ll lose the granular data on loss & latency.

Lastly, and another word of warning, when you’re actively viewing the MG monitoring page - this also consumes additional data. To demonstrate - I’ll post a snippet of the screenshot from earlier:

If you notice, all the way on the far left there was barely any activity. However, once I loaded the MG monitoring page - you begin to see minor spikes in data usage as the Meraki Cloud starts actively polling the MG for data.

In my experience so far, this isn’t a ton of data. I’ve checked in to see how the MG has been performing a few times this week, and each time has totaled around 10-15Mb of data usage.

To sum up - I’m cheap and want to avoid excess data usage. Just wanted to provide some of that info as something to be aware of.

Final Thoughts

I’m only a week in, but pretty pleased with the MG’s performance. It’s maintained a very solid & stable connection compared to the NetGear modem it replaced. The device is intended to provide LTE connectivity or backup service for business networks, so I would certainly hope it would meet my home needs :)

Outside of that, I do wish there was a little better documentation & clarity from the Meraki team on data usage. Right now their documentation only mentions the 6-8Mb of usage due to backend data to/from the Meraki Cloud:

I would be happy to see additional settings on the uplink monitor to allow me to choose the polling frequency. I feel like throttling down the amount of requests could reduce data to a point where I would be comfortable re-enabling that feature.

Oh - and currently there are no native email alerts for the MG. So if the MG goes offline, etc… there is no alerting from the Meraki Dashboard. This kinda sucks. I’m sure this is coming soon, but for the time being I’m inclined to write my own monitor using the Dashboard APIs. (see note below!)

Overall, I’m happy with the device. Definitely looking forward to future feature & firmware updates to see where the Meraki team takes this platform!

Update 08/28/2020 - Looks like in the week since I posted this, Meraki added alerting for the MG! Now you can be notified if the cell gateway goes offline:

In a recent post, I walked through setting up a Netgear LTE cellular modem with Google Fi. Might seem random - but I had a plan for this!

In trying to ensure I keep a stable internet connection for work, I eventually led myself down the path of buying a cellular modem. That was part one. The next step was being able to somewhat-intelligently monitor my home internet connection, and then fail over to the cell modem when connectivity was poor (think lazy SDWAN).

At first I figured this would be easy…. but of course nothing is :)

I currently have a Cisco FirePower 1010 appliance that I’m using for a home firewall. Unfortunately, while this thing does support IP SLA - It wouldn’t quite accomplish what I wanted to do. And to be fair, this box is intended to be a security appliance - not SDWAN.

Anyways - I talked myself into just writing some Python automation to monitor my home internet, and dynamically inject/remove static routing entries. I needed a new project anyways

For those of you wanting to jump straight to the code, the GitHub repo is here

Part 1 - Monitoring Packet Loss & Latency

So first thing - I regularly experience both complete internet outages (always at the worst times), as well as very high latency. Packet loss seems to be less common with my ISP - but hey, it happens sometimes too.

I opted to use the pythonping module as an easy way to collect some of the info I needed.

After importing the module, it’s as easy as specifying a destination, packet size, and number of ICMP messages to send:

from pythonping import ping
result = ping('8.8.8.8', size=2, count=10, verbose=True)

To simplify at least one part of the operation, pythonping has a built in function to return the average round trip:

>>> print(result.rtt_avg_ms)
74.23

The actual contents of our ping results are going to be in the following format: Reply from 8.8.8.8, 10 bytes in 43.82ms

So a quick way for me to figure out packet loss was to simply search for the “Reply from” text in each response:

lost = 0
for packet in result:
    if "Reply from" in str(packet):
        pass
    else:
        lost += 1

Easy enough - and we can use that to calculate the amount of packet loss against the 10 total packets sent:

lossperc = 100 * lost / 10

Once all that is figured out, we can use a simple expression to determine whether or not the loss or latency thresholds have been exceeded:

if response >= MAX_LATENCY or loss >= MAX_LOSS:
    print("Loss/Latency violate thresholds.")
    return True
else:
    print("Loss/Latency within thresholds.")
    return False

The code I wrote then takes one of two paths.

• If the loss and latency is ABOVE our thresholds, call to the FirePower module to inject a static default route over the LTE modem
• If the loss and latency is BELOW our thresholds, call to the FirePower module to delete the static default route - which returns traffic to the primary internet

These functions within the FirePower module were written so that each change will only be made once. For example, if the script checks and finds that the loss/latency is bad, but the static route towards the LTE modem already exists - then no additional changes are made.

So next - Let’s take a look at the FirePower module.

Part 2 - Authenticating to FDM & Initial Checks

This was my first time getting into the FirePower FDM APIs - but they ended up being fairly straightforward to use.

Turns out that FDM has built-in API documentation, which is extremely helpful. This can be found at https://<FDM IP>/#/api-explorer

Okay - So first thing’s first. Authenticating to the firewall:

oauth_data = {
    "grant_type": "password",
    "username": "admin",
    "password": "Cisco1234!"
}
headers = {
          "Content-Type": "application/json",
          "Accept": "application/json"
}
baseurl = "https://" + FDM + "/api/fdm/latest"

authurl = baseurl + "/fdm/token"
print("Posting AUTH request to FDM")
# POST request to FDM with headers / oauth data
resp = self.s.post(authurl, headers=headers,
                   data=json.dumps(oauth_data),
                   verify=False)
# If success - only pull out & return the auth token
if resp.status_code == 200:
    print("Auth success - got token.")
    return json.loads(resp.text)['access_token']
else:
    print("Authentication Failed.")
    print(resp.text)

In the code above - we take the headers and some basic authentication info (username, password, grant type) and send it in an HTTP POST request to https://<FDM IP>/api/fdm/latest/fdm/token

This returns an authentication token, which we’ll need to include in any further HTTP request to the firewall. This can be done by sending a new set of headers in the future:

headers = {
          "Content-Type": "application/json",
          "Accept": "application/json",
          "Authorization": "Bearer " + self.token
}

Next we need to figure out which routing table to insert the route into. Since I am only using the default routing table, the script will just grab the UID for the default global routing table:

vr_url = baseurl + "/devices/default/routing/virtualrouters"
routing_table = requests.get(vr_url, headers=headers)

I mentioned earlier that the script will not make any changes if the firewall is already in the desired state. So we will take time now to check what state the firewall is in, before attempting to make any changes.

This is accomplished by making our first primary action scanning the routing table to see if our route already exists:

route_url = baseurl + "/devices/default/routing/virtualrouters/" + \
            self.globalVR + "/staticrouteentries"
route_data = requests.get(route_url, headers=headers)

# Convert returned JSON object
current_routes = json.loads(route_data)['items']
# Run through each route returned and see if it matches the one we're looking for
for route in current_routes:
    # For each route, we need to manually go look up the uid of our gateway & network objects
    gateway = self.getNetworkObject(route['gateway']['id'])
    dest_network = self.getNetworkObject(route['networks'][0]['id']).split('/')[0]
    # Match based on route prefix & upstream next hop gateway
    if gateway == GATEWAY and dest_network == ROUTE.split('/')[0]:
        print("Found route to %s via %s" % (dest_network, gateway))
        return route

Depending on whether we were looking to add or remove a route, the script may proceed with doing so - or just quit if the action has already been taken previously.

Part 3 - Failover (Add Static Route)

Assuming that we need to Add a route, we need to sent a POST request to https://<FDM IP>/api/fdm/latest/devices/default/routing/virtualrouters/<Virtual Router ID>/staticrouteentries with some required information (like subnet info, next-hop, etc)

Unfortunately, we can’t just send a POST with the target subnet & gateway. Those need to be created as network objects on the FirePower box beforehand. For each network object, we need to build a quick JSON object with the required info:

host_data = {}
host_data['name'] = name
host_data['description'] = "Created by ISP Failover automation"
host_data['subType'] = subtype
host_data['value'] = address
host_data['type'] = "networkobject"

Then we can send a POST to the FDM API to create the object with our supplied parameters:

posturl = baseurl + "/object/networks"
requests.post(posturl, headers=headers, json=host_data)

Once we have those objects created, we can compile all of that info into a JSON object with all the components needed to create a static route entry:

routeobject = {}
routeobject['name'] = "route_BACKUP"
routeobject['description'] = "Created by ISP Failover automation"
routeobject['iface'] = {}
routeobject['iface']['id'] = iface_ID
routeobject['iface']['type'] = "physicalinterface"
routeobject['iface']['name'] = iface_name
routeobject['networks'] = [{}]
routeobject['networks'][0] = {}
routeobject['networks'][0]['id'] = network_ID
routeobject['networks'][0]['type'] = "networkobject"
routeobject['networks'][0]['name'] = network_name
routeobject['gateway'] = {}
routeobject['gateway']['id'] = gateway_ID
routeobject['gateway']['type'] = "networkobject"
routeobject['gateway']['name'] = gateway_name
routeobject['metricValue'] = 1
routeobject['ipType'] = "IPv4"
routeobject['type'] = "staticrouteentry"

Then pass all that into a POST request to create the route object:

add_url = baseurl + "/devices/default/routing/virtualrouters/" + self.globalVR + "/staticrouteentries
requests.post(add_url, headers=headers, json=routeobject)

Success? Well not quite yet.

FirePower requires all changes to be deployed (or applied) to the system before they take effect.

So next we need to initiate a deployment task - and periodically check on it. Deployments can take a while to complete, depending on number of changes, processing power of the appliance, etc.

# First send a POST to the deployment endpoint:
deploy_url = baseurl + "/operational/deploy"
deploy_response = requests.post(deploy_url, headers=headers)
# Grab the deployment task UID:
deploymentID = json.loads(deploy_response.text)['id']

# Loop while deployment is running:
while deployed is False:
    # Sleep for a few seconds:
    sleep(5)
    # Get list of all deployment tasks:
    tasklist = json.loads(requests.get(deploy_url).text)
    for task in taskList['items']:
        if task['id'] == deploymentID and task['state'] == 'DEPLOYED':
            print("Deployment status is: " + task['state'])
            deployed = True
            return True
        elif task['id'] == deploymentID and task['state'] != 'DEPLOYED':
            # If changes not yet deployed, check again momentarily
            print("Deployment status is: " + task['state'])
            deployed = False

So in the above block, we POST a request to deploy changes. The response of that POST request contains our deployment ID, which we keep to check the status later.

Then the script waits a few seconds, and pulls a list of all deployment tasks. Search through that list for our deployment ID - and check to see if it has been completed yet. If not, wait and run the loop again.

And that’s it! Once we get confirmation that our changes have been deployed - we’ve now routed traffic over the secondary connection (an LTE modem, in my case).

Part 4 - Fail-Back (Delete Static Route)

Okay - this section will be fairly quick. We’ve already looked at authenticating, checking if our route already exists, and how to add a new route. So the only thing remaining is removing our route if we wanted to migrate traffic back to the primary connection.

Same logic applies as earlier. If our path monitoring script runs and finds that loss & latency are within the thresholds we set, then we make a call to the firepower module. First we check to ensure there is a route for us to delete, and if so - proceed with deleting it.

Adding a route is a little more work, since we may need to create network objects. However, when we delete the route - we’ll just leave those objects on the FirePower box. They’ll be there for the next time we need them (which also speeds up deployment times).

To get started, we just need the UID for the route we want to delete.

Remember that code I showed above, where we check to see if the static route exists? And match on our intended subnet / gateway pair? Well, I just made that into a function that will return the UID of the route if it finds it. Easy enough!

Next, we just send an HTTP DELETE to the static routing endpoint - and reference that UID:

del_url = baseurl + "/devices/default/routing/virtualrouters/" + \
          self.globalVR + "/staticrouteentries/" + route['id']
requests.delete(del_url, headers=headers)

Once that succeeds - we use the same logic as earlier to deploy the changes.

After that, our route is removed & traffic should be flowing over our primary connection again.

If it helps anyone - I also threw together a quick diagram to explain the flow of operations here:

Well - that’s it. This was a fun side project to work on over the past week or two. I hadn’t spent much time with the FirePower/FDM APIs just yet. While there was a bit of work in creating all the necessary network objects, the overall process was fairly simple.

It helped tremendously that the FDM API explorer exists, and is available on-box. This utility allows you to see all the available API calls, what parameters they require, and even run test calls from the web UI. This greatly reduced the time needed to figure this out.

Hope this was interesting. If you would like to see the whole project - check out the repo on my GitHub page.

^{Note: I may receive commissions for purchases made through links in this post. This is to help support my blog and does not have any impact on my recommendations.}

It’s been a crazy couple of months recently. With what began as a temporary lockdown & mandatory work-from-home has now become a question of “Will we ever return to the office”?

Luckily I have a decent home office setup, as my job is already primarily work from home. That being said - it’s more important than ever to have a stable, reliable internet connection available. Can’t run the risk of being in an important meeting with a customer and drop from the video conference (or have audio issues, etc).

Needless to say, my internet provider is far from perfect. While it does perform reasonably well most days, the spikes in latency & packet loss have increased substantially with everyone in the neighborhood working from home.

I’ve been debating for a while on whether or not to pick up a cellular modem. Originally I had planned on having one handy for a complete internet outage (which happens). But more recently I began to wonder if it might be worth it for routing around my primary provider when the connectivity is poor.

To accomplish this, I decided to use Google Fi. I’ve already used the service as my cellular provider for a number of years now and I’ve been extremely pleased with the service. One of the awesome benefits they offer for subscribers is free data-only SIM cards. So I could pay nothing for the SIM card itself (no plan/recurring costs either) - then only pay for data as I use it.

First, I went ahead and tested what typical latency & performance was over my cell connection. Occasionally my home internet provider would see latency spikes up to 1200-1800ms. Checking the cell connection - it seemed pretty stable in the 200-300ms range. Still pretty high, but much more bearable than ~1200ms. Seemed like it should work reasonably well for what I wanted.

Then, I had to double check compatability. Google Fi runs over multiple carriers on the backend - as long as you’re using a Fi-compatible device. If not, it seems they default to only using T-Mobile’s network. So in that case, Fi requires that our device supports LTE bands 2 & 4 at a minimum. The Netgear LB1120/1121 modems do, so we’re good there.

Next, I went ahead and ordered a NetGear LB1121 LTE Cellular modem. I opted for the power over ethernet model, since I already had a Cat5 run over to where the modem would be placed. The switch I’m using is also being backed up by a UPS, so in the event of a power outage the LTE modem would stay online.

Finally - I placed my order for the Google Fi SIM card. Easy process - $0 for the SIM, $0 for shipping, and it arrived in only a few days. As an additional note - the modem requires a micro-SIM, but Fi provides nano-SIM cards. I also had to pick up a cheap adapter kit to make this work.

Getting the Modem Set up

Once the modem & SIM card arrive, it’s time to get started. This will be a fairly quick and straightforward process.

On the bottom of the LTE modem, the device will list it’s default IP address & the password to log in. Take note of that, as you’ll need it later.

Also on the bottom is the slot to insert the SIM card. Remove the cover, insert your SIM card into the micro-SIM adapter, then insert into the cellular modem.

Next, go ahead and power on the modem & get it plugged into a PC.

Once the modem boots up - we should be able to reach it via a web browser @ 192.168.5.1.

Log in with the credentials from the bottom of the modem.

Upon landing on the main page, you may see that the modem auto-connects to T-Mobile’s network. While we might be able to use this as-is, ideally we need to change our connection to specifically use Google Fi’s information.

Head over to Settings > General > APN:

Click Add to create a new cellular profile

For the network Name, enter Google Fi. For the APN enter h2g2.

Click Save to add the profile

Then make sure to select the correct radio button to activate the new profile:

At this point, I would recommend deleting the old/default T-Mobile profile. I would also suggest rebooting the device to force it to switch to Google Fi.

If all is well - You’ll log back in and see that the modem has now connected to the appropriate network:

Note: I’ve had some issues where the carrier name doesn’t display correctly. It may take a few minutes to finally display “Google Fi”. However, at this point you should be connected to the correct network and be able to use it with no issues

Additional Settings

One other thing that may be worth noting. Depending on how you’re using the modem, you may want to change the way the modem operates.

Let’s go ahead and check out a few options on the LAN settings page. This can be found by going to Settings > Advanced > LAN.

By default, the modem operates in router mode. What this means is that the modem terminates the connection from Google Fi - and acts as a gateway between your devices & the carrier network.

You would want to use router mode if you have multiple devices you want to use with the modem & you do not have an additional router or firewall. In this case, the modem will handle providing IP addresses to the client devices. In addition, it will act as a firewall and allow client devices to the internet, while blocking any external devices from accessing the clients directly.

The other available mode is bridge mode. In this case, the modem operates as a pass-through device & assumes that there is another device behind the modem that is providing the routing/firewalling functionality.

You may want to use bridge mode if you already have a router or firewall you want to use - and you want to treat the modem as a second internet connection. When the modem connects to the provider, it will automatically pass the public IP address & configuration straight through to whatever device is plugged into it.

If you’re interested in how I configured the auto-failover to the cellular modem - check back in a bit. I wrote some custom automation to monitor my primary connection & conditionally inject a route over the LTE modem.

I’m just wrapping up some of that, and should have something posted here shortly!

Hey there! First thing’s first - Hope all is well with everyone. The past few months have seen all sorts of craziness going on in the world. I’ve been lucky in the sense that my current job role was already well set up to make an easy transition to working from home all the time. That being said, I know it hasn’t been easy for everyone - and I know quite a few people who have lost their jobs, etc.

Second thing! As I’m sure you may have already seen above - I opted to spend some of my new-found free time trying out a new format. I’ve had a couple of ideas in the past for making short videos, but never forced myself to sit down and give it a shot. So here we are - after about a month of on-and-off work - I finally have something to share. Please give it a look if you’re interested, and I would appreciate any comments & feedback. I still have a few other ideas - so if this one goes well I may pursue producing a few more of these.

Okay - Now onto the real content!

A few months ago I had the chance to pick up a pair of Cisco Catalyst 9100 series access points from work. My home network has been running on Ubiquiti APs for years - but they’re getting old and I desperately needed to replace them. So along comes the Catalyst 9100 APs, which are capable of supporting 802.11ax clients - and why not take the time to upgrade & future proof? I needed some practice anyways, as I have a few customers at work that I think will be interested in these in the near future. I ended up with two 9120AXI APs.

So in the process of trying to dive on the opportunity to play with something new - I completely missed the fact that the 9100 APs have two different product SKUs. One which ships the AP pre-configured with the standard lightweight AP code that you might use if you had an external wireless LAN controller (WLC). And a second SKU that ships the AP with the Catalyst 9800 Embedded WLC (eWLC) software loaded. Of course, I grabbed the lightweight APs without checking.

Product SKUs (Example, using 9120AXI):
 - Standard Lightweight AP: C9120AXI
 - Pre-load with Embedded WLC: C9120AXI-EWC

I wrote in a previous blog about getting MAC address filtering set up on the Catalyst 9800 WLC. When I wrote that blog I was actually using the C9800-CL, which is a virtual machine version of the Catalyst 9800 controller software. I was originally excited to run the controller as a VM and not need a hardware appliance - but then got to thinking that maybe I should try and save VM resources as well. Which led me to looking at the 9800 eWLC.

Finding the Software Image

In order to convert an existing lightweight access point to one running the embedded WLC software - we first need to grab a copy of the software images from Cisco.com. Search for the model of your 9100 access point (in my case, the 9120AXI).

You’ll see two options for Software Type - and it may not immediately be obvious - but we’ll need to look under IOS XE Software to find the eWLC images:

Then we’ll grab the EWC AP image bundle. In my case, I was waiting on a specific feature set that wasn’t available until 17.x - so I downloaded the 17.2.1 software:

Okay - after we’ve done that - let’s drop everything onto a server/laptop/etc which has console connectivity to our AP and a running TFTP server.

Once you un-zip the contents of that AP image bundle, you’ll notice there are quite a number of files. We’ll only need two of them - one image to load onto the AP itself, and one that will get loaded for the WLC software container. Within the image bundle, there will be a readme.txt file that will tell you which image to use with your AP:

ap1g5 : AP1815, AP154x
ap1g4 : AP180x, AP183x, AP185x
ap1g7 : C9115, C9120
ap1g6 : C9117
ap1g6a : C9130
ap3g3 : AP380x, AP280x, AP156x

So in my case, I would use the file named ap1g7 since I had the 9120 APs. In addition, you should also see a file named C9800-AP-iosxe-wlc.binwhich we’ll need to load the controller.

What’s with that second image? Well - the Catalyst 9100 APs include a feature called Application Hosting (also found on some of the Catalyst 9000 series switches). This is equivalent to running a Linux container or Docker container directly on the AP hardware. At time of writing, this is only available for the embedded WLC software. However, there will be a future software update that will allow you to provision other software containers as well (according to the data sheet).

Converting the Access Point

Now we can get started on the actual conversion process. If it’s a new AP, we can just go ahead and boot it up with a console cable connected. If it’s already running something, you’ll likely want to factory reset the AP first. You can find detailed instructions here, but a quick summary - power off the AP, hold the mode button while plugging the AP back in, and keep the mode button held for at least 20-30 seconds. If you’re logged into the console during this time - you’ll actually see the AP counting how long you’ve held the button for. Once that counter shows at least 20 seconds, release the modebutton and allow the AP to reboot.

Once the AP is booted. The default login is Cisco / Cisco. The enable password is also Cisco. You may also want to quickly issue a show version, and take note of the current software version running on the AP (hint: we’ll need that later).

Next - it’s important to note that after we convert the AP - our default CLI will actually be the WLC, not the AP (even when connected via console). So we should configure a hostname and IP address for the AP now:

0xAP# capwap ap hostname <hostname>
0xAP# capwap ap ip <ip-addr> <netmask> <gateway>

Then we can load the new AP image and WLC image. Depending on which software version you’re running today, there is a different command to load the image:

0xAP# ! If you're running software 8.9 or lower, use the following:
0xAP# ap-type mobility-express <TFTP path to AP image> <TFTP path to WLC image>
0xAP# ! If you're running anything above 8.9, use the following:
0xAP# ap-type ewc-ap <TFTP path to AP image> <TFTP path to WLC image>

Once those commands are submitted, the AP will begin copying the software from the TFTP server. The AP will automatically reboot after the image load is completed.

Quick note: For one of my APs, the WLC software didn’t load correctly - and when my AP rebooted I was left with an error that said “EWC-AP in Recovery Mode”. Re-copying the WLC image fixed it without much trouble. If this happens, the AP will print out the command to re-image the WLC software, but I’ll also put it here for reference:

0xAP# ! If you end up in "EWC-AP Recovery Mode", run the following:
0xAP# archive download-sw ewc-ap <TFTP path to WLC image>

Okay - Once our AP has come back from loading up all the new software, we can get on with a bit of minimal config required to access the WLC web UI.

First, we’ll configure the WLC hostname & our local user account. This can be switched over later to your choice of directory service:

0xC9800eWLC# config t
0xC9800eWLC(config)# hostname <hostname>
0xC9800eWLC(config)# user-name <admin username>
0xC9800eWLC(config-user-name)# password <admin password>
0xC9800eWLC(config-user-name)# privilege 15

Next we’ll also provide the WLC with the administrative credentials used to manage the connected APs:

0xC9800eWLC(config)# ap profile <profile-name>
0xC9800eWLC(config-ap-profile)# mgmtuser username <AP admin user> password 0 <AP admin password> secret 0 <AP admin secret>

After that, we’ll go ahead and configure our basic network settings. This will be the IP address & gateway info that will be used to connect to the WLC web UI and SSH:

0xC9800eWLC(config)# interface gigabit 0 
0xC9800eWLC(config-if)# ip address <managemnt IP> <Network mask>
0xC9800eWLC(config-if)# no shut
0xC9800eWLC(config)# ip default-gateway <Gateway IP address>

Lastly - we’ll need to actually enable the web server. In my case, I opted to not enable the standard HTTP server. I only enabled the encrypted SSL-enabled web server:

0xC9800eWLC(config)# ! I only enabled the HTTPS server:
0xC9800eWLC(config)# ip http secure-server
0xC9800eWLC(config)# ! If you wanted to enable the plain-text / unencrypted web server:
0xC9800eWLC(config)# ip http server
0xC9800eWLC(config)# 
0xC9800eWLC(config)# ! Finally - Save the config
0xC9800eWLC(config)# exit
0xC9800eWLC# wr mem

When we save our config for this first time, the AP will verify that the initial configuration pieces have been completed. There is a bit of background cleanup that is performed - which removes any factory config that assisted with provisioning. Once that’s all done - we’re free to log into the web UI!

Embedded WLC Web Interface

I won’t spend a lot of time here, as there are far too many things to cover with the new 9800 series controllers. However, I did want to point out a few things that stand out regarding the Embedded WLC.

When we log into our WLC web UI - we’ll be able to use the username & password combo that we configured previously. Then we’ll be dropped into our WLC dashboard.

*Note: Screenshot taken a bit after I did my initial config. This has been running a bit now…

If we click on the number under the Access Points heading, we’ll be taken to a quick monitoring view of the current connected APs:

Here we can see the APs we have configured, along with their IP address & status info. We will also see what our Current Active WLC controller is, and what our Current Standby / Preferred Active if we have either configured.

By default, if we boot up a second AP running the embedded WLC software, it will automatically join the WLC cluster as the secondary node. Any WLC config/settings will be copied, then it’s ready in case the primary controller fails. In the event of a failure, the secondary WLC will take over - and be accessible using the same IP & login info that we used on the primary.

In my case, I’ve also configured one of my APs as a preferred primary controller. For me, this AP is connected to a switch that has a short battery backup - so it’s less likely to experience failure. This option can be configured on the individual AP itself. In the left-hand menu, drop into the Configuration section, then click on Access Points under the Wireless header. We’ll see a very similar screen to the AP monitoring screen from above. Click on the AP that we want to make our preferred primary, then jump over to the Advanced tab. There will be a checkbox for Preferred Controller:

WLC Image Repository

Last thing - and this one is important. Normally the WLC will have dedicated storage to keep AP software images, which it distributes to new APs when they come online. Unfortunately, while we gain the flexibility and limited footprint of the embedded WLC - we also lose that dedicated storage space. As you might imagine, storage on the AP is limited.

So we’ll need to make sure that the WLC has an external image repository that it can use. This configuration can be done via teh web UI and CLI. I’ll cover both very quickly here.

From the web UI - We’ll go to Administration > Software Management:

Here we’ll see our options for where to get images, as well as an inventory of current APs & their images. In the Mode drop-down, we’ll have the option of tftp, sftp, CCO, and Desktop. TFTP / SFTP are what we’re used to. With Desktop, we would just upload images directly from our laptop / PC. Interestingly though, CCO allows us to provide our Cisco.com credentials - and the controller can pull images directly from Cisco. We’ll even have the option to enable automatic software downloads - and specify whether we want to auto-download the latest software release, or only the latest recommended release.

On the CLI side of things - we can accomplish the same using the following commands on our WLC CLI:

0xC9800eWLC(config)# wireless profile image-download <profile name>
0xC9800eWLC(config-wireless-image-download-profile)# image-download mode tftp
0xC9800eWLC(config-wireless-image-download-profile-tftp)# tftp-image-server <IP address>
0xC9800eWLC(config-wireless-image-download-profile-tftp)# tftp-image-path <path to image files>

With that, any new APs that join our WLC should be able to auto-load the software necessary.

Thanks for reading!

Update 2020 / 05 / 19 - I’ve added a video above that walks through the steps detailed in this blog post.

If you’re using the ‘Basic’ Wireless setup, you may see an error when trying to apply the policy: “switch 1 dbm wireless Use of default ACL preauth v4 is not permitted”

If you come across this error, it’s a known bug (CSCvt18875) specific to only the ‘Basic’ setup wizard (which is what I used in this post below). If so, check out the video above which walks through the ‘Advanced’ setup and bypasses this error.

I’ve been spending a bit of time over the past few weeks building up a wireless lab. Trying to get a good understanding of how the new Catalyst 9800 wireless controller works, and how it differs from some of the previous iterations.

In order to play around with the new controller, I decided to try to build a new configuration that mimics my current home wireless. Today I am using Ubiquiti APs, which come with their own free controller software. Most of my current config is fairly straightforward - a few SSIDs, two APs, and a guest network with captive portal. One of my SSIDs is dedicated to any IoT devices and is more restrictive than the other networks. This network uses both a pre-shared key for authentication as well as MAC-based filtering.

In this post - we’ll walk through how to set up a new SSID with client MAC filtering.

Note: This was written using Catalyst 9800-CL version 16.12.1s. APs are configured in flexconnect with local authentication (no AAA, ISE, etc)

While this post is not focused on in-depth WLAN config, we will start by quickly setting up a new network.

Once you get logged into the controller - we’ll click on the Wireless Setup icon in the upper right-hand side. Then drop down to the Basic option:

This takes us through a pretty quick and easy wizard to set up our new location & wireless networks. At first, we will have no locations configured - so we will click Add:

We’ll start off building our location by giving it a name and description. These are used for some naming of policy objects within the WLC, so make sure to use a name that makes sense.

In my case, I’m also going with a flexconnect deployment - so we’ll select that option and also provide the AP native VLAN.

Next, we’ll click over to the Wireless Networks tab. This is where we will create our WLAN and apply the initial configuration. We don’t have any networks yet, so click Add

The two primary things we need to address are highlighted in red below. We need to create a WLAN and assign it to a VLAN. Let’s start with the WLAN by clicking Define new.

On the General tab, we’ll give this WLAN a profile name and a SSID.

Next, we’ll hop on over to the Security tab, and focus on the Layer 2 sub-tab. The first thing I want to point out - is that at this point, we will not be enabling the MAC Filtering checkbox. We’ll need some additional config first, then come back to this later.

Scroll down to the bottom of the window and there will be some settings for your authentication. By default, 802.1x will be enabled. This post won’t cover how to setup/configure that. Instead, we’ll be deselecting 802.1x and checking the box for PSK. Then a text field will appear for us to enter the Pre-Shared Key.

Once we click Apply to Device, we’ll finish up by assigning our VLAN or VLAN Group. In this case, I have already created a VLAN named IoT. If you haven’t created a VLAN yet, you can do so by going to Configuration > Layer 2 > VLAN. Then add a new VLAN under the VLANtab.

Click Add, then we’ll be back to the wizard and see the new WLAN we just created.

In order to finish up with the wizard, we just need to assign our Access Points. Click on the AP Provisioning tab. If you have already configured APs to join to this controller, you will see them on the left side under Available APs. Check which ones to apply this WLAN to, then click the arrow to move them to APs on this location.

Click Apply, and that will finalize all of the configuration we just did - then drop us back to the Wireless Setup page.

Okay - Now that we have that completed, we can move onto creating our MAC filtering policies.

Back in the menu - Let’s go to Configuration > Security > AAA

In this section - we first need to create an Authorization policy. Select the AAA Method List tab, then Authorization, then Add to create the new policy.

In here we’ll specify a name, then select Type: network, and Group Type: local. Then go ahead and Apply to Device

Once we have that, let’s go over to the AAA Advanced tab, and click Add in the Attribute List Name section.

Here we need to provide the SSID we want our MAC policy to apply to. Under Attribute Type, select SSID. Then under Attribute Value, select the target SSID that our policy will be tied to.

Don’t forget to click Save on the attribute before clicking Apply to Device!

Time to input our list of device MAC addresses! Drop into the Device Authentication section, and click Add - or upload a CSV file if you have one.

Input the device MAC Addrees and select the Attribute List Name that we configured just a minute ago. Then Apply to Device

After that - we should have our completed list of MAC addresses which will be permitted to join our wireless network. All we need to do is go back to our WLAN and enable MAC filtering.

Let’s go to Configuration > Tags & Profiles > WLANs

This should give us the list of any configured WLANS - including the one we created earlier. Go ahead and click on it to edit.

Next, we’ll jump straight to the Layer 2 section under the Security tab. Check the box for MAC Filtering and select the Authorization List we created from the drop down.

And we’re done! Clients that want to join our newly created SSID will need the pre-shared key we configured, but they will also need to be manually added to our MAC address filter as well. While this isn’t a perfect security measure since MAC addresses can be easily spoofed - it does add an extra layer of protection to keep unauthorized devices from inadvertently being able to join this specific WLAN.

In the event that a client is NOT on the authorized list, you may see the following logs in a client debug:

2019/10/25 22:26:12.327 {wncd_x_R0-0}{1}: [client-orch-sm] [22573]: (note): MAC: aaaa.aaaa.aaaa  Association received. BSSID aaaa.aaaa.b34d, old BSSID 0000.0000.0000, WLAN SuperSecret, Slot 1 AP aaaa.aaaa.b340, AP_3802-1F01
2019/10/25 22:26:12.327 {wncd_x_R0-0}{1}: [client-orch-state] [22573]: (note): MAC: aaaa.aaaa.aaaa  Client state transition: S_CO_INIT -> S_CO_ASSOCIATING
2019/10/25 22:26:12.328 {wncd_x_R0-0}{1}: [client-orch-state] [22573]: (note): MAC: aaaa.aaaa.aaaa  Client state transition: S_CO_ASSOCIATING -> S_CO_MACAUTH_IN_PROGRESS
2019/10/25 22:26:12.328 {wncd_x_R0-0}{1}: [client-auth] [22573]: (note): MAC: aaaa.aaaa.aaaa  MAB Authentication initiated. Policy VLAN 0, AAA override = 0, NAC = 0
2019/10/25 22:26:12.329 {wncd_x_R0-0}{1}: [ewlc-infra-evq] [22573]: (note): Authentication Success. Resolved Policy bitmap:11 for client aaaa.aaaa.aaaa 
2019/10/25 22:26:12.329 {wncd_x_R0-0}{1}: [ewlc-infra-evq] [22573]: (ERR): SANET_AUTHC_FAILURE - AAA Server Down username ac37434a673a, audit session id 000000000000006B3E9D2A55, 
2019/10/25 22:26:12.331 {wncd_x_R0-0}{1}: [client-orch-state] [22573]: (note): MAC: aaaa.aaaa.aaaa  Client state transition: S_CO_MACAUTH_IN_PROGRESS -> S_CO_ASSOCIATING
2019/10/25 22:26:12.331 {wncd_x_R0-0}{1}: [dot11] [22573]: (ERR): MAC: aaaa.aaaa.aaaa  Failed to assoc failure tr state entry. Incorrect validation status value :1
2019/10/25 22:26:12.331 {wncd_x_R0-0}{1}: [dot11] [22573]: (ERR): MAC: aaaa.aaaa.aaaa  Dot11 update co assoc fail. Sent assoc failure to CO. delete reason: 9, CO_CLIENT_DELETE_REASON_MAB_FAILED
2019/10/25 22:26:12.331 {wncd_x_R0-0}{1}: [client-orch-sm] [22573]: (note): MAC: aaaa.aaaa.aaaa  Client delete initiated. Reason: CO_CLIENT_DELETE_REASON_MAB_FAILED, fsm-state transition

On the opposite side, when a client is successfully able to pass MAC authentication - the logs will show the following:

2019/10/25 21:25:53.148 {wncd_x_R0-0}{1}: [client-auth] [22573]: (note): MAC: aaaa.aaaa.aaaa  MAB Authentication initiated. Policy VLAN 0, AAA override = 0, NAC = 0
2019/10/25 21:25:53.150 {wncd_x_R0-0}{1}: [ewlc-infra-evq] [22573]: (note): Authentication Success. Resolved Policy bitmap:11 for client aaaa.aaaa.aaaa 
2019/10/25 21:25:53.151 {wncd_x_R0-0}{1}: [client-auth] [22573]: (note): MAC: aaaa.aaaa.aaaa  MAB Authentication success.
2019/10/25 21:25:53.151 {wncd_x_R0-0}{1}: [client-orch-state] [22573]: (note): MAC: aaaa.aaaa.aaaa  Client state transition: S_CO_MACAUTH_IN_PROGRESS -> S_CO_ASSOCIATING

2019/10/25 21:30:08.626 {wncd_x_R0-0}{1}: [client-orch-sm] [22573]: (debug): MAC: aaaa.aaaa.aaaa  Received Dot11 association request. Processing started,SSID: SuperSecret2, Policy profile: Home_WLANID_3, AP Name: AP_3802-3F01, Ap Mac Address: aaaa.aaaa.c9a0 BSSID MAC aaaa.aaaa.b34d wlan ID: 3RSSI: 0, SNR: 32
2019/10/25 21:30:08.626 {wncd_x_R0-0}{1}: [client-orch-state] [22573]: (note): MAC: aaaa.aaaa.aaaa  Client state transition: S_CO_L2_AUTH_IN_PROGRESS -> S_CO_L2_AUTH_IN_PROGRESS
2019/10/25 21:30:08.626 {wncd_x_R0-0}{1}: [dot11] [22573]: (info): MAC: aaaa.aaaa.aaaa  DOT11 state transition: S_DOT11_ASSOCIATED -> S_DOT11_MAB_PENDING
2019/10/25 21:30:08.626 {wncd_x_R0-0}{1}: [client-orch-state] [22573]: (note): MAC: aaaa.aaaa.aaaa  Client state transition: S_CO_L2_AUTH_IN_PROGRESS -> S_CO_MACAUTH_IN_PROGRESS
2019/10/25 21:30:08.626 {wncd_x_R0-0}{1}: [client-auth] [22573]: (info): MAC: aaaa.aaaa.aaaa  Client auth-interface state transition: S_AUTHIF_ADD_MOBILE_ACK_WAIT_KM -> S_AUTHIF_MAB_AUTH_DONE
2019/10/25 21:30:08.626 {wncd_x_R0-0}{1}: [client-orch-sm] [22573]: (debug): MAC: aaaa.aaaa.aaaa  Processing MAB authentication result status: 0, CO_AUTH_STATUS_SUCCESS
2019/10/25 21:30:08.626 {wncd_x_R0-0}{1}: [client-orch-state] [22573]: (note): MAC: aaaa.aaaa.aaaa  Client state transition: S_CO_MACAUTH_IN_PROGRESS -> S_CO_ASSOCIATING
2019/10/25 21:30:08.626 {wncd_x_R0-0}{1}: [dot11] [22573]: (debug): MAC: aaaa.aaaa.aaaa  dot11 send association response. Sending association response with resp_status_code: 0 
2019/10/25 21:30:08.627 {wncd_x_R0-0}{1}: [dot11] [22573]: (info): MAC: aaaa.aaaa.aaaa  dot11 send association response. Sending assoc response of length: 137 with resp_status_code: 0, DOT11_STATUS: DOT11_STATUS_SUCCESS
2019/10/25 21:30:08.627 {wncd_x_R0-0}{1}: [dot11] [22573]: (note): MAC: aaaa.aaaa.aaaa  Association success. AID 1, Roaming = True, WGB = False, 11r = False, 11w = False
2019/10/25 21:30:08.627 {wncd_x_R0-0}{1}: [dot11] [22573]: (info): MAC: aaaa.aaaa.aaaa  DOT11 state transition: S_DOT11_MAB_PENDING -> S_DOT11_ASSOCIATED

The config I used is based on this blog post. It’s a great write-up on how to back up F5 configurations using CatTools. I don’t want to replicate what was written over there - but I did hit some issues that were specific to my use-case that I wanted to share.

While I was happy to find the article linked above, the immediate results didn’t work so smooth for me. This may be due to some key configuration differences that I face in my network:

• All the F5’s are LDAP integrated - so there isn’t an easy way to provide LDAP users with direct bash access
• All of my F5’s are remote appliances, where the backup configuration is being copied across the WAN

Getting around the first problem was my biggest challenge. CatTools is a very command/response-oriented application. Any remote LDAP authenticated users are immediately dropped into F5’s shell: tmsh. To get from there to their ‘advanced shell’ is as simple as typing bash. However, When the terminal prompt changes, it often throws CatTools into a state of “I didn’t receive the response prompt I expected, therefore kill the job - something went wrong”. I spent a bit more time on this than I wanted to - but the underlying problem was that the “F5.BigIP” device type was specifically looking for the tmsh shell and couldn’t handle the prompt change. The fix? Switch the device type to “Linux.RedHat.Bash”, then add the bash command to the first line of your backup script.

The next problem was using TFTP to copy the backup archives over the WAN. Even some of the new F5’s with minimal configuration still generate a 10Mb file. Doesn’t seem like much, but when you’re copying that over a WAN between two datacenters, that turns into a ~5 minute file transfer. CatTools by default will only wait 30 seconds after executing a command before it expects a response. So every time I tried to run the job, CatTools would kill it only 30-seconds into the file transfer. Luckily enough, they support a utility command that can alter the normal timeout:

%ctUM: Timeout 600
tftp -m binary 192.168.1.10 -c put $filename
%ctUM: Timeout 0

The command %ctUM: Timeout 600 changes the timeout value to 600 seconds, or 10 minutes. The TFTP file transfer command is next, which is now permitted up to 10 minutes to finish. The last command resets the timeout back to the default (30 seconds).

I also realized that the original script doesn’t purge the backup archive afterwards. For my use case, I would much rather automatically clean up the backup files once they’ve been transferred to a central location.

So after all that, here is the version of that script that I’m using:

export date=`date +"%y%m%d"​`
export filename=$HOSTNAME.$date.ucs
tmsh save /sys ucs /var/local/ucs/$filename
cd /var/local/ucs
%ctUM: Timeout 600
tftp -m binary 192.168.1.10 -c put $filename
%ctUM: Timeout 0
rm -f $filename

Thanks again to the original blog post for getting me on the right track with this! I hope that my ramblings here are helpful to anyone with a similar deployment scenario.

What is an etherchannel? It’s a way of taking multiple independent links and bundling them together, so that they appear as one logical connection between two devices. Etherchannels are commonly used between two switches, or between a switch and a host - which allows for both additional bandwidth and fault tolerance/redundancy. In the example today, we’ll be using an etherchannel protocol called Link Aggregation Control Protocol (LACP). LACP is an IEEE standard (802.3ad).

You might be thinking “Wait, wouldn’t multiple links cause a loop? Or trigger Spanning-tree to block ports?”. Not in this case! Etherchannel technologies work around those problems by creating a single logical interface for spanning-tree to worry about. The etherchannel protocol itself worries about loop prevention in between the two devices, so we get multiple ports of non-blocking bandwidth.

For everything we cover in this example, we’ll be using the following topology:

So we have two switches, which are connected together via Eth0/0 and Eth0/1. Each switch has three VLANs configured - 10, 20, and 30.

Configuring an Etherchannel

I’ll only be showing the configuration from the perspective of 0x2142-SW1 - but all configuration is replicated on 0x2142-SW2.

! We'll use the interface range command to apply the etherchannel configuration to
! both Eth0/0 and Eth0/1 at the same time:
0x2142-SW1(config)#int range Eth0/0 - 1

! We specify which etherchannel protocol to use by configuring 'channel-protocol'
! PAgP is a Cisco Proprietary protocol, but we'll be using LACP for this example:
0x2142-SW1(config-if-range)#channel-protocol ?
  lacp  Prepare interface for LACP protocol
  pagp  Prepare interface for PAgP protocol
0x2142-SW1(config-if-range)#channel-protocol lacp

! Next we need to specify a channel-group and mode:
0x2142-SW1(config-if-range)#channel-group 1 mode ?
  active     Enable LACP unconditionally
  auto       Enable PAgP only if a PAgP device is detected
  desirable  Enable PAgP unconditionally
  on         Enable Etherchannel only
  passive    Enable LACP only if a LACP device is detected

0x2142-SW1(config-if-range)#channel-group 1 mode active
Creating a port-channel interface Port-channel 1

0x2142-SW1(config-if-range)#
*Jan 26 01:03:04.532: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel1, changed state to up

Let’s talk through a few notes about the above configuration. In order to enable etherchannel, we only need to configure two commands: channel-protocol and channel-group. The channel-protocol command tells the switch which etherchannel protocol to use for negotiation (LACP in this case). The channel-group command provides two necessary components: the group number and mode. The group number is just a device-local identifier for which group to add these ports to. When we specified group 1, the switch adds both Eth0/0 and Eth0/1 into the new logical interface Port-Channel 1.

The etherchannel mode is also important. The two primary options we want to look at for LACP are active and passive. Active tells the switch to preemptively send out LACP negotiation packets. In this case, the switch really wants the ports to become a bundle and will ask it’s partner device for an etherchannel to be formed. Passive mode tells our switch to only negotiate if another device wants to. In this mode, our switch won’t send out any etherchannel negotiation packets unless its partner device does so first.

Generally speaking, the most common configuration is to set the mode on both devices to active. This ensures that both devices actively participate in trying to establish an etherchannel. Placing one device in active and one in passive will work as well. However, if both devices are placed into passive mode, an etherchannel will never form.

Validation

So how do we validate that the etherchannel has formed correctly? One way is using the show etherchannel summary command:

0x2142-SW1#show etherchannel summary
Flags:  D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      N - not in use, no aggregation
        f - failed to allocate aggregator

        M - not in use, minimum links not met
        m - not in use, port not aggregated due to minimum links not met
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port

        A - formed by Auto LAG

Number of channel-groups in use: 1
Number of aggregators:           1

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Et0/0(P)    Et0/1(P)

From the output above, we see that there is one group configured with the group ID of 1. It shows that both Eth0/0 and Eth0/1 have been added into the Port-channel 1 interface. The (SU) next to the Port-channel interface indicate that the etherchannel is up (U) and configured for layer 2 (S). I mentioned earlier that spanning-tree only worries about the port-channel interface, not the individual member ports. We can also check that out by using the show spanning-tree command:

0x2142-SW1#sh spanning-tree vlan 20
VLAN0020
  Spanning tree enabled protocol rstp
  Root ID    Priority    32788
             Address     aabb.cc00.1000
             This bridge is the root
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32788  (priority 32768 sys-id-ext 20)
             Address     aabb.cc00.1000
             Hello Time   2 sec  Max Age 20 sec  Forward Delay 15 sec
             Aging Time  300 sec

Interface           Role Sts Cost      Prio.Nbr Type
------------------- ---- --- --------- -------- --------------------------------
Et0/2               Desg FWD 100       128.3    Shr
Et0/3               Desg FWD 100       128.4    Shr
<-- Output omitted -->
Po1                 Desg FWD 56        128.65   Shr

Making Configuration Changes to an Etherchannel

Now that we have a working etherchannel - We have a few things that need special attention. The individual port configurations, Eth0/0 and Eth0/1 in this case, need to match at all times! Port configuration mis-matches are going to be an easy way to inadvertently bring down the port-channel. The good thing is that we now have a convenient Port-Channel interface which we can use for configuration. This logical port will replicate any configuration changes to all member ports.

! Let's jump into our Port-Channel 1 interface and configure a trunk for VLAN 20
0x2142-SW1(config)#int po1
0x2142-SW1(config-if)#switchport mode trunk
0x2142-SW1(config-if)#switchport trunk allowed vlan 20
! Now we can check the individual port configs:
0x2142-SW1(config-if)#do sh run int e0/0
Building configuration...

Current configuration : 176 bytes
!
interface Ethernet0/0
 switchport trunk allowed vlan 20
 switchport mode trunk
 channel-protocol lacp
 channel-group 1 mode active
end

0x2142-SW1(config-if)#do sh run int e0/1
Building configuration...

Current configuration : 176 bytes
!
interface Ethernet0/1
 switchport trunk allowed vlan 20
 switchport mode trunk
 channel-protocol lacp
 channel-group 1 mode active
end

Easy enough, right? The configuration changes for the trunk are now on both Eth0/0 and Eth0/1.

Troubleshooting Etherchannels

There is always a possibility that something goes wrong - so let’s take a quick look at some common problems and how to fix them.

Remember how I said that the member port configurations had to match? Here’s what happens if we make a configuration change on only one of the two member ports:

0x2142-SW1(config)#int eth0/1
0x2142-SW1(config-if)#switchport trunk allowed vlan 30
0x2142-SW1(config-if)#
*Jan 28 20:43:55.458: %EC-5-CANNOT_BUNDLE2: Et0/1 is not compatible with Et0/0 and will be suspended (vlan mask is different)

Eth0/1 immediately gets put into a suspended state, and is no longer active in the port-channel interface. In this case the switch gives us a good hint as to what’s wrong - vlan mask is different. Error messages will vary slightly, but a suspended port is easy to fix by comparing individual port configurations and fixing the mismatch.

Here’s another one:

*Jan 28 21:06:07.346: %EC-5-L3DONTBNDL2: Et0/0 suspended: LACP currently not enabled on the remote port.
*Jan 28 21:06:08.009: %EC-5-L3DONTBNDL2: Et0/1 suspended: LACP currently not enabled on the remote port.

This error message can mean a few things - the common one being exactly what it states! Check both sides of the connection, and ensure that LACP is configured on each device. This error message can also occur on certain mismatches - like if one side is running as a Layer 2 etherchannel, but the other side is running as Layer 3.

One more:

Jan 28 20:83:55.458 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel1 is down (No operational members)

The above message is also somewhat self-explanatory. In this case, the switch is unable to bring up the port-channel interface, because none of the underlying member ports are coming online. Troubleshoot what might be wrong with those ports first, then the port-channel should come up.

Hope this was useful! In a later post, we’ll dig into more configuration and considerations - like packet hashing, layer 3 etherchannels, and how packets are weighted between interfaces.

Questions? Drop them in the comments below!

Redundant Ethernet Interfaces

So first thing is first - Once you have a cluster configured, you’ll probably want to configure a few sets of redundant ethernet interfaces. These interfaces are also often referred to as reth interfaces. This will create a shared interface between your SRX pair, where you can configure IP address and VLAN information to be shared between the two. Let’s say that we have a Juniper SRX 1500 cluster, and we want to create a redundant interface for one of our 10Gb ports. Here is how we would do that:

root@testsrx# set interfaces xe-0/0/16 gigether-options redundant-parent reth1
root@testsrx# set interfaces xe-7/0/16 gigether-options redundant-parent reth1
root@testsrx# set interfaces reth1 redundant-ether-options redundancy-group 1

In the config above, we first take both of our interfaces (xe-0/0/16 on node0, and xe-7/0/16 on node1) and tell them that they now belong to a redundant interface group (reth1). Next, we enter into the reth1 config, and associate it to a redundancy group.

You’re also going to need to keep in mind that the SRX requires you to specify how many redundant ethernet interfaces will be configured. This is likely a memory thing, since each SRX also has a different maximum number of reth interfaces that can be configured. For example, if you tell the SRX that you need 5 reth interfaces, then the SRX will allocate system resources to manage those interfaces. In order to set the number of available reth interfaces, we’ll use the following command:

root@testsrx# set chassis cluster reth-count 5

Redundancy Groups

A redundancy group, or RG, is used as a container for logically grouping redundant interfaces/virtual routers which must fail over together. A single RG can be configured as primary on one of the two active SRX firewalls is a cluster - with the ability to fail over to the other node. For example, we might want be planning on only using one virtual routing instance on our SRX - so we would create RG1 and assign out interfaces to belong to it.

A quick note - all interfaces in a single virtual router must belong to the same RG. This way the virtual routing instance and all of it’s associated interfaces will always run on the same SRX node. In order to achieve an active/active firewall configuration, you would need to create two separate virtual routers, each with their own reth interfaces and different RGs. Then you would make RG1 primary on node0, and RG2 primary on node1.

In most configurations, dumping all of your reth interfaces into RG1 will be sufficient. You’re likely going to want to set up a priority for each RG - and maybe even preemptive fail-over. In order to do that - you’ll have to configure each cluster member with a priority:

root@testsrx# set chassis cluster redundancy-group 0 node 0 priority 200
root@testsrx# set chassis cluster redundancy-group 0 node 1 priority 50
root@testsrx# set chassis cluster redundancy-group 1 node 0 priority 200
root@testsrx# set chassis cluster redundancy-group 1 node 1 priority 50
root@testsrx# set chassis cluster redundancy-group 1 preempt

The higher priority wins here - so if you set node0 to a higher priority and preempt is enabled, then node0 will actively try to take ownership of RG1. I would rather not set preempt on RG0 for a few reasons - which we’ll cover in the next section. Priorities can also be modified using interface monitoring, so if a particular interface goes offline we can decrement the priority of that node (also covered below).

A Note About RG0

You might notice from the last post, that you’re output of show chassis cluster status already showed two redundancy groups: RG0 and RG1. RG0 is only used for management traffic and manages the routing engine for your SRX. Unfortunately, this can lead to some weird behaviors that you might not be expecting.

For example, whichever node is primary for RG0 is the only node that collects interface and monitoring statistics. If you’re using a monitoring tool that polls data from both of your SRXs, then the secondary for RG0 will report nothing about it’s interfaces, CPU, etc. This is also true if you log into the actual SRX itself - a show interfaces will actually return a bunch of default values, including showing that your ports are half-duplex. Don’t panic though, this is just an oddity of RG0. If you log back into the primary node for RG0, then it will show all of the proper statistics for both SRX firewalls.

Due to these weird things about RG0 - I prefer to always leave it on node0. Therefore I know which one to log into whenever I need to look at something, or which SRX to check in our monitoring tools. It’s also worth noting that whichever SRX is primary for RG0 is also the node you’re going to need to log into for configuration changes - even if all of your other redundancy groups are the other SRX.

Weird, right?

Oh, and be warned that since RG0 controls the routing engine, a failover of this RG can cause brief outages. This is primarily because the routing table and firewall state information will be lost. The secondary node has to spin up new processes for the routing engine, and at least currently there isn’t a graceful sync of all of that data.

Interface Monitoring

I mentioned setting device priorities a bit earlier. Setting interface weights is going to be the primary method for dynamically affecting those priorities, and therefore possibly causing a preemptive failover. One example might be that you’re using an SRX cluster for your edge firewall, and you want it to automatically fail over if the primary loses it’s internet uplink.

Note that you must configure the physical interfaces here, not the redundant ethernet interfaces:

root@testsrx# set chassis cluster redundancy-group 1 interface-monitor xe-0/0/16 weight 160

Remember when we set the priorities of our firewalls earlier? Node0 was set to 200, and node1 at 50. So here we are saying that xe-0/0/16 on node0 is worth 160 points. So if xe-0/0/16 goes down, then node0 will decrement it’s priority by 160 - which will be 40. This will trigger a preemtive failover by node1. The reverse is also true - when xe-0/0/16 comes back up, then node0’s priority will go back up to 200. Then node0 will take back ownership of RG1.

Manual Failover

There is a pretty good chance at some point you might need to perform a manual failover of your SRX redundancy groups. Maybe you need to do some maintenance or upgrades, or you just want to make sure failover works as you expect. In either case, the commands to do this are pretty straightforward:

root@testsrx> show chassis cluster status
Cluster ID: 5
Node       Priority       Status       Preempt       Manual failover
Redundancy group: 0 , Failover count: 0
node0      200            primary      no            no 
node1      50             secondary    no            no

Redundancy group: 1 , Failover count: 0
node0      200            primary      yes           no 
node1      50             secondary    yes           no

root@testsrx> request chassis cluster failover redundancy-group 1 node 1 

root@testsrx> show chassis cluster status 
Cluster ID: 5
Node       Priority       Status       Preempt       Manual failover
Redundancy group: 0 , Failover count: 0
node0     200             primary      no            no 
node1     50              secondary    no            no

Redundancy group: 1 , Failover count: 1
node0     200             secondary    yes           yes
node1     255             primary      yes           yes

Okay - so let’s talk about a few things that have happened here. I always recommend that you run a show chassis cluster status first, so you know where things already stand. Then we can proceed by requesting a failover. To do this, you have to specify which redundancy group you want to fail over, and which node you want to become the new primary. So in this case, we made node1 the new primary of RG1.

You might also notice that the priorities have changed, and the devices are marked as being in a manual failover state. This is important, because you cannot manually fail back until you reset this state. That’s right - if you tried to run the failover command again to move RG1 back to node0, it will not work. An automatic failover due to hardware failure or interface monitoring will still be permitted. In order to perform a manual fail-back to node0, we have to run the following reset command:

root@testsrx> request chassis cluster failover reset redundancy-group 1

Hopefully between last weeks post and this one, you should have a good handle on the basics of configuring a chassis cluster on your new pair of Juniper SRX firewalls. Let me know in the comments below if this helped you!

So let’s take a look at what we need to do!

Physical configuration

First thing we need to do is get both devices unboxed and cabled appropriately. In order to get a successful cluster configured, we will need to get two critical ports connected: the HA control port and the cluster fabric port. Technically only the HA control port is required to get a cluster working, but you’ll want to get the fabric port working as well - here is what both ports are used for:

HA Control Port - This is used for communication between the cluster members. This connection is used just for control-plane stuff - like keepalives/heartbeats and config sync between the two nodes.

Fabric Port - This port is used for data sync between the cluster members. All routing/firewall state information is synced using this port, and any cross-cluster traffic is also transferred using this port (for example, if one SRX was primary for a redundancy group, but the secondary was the active BGP speaker for your upstream connection - then the traffic would come in through the secondary and cross this link to reach the primary RG)

The fabric port is the easiest to connect - because you can use any port you like, then specify which port to use in the CLI. The control port, however, must be the assigned port that Juniper allocates for this use. Unfortunately, this port varies between device models. The two most common SRXs that I’ve deployed are the 345 and 1500. The SRX 1500 has a dedicated 10G HA control port, but the SRX 345 actually uses ge-0/0/1 on both nodes for this. Juniper lists what all those port assignments are over on this page.

Once those ports are connected, go ahead and power on both devices!

JunOS Config

Okay - Once the physical configuration has been completed, there are a few things that need to be configured on both devices before you can establish a cluster.

When you first boot each device, you’ll log in with root and no password. Then you’ll be dropped into the JunOS shell, and you’ll need to type cli to start the JunOS command-line interface. Then type configure to get into the configuration mode.

root@testsrx% cli
root@testsrx> configure
root@testsrx#

In the config mode, we’ll need to set a root password before we can enable the clustering. This password must match on both devices!

root@testsrx# set system root-authentication plain-text-password
New Password: <type your root password here>
Retype new password: <and again...>

For the SRX 1500 series, where there is a dedicated HA Control port, this is enough to get the cluster working. But for some of the branch SRXs, like the 300 series, you’ll need to make a few additional changes. These devices come with a default config, which includes IP addresses on certain interfaces. Unfortunately, this will conflict with your cluster config and will not allow your cluster to reach a healthy state.

In my config, I already plan on re-configuring all of the interfaces and security-zones to fit my needs - so I will just delete those entire config sections:

root@testsrx# delete interfaces
root@testsrx# delete security

After all that is done, we need to commit our changes:

```text
root@testsrx# commit

Finally we can go ahead and set up the cluster! This config is actually done outside of configure mode, so you will need to exit that.

So one thing to note here - each cluster will be configured with a cluster-id. This MUSTbe unique across any layer 2 subnet. So if we had multiple SRX clusters within a single broadcast domain, we would need to assign each one a different cluster ID.  I’ll use cluster-id 5 in this example.

On whichever SRX you want to be the primary node:

root@testsrx# exit
root@testsrx> set chassis cluster cluster-id 5 node 0 reboot

I personally like to give the primary a minute or to into the boot process before I configure the secondary, but we’ll do so with a similar command (just specifying node 1 instead of 0):

root@testsrx> set chassis cluster cluster-id 5 node 1 reboot

After both nodes come back online, log into node 0 and run the following command:

root@testsrx> show chassis cluster status 
Cluster ID: 5
Node       Priority      Status      Preempt      Manual failover

Redundancy group: 0 , Failover count: 0
node0      100           primary     no           no 
node1      100           secondary   no           no

Redundancy group: 1 , Failover count: 0
node0      100           primary     yes          no 
node1      100           secondary   yes          no

Perfect! Now let’s go configure our fabric ports! Interface fab0 will be configured as the fabric port on node0, and interface fab1 will be configured as the fabric port on node1.

root@testsrx> configure
root@testsrx# set interface fab0 fabric-options member-interfaces ge-0/0/10
root@testsrx# set interface fab1 fabric-options member-interfaces ge-5/0/10
root@testsrx# commit

Now that we’re in a cluster, all of this configuration can be done on node0 - but note that in this case the secondary device’s ports all start with ge-5/x/x. This is another oddity of JunOS - that numbering scheme isn’t always the case. In the SRX1500s, the node1 ports all start with ge-7/x/x - so this will vary depending on what devices you’re working with. If you ever need to check this - you can run show interface terse to list all interfaces in the cluster.

As a final verification that all our ports are up, drop out of config mode and run show chassis cluster interfaces:

root@testsrx> show chassis cluster interfaces
Control link 0 name: ge-0/0/1
Control link status: Up

Fabric interfaces:
Name Child-interface Status
fab0 ge-0/0/10 up
fab0
fab1 ge-5/0/10 up
fab1
Fabric link status: up

Hooray! We now have a functioning SRX cluster!

Sometimes if this doesn’t work, the output of show chassis cluster status will show the secondary node as disabled or lost. I’ve found that lost usually indicates a conflicting configuration on the cluster interfaces (like leaving the default IPs configured). If you see disabled, try rebooting the secondary node again - and if that doesn’t work, then you may need to disable clustering on both nodes and re-configure. This can be done using the set chassis cluster disable reboot command.

Next week, we’ll look at redundancy-groups, performing manual failovers, and setting up interface monitoring for automatic failovers. Hope this was helpful!

Securing your BGP peering (Know who you’re connecting to)

BGP is a little different from most other routing protocols, since it uses a single unicast TCP connection between peers to exchange routing updates. Lucky for us, that means that we can easily filter traffic from only known peers. Once you have direct connectivity up between your edge router/firewall and your direct peer, lock down that connection with an ACL. Permit TCP port 179 traffic ONLY from your directly connected peer IP - no one else.

While you’re at it, let’s take it another step further: Request that your ISP set up BGP authentication. Sure, a majority of BGP implementations today still require use of MD5 for auth (which is terrible) - but some authentication is still better than none. This can usually be arranged at the time of turning up peering. Both sides configure the same authentication password and with any luck the peering still establishes.

BGP by nature is unfortunately not the most secure protocol - but a few simple steps like this will help ensure you’re only connecting out to authorized peers.

Route filtering (Don’t trust anyone)

Usually when you’re filling out the BGP peering paperwork for your service provider, they will ask you what kinds of routes you want. In most cases, you should be able to request one of the following:

Default only - Exactly what it sounds like. Your provider will only advertise a route for 0.0.0.0/0. In many cases, this is probably what you’re going to want. With this type of advertisement, each upstream provider will just give us the same default route to the internet. From there we can weight which one we want to use, and traffic will automatically fail-over to the secondary connection should the primary fail.

Partial - If for any reason you want to weight routes to certain destinations differently, then we might request this. In this case, you’re probably going to still receive 0.0.0.0/0 plus any specific routes you ask for. A good example of this is if we wanted to specifically manipulate routes for a remote office we have. Maybe we want to weight Internet traffic for one uplink, and VPN traffic to a remote office on the other uplink.

Full - In 99% of typical business cases, this won’t be required. This option means the upstream providers will be dumping the entire Internet routing table on you. While this offers you a ton of control over path manipulation, it also requires significant memory resources on your routers in order to maintain that routing table.

After we figure this out, the next step is to make sure we are filtering the routes we accept from the upstream provider. Wait - didn’t we just tell them exactly what routes to send us? Why do we need to filter them? Well you can never be too safe here - and we would rather perform an unnecessary filtering than have an ISP accidentally misconfigure route advertisements. So if you’re only expecting a route for 0.0.0.0/0, then filter your inbound route advertisements so you only accept that route.

Same thing goes for outbound route advertisement  - if we own a /24 of public IP space, then we only want that range to be advertised out. Some providers may already filter this on their end, but again it doesn’t hurt here to be extra cautious. If we are accepting anything other than a default route from our provider, then we run the risk of leaking those additional routes between the two providers - which would lead to inadvertently becoming a transit AS. Chances are pretty good that you don’t want that, so make sure you configure filtering for all outbound route advertisements.

Minimum Advertisement (Oh no, we have to re-address everything)

I mentioned this in the original post - but typically when you are peering with two separate upstream providers, you need to advertise no less than a /24. We ran into this at my last job, where we had been provided a /25 by AT&T but we needed to bring in a second carrier via BGP. The reasoning behind this is to keep global routing tables as small as possible, by not allowing them to end up flooded with a ton of routes for smaller subnets. It makes sense, but on the other hand I feel like requiring a /24 in all cases can be a bit wasteful. My last job only required maybe 30 publicly addressable hosts - which meant that the remaining addresses went unused.

At any rate - should you find yourself in this scenario then you’re going to have to face the inevitable: Renumbering into a new IP space. Any time you have to do this, it’s going to be a bit of a pain - but for external addressing like this it might be easier. So in our case, the entire /25 space was hosted on our external firewall then NAT’ed into DMZ servers.

Here is the quick steps that I used to do a side-by-side migration without taking any significant downtime:

• Get the new subnet up and running - assign the interface addresses on your firewall and BGP up and running
• Assign new IP addresses to all of your existing services
• Configure NAT rules for the new external IP addresses to the DMZ hosts - while leaving the existing NAT rules for the old subnet (Also make sure your firewall rules permit the same traffic to either IP)
• Migrate DNS entries externally to point to the new IP space
• Once traffic stops flowing to the old IP, remove the old NAT

As a side note - if you procure redundant internet connections through the same upstream provider, then you might be able to work out something else. They may be able to provide you a private ASN to use, and they will likely accept any minimum advertisement - since they will be summarizing upstream within their network anyways.

I had a few more things I originally intended to cover here - but it seems that these topics are filling way more space than I thought they would. Specifically, I’m thinking about a dedicated post to BGP path manipulation - which is probably something you’re going to want to implement after peering is established. Hopefully these tips help! If you have any questions, throw them in the comments below.

So the concept of port security is fairly simple - We want to secure each individual switch port to a physical layer 2 MAC address, or at least limit how many unique MAC addresses might be learned on an individual port. The technology could be used to just limit the number of simultaneous devices on a port - by just setting a MAC threshold. Or we can also take it to the extreme and lock down each port to a hard-coded MAC address - which will never allow another device to connect. You might be thinking that the second method is absolutely ridiculous, but it really depends on the business needs.

First, let’s take brief look at the typical port security configuration and some of the options available.

SecureSwitch(config)# interface x/x  ! Whichever interface we want to lock down
SecureSwitch(config-if)# switchport port-security max xx  ! Max number of MAC addresses that can be learned
SecureSwitch(config-if)# switchport port-security violation xxxxx  ! Choose to either restrict or shutdown the port (description below)
SecureSwitch(config-if)# switchport port-security  ! This actually enables the port security config

Fairly straight-forward, right? We choose a port (or you could do a range) and set a few options. The default number of MAC addresses able to be learned on a port is 1, so it’s likely you’re going to want to change this - unless 1 is all you need. Port security can only be enabled on access ports, so 1 MAC address works in most cases  - except where you have a PC daisy-chained off of an IP phone (in which case this will need to be set to 2 or 3).

Next we set our preferred violation action. This step is pretty important because it defines what happens when the port exceeds it’s MAC count. Restrict is the passive approach. If we have two PC’s plugged into a single access port (maybe using an unmanaged switch), then the second PC will just never be able to work as long as our max MAC limit is 1. The first PC to connect will be fine, and the switch will log a message and send an SNMP trap when the second MAC is picked up. Shutdown is the more forceful approach. Once that second MAC address turns up on the port, the switch puts the port in an err-disabled state - which shuts down the port to all traffic. This event is also logged and generates an SNMP trap - however the port will not come back online until an administrator manually re-enables it.

Now that we see a basic config, let’s take a look at a few different use cases for this feature. In one of my previous jobs, I worked as a network admin for a local government organization. Port security configuration in that environment was extremely strict. Each switchport was configured to permit only one MAC address, shutdown upon violation, and the switchport port-security mac-address sticky command was also used. This command takes the first MAC address learned on the port and commits it to the running configuration, which means that this MAC is essentially hard-coded to be the only MAC permitted on the port. So in this environment, a single PC was tied to a single port - nothing else could ever be plugged into that port without either shutting down the port or administrator intervention. In a government office, this was absolutely necessary because every device on the network needed to be tracked and personal devices were not permitted to be connected. We needed to know if anything was ever plugged in that wasn’t an authorized device - so manual intervention and investigation was a requirement.

In a more typical office environment - port security configurations can just be a good security practice without going overboard with it. We never want a user to plug in a rouge switch into our network without our knowledge, right? So maybe we assume each user has an IP phone and PC, and limit the port to 2 MAC addresses. In this case, we can go ahead and just set the port to restrict. We don’t want to prevent the user from working if a port violation occurs, nor do we want to spend time resetting the port for them - but we might still want to be notified, especially if it happens often. In addition, port security is an excellent way to secure ports public areas. For example, maybe we have an IP phone or kiosk PC in our lobby. These need access to the network, but we don’t want anyone to be able to unplug that device and gain access into our network. In cases like this, it would actually make sense to have the switch only permit access from that single MAC address.

Outside of the ‘practical’ use cases, there is also the strictly security side of things. I’ve touched on a few considerations already - but there are also certain types of attacks that can be defeated by port security. One of those would be exhausting the CAM table resources. A malicious person could use publicly available tools to spoof MAC addresses in the packets they send to the switch. Tools like this force the switch to learn hundreds of thousands of MAC addresses, which eventually will overload the CAM table. When a switch CAM table becomes full, the switch begins flooding packets out all interfaces. This is because the switch can no longer assign mappings between MAC addresses and the ports they originate from - so the switch has no choice but to flood everything and hope the correct recipient receives the data. For the attacker, this means they can run a packet capture on the port and collect information they wouldn’t have otherwise needed to. This scenario could be prevented by implementing port security, which could simply restrict the number of MAC addresses learned off of any individual interface.

Port security configuration can be implemented in a few different ways depending on your use case. Overall though, it can prove to be a useful way to help implement security controls on your network.  What do you think about port security? Extremely useful or does it just get in the way? Comment below and tell me how you have implemented it!

The problem here stemmed from the fact that there were no documented standards in place. An engineer was given a device to configure, and it was configured depending on who did it and what they felt needed configuring. In a few cases, this actually led to unnecessary security risks being introduced into the environment because something was left enabled. In one instance, this included open root SSH logins via the Internet to a production firewall. Scary, huh?

So how do we go about changing this? Here is a quick little guide I threw together on my method for tackling the situation:

1. Define a standard

Begin creating a baseline document, whether it be a spreadsheet, word doc, or a wiki page. Start small and choose a single system, like your external firewalls for example.

2. Research best practices

Check out the vendor’s website to see what they recommend. There are also some amazing free resources out there like the Center for Internet Security’s configuration benchmarks, Do your research - there is plenty available to help you.

3. Figure out what’s best for your network

Not all of the best practices or security hardening guides will be a perfect fit for your environment. So it will take a little manual review to see what actually fits. For example, many of these guides recommend disabling local authentication in exchange for something centralized like TACACS+ or RADIUS. But if you don’t have that available, then you’re going to stick with local authentication. This can still be a great time to find room for future improvement projects though.

4. Test

If you have a development or test environment available, then run a device or two through your checklist and make sure there are no big issues. If you don’t have a dedicated test area, then try and choose a low-impact device - where not much will be impacted if the changes go wrong.

5. Roll out the changes

Make sure you have a list of every device that needs to be touched, so that you have a way to validate. Then make the configuration changes to get each device into compliance with your new standards. Have a validation/testing checklist ready, so that you can quickly ensure that no production traffic was impacted

6. Train your peers

Configuration standards only work well as long as everyonefollows them. It only takes one person to ignore the checklist and potentially expose a vulnerability. So take an afternoon, schedule a training session with your team. Help them understand the importance of maintaining these standards, and train them on how to apply the changes (if necessary).

7. Automate

This part is optional, but highly recommended. If nothing else, spend the time to automate verification of the standards - which will make it easy to locate a device that falls out of compliance. If you or your team have the skill set, then automate the entire process from initial deployment to continuous validation. Why is this the last step, instead of being included with the roll out? I am a firm believer that you should completely understand how your device functions and reacts to changes before automating those changes.

So that’s more or less how I worked to implement a standardized configuration at my current job. I began with a completely new device platform that we were integrating into our environment, then began to go back to older device platforms. It might be a lot of upfront work, but it certainly helps me sleep better at night not having to wonder if there might be one device out there that’s misconfigured (and will cause an issue later, due to that misconfiguration).

So let me know in the comments below - have you ever implemented something like this? If so, what did you do differently? If not, then let me know if you give this a try!

The easy part of the whole process is the first step - ordering a second Internet connection. Our CIO at the time placed a few calls and had a quote back pretty quickly. A local carrier was willing to run new fiber cables to our building in less than a month. Depending on how important uptime is to your organization, this is the point where you might want to ask about a diverse path into the building. If both connections run though the same physical paths, then a single incident could still cause an outage. For example - I once worked somewhere where the redundant Internet connections shared the same telephone poll across the street. So even though the connections were redundant, a single accident involving that telephone poll and both connections were severed.

Next - Ask about IP space. In terms of IPv4, the general rule for external BGP peering is that ISP’s don’t like to accept any prefixes smaller than a /24. In our case, we had a single /25 block already allocated by our current provider - which wasn’t going to work. Luckily, the new service provider offered to give up a free /24 block along with the installation costs. Unfortunately, this meant that we had to re-address all of our public-facing services, which is almost always a pain to do. I have a few tips for this, which helped us to minimize downtime - but that’s a story for another time.

Next, we need to obtain a globally unique Autonomous System (AS) number, which will be used to advertise our network to the world. Since we were located in North America, we went though ARIN for this process - which was fairly painless. Sign up for an account, prove that you’re associated with the business, fill out a few forms to justify your need, and then just wait for the approval. One thing to watch out for is 2-byte vs 4-byte AS numbers. 2-byte is the standard and has been around forever, but only allows for up to 65,535 unique IDs. A 4-byte ASN allows for significantly more unique IDs, but I have actually run into instances where an ISP doesn’t support these. I would hope that in most cases a 4-byte ASN will be just fine, but it might be worth asking your ISP just in case.

At this point, you should be ready to hit the ground running as soon as that second Internet uplink is installed. This is also assuming you already run a router or multilayer switch on the edge of your network, which also has BGP capabilities. So let’s get down to the fun stuff - an extremely basic configuration to peer between two ISPs. I’ll dedicate another post to additional recommended settings and configurations - but for now let’s focus on getting this running. The configuration sample below is aimed at Cisco devices, but the same concepts apply to most vendors:

EdgeRouter(config)# router bgp *<YOUR AS NUMBER>  *! The AS number provided by ARIN
EdgeRouter(config-router)# network *<YOUR LOCAL SUBNET>*   ! The subnet we need to advertise out both ISPs
EdgeRouter(config-router)# neighbor *<ISP1 PEER IP>* remote-as *<ISP1 ASN>* ! Provided by the first ISP - Their remote peer IP and ASN
EdgeRouter(config-router)# neighbor *<ISP2 PEER IP> *remote-as *<ISP2 ASN> *! Provided by the second ISP

As I mentioned, this config is very basic and will just accomplish what we need to get going. Follow up with a quick show ip bgp neighbors and hopefully you’ll see two peers in the established state. Any other state indicates a problem bringing up the peer connection. I won’t get into too much detail here - but check the physical connection, ping the peer, and make sure there are no firewalls blocking TCP port 179 between the peer addresses.

Hope this was helpful! Comment below and let me know how your experiences have gone with this type of setup - and look forward to a few more posts regarding BGP peering setup with multiple ISPs.