With a lot of the recent nonsense going on after VMware was acquired last year, I’m seeing people start to look for ESX alternatives - especially for home labs. In fact, I also just migrated my home lab from ESX to Proxmox.

One of the things I needed to keep running was Cisco Modeling Labs (CML). I often use CML to build quick virtual topologies for testing things, so it was important for me that it worked on Proxmox.

It’s worth noting that Proxmox isn’t an officially supported platform. The docs state that only bare metal & VMware platforms are supported.

That being said, I’m happy to say that Proxmox has worked just fine for me - it just needed a few tweaks. Though just be warned that you may still run into issues and it is technically an unsupported configuration.

In the post below, I’ll share the steps that I used to get CML up & running on Proxmox.

Prerequisites

To start with, we’ll need a few things:

• A server running Proxmox
  • I’m currently using version 8.1.3
• A CML install ISO, found here
  • We’ll be using version 2.6.1 for this guide
• CML’s reference platform ISO, also found at the link above

Our Proxmox machine will still need enough resources to build a VM with the minimum requirements listed here.

The CML & refplat ISOs should be placed into a Proxmox ISO storage location.

Creating the VM

Next we can get to the fun part: Setting up the VM in Proxmox.

We’ll first locate a node in Proxmox to place our VM. Then right-click that node & select Create VM:

Then, we’ll give our VM a name & ID. I’ve named mine CML:

On the next page, we can select our CML ISO from the correct storage device. We can leave the Guest OS as Linux and 6.x - 2.6 Kernel:

Next, we’ll need to make some minor adjustments on the System tab.

The default BIOS will likely be set to Default (SeaBIOS). We’ll change that to OVMF (UEFI), which will also give us a few additional options.

We’ll keep Add EFI Disk checked, and select a storage location for that EFI disk.

On the Disks tab, feel free to increase the size of the VM disk. While 32GB is the minimum required, you’ll likely want more than that. I’ve increased mine to 50G for now. Keep the image format as QEMU.

We’ll also want to update the Async IO setting to native. This is hidden under the Advanced settings:

On the CPU tab, we’ll update the core count to a minimum of 4 per the requirements. Of course, you’re welcome to increase this as needed.

More importantly however, we’ll need to set the CPU type to host. This allows the VM to use the underlying nested virtualization features:

Next we can assign memory to our VM.

The requirements state a minimum of 8GB, however this will likely only accommodate simpler labs. Some individual CML nodes require more than that to start.

In my case, I’ll start with 32GB - and we can always increase this later:

Lastly, on the Network tab, no changes are necessary. Of course, you can assign a VLAN if needed.

After that, we can head over to the Confirm tab & finish up the wizard.

Add Refplat ISO

Next, we’ll need to make sure our reference platform ISO is connected to the VM.

We’ll head over to the VM’s hardware tab, then click Add and select CD/DVD Drive:

Then we’ll select the appropriate ISO storage, and pick our refplat ISO file:

Install CML

Once that’s done, we can go ahead and power on our VM & pop open the console!

If you’ve installed CML previously on another platform, this process is just the same.

At boot, we’ll select Install CML.

After a moment & a few screens, we’ll start the system configuration.

First, we’ll input a system hostname:

Then, we’ll input credentials for the system management user.

Note: This user isn’t used to log into the CML software. Rather, it’s used for system administration tasks via the Cockpit UI, such as: Installing updates, checking logs, joining a domain, or powering off / rebooting the system.

After that, we can configure our CML admin user:

Next we have our network config. By default, CML will prompt to use DHCP, but we’ll likely want to change this to a static IP assignment:

After that, CML will prompt us to confirm our settings - then it will begin the installation. Since a lot of the VM images come from the reference platform ISO, this process may take a while as each of those images are copied over to the system.

When CML is ready, we’ll see something similar on the console telling us how to reach the web UI:

Again for reference, the main UI is reachable at https://<CML IP> and the Cockpit management UI is at https://<CML IP>:9090.

At this point, we can log in & start building labs!

The process for setting up CML on Proxmox isn’t super crazy, but there are just a few tweaks that need to be made during setup. I wrote this hoping that if anyone else out there is trying to set this up, maybe it would help.

As always, Thanks for reading!

In this blog post, I wanted to spend just a few minutes going over how to solve an issue one of my peers ran into recently.

The Issue

Let’s say we were trying to use Ansible to update the interface admin state on a Cisco Nexus switch. In this case, I’m using a virtual Nexus 9000 which is running NX-OS version 10.2(6).

This switch has a loopback interface (lo100), which we want to disable.

Reading the Ansible documentation for the NETCONF module, we might assume that the following would be enough to get the job done:

- name : NX-OS Interface Updates
  hosts: all
  connection: netconf
  gather_facts: no

  tasks:
    - name: Update Interface State
      ansible.netcommon.netconf_config:
        content: |
            <config>
              <System xmlns="http://cisco.com/ns/yang/cisco-nx-os-device">
                <intf-items>
                  <lb-items>
                    <LbRtdIf-list>
                      <id>lo100</id>
                      <adminSt>down</adminSt>
                    </LbRtdIf-list>
                  </lb-items>
                </intf-items>
              </System>
            </config>

However, once we run that playbook, we’ll get an error back:

(ansible) matt@ansible-dev:~/ansible/nxos$ ansible-playbook nx-update-interface.yaml -i inventory.yml

PLAY [NX-OS Interface Updates] **********************************************************************************************************

TASK [Update Interface State] ***********************************************************************************************************
fatal: [10.122.2.229]: FAILED! => {"changed": false, "msg": "Request without namespace and filter is an unsupported operation"}

PLAY RECAP ******************************************************************************************************************************
10.122.2.229               : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

The Fix

Luckily, this has an easy solution.

According to the Cisco NX-OS Documentation, there was a change in version 9.3(1) that requires a filter to be sent along with any GET or GET-CONFIG request.

But wait - aren’t we updating the config via an EDIT-CONFIG request? We shouldn’t need to send a filter, since it’s not a GET or GET-CONFIG, right?

Well because of the way Ansible works, we actually send two GET-CONFIG requests along with our EDIT-CONFIG. One before the change, and another one after making the change.

These are used to validate that we actually updated the config. Ansible uses these two GET-CONFIG requests to compare & see if there is actually any difference between the two.

So to address our error above - when we send our config change in Ansible, we also need to include the get_filter parameter. This will give Ansible a NETCONF filter use when it issues the GET-CONFIG requests to validate our changes.

In our case, that GET-CONFIG filter could look something like this:

<System xmlns="http://cisco.com/ns/yang/cisco-nx-os-device">
  <intf-items>
    <lb-items>
      <LbRtdIf-list>
        <id>lo100</id>
        <adminSt/>
      </LbRtdIf-list>
    </lb-items>
  </intf-items>
</System>

That filter would pull back just the admin state for our lo100 interface, which then Ansible could use to compare after we issue our request to disable the interface.

For what it’s worth, we don’t necessarily have to be that specific with our filter. We just need to ensure that whatever config returned by the filter would also include the section we intend to change.

So for example, we could actually just filter by System and this would still work:

<System xmlns="http://cisco.com/ns/yang/cisco-nx-os-device">
</System>

The main difference here is that the switch would return a much larger amount of data, which then Ansible would also need to process to detect the change. So for efficiency & accuracy, it’s probably still helpful to keep our get_filter scoped to a reasonably narrow amount.

With all that covered, here’s the example of our final playbook with the new filter:

- name : NX-OS Interface Updates
  hosts: all
  connection: netconf
  gather_facts: no

  tasks:
    - name: Update Interface State
      ansible.netcommon.netconf_config:
        get_filter: |
          <System xmlns="http://cisco.com/ns/yang/cisco-nx-os-device">
                <intf-items>
                  <lb-items>
                    <LbRtdIf-list>
                      <id>lo100</id>
                      <adminSt/>
                    </LbRtdIf-list>
                  </lb-items>
                </intf-items>
              </System>
        content: |
            <config>
              <System xmlns="http://cisco.com/ns/yang/cisco-nx-os-device">
                <intf-items>
                  <lb-items>
                    <LbRtdIf-list>
                      <id>lo100</id>
                      <adminSt>down</adminSt>
                    </LbRtdIf-list>
                  </lb-items>
                </intf-items>
              </System>
            </config>

And, if we try to run the playbook again:

(ansible) matt@ansible-dev:~/ansible/nxos$ ansible-playbook nx-update-interface.yaml -i inventory.yml

PLAY [NX-OS Interface Updates] **********************************************************************************************************

TASK [Update Interface State] ***********************************************************************************************************
changed: [10.122.2.229]

PLAY RECAP ******************************************************************************************************************************
10.122.2.229               : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

That looks much better!

Thanks so much for reading this blog post. I really hope it was helpful to you!

If you found value in this, please let me know! Feel free to leave me a comment below, or follow me on YouTube!

Oh, and for those of you on Mastodon - You can find me at @matt@0x2142.com.

Thanks again!

In this post, we’ll walk through how to connect an OPNsense firewall to Mullvad’s wireguard VPN. This can be used in various deployments to help protect your home network traffic.

As a quick note, this post is not sponsored by Mullvad - I just happen to really like their service & appreciate their approach to privacy and security. In fact, they currently don’t offer any incentives or paid promotions of their product. More info on their current policy can be found here.

Creating an Account

Mullvad is unique in that they don’t really require any sign up to get started with their service. Instead, when you sign up for new service, they randomly generate an account number. That’s it. They don’t require any additional information from you.

So first we can hit their new account page here - then click to generate a new account number.

Keep this account number somewhere safe. Mullvad has very limited ability to help you recover the account number.

Once we have that, we can click the button to Add time to our account.

Mullvad doesn’t support any recurring subscriptions, so that they are able to keep less data about their customers. Instead, we just add time increments in months for how long we would like to use the service. This can be anywhere from a single month up to a year.

In terms of payments, they do offer a number of options, including the usual PayPal/credit cards - or if you’re truly concerned about privacy, they accept a few cryptocurrencies or you can actually mail them a cash payment as well.

After a payment has been made & time added to the account, we can quickly jump into the configuration side of things.

Config: Mullvad Side

First we’ll take a quick look at the configuration required on the Mullvad VPN side of things.

On the left side of the account page, we’ll click on WireGuard Configuration.

First we’ll select Linux as our platform (In this context, what is selected here isn’t really important) - and then click the button to generate WireGuard keys:

Note: Alternatively, we could have generated our WireGuard keys on our OPNsense firewall - then applied them here. It’s up to you on which method you prefer!

Next, we’ll take a look at which server(s) we would like to connect to.

When selecting a server, we have the option to pick our desired country & location - as well as picking a specific server to connect to if we choose. There is also an option to select all servers - in which case the config generator will create a WireGuard configuration file for each server.

For the purpose of this walk through, we’ll keep things simple & only select a single server. So in the screenshot above, I’ve selected us-qas-wg-004.

We’ll also take a quick look at the advanced options, which give us a little flexibility if we need it. For most people, these additional options will not be necessary to change or modify.

We can enable Multihop functionality, so long as we only selected a single server to connect to. So if you selected the All Servers option, this won’t be available. This allows us to specify an entry & exit server for our VPN. In other words, our device would directly connect to the entry server we select - then Mullvad would tunnel our traffic across their network to the exit server, where our traffic would be decrypted & forwarded out to the internet. Depending on your privacy & security desires, this is a really nice option to have the ability to enable.

Next, we have options on what type of connection we would like & which traffic to forward.

Server connection protocol will specify whether we are using IPv4 or IPv6 between our device & our VPN server. Most likely you’ll want to leave this at IPv4, unless you have an IPv6-only internet connection (or if you just would prefer to use IPv6 anyways!).

Tunnel traffic is how we can specify whether we would like IPv4 or IPv6 client-side traffic to be forwarded over the VPN. So this would depend on whether the clients on our network have IPv4 vs IPv6 connectivity, or both - and whether we prefer to only forward certain types of traffic over the VPN. I’ll be leaving this setting as the default: Both.

Next we can specify a custom port if we would like. By default, wireguard will use UDP port 51820 - and we probably won’t need to change this unless the port is being blocked upstream.

Lastly, we also have the option to enable content blocking across the VPN. Mullvad accomplishes content filtering through DNS-level blocking - and when we finish generating a configuration file, the file will include DNS servers to use. This is okay to use if you were connecting a single client to their VPN service. However, if you’re using a router or device like OPNsense, you would need to update the DNS on all clients on your network to make this work. This is possible by updating DHCP on our router with the new DNS server address - or configuring a DNS rewrite. We won’t get into either of those in this post - so for now I will leave the content blocking options unchecked.

Once we’re good with our configuration - we can click the Download File button. We’ll get a standard WireGuard config file, that looks like this:

At this point, we’re good to move onto the next part!

Config: OPNsense Side

Okay, so now that we have everything ready to go on the Mullvad side - we can configure our OPNsense device.

Make sure you already have WireGuard installed on OPNsense. This can be done by navigating to System > Firmware > Plugins then searching for wireguard & clicking the install button.

Next, we’ll enable Wireguard by navigating to VPN > Wireguard and checking the box to Enable WireGuard, then Apply.

Then we’ll hop over to the Endpoints tab & configure our Mullvad VPN peer.

For this part of the configuration, we’ll just copy our public key, allowed IPs, endpoint address, and endpoint port from our Mullvad config file. In the screenshot below, I also named my endpoint with the specific Mullvad server I’ll be connecting to:

Then we can click Save and Apply.

If you wanted multiple Mullvad servers configured, just create a new endpoint for each one. Then, make sure that you select all of the Mullvad peers on the next step below.

After that, we can move over to the Local tab to define our OPNsense tunnel configuration.

Click the button to add a new peer, then we’ll fill in our private key and tunnel address(es) from the Mullvad config file. Under Peers, we’ll also select our Mullvad VPN peer that we configured just a moment ago:

UPDATE: Looks like with a recent OPNsense update, they now require you to enter both the WireGuard private & public key into the local config (shown above). In the the screenshot, I only show entering the private key - since this was all that was required at the time. Two ways to get your public key:

1. Log into Mullvad & check the “Devices” tab under “Account Management”. This will show your device public key (They don’t keep your private key after generating it for you, only the public).
2. If you have wireguard installed somewhere else, you can use the “wg pubkey” command to derive a public key from your private key. Command: echo <private_key> | wg pubkey

Note: By default with the configuration we’ve applied so far, this VPN will forward ALL traffic on our network to Mullvad. If we would prefer to selectively choose which traffic to send over the VPN, we can check the box for Disable Routes - then use policy routing to forward specific things to Mullvad. We’ll take a look at how to do this later in the post - but for now just be aware that our current configuration will forward ALL traffic.

Okay, with that all done we can click Apply and Save here as well.

With any luck, we can check the Status tab & see that there is data being transmitted & successful WireGuard handshakes:

However, before our clients traffic can be forwarded over the VPN, we’ll need to create a firewall rule to permit traffic & a NAT rule to translate our client addresses to our Mullvad IP.

We’ll navigate to Firewall > Rules > WireGuard (Group). Then we’ll click to create a new rule.

Within this new rule, I’ll update Direction to Out and change TCP/IP Version to IPv4+IPv6. I’ll leave the source as Any:

We can also leave the Destination as Any, but I’ll update the rule to enable logging:

This rule will allow any clients behind our OPNsense firewall to reach anything on the internet.

Next, we’ll have to create a NAT rule. This ensures that our client addresses on our network get appropriately translated to the tunnel IP address that Mullvad has assigned us.

We’ll navigate to Firewall > NAT > Outbound. By default, OPNsense will be set to Automatic outbound NAT rule generation. We’ll need to update this to Hybrid outbound NAT rule generation to allow custom NAT rules. Then we can click Save and Apply.

Next, we’ll create a new Manual NAT rule, where we’ll update our Interface to WireGuard (Group):

Then we’ll make sure our Translation / target is set to Interface Address - and again, I’ll enable logging:

After we click save, we should have a NAT rule that looks like this:

At this point, we should be good to test our clients!

Testing

Of course, it’s easy enough to use one of our clients to check that we still have internet access - but how can we be sure that they’re using the VPN?

The easiest way might be to check the mullvad.net, where they do have a quick validation at the top of the page:

So according to Mullvad, it looks like we’re connected & they even show which server we’re connecting from.

We can also double check using a traceroute or tracepath command:

In this case, we can see that our traffic to Google hits our OPNsense gateway, then the Mullvad VPN gateway followed by another external address owned by Mullvad.

So based on some quick testing, it looks like we’re all good!

Policy Routing

So in the above walkthrough, we configured a Mullvad VPN from our OPNsense firewall - but it is forwarding ALL of our network clients over the VPN. What about if we only wanted certain clients to use the VPN? Or all clients to use it, but only for certain destinations?

We can accomplish this through policy routing.

So the first thing we’ll do is go back to our WireGuard config, then under the Local tab. We’ll edit our configuration here, and check the box for Disable Routes.

By default, OPNsense / WireGuard will install routes for any IPs listed in the AllowedIPs field for each peer. In our set up, we configured 0.0.0.0/0 - which matches all traffic. By checking the box for Disable Routes, we prevent OPNsense from installing that default route - and instead we can manually specify our own.

If you’re curious to double check this, you can try hitting Mullvad’s website after changing this setting - and it should show that you’re no longer connected.

Then, we’ll need to set up our WireGuard configuration to use a dedicated, named interface so that we can create a static gateway.

We’ll head to Interfaces > Assignments - and create a new interface. From the drop-down, we’ll select our WireGuard interface - in my case this was wg1. Then we can assign it a name:

Then click the + icon to add, and Save.

Then we can navigate to the interface name under the Interfaces menu - and enable the new interface:

Next we can create a gateway to route traffic through. Navigate to System > Gateways > Single.

Create a new gateway & give it a name. Then we’ll enter our Mullvad VPN gateway IP address, which in my case was 10.64.0.1. How did we find this? Well earlier when we tested our VPN connectivity - we performed a tracepath. In the output of this tracepath, our second hop (the one right after our OPNsense firewall) would be the Mullvad VPN gateway. So this is the address we’ll use for our gateway here:

Then clic Save and Apply.

Note: You may get an error here, like “The gateway address “10.64.0.1” does not lie within one of the chosen interface’s IPv4 subnets.” To resolve this, we’ll temporarily change our interface address. Navigate to the WireGuard local config, and re-enter your tunnel address excluding the “/32”. For example, if your tunnel IP was 10.10.10.10/32, change this to just 10.10.10.10 You should be able to set the gateway address now. Be sure to change your tunnel IP back afterwards!

Next, we’ll create a firewall rule for each set of sources or destinations we would like to manipulate.

So we can navigate to Firewall > Rules > Floating (or select a specific interface for clients, like LAN).

In here, we’ll set whichever parameters we would like to match. So for this example, let’s say I have a client PC at 10.100.100.10 and I only want traffic to 8.8.8.8 to use Mullvad. All other traffic can use the normal internet connection & not use the VPN.

In that case, I’ll set my source address to 10.100.100.10/32 and my destination to 8.8.8.8/32:

Then all we need to do is update our Gateway to use the Mullvad gateway we just created:

Now, if we hop over to our client PC - Mullvad’s website will say that we’re not connected to the VPN. However, we can check that traffic to 8.8.8.8 is actually being sent through Mullvad using tracepath again:

In the screenshot above, I also included a tracepath to 8.8.4.4, just to show that it is going through my normal internet connection & not Mullvad.

This was just one example, but you could easily create multiple firewall rules for different sources and/or destinations to control where traffic is sent. Aliases can also be used to group sources or destinations, so that multiple can be added to a single firewall rule.

Okay - I think that’s about all I wanted to cover in this post.

Hope it is helpful! Feel free to leave a comment below - or follow me on YouTube 😊

In this post - we’ll take a look at how to set up & configure AdGuard Home on OPNsense.

Please note that the AdGuard Home plugin for OPNsense is a community built plugin, and not officially supported by OPNsense.

What’s AdGuard Home? Why use it?

Almost every website we visit these days is loaded with additional components for advertisements, analytics, and engagement tracking. One one side, these tools can be very helpful for the company or website owner to monetize their platform and/or track & understand their audience’s interests.

However, it’s also becoming more popular to want to avoid being tracked on every website, or reduce the amount of advertisements you see. Unfortunately, a lot of these scripts & code snippets are automatically embedded in websites and most don’t allow you to opt-out.

A while back, there were a few browser extensions that became popular by automatically blocking the advertisement & tracking elements from loading. These were great (and still are!), but a lot of website owners have been fighting it & making it harder to block their content. In addition, these types of extensions operate at your web browser level - meaning that your computer has already made a few calls out to the internet before the extension even has a chance to block something.

Here’s where we’ve started to see more ad blockers come out that operate at the network level. AdGuard Home is one of them, but you also may have seen similar packages like Pi-hole or NextDNS. These are typically packages that you install on your home network & run as a local Domain Name System (DNS) server.

Each time your browser needs to load something from the web, the first step is figuring out what IP address to connect to. For this, the computer reaches out to it’s configured DNS server and provides the website name (like 0x2142.com). The DNS server looks up where that lives & provides the computer with the IP address (like 203.0.113.52). Then your computer can load the website by connecting to that address.

With a DNS-level blocker, like AdGuard Home, we can block your computer from ever trying to establish that connection. If you tried to go to a website (like 0x2142.com), and there was an embedded advertisement or tracking, AdGuard would tell your computer that the domain hosting the advertisement doesn’t exist (usually via returning a 0.0.0.0 or NXDOMAIN response). So your browser would still be able to load the main site (0x2142.com), but it would never even try to establish a connection to the advertisement or tracking components.

So we gain a few benefits here - the big ones being some level of privacy & reduced advertisement noise when browsing the web. But also since we block so much of that noise early in the process, your computer never has the opportunity to load that content - meaning that we also save on bandwidth usage & data costs. There may also be small performance improvements since each site has less content that needs to be loaded.

The other bonus worth considering is security. There are quite a handful of DNS blocklists that are constantly updated with the latest malicious or suspicious domains. The quicker we can block & stop clients from potentially connecting to those domains, the better off we are!

Is there a down side? Yeah, of course there is! A lot of these DNS-level blockers pull from varying website blocklists - which are not always 100% accurate. So sometimes you may still see advertisements or get tracked. It’s not a perfect system. In addition, you may also (and sometimes often) see the reverse - parts of websites being blocked that are legitimate. And there are quite a handful of websites these days that won’t work correctly unless they can load 3rd party components. Most of the time everything will be fine, but just be aware that there may be some time spent troubleshooting & manually unblocking website components.

Do I have to install this on OPNsense?

Nope. AdGuard Home has a number of packages & ways to get running. Check out their GitHub repo.

If you’re already running OPNsense, it’s easy to install this as an add-on package & not have another system to manage. However, if you prefer to set up AdGuard (or Pi-hole, or others) elsewhere, that’s fine too. You’ll just need to update your client network’s DHCP options to use the new DNS servers. See the last section below on how to do that.

Okay - Let’s get started with setting this up!

Topology

For the purposes of this walkthrough, we’ll be using a fairly simple & straightforward topology. A single OPNsense appliance connected to the internet via it’s WAN port, as well as a single client PC connected via the LAN port.

In this setup, the OPNsense appliance is configured to provide IP address & DNS information to our client PCs via DHCP.

Adding the Community Repository to OPNsense

So by default, AdGuard Home is not included in the available plugins to download/install in OPNsense. However, someone built a community plugin repository that includes a small handful of additional packages.

Before we can install the AdGuard Home plugin, we will need to setup & install that community repository.

To do this, we’ll need direct SSH or console access to our OPNsense appliance.

SSH is disabled by default, but we can enable it quickly by navigating to System > Settings > Administration and then scrolling down to the Secure Shell section.

We’ll need to check the box for Enable Secure Shell and Permit Password Login. If you’re logging into OPNsense with the root account, you’ll also need to select Permit root user login.

Then scroll down to the bottom of the page & click Save.

Note: By default OPNsense will also have the SSH Listen Interface set to All. I would highly recommend setting this to only enable on your LAN interface Also: If you don’t need SSH access all the time, please remember to disable this service once you’re finished setting this up!

Okay, now that’s enabled - we can connect to our OPNsense appliance using your preferred SSH client (like PuTTY).

If you’re using the root account, you’ll likely be dropped into the OPNsense shell - but you can select option 8 here to access the underlying FreeBSD shell.

In order to install the community repository, we’ll pull down the repository config file using the following command:

fetch -o /usr/local/etc/pkg/repos/mimugmail.conf https://www.routerperformance.net/mimugmail.conf

Then, we’ll need to ask OPNsense to update it’s local cache with the new repo - so it knows what packages are hosted there:

pkg update

If everything is successful, you’ll see output similar to below - which lists the mimugmail repository now:

root@0xOPNsense:/home/matt # pkg update
Updating OPNsense repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%  229 KiB 234.3kB/s    00:01
Processing entries: 100%
OPNsense repository update completed. 822 packages processed.
Updating mimugmail repository catalogue...
Fetching meta.conf: 100%    163 B   0.2kB/s    00:01
Fetching packagesite.pkg: 100%   54 KiB  54.8kB/s    00:01
Processing entries: 100%
mimugmail repository update completed. 177 packages processed.
All repositories are up to date.

Installing the AdGuard Home Package

Now that the additional package repository is set up, we can download & install the AdGuard Home plugin via the OPNsense web interface.

So back in our browser, we can nagivate to: System > Firmware > Plugins. On this page we can search for adguard or scroll through the list to find it.

Then we just click the plus icon on the right side to install (not shown in the screenshot above).

This should install pretty quickly:

***GOT REQUEST TO INSTALL***
Currently running OPNsense 22.7.9 (amd64/OpenSSL) at Sun Dec  4 12:48:38 EST 2022
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
Updating mimugmail repository catalogue...
mimugmail repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
 os-adguardhome-maxit: 1.8 [mimugmail]

Number of packages to be installed: 1

The process will require 35 MiB more space.
7 MiB to be downloaded.
[1/1] Fetching os-adguardhome-maxit-1.8.pkg: .......... done
Checking integrity... done (0 conflicting)
[1/1] Installing os-adguardhome-maxit-1.8...
[1/1] Extracting os-adguardhome-maxit-1.8: .......... done
Stopping configd...done
Starting configd.
Migrated OPNsense\Adguardhome\General from 0.0.0 to 0.0.1
Reloading plugin configuration
Configuring system logging...done.
Reloading template OPNsense/Adguardhome: OK
Checking integrity... done (0 conflicting)
Nothing to do.
***DONE***

Now all we have to do is enable the plugin.

So we’ll navigate down to Services > Adguardhome > General. Our only option here will be an Enable checkbox, so we’ll select that & Save.

The rest of the setup & initial configuration will be done directly from the AdGuard-specific web interface.

Initial Setup

By default, the AdGuard Home web interface will run on port 3000 & is not HTTPS-enabled. So if your OPNsense firewall is at https://192.168.1.1, you’ll need to connect to http://192.168.1.1:3000.

As long as that works - we’ll see the initial setup prompt below:

We’ll click on Get Started.

Now we’ll be asked to configure the Admin Web interface (the interface we’re connected to now) and the DNS server interface (which clients will use to resolve domain names).

By default, AdGuard home will try to set both of these to listen on All interfaces - and set the web on port 80 & DNS on port 53.

I would recommend setting the Listen Interface on both of these to only your LAN-side networks. There is no reason to enable them on your WAN, and it can be a security risk to do so.

You may also get warnings that port 80 & 53 may already be in use. For the web interface, we could change 80 to 3000 & just keep what we’re using now.

However, if we change the default DNS port, that will cause some additional problems since client machines will query port 53. Likely if port 53 is already in use, it’s because another service on OPNsense (like Unbound DNS) is already enabled. In my case, I disabled this in favor of using AdGuard. However, if you want to use both - you can change the default DNS port in AdGuard to something like 65353, then have Unbound forward requests to AdGuard (More on this down below).

So here’s what my set up looks like so far, with 192.168.1.1 being my LAN side interface:

On the next page, we’ll be prompted to set up an administrative user & password for logging into AdGuard.

Next we’ll be given instructions on how to set up client devices. In my lab network, the OPNsense firewall is providing DNS server configuration via DHCP - so we’ll get to that configuration shortly.

For now, we’ll just click Next.

On the last screen, we’ll just get a message saying that setup is complete & a link to open the dashboard:

And now we can log in:

AdGuard Home Configuration

After logging in, the first thing we’ll see is a pretty empty dashboard. We don’t have any clients configured to use this yet, so there isn’t anything to report on.

Blocking Domains

First thing we’ll look at is our DNS blocklists. We’ll navigate to Filters > DNS blocklists.

Here is where we can ask AdGuard to query lists of what domains to block. By default, AdGuard does include two - but we can add more if we want:

If we want to add to the configured blocklists, we can do so by clicking the Add Blocklist button. This will prompt us whether we want to choose from a pre-populated list, or supply our own custom list:

The easy option will be selecting from the provided lists:

There are a ton of different curated block lists available depending on what you’re trying to block. If we wanted to use a custom list, a lot can be found on GitHub just by searching for PiHole or Adguard blocklists.

How to pick a blocklist will be up to you. There are blocklists that focus on advertisements, tracking & analytics, parental controls, etc. So it just depends on what areas you want to focus on.

Allowing Domains & Custom Filtering

If we have a list of known services that we want to ensure are never blocked, we can pull those lists via Filters > DNS allowlists. However, it’s more likely you’ll find a handful of domains you want to unblock, rather than a whole list.

For that - we can go to Filters > Custom filtering rules. At the bottom of this page there is a tool to check filtering, where we can enter a domain name & instantly see what the result is.

For example, with the default ruleset I’ll check to see if 0x2142.com is filtered:

So by default that domain isn’t found anywhere, so it will be permitted. The tool also gives us a convenient button to quickly block a domain.

We can click that button, or add the syntax ||0x2142.com^ to the custom filtering rules at the top of the page (and saving via the Apply button). Now if we check the results again - the filter check will show the domain is blocked:

And of course, we don’t want to block 0x2142.com!! So let’s add this to our allowlist instead, so that it can never be blocked 🙃. We can do that by adding @@||0x2142.com^ to the custom filtering.

And now we’ll see a green box that shows that the domain is permitted via an allowlist:

Blocking Known Services

The other option worth mentioning is the ability to block certain known services, like WhatsApp, Twitter, Reddit, etc. This can be great if there are certain services you want to block, or for use as parental controls.

This can be found on the Filters > Blocked Services page.

This way we can select a service to block, rather than having to know all of the individual domains that service uses. For example, I’ll go ahead and select YouTube to block - and we’ll check that later on after we configure our clients.

Configure OPNsense DHCP to use AdGuard

Now that we’ve taken a quick look at the AdGuard Home settings & have a few things configured - let’s look at setting up our clients to use our new DNS server.

In the lab environment I’m using, the OPNsense appliance is providing client IP address configuration via Dynamic Host Configuration Protocol (DHCP).

By default, if a specific DNS server is not configured for your client DHCP settings, then OPNsense will provide the clients with the same DNS server it uses. This could have been a DNS server that was configured when you set up OPNsense, or it also can use DNS servers that are provided by your internet service provider.

So to update our LAN DHCP configuration, we’ll head back to our OPNsense web interface. From there, we’ll navigate to Services > DHCPv4 > [LAN].

In the configuration, there is an open option for DNS Servers. We’ll set this to our OPNsense LAN IP address. In my case, that is 192.168.1.1. Then scroll to the bottom of the page & click Save.

Client Testing

Now we should be all set up! However, it’s important to note that because of the way DHCP works, clients may not pick up the new configuration immediately. When DHCP assigns an IP address, it also tells the client how long it can use that address for. So if a client stays powered-on & connected, it won’t ask for new configuration until that timer expires.

We can speed that up by resetting the network interface on our clients. This can be done in a number of ways including rebooting the client or simply disconnecting from wifi/ethernet & reconnecting.

I’m using a Linux computer as my test system, so first I’ll check via the nslookup command - which will query our configured DNS server & return the resolved IP addresses.

If you remember, I blocked all of YouTube’s services earlier:

As we can see, we did get the correct IP addresses - which means our filtering isn’t working yet.

I’ll reset the network adapter on the test PC, which will refresh the DHCP configuration - then try again:

Now that’s the result we want! By returning the 0.0.0.0 result, our client can no longer resolve that domain. So if this was an advertisement or tracking domain, it’s now blocked from loading.

And sure enough, if we now try to browse to that site via a web browser - we don’t be able to access it:

Troubleshooting Blocked Domains

Okay, so now we know our blocking works…. But now someone in our home is trying to access YouTube & it’s not working. How can we tell if that’s our AdGuard service?

Our first stop might be the AdGuard query log. Opening this log, we can filter by domain name or client - or show only blocked queries if we like.

Pretty quickly we can see the issue - we blocked YouTube’s services:

Now we know how to fix the issue, which would be to unblock that service. However, if it was just a specific domain that was blocked, we would likely want to add it to our custom filtering as we showed earlier.

Reporting

Last but not least, we can also check our AdGuard Home dashboard again, which should be much more interesting than before:

Here we can quickly see how many queries have been made & how many were blocked for various reasons. We’ll also see what clients are using our DNS server, and which are making the most queries.

Most interesting (at least to me), is being able to see the top domains that were queried or blocked. Here’s where you might find some interesting information. For example, on my test machine - it’s a fresh installation of Ubuntu & we used FireFox to test. But we can see that even during the brief time it’s been set up, almost all of the highest queried domains belong to Mozilla’s analytics services. So it may be tempting to add those to our custom blocklists.

Additional Info

What if I have AdGuard running on a different server? Or want to keep using Unbound DNS?

Sure - we can make both of those work.

For the first scenario, maybe we have AdGuard Home installed & set up on a Raspberry Pi on our network. For that, all we need to do is set that Raspberry Pi as the DNS server in our DHCP configuration on OPNsense. See above where we did that for our on-box AdGuard setup.

For the other situation, perhaps you want to use Unbound on OPNsense, but also AdGuard. There might be reasons for this - like even though Unbound does support DNS blocklists, AdGuard has better reporting tools. But on the other hand, Unbound has more features & configuration options for DNS than Unbound.

In this case, we would want to run AdGuard on a different DNS port (like 65353), then have Unbound forward those to AdGuard. See below if you need to change the port AdGuard uses for DNS.

Within OPNsense, we could go to Services > Unbound DNS > Query Forwarding. Then add a new custom forwarding entry. Here we can forward requests for specific domains if we want - or if we want to forward all DNS requests, we can leave the domain field empty. Then fill in the AdGuard information - so in my example this would be 192.168.1.1 and port 65353.

Then click Save and Apply!

How do I change the interface / port for the Web UI or DNS?

So perhaps we mis-typed something when configuring AdGuard. Or just wanted to change the interface IP address AdGuard listens on. No problem!

Unfortunately, since this is a community plugin - there is no configuration for the plugin within the OPNsense interface.

We’ll need to reconnect to the OPNsense command line to make some additional configuration changes. This can be done via SSH or the device console.

Once there, we can use the command edit /usr/local/AdGuardHome/AdGuardHome.yaml.

That config file looks like this:

At the top, bind_host & bind_port pertains to the admin web interface. A little below there, under the dns section - you’ll see another bind_hosts and port config. Those ones are specific to the DNS server side of things.

Once done, save the config file by pressing Esc then selecting to quit the editor & save the file.

Lastly - Go back into the OPNsense web UI & restart the AdGuard Home service for the changes to take effect.

So in my last post, I picked up a Qotom Mini-PC to run OPNsense on. After a few months, the device has been running well & I’m very happy with it.

One of the new things I got to try out with OPNsense was Wireguard VPN. I had previously been using something else for VPN connectivity back to my home network - but I heard good things about Wireguard & wanted to give it a try.

So far my experience has been good! I’ve been pleasantly suprised with how easy it is to configure & get running. In addition, the performance & overall experience has been very positive. The VPN connects quicker than anything I’ve used in the past, and just simply works without issue.

All that being said - I wanted to put together a quick guide on how to configure Wireguard on OPNsense. Specifically, this configuraion will be for remote-access VPN - where clients will connect to a VPN headend. We’ll walk through the OPNsense configuration & a few clients as well. So let’s dig in!

Topology

For the purpose of this blog post, we’ll be using the lab topology below:

I will be using the reserved IP range 203.0.113.0/24 for the WAN-side addressing. I’ll have one Windows & one Android client that we’ll walk through & connect to the VPN.

Installing the Wireguard Plugin

To get started, first thing we will want to do is install the Wireguard plugin for OPNsense. By default, OPNsense will have standard IPSec & OpenVPN already available - but other VPN options can be enabled easily.

So in OPNsense, we’ll navigate down to System > Firmware > Plugins, then search for wireguard and click the plus icon.

This should pull down the package & install pretty quickly. No reboot required here!

Once installed, you may have to refresh the page or navigate to a new page so that the menu bar has a chance to reload. Then we’ll have a new option under VPN:

Wireguard Tunnel Configuration

Next we’ll begin configuring Wireguard on the OPNsense side.

There is a little bit of a chicken & egg scenario here since everything is based on cryptographic keys. We’ll need to generate keys on the firewall, which we need to enter on the client - but we also need the client keys to enter on the firewall. A bit of bouncing between the two - but for now we’ll try to complete as much as we can on the firewall side.

We’ll enable Wireguard by dropping down to VPN > WireGuard then clicking Enable and Apply

Next we’ll set up the Wireguard tunnel interface on OPNsense. This will be a virtual tunnel interface that will be created as interface wg<instance number>.

To do this, we’ll navigate to the Local tab, and click the plus icon to add a new tunnel.

In the above screenshot, I’ve filled in just a few details.

For Name, I’ve entered our virtual interface name wg1. Since OPNsense shows the Instance as 1 - it will create a wg interface with that instance number.

We’ll leave Public Key & Private Key blank for now. OPNsense will auto-generate these keys once we save this config.

For Listen Port, I’ve set this to 51820 which is the default for Wireguard. It’s not stated here, but this is a UDP tunnel.

For Tunnel Address - this is where we add the IP address of the virtual tunnel interface. This will be the gateway for our remote clients. In my lab I’ll be using 10.50.50.1/24 here.

We have no peers configured yet - so we can’t select any here. We’ll leave this blank for now, but come back later.

We will leave Disable Routes unchecked. By default, OPNsense will add static/connected routes for any client via the tunnel interface. You might not want this behavior if you wanted to do custom routing - for example, in a site-to-site VPN connection - but we’ll leave this enabled.

Once we’re done, we’ll click Save.

Like I mentioned before, OPNsense will now auto-generate our crypto keys for the tunnels. So if we edit our tunnel, we’ll now see those fields populated:

We’ll want to copy the Public Key & save it for later. This will need to be imported onto our clients, so that they can communicate securely with our firewall.

Wireguard Interface Assignment

Now that we have our headend tunnel interface defined, we can map our wg1 interface to an OPNsense interface. The OPNsense documentation suggests this is optional, but I would recommend it since it will allow us to create firewall rules to permit/deny access to clients.

We’ll navigate to Interfaces > Assignments, and we should see a New interface available: our wg1 tunnel.

We can assign this a name, then click the plus icon & Save.

Next we’ll enable the interface by navigating to Interfaces > WG1. Here we’ll only need to click Enable & save the change - nothing else is necessary.

Of course, we’ll be prompted to apply the changes - which we will do:

By default, all traffic through our WG1 firewall interface will be blocked - so please make sure to configure a firewall rule to permit traffic from the Wireguard clients.

Firewall Rules: Allow Inbound Wireguard Traffic

Next we need to permit the Wireguard traffic into our firewall. By default the WAN interface will block all traffic that isn’t explicitly allowed - including our Wireguard traffic.

For this, we’ll navigate to Firewall > Rules > WAN. Then click the plus icon to add a new rule.

The screenshot above shows what our firewall rule will look like.

Here’s the summary:

• Action: Pass
• Interface: WAN
• Direction: In
• TCP/IP Version: IPv4
  • You can enable IPv6 as well, if you have IPv6 connectivity (this is a lab box, which does not)
• Protocol: UDP
• Source: Any
  • This will allow anyone on the internet to reach our VPN. We could restrict source IP addresses, if our clients had permanent, static IPs.
• Destination: WAN Address
  • The firewall itself is the destination for this traffic
• Destination Port Range: (other) / 51820

I also enabled logging & added a quick description. After we have all this configured, we can click Save - then Apply Changes.

Client Setup - Windows

So first we’ll start with an easy configuration on a Windows client. Wireguard client software can be found on the Wireguard site here.

For the sake of the walkthrough, we will manually configure each client. However, this can be a difficult task if there is a large number of clients. Wireguard does support importing configurations, and there are a number of free tools available to help automate generating config files for clients - including some which will generate QR codes for easy import on mobile clients.

So on my lab Windows machine, we’ll open up the Wireguard client & click Add Empty Tunnel:

Then we’ll be given a blank config file, with only the devices public & private key pair generated for us.

We’re going to fill in details similar to the below screenshot:

The configuration under [Interface] is the local, client-side configuration. I’ve added the client’s tunnel address - which will be 10.50.50.15/32. I’ve also configured DNS servers which the client can reach via the VPN.

Then we’ll add the [Peer] section, which contains info about our VPN headend. Here’s where we’ll need the public key from our OPNsense firewall. We’ll also specify the Endpoint address, which is the IP or hostname of our VPN headend & the port (which by default is 51820).

We’ve also configured AllowedIPs as 0.0.0.0/0. This will force all client traffic over the VPN tunnel - including general internet traffic. However, we could limit this to specific subnets. For example, let’s say your network only used 172.16.90.0/24 & 10.1.1.0/16 subnets & we only wanted the user to be able to access those. We would configure the following: AllowedIPs = 172.16.90.0/24, 10.1.1.0/16. In this case, only traffic for those subnets would be routed over the VPN - any other traffic would use the devices default internet connection.

Now, we’ll save this - and again need to copy the device’s Public Key, which we’ll need to enter on the OPNsense firewall.

Client Setup - Adding Clients to OPNsense

In order for the Windows machine to connect to OPNsense, we’ll also need to configure a client profile on the firewall.

In OPNsense, we’ll navigate back to VPN > WireGuard, then click on the Endpoints tab.

Here we’ll configure a name for our client & paste in the client’s Public Key.

We’ll also set AllowedIPs to the client’s IP address, which we have configured as 10.50.50.15/32. This controls what IP addresses are reachable via this endpoint.

We do have some additional fields available, which we will leave blank. For example - Endpoint Address & Endpoint Port would be used to define a public IP that we expect our client to connect from. Since a remote-access client could connect from any IP, we leave those fields blank to allow this.

Once we’re done, we’ll click Save then Apply.

Next we’ll jump back to the Local tab, and edit our headend tunnel configuration.

We should now see our windows client in the Peers dropdown. We’ll select that client, so that Wireguard will permit that client to connect via this tunnel interface.

Then, as always, click Save & Apply

Last but not least - we should restart our Wireguard server on OPNsense. This can be done by either disabling & re-enabling Wireguard - or by navigating back to the OPNsense dashboard & clicking the restart icon next to the wireguard-go service.

Testing The Connection

Okay - now that we have all that completed, it’s finally time to test connectivity from our client.

On my Windows client, I’ll just click the Activate button for the tunnel:

And we’ll see that the Status shows Active, and we’ll start to see updates to the Last Handshake & Transfer fields - indicating we are connected & sending data.

To further validate, we can check the Log tab:

The key things to look for here, are the following messages:

• Receiving handshake response which indicates our firewall responded to our request to connect
• Keypair 1 created which indicates that our connection is healthy to our peer
• Receiving keepalive packet from peer - we should see these periodically to maintain our connection. If keepalives stop flowing, then we may have a break in connectivity between client & peer.

We should also be able to reach out wg1 tunnel address from the Windows client:

On the OPNsense side of things, we can check what client is connected via the List Configuration tab, under VPN > Wireguard:

On this screen, we can see we have 1 peer connected - which matches the IP & public key of our Windows client. We’ll see similar output like the Windows client, where we can see the latest handshake & data transfer.

Additionally, for easier access we can add the Wireguard widget to the OPNsense dashboard.

If we navigate to our dashboard, then click Add widget - we can add the Wireguard widget.

Here’s what that looks like:

Additional Client Setup - Android

Let’s also take a quick look at a mobile client. I’ll get through this pretty quickly, since the configuration will be very similar to our other client - just a different UI.

On my Android device - if I open up the Wireguard app, I have a few options for creating or importing a tunnel:

In this case, we’ll again walk through creating a tunnel manually. Again, it’s worth mentioning that there are 3rd party apps available to auto-generate config imports or QR codes to make this easier.

First we’ll have a handful of fields to fill in about the Android-side configuration. This includes a name for the tunnel, an address, and DNS servers.

We can click on the refresh icon in the Private Key field to auto-generate our key pairs:

One of the interesting parts of the mobile app is the ability to permit/exclude individual apps from traversing the VPN tunnel.

Here’s a quick screenshot, where we can pick either to allow only certain apps to use the VPN - or permit all except a few we choose to exclude:

In my case, I’ll keep this set to allow all applications.

Next we’ll work on the peer config:

After that, all we need to do is save our VPN configuration - then we can toggle the tunnel on or off:

And similar to the Windows client, we can click on the tunnel itself to see current status / data transfer:

So it looks like we should be connected & can try accessing VPN resources!

We can also use the same methods as earlier to check connectivity from the OPNsense side.

Okay - that’s all I wanted to share today. I’ve been quite pleased with how easy to use WireGuard has been - and how well it performs! I hope this blog post was helpful if you’re interested in trying it out.

Thanks!!

^{Note: I may receive commissions for purchases made through links in this post. This is to help support my blog and does not have any impact on my recommendations.}

One day I would love to have symmetric internet speeds, but for the meantime I have to work with what I have - which is dismal upload speeds that don’t come anywhere close to the download speed.

So I’m currently at 200Mb/s down & 10Mb/s up - and I would prefer to have higher upload speeds. However, even just 20Mb/s upload requires me to purchase a 500Mb/s download plan. 🙄

I’ve been hesitant to upgrade, since my current Meraki MX64 tops out at ~250Mb/s throughput. I would have to upgrade my firewall or half of my new download speed would be unusable.

While Meraki does have faster firewalls, they’re a bit expensive & at the moment somewhat difficult to obtain. So instead I searched around for alternatives, and came across the Qotom Q750G5 - which is a barebones mini-PC that I could load pfSense or OPNsense onto.

So in this blog post - I wanted to talk about the device & some of the performance testing I did.

Qotom Q750G5 - Hardware

First let’s take a quick look at the hardware. What got my attention quickly was the five 2.5GbE ports, but this small device offers quite a lot for how inexpensive it was.

A quick look at the specs:

• Intel Celeron J4125 Quad Core 2.0Ghz (Burst 2.7Ghz)
• Five Intel I225-V 2.5Gb Ethernet ports
• Optional WiFi slot
• Optional 3G Cellular & SIM card slots
• 1 RAM slot
• 1 mSATA slot
• 1 2.5in SATA HDD slot
• 2x USB 2.0 & 3x USB 3.0 ports

By default it seems this is a barebones kit, however I did opt to buy a model that included 16GB RAM & a 256GB mSATA SSD. It arrived with a single 16GB TeamGroup DDR4 3200Mhz module & 256GB Hoodisk mSATA SSD.

Are the specs a bit overkill for an embedded firewall? Yeah definitely. But the whole kit was only around $230 USD - which seemed crazy to me.

The big thing that pulled me toward this model (besides price) was the 2.5GbE ports. Since a lot more places have gigabit residential internet these days, and some lucky places are starting to see 2Gb/s - I was hoping that this little box would offer me a bit of future-proofing.

Of course, I was a little curious about what performance this device could realistically achieve - but more on that later!

I wanted to provide a few photos of the unit as well. The casing itself is pretty minimal, but the box is definitely heavier than you would expect!

Front Photo:

On the front there isn’t a whole lot to see. Power button, reset button, status LEDs, and USB ports. Oh - and there is an HDMI port as well, which comes in handy when installing an operating system (or if you’re using this as an actual PC).

Back Photo:

The back shows off the five 2.5Gb Ethernet ports, as well as the power plug. While I didn’t get a WiFi-enabled unit, the back faceplate still has cutouts for wireless antenna. I could opt to add wireless to this later, but I do wish the faceplate came with plugs or something to cover the holes in the meantine.

Inside Photo:

Now here’s where things get interesting! This box surprisingly packs a lot in it’s tiny size.

In the upper right, there is a PCI slot for WiFi. There’s also a WiFi/3G PCI slot in the lower left.

Just above the 3G slot is the SSD mSATA slot. When a drive is installed, it does cover the SIM card slot - meaning that the SSD would need to be removed to change SIM cards.

Also on the left, mostly out of the photo, is a connector for a 2.5in HDD. While I didn’t snap a photo of the bottom of the unit, the drive would screw into the bottom plate.

Interestingly enough, the CPU is on the other side of the motherboard. While you might think that the overall unit kinda looks like a heatsink - I was surprised to see that’s exactly what it is. The CPU is located right under the four screws with black washers and has thermal paste pre-applied. It rests up against the top of the case & uses the top as a heatsink.

As a last note about the CPU - it does get quite toasty at times. During one of my performance tests, the CPU reached ~90°C - and the top of the unit was very hot to the touch. It may not act as the world’s best heatsink, so I would avoid placing anything on top of the unit - or resting your hand there for too long 🙂.

Performance Testing

As a quick note before we get to the good stuff: All of my tests were performed using iPerf3. This may not show us the best real-world throughput tests, but I had quite a difficult time getting dpdk drivers to compile correctly for the Intel 2.5GbE ports (so I could use something like TRex). I may attempt to go back to this later, but the iPerf data is what I’ll provide for now.

The Testbed

For the performance testing, I used the two Intel NUC11 PCs that I purchased recently for my VMware lab. These already come with a 2.5GbE port, which I am currently using for vMotion between the two. I disconnected the vMotion port temporarily, and instead enabled PCI-passthrough to connect the ports directly to a VM.

I built a VM on each NUC with the following specs:

• Debian 11
• 8x vCPU
• 16GB RAM
• PCI-passthrough to 2.5GbE adapter

iPerf 3.9 was used for all tests. The iPerf server was enabled with the iperf3 -s command, and the client tests were run with iperf3 -c <client ip> -P 8 -t 600. Each test was run for 10 minutes.

The devices were connected & configured with the following topology:

Note: In the below paragraphs, I will talk about the configuration I used for each test. If you’re interested in more detail, please check out the video at the top of this blog post. In the video, I walk through the configuration & setup for most of the tests below.

Test 1: No Firewall

Avg: 2.35Gb/s

Okay, so expecting that I likely wouldn’t get the full 2.5Gb speeds once I added the firewall - I wanted to perform a baseline test with the VM’s directly connected via the 2.5GbE passthrough port.

In each of these tests, I was able to reach an average speed of: 2.35Gb/s

Test 2: Routing, NAT, Simple Firewall rules

Avg: 2.35Gb/s

In this test case, OPNsense was configured to match the diagram above. The iPerf server was located on the LAN segment, with an IP of 10.2.2.1 & a default route toward the LAN interface of the OPNsense box (10.2.2.2). The client is connected via the WAN interface, at 203.0.113.50.

I created a proxy-ARP virtual address for the 203.0.113.25 IP address, then created a NAT rule to forward traffic sent to that address to 10.2.2.1.

Next, there was a single firewall rule created to permit TCP port 5201 inbound from the WAN. This is the default port that iPerf will use.

During these tests, I averaged about 2.35Gb/s and relatively low CPU usage around 10-20%.

As an interesting side note to this: Originally I didn’t use the default LAN & WAN ports (ports 1 & 2), but instead used ports 3 & 4. During testing between those ports, I could only reach ~1.7Gb/s. Once I switched to ports 1 & 2, I could easily hit 2.35Gb/s. I’m curious to dig in later & see if this is a hardware issue, or possibly OPNsense is prioritizing the LAN/WAN traffic differently

Test 3: Large Firewall Ruleset

Avg: 2.35Gb/s

While my home network will likely have a somewhat minimal firewall ruleset, I was curious to see how well the device would perform with a large ruleset.

So I wrote a script to auto-generate ~1,200 firewall rules with random IP addresses, ports, and a mix of permit/block actions. I applied the ruleset inbound on both the LAN & WAN ports.

Under this test, I was still able to reach the 2.35Gb/s speeds - but now the CPU was creeping up to 30-40%.

So what next? Well I decided to see if I could stress the box a little - and increased the auto-generated ruleset to just over ~15,000 rules.

The increased ruleset took about 5-10 minutes to load into the system, and CPU was pegged at 100% the whole time. In fact, the CPU never really came back down. Even after the rules had loaded, the CPU was flat at 100% - and it took anywhere from 2-4 minutes to navigate between pages in the web GUI. (This is the part where my CPU temps went from <50°C to over 90°C 🙃)

I ended up having to re-image the box.

Test 4: Suricata IPS

Avg: 2.35Gb/s (But sometimes much, much less)

Next I loaded up the built-in IDS/IPS, which uses suricata. I downloaded all available free rulesets, and enabled all of them. Ideally, you wouldn’t necessarily enable everything - but I wanted to start off with a full load test.

My experience with these tests was highly variable. Most times, I was surprised to see that I could still push 2.35Gb/s through without any issue. During these tests, the CPU usually bounced between 50-80% usage.

Strangely, every so often I would test and get significantly less than that. Most times when this happened, I would instead only see somewhere between 400-700Mb/s - but on one test the box slowed to just over 200Mb/s. Each time this happened, the performance tests would remain degraded until I restarted the suricata service - then I would be able to reach the 2.35Gb/s again.

My best guess is that something is getting stuck. I know some older IPS systems I’ve worked with in the past had issues with CPU-pinning, and you might get garbage performance if your traffic hit the wrong CPU core. I didn’t spend a lot of time digging into this, but it feels a bit similar.

Test 5: Wireguard VPN

Avg: 800Mb/s, 650-700Mb/s with IPS also enabled

I also wanted to try VPN performance. There are a lot of options for VPN services within OPNsense, including standard IPSec and OpenVPN. However, I opted to try out wireguard - since it’s really easy to set up & get running.

In this case, I definitely hit a limitation with CPU-pinning. With my initial tests, I could only reach about ~450-500Mb/s - but the Qotom CPU was only spiking up to 70%.

Seems like the wireguard client (at least for same source/destination/port traffic) does pin everything to a single CPU. So my client CPU, between encrypting the traffic & generating it, was actually maxing out long before the Qotom was.

I ended up spinning up a second VM on the client side (which required disabling PCI passthough & adding both VMs to a shared vSwitch). With both of these running, I was able to max out the Qotom CPU at 100% - and hit a fairly consistent 800Mb/s VPN throughput.

As a final test, I enabled the suricata IPS on top of the VPN as well. Now the traffic would need to be decrypted & inspected before reaching the server. With a single client, I averaged ~400-450Mb/s - and with both clients it was around ~650-700Mb/s

Overall I’m really pleased with how well this thing performs, especially for how relatively inexpensive it was. I also expected OPNsense to have a steep learning curve, but it was surprisingly easy to get up and running very quickly.

Next I’ll be working on getting this firewall up & running at home. I may be tempted to write up some more on OPNsense - so if there are any questions or specific configurations you might like to see, please leave a comment!

I’ve been getting a handful of questions lately on the process of bringing a Cisco Catalyst 8000v or CSR 1000v into an SD-WAN environment. So I figured maybe I should put something together to share.

On a related note - I’ve been debating for a while doing a blog and/or video on building out a Cisco Viptela SD-WAN lab in EVE-NG. This would (tentatively) include everything from building controllers, bringing up remote sites, and template/policy configs. If this is something you might get value from, please let me know!! I’m looking for some motivation :)

Okay - all that being said, let’s go ahead and get started.

Note: This guide should work with CSR 1000v devices as well. But it will NOT be 100% accurate for physical ISR/IOS-XE routers, as there are some additional steps with certificates & the Plug and Play portal to get those running.

Topology

So to start with, figured I would share the topology that I’m working from. If you read my last post you might recognize this, but with an added location. This new location (site id 400) contains our Catalyst 8000v VM, running IOS-XE version 17.04.01a.

Controller or Autonomous Mode?

Back in the earlier days of IOS-XE SD-WAN, there used to be two separate software images to load on your network appliance - one for traditional IOS-XE, and one for SD-WAN code.

With the newer releases of IOS-XE, we’re now getting a unified image that contains both software sets. So our options now are two modes: autonomous (traditional IOS-XE) or controller (SD-WAN).

One way we can check this, is by running a show version and looking for Router operating mode:

Router# show version
<-- Output omitted -->
cisco C8000V (VXE) processor (revision VXE) with 2035355K/3075K bytes of memory.
Processor board ID XXXXXXXXXXX
Router operating mode: Autonomous
4 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
3965744K bytes of physical memory.
5234688K bytes of virtual hard disk at bootflash:.

Configuration register is 0x2102

Okay, and looks like the Catalyst 8000v image I’m using booted up in autonomous mode. No big deal, we can change modes pretty easily!

So in normal exec-mode, we’ll use the command controller-mode enable:

Router# controller-mode enable
Enabling controller mode will erase the nvram filesystem, remove all configuration files, and reload the box!
Ensure the BOOT variable points to a valid image
Continue? [confirm]

As noted in the snippet above - this command will erase the config! So you will want to take a backup snapshot of your existing configuration, if this is already being used. In my case, it’s a brand new VM - so we’ll continue on.

Once the device is back online, we’ll log in with the default login of admin/admin. Note that you will be forced to change this upon first login.

User Access Verification

Username: admin
Password:

Default admin password needs to be changed.

Enter new password:
Confirm password:
Router#

And just for fun, we’ll run a show version again to ensure we’re in controller mode:

Router# show version 
<-- Output omitted -->
cisco C8000V (VXE) processor (revision VXE) with 2035355K/3075K bytes of memory.
Processor board ID XXXXXXXXXXX
Router operating mode: Controller-Managed
4 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
3965756K bytes of physical memory.
5234688K bytes of virtual hard disk at bootflash:.

Configuration register is 0x2102

Catalyst 8000v Initial Config

So since this is a lab environment, and I’m not trying to provide any Day0 provisioning files - I’ll have to complete some manual configuration to get started.

First, we’ll start with some SD-WAN specific config (site id, org name, etc). I’ll be using the values that apply to my lab, so be sure to change these in yours!

Note: If you haven’t used IOS-XE SD-WAN previously, be aware that “conf t” doesn’t work! In SD-WAN/controller mode, you’ll use “config-transaction”. In this mode, all changes will need to be committed before they’re applied to the device.

Router# config-transaction
 ! Set hostname
Router(config)# hostname Cat8k-Site400
 ! Required SD-WAN system configs
Router(config)# system
Router(config-system)# organization-name "SDWAN-LAB"
Router(config-system)# site-id 400
Router(config-system)# vbond 192.168.99.241
Router(config-system)# system-ip 10.10.10.237
Router(config-system)# commit
Commit complete.

Cat8k-Site400# 

Once we have those basics configured, at a minimum we’ll also need to configure our internet-facing tunnel interface. This allows our Catalyst 8000v to communicate with our control plane for bring-up.

Cat8k-Site400#config-transaction
 ! Config physical interface (This is a lab, so I'm using a static IP)
Cat8k-Site400(config)# interface GigabitEthernet1
Cat8k-Site400(config-if)# ip address 192.168.99.237 255.255.255.0
Cat8k-Site400(config-if)# no shut
 ! Config tunnel interface
Cat8k-Site400(config-if)# interface Tunnel1
Cat8k-Site400(config-if)# no shut
Cat8k-Site400(config-if)# ip unnumbered GigabitEthernet1
Cat8k-Site400(config-if)# tunnel source GigabitEthernet1
Cat8k-Site400(config-if)# tunnel mode sdwan
Cat8k-Site400(config-if)# exit
 ! SD-WAN tunnel config
Cat8k-Site400(config)# sdwan
Cat8k-Site400(config-sdwan)# interface GigabitEthernet1
Cat8k-Site400(config-interface-GigabitEthernet1)# tunnel-interface
Cat8k-Site400(config-tunnel-interface)# encapsulation ipsec
Cat8k-Site400(config-tunnel-interface)# color biz-internet
Cat8k-Site400(config-tunnel-interface)# exit
 ! Default route to our internet gateway
Cat8k-Site400(config)# ip route 0.0.0.0 0.0.0.0 192.168.99.1
Cat8k-Site400(config)# commit
Commit complete.

Cat8k-Site400#

A Note on Certificates

Since this is a lab environment, I’m using self-signed local certificate authority to provision all of my certificate infrastructure. Because of this, I’ll need to install my local CA certificate on the Catalyst 8000v. If you’re using the default Cisco-provisioned certificate setup, you won’t need to do this.

Since my SD-WAN lab doesn’t have direct access to my local TFTP server - I do have an out-of-band management interface connected to my Cat 8000v. We’ll start by configuring that interface:

Cat8k-Site400# config-transaction
Cat8k-Site400(config)# vrf definition 512
Cat8k-Site400(config-vrf)# address-family ipv4
Cat8k-Site400(config-vrf)# exit
Cat8k-Site400(config)# interface GigabitEthernet 3
Cat8k-Site400(config-if)# vrf forwarding 512
Cat8k-Site400(config-if)# ip address dhcp
Cat8k-Site400(config-if)# exit
Cat8k-Site400(config)# ip tftp source-interface GigabitEthernet 3
Cat8k-Site400(config)# commit

The standard management VRF/VPN for SD-WAN is 512, so I kept that config to match when I configured this management interface. This will all be over-written anyways once we get connected to vManage & configure/push our template configs.

Once that’s done, we can go ahead and copy our CA certificate to bootflash.

Cat8k-Site400# copy tftp://10.0.0.2/cacert.pem bootflash:
Destination filename [cacert.pem]?
Accessing tftp://10.0.0.2/cacert.pem...
Loading cacert.pem from 10.0.0.2 (via GigabitEthernet3): !
[OK - 1406 bytes]

1406 bytes copied in 0.112 secs (12554 bytes/sec)

Cat8k-Site400# dir bootflash: | inc cacert.pem
16      -rw-             1406  May 14 2021 14:41:27 +00:00  cacert.pem
```text

Then we'll use the command below to install the CA certificate:

```text
Cat8k-Site400# request platform software sdwan root-cert-chain install bootflash:cacert.pem
Uploading root-ca-cert-chain via VPN 0
Copying ... /bootflash/cacert.pem via VPN 0
Updating the root certificate chain..
Successfully installed the root certificate chain

And we can validate by using the show sdwan certificate root-ca-cert command:

Cat8k-Site400 show sdwan certificate root-ca-cert
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            11:3b:0a:12:17:b0:e0:b5:4b:fa:c2:e9:2c:9c:12:84
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: DC = local, DC = 0x2142, O = SDWAN-LAB, CN = 0x2142-0XWIN1-CA-2
        Validity
            Not Before: Nov 29 20:00:11 2018 GMT
            Not After : Nov 29 20:10:11 2043 GMT
        Subject: DC = local, DC = 0x2142, O = SDWAN-LAB, CN = 0x2142-0XWIN1-CA-2
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
        <-- Output omitted -->

Activating the Catalyst 8000v

Almost there! Now that we’ve finished our pre-config & added our root CA certificate - we’re ready to join the Catalyst 8000v to the SD-WAN fabric.

We’ll start over in vManage - by going to Configuration > Devices.

Then we’ll find our target, unused Catalyst 8000v device. Click the ellipsis on the right side, then select Generate Bootstrap Configuration

This will give us a prompt to select which configuration style to generate. We’ll leave this on “Cloud-init”:

Once we hit okay - we’ll be presented with the info we need. We won’t necessarily need all of this information, but we’ll want to take note of our uuid and otp:

We’ll drag this info back over to our Catalyst 8000v, and we can now use it to activate the device & join to our SD-WAN fabric.

For the command below, chassis-number will be our uuid value - and token will be our otp.

Cat8k-Site400# request platform software sdwan vedge_cloud activate chassis-number C8K-178BXXXX-XXXX-XXXX-XXXX-XXXXXXXXBC24 token 421ecxxxxxxxxxxxxxxxxxxxxxbd53b5

Validation

After a few moments, we’ll see some log messages start to appear showing our control connections coming up. You might see these on the terminal if you’re using the console port, or you can use show log:

*May 14 15:39:06.910: %Cisco-SDWAN-Cat8k-Site400-OMPD-3-ERRO-400002: R0/0: OMPD: vSmart peer 10.10.10.242 state changed to Init
*May 14 15:39:07.980: %DMI-5-AUTH_PASSED: R0/0: dmiauthd: User 'vmanage-admin' authenticated successfully from 10.10.10.240:48140 and was authorized for netconf over ssh. External groups:
*May 14 15:39:08.798: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400002: R0/0: OMPD: vSmart peer 10.10.10.242 state changed to Handshake
*May 14 15:39:08.804: %Cisco-SDWAN-Cat8k-Site400-OMPD-5-NTCE-400002: R0/0: OMPD: vSmart peer 10.10.10.242 state changed to Up
*May 14 15:39:08.808: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400005: R0/0: OMPD: Number of vSmarts connected : 1
*May 14 15:39:09.827: %Cisco-SDWAN-Cat8k-Site400-OMPD-3-ERRO-400002: R0/0: OMPD: vSmart peer 10.10.10.243 state changed to Init
*May 14 15:39:10.584: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
*May 14 15:39:11.756: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400002: R0/0: OMPD: vSmart peer 10.10.10.243 state changed to Handshake
*May 14 15:39:11.762: %Cisco-SDWAN-Cat8k-Site400-OMPD-5-NTCE-400002: R0/0: OMPD: vSmart peer 10.10.10.243 state changed to Up
*May 14 15:39:11.762: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400005: R0/0: OMPD: Number of vSmarts connected : 2
*May 14 15:39:12.738: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400007: R0/0: OMPD: Using policy from peer 10.10.10.242
*May 14 15:39:13.570: %Cisco-SDWAN-Cat8k-Site400-FTMD-6-INFO-1000020: R0/0: FTMD: SLA class added : class 'Default' at index '1' loss = 25%, latency = 300ms, jitter = 100ms, app-probe-class = None
*May 14 15:39:14.738: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400007: R0/0: OMPD: Using policy from peer 10.10.10.242

We can also see our control connections using the command show sdwan control connections:

Cat8k-Site400# show sdwan control connections
                                                                                       PEER                                          PEER                                          CONTROLLER
PEER    PEER PEER            SITE       DOMAIN PEER                                    PRIV  PEER                                    PUB                                           GROUP
TYPE    PROT SYSTEM IP       ID         ID     PRIVATE IP                              PORT  PUBLIC IP                               PORT  ORGANIZATION            LOCAL COLOR     PROXY STATE UPTIME      ID
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
vsmart  dtls 10.10.10.242    100        1      192.168.99.242                          12446 192.168.99.242                          12446 SDWAN-LAB  biz-internet    No    up     0:00:00:51  0
vsmart  dtls 10.10.10.243    100        1      192.168.99.243                          12446 192.168.99.243                          12446 SDWAN-LAB  biz-internet    No    up     0:00:00:49  0
vmanage dtls 10.10.10.240    100        0      192.168.99.240                          12646 192.168.99.240                          12646 SDWAN-LAB  biz-internet    No    up     0:00:00:52  0

Of course, we can also check to see our device status in the vManage dashboard as well.

Over on the Monitor > Network page, we can see that our new Catalyst 8000v is now online:

Extra: How do I check the routing table on an IOS-XE SD-WAN device?

So - if you’ve only used the vEdge software devices, you may be used to using the show ip route or show ip route vpn 10 commands.

In the IOS-XE world, most SD-WAN commands are prefixed with the sdwan keyword. For example: show sdwan bfd sessions (where on vEdges, it would just be show bfd sessions)

This might lead you to believe that you can use show sdwan ip route or show sdwan ip route vrf 10 - but these won’t work! In fact, you’ll get the following message:

Cat8k-Site400# show sdwan ip route
% Error: This command is not supported

For the IOS-XE based devices, they actually just use the standard IOS-XE routing table and VRF constructs.

So on a vEdge, you would have VPN 0 as your transport VPN. On IOS-XE, this is just the default global routing table, shown with show ip route.

But what about our LAN-side service VPNs? In this case, our routes are being dumped into a VRF on the IOS-XE device.

So for example, I have VPN 10 in my lab which is used for LAN-side clients. We can use the show vrf command to see that this exists, and then show ip route vrf 10 to see the routes from our other SD-WAN locations:

Cat8k-Site400# show vrf
  Name                             Default RD            Protocols   Interfaces
  10                               <not set>             ipv4        Gi2
  512                              <not set>             ipv4        Gi3
  65528                            <not set>             ipv4        Lo65528

Cat8k-Site400# show ip route vrf 10

Routing Table: 10
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
       n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       H - NHRP, G - NHRP registered, g - NHRP registration summary
       o - ODR, P - periodic downloaded static route, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR
       & - replicated local route overrides by connected

Gateway of last resort is not set

      10.0.0.0/24 is subnetted, 2 subnets
m        10.2.2.0 [251/0] via 10.10.10.235, 00:09:02, Sdwan-system-intf
m        10.3.3.0 [251/0] via 10.10.10.236, 00:09:02, Sdwan-system-intf

Okay, that’s it! Pretty quick process overall - and now we can get into applying our device/feature templates.

Hope this was helpful!!

In this post, we’ll walk through configuring a Cisco Viptela SD-WAN network to integrate with Cisco Umbrella’s Secure Internet Gateway (SIG) & Secure Web Gateway (SWG).

If you’re interested in reading more about what SIG & SWG are, please check out my last post where I walked through this integration with a Meraki MX firewall.

For additional reading, there’s also a good Umbrella blog post on all the components of Cisco’s SASE architecture, which you can find here.

Okay, with that being said - let’s get started!

Step 1: Umbrella API Keys

Okay, so we’ll start on the Umbrella side. We’ll need to generate a set of API keys, which we’ll push out to our WAN edge devices. These API keys will allow our remote devices to reach out to Umbrella & auto-configure their SIG tunnels.

We’ll log into our Umbrella dashboard at https://login.umbrella.com/

Once we log in, we’ll hop over to Admin > API Keys. Then up in the upper-right corner, click Create.

For this integration, we’ll need to select Umbrella Management to make sure our API keys have the access they need. Then click Create. Easy enough, right?

Once we do that, the Umbrella dashboard will display our API keys - which we’ll need to copy over to our Viptela SD-WAN environment.

We’ll also need to collect our Umbrella Organization ID. This is actually just embedded within the Umbrella Dashboard URL, and should be a seven-digit number. For example, if we look at our current URL on the API keys page:

https://dashboard.umbrella.com/o/<ORG ID>/#/admin/apikeys

And that’s all we need for now from the Umbrella side. So let’s hop over to Viptela.

Step 2: Viptela Templates & Config

For this post, I’ll be using my SD-WAN lab in EVE-NG. Currently I just upgraded the lab to run Viptela version 20.4.1, though the SIG integration has been supported since 20.1.

Just for reference, here is the lap topology that I’m working with:

This lab includes a bridged connection to allow direct internet access from the lab network.

To configure our SIG automatic tunnels, we’ll need to create / update a few templates:

• Create a SIG Credentials feature template
• Create a SIG feature template
• Assign SIG templates to device templates
• Edit Service-side VPN Template to inject a service route

2a: Creating a SIG Credentials Template

First we’ll create a template which will store our Umbrella API credentials.

In the vManage dashboard, we’ll go to Configuration > Templates, then drop into the Feature tab, and click on Add Template.

Once we get to the next screen, we’ll select our device type. In my case, I’m using a vEdge Cloud.

A list of templates will pop up - we’ll drop down to SIG Credentials, which will be near the bottom under the Other Templates header.

We’ll then be taken to the page where we create our template.

We’ll fill in our template name & description, then paste in our Umbrella details (Org ID, API Key, API Secret).

Once we’re done - We’ll hit Save.

Note: At time of writing, the “Get Keys” button only functions if you’ve purchased your SD-WAN subscription with DNA Premier licensing. This license level includes Umbrella SIG, and allows this 1-click integration that pulls your Umbrella API keys automatically.

2b: Creating a SIG Tunnel Template

Okay, next we’ll need to create another feature template to specify our IPSec tunnel configuration.

Just like before, we’ll head back over to Configuration > Templates > Feature > Add Template.

This time, after selecting our device type, we’ll choose Secure Internet Gateway (SIG) / WAN - which is located under the VPN header.

In the template configuration, we’ll give the template a name & description as always. Then we’ll jump down to the tunnel config.

We’ll select Umbrella for SIG Provider, then click Add Tunnel.

Within the tunnel config, we’ll specify an interface name - I’ll name mine ipsec1 (and I’ll create an ipsec2 shortly). We’ll also specify a Tunnel Source, which in my lab is ge0/0 for the biz-internet VPN 0 interface.

We’ll keep Data-Center as Primary, then click Add. Then - we’ll add a second tunnel configuration, but using Data-Center as Secondary and the interface name as ipsec2.

When all that is done, we should have the following:

If we scroll down to the bottom of the template config, we’ll have some settings for High Availability. Here, we’ll specify what our active & backup IPSec tunnels are, and their weights. I’ll specify ipsec1 as active & ipsec2 as backup. Then click Save to finish our template.

Note: Weight settings are only available starting with 20.4.1 & allow for ECMP routing to SIG. Not shown above, you can create up to four HA tunnel pairs. Depending on weighting, you can equal-cost or unequal-cost load balance between those pairs.

Why would you want to load balance across multiple tunnels? At time of writing, each IPSec tunnel is limited to a maximum 250Mb/s throughput. By creating multiple tunnels & load balancing, we can overcome this limitation if we need higher bandwidth.

2c: Attaching SIG to Device Templates

After we’ve built out our two SIG templates, we can now attach them to our device templates.

For me, I currently only have a single device template which is applied to all of my remote WAN edge devices.

So we’ll go back to Configuration > Templates and find whichever device template we want to use - then click the ellipsis on the far right, end select Edit.

In the device template, we’ll scroll down and look for our VPN 0 configuration under Transport & Management VPN. We’ll attach our SIG tunnel template to VPN 0, since that’s where those IPSec tunnels are being sourced from.

On the right side, under Additional VPN 0 Templates, we’ll click Secure Internet Gateway to add our template. Then from the drop-down, we can select the feature template we just created:

We’ll also include our SIG credentials template, which we can find at the bottom of the page, under Additional Templates:

Once we’re done, we can click Save at the bottom. This will take us through the process of pushing out these changes to our remote WAN edge devices. When this configuration is deployed, each vEdge will now reach out to Umbrella & auto-configure an IPSec tunnel.

Note: While the IPSec tunnels will be established when this configuration is pushed - no traffic will flow over them yet. We’ll need to add a service route for that, which we’ll do next!

2d: Injecting a Service Route

Now we have our IPSec tunnels deployed - all we need to do is start routing traffic out to Umbrella. We’ll accomplish this using a service route on our LAN-side VPN.

So we’ll go over to Configuration > Templates > Feature - and we’ll scroll down & find whichever template we’re currently using on our LAN side. For me, I want VPN 10 to route over the tunnels, so I’ll be editing my VPN 10 template.

Within the template, we’ll scroll down to the Service Route section, click New Service Route, and we’ll enter our desired prefix. This will be what traffic will be routed over the IPSec tunnel to Umbrella. Since this is intended to be an internet gateway, I’ll enter 0.0.0.0/0 to inject a default route to Umbrella.

Service should be pre-selected as SIG, so we’ll click Add, then Update & deploy our changes to the remote edge devices.

Step 3: Validation & Troubleshooting

Awesome! Now we should have everything configured & working. So let’s jump through some initial testing and validation we can do.

Within vManage, we can check the status of our IPSec tunnels. We’ll go over to Monitor > Network then select one of our WAN edge devices.

We’ll click on the Interfaces tab on the left side - and we should be able to see a list of all interfaces on our device. This should include an ipsec1 & ipsec2 interface.

I’ve also selected the Real Time monitor on mine, and filtered to just show traffic on the ipsec1 interface:

You might recall from my topology above, that I have a linux VM running behind each of the two vEdge Cloud appliances. Using these, I can also test web access & see what my external IP is:

And sure enough, I am getting a 146.112.x.x address - which belongs to the Umbrella datacenter.

If we log into one of our vEdge devices, we can check the routing table with show ip route vpn 10 (or whichever VPN you’re using for the LAN-side). We should see our default 0.0.0.0/0 route via ipsec1, with a next-hop-VPN of VPN0:

vEdge-01# show ip route vpn 10

     ADDRESS               PATH             PROTOCOL          NEXTHOP  NEXTHOP                         NEXTHOP
VPN  FAMILY   PREFIX       ID    PROTOCOL   SUB TYPE  METRIC  IFNAME   ADDR     TLOC IP  COLOR  ENCAP  VPN      STATUS
------------------------------------------------------------------------------------------------------------------------
10   ipv4     0.0.0.0/0    0     std-ipsec  -         0       ipsec1   -        -        -      -      0        F,S
10   ipv4     10.2.2.0/24  0     connected  -         0       ge0/2    -        -        -      -      -        F,S

We can also check specifically our SIG tunnel status using the command show secure-internet-gateway tunnels:

vEdge-01# show secure-internet-gateway tunnels

TUNNEL                                                        API   LAST
IF                                                            HTTP  SUCCESSFUL     TUNNEL
NAME    TUNNEL ID  TUNNEL NAME                     FSM STATE  CODE  REQ            STATE
-------------------------------------------------------------------------------------------
ipsec1  530233543  SITE200SYS10x10x10x235IFipsec1  st-tun-up  200   create-tunnel  -
ipsec2  530233542  SITE200SYS10x10x10x235IFipsec2  st-tun-up  200   create-tunnel  -

In that output above, we’ll see that we have both ipsec1 & ipsec2 tunnels shown. The last API queries to Umbrella have a HTTP 200, which is good. We’ll also see our tunnel names, which we can use to find our device tunnel configuration in the Umbrella dashboard.

The tunnel name might look like a mess at first, but it’s a unique identifier to represent each tunnel that was created. So if we break it down for this vEdge:

• This vEdge is at Site ID 200
  • Shown as “SITE200”
• This vEdge has a system IP of 10.10.10.235
  • Shown as “SYS10x10x10x235”
• This vEdge has two IPSec interfaces, ipsec1 & ipsec2
  • Shown as “IFipec1” and “IFipsec2”

If we jump over to the Umbrella dashboard, we’ll see the same. Within the Umbrella dashboard, we can jump to Deployments > Network Tunnels.

On this page, we should see a total of four tunnels listed (two from each vEdge appliance):

And with everything configured & validated - now we can move onto configuring firewall and web filtering policies within Umbrella!

If you’re interested in seeing how to configure Umbrella’s cloud firewall & web filtering policies - please check out my YouTube Video above!

In today’s blog post - we’ll be checking out how to integrate a Meraki MX firewall to Cisco’s Umbrella Secure Internet Gateway (SIG) & Secure Web Gateway (SWG) services.

Interested in seeing how to do this with Cisco Viptela SD-WAN? Check out this post

Note: This integration is still rather new - so I anticipate some of the screenshots & steps below may differ as this post ages.

What are SIG & SWG?

Good question! A lot of us are probably familiar with Cisco Umbrella (formerly OpenDNS) for the DNS-layer security products. Using Cisco Umbrella, we can configure our end PCs or DNS forwarders to use the Cisco DNS servers. Then, we apply security policies to allow/deny DNS queries based on our policy.

Now applying security policy at the DNS level is great! Typical web filtering only inspects HTTP/HTTPS traffic, but inspecting DNS traffic allows us to stop threats that might not use those typical ports. While there may be some applications or malware that use hard-coded IP addresses, a good majority use a domain name that requires a DNS lookup.

But what if we wanted to still do web filtering too? Or maybe we want a centralized, cloud-hosted firewall service? That’s where Umbrella SIG & SWG come in.

Secure Web Gateway (SWG) is pretty much exactly what it sounds like. It’s a cloud-hosted web filter or web proxy, where we can set URL-based policys to permit or deny access. It supports the usual allow-lists, deny-lists, HTTPS inspection, file inspection, and policies based on user/group identities.

Secure Internet Gateway (SIG) is a large, cloud-hosted firewall service. What we can do with SIG is tunnel all of our traffic to one centrallized location to apply firewall policies (permit/deny/etc).

Why would we want either of these functions to be cloud-hosted? Well we might not have the resources at every remote site to perform firewalling & web filtering - or we don’t want to invest in beefy hardware in our corporate datacenter, and backhaul all of our remote traffic back for inspection. The cloud-hosted piece also means a central management point, so only one place to make change for all of our remote sites - rather than having to touch dozens or hundreds of remote devices…..Did someone say SASE? 🙃

With all that said - Let’s get into making this work!

Note: There are many ways to use Umbrella SIG/SWG (IPSec tunnel, PAC file, Anyconnect, etc). This particular post will only cover IPSec via Meraki MX.

Step 1: Getting our Keys

First thing we’ll need to do is on the Umbrella side. We’ll need to generate tunnel keys for our Meraki MX to use for IPSec negotiation.

We’ll log into our Umbrella dashboard at https://login.umbrella.com/

Once we’re in, we’ll navigate to Deployments > Core Identities > Network Tunnels. Shown below, we currently have no tunnels configured:

In the upper right corner, let’s click the Add Tunnel button.

We’ll now see the following screen to begin creating our IPSec tunnel:

Here, we’ll need to specify a name for our tunnel & which device type. The name can be anything we choose that helps us identify what locations are using this tunnel. I’ve specified MX64 as my tunnel name, since I only have one MX that I’ll be testing this with. We also have Meraki MX as an option in the Device Type drop down - so we’ll select that as well. Then, we’ll click Next.

Then we’ll need to configure our Tunnel ID and Passphrase:

The Tunnel ID we’ll be sending later, as part of our IPSec local ID. The passphrase will be used as our pre-shared key for the tunnel config. Once we’re done with that, click Save.

As long as our ID & passphrase are all good, we’ll be presented with a box to easily copy these parameters into the Meraki config:

Finally, we’ll be dropped back to our Network Tunnels page - where we can see that our tunnel is configured, but not yet established. Let’s fix that & hop over to the Meraki dashboard!

Step 2: Meraki MX IPSec Configuration

Okay, now onto the fun stuff.

We’ll log into our Meraki Dashboard at https://account.meraki.com/secure/login/dashboard_login

Once we’re in, you’ll need to select the network with the MX you want to connect. For me, I currently only have a single network, which contains my MX firewall.

Then we’ll navigate to Security & SD-WAN > Configure > Site-to-site VPN.

Assuming we have no other VPNs configured, you’ll have to change the VPN Type to Hub (Mesh) or Spoke. In my case, Umbrella SIG will be the only VPN I’ll have configured - so I’ll go ahead and use Hub (Mesh). Don’t mind the error regarding exit hubs, as we’ll be configuring a non-Meraki peer below.

Okay, scrolling down the page just a bit - we’ll need to specify which VLANs/internal subnets will be routed across the VPN & pushed through Umbrella for SIG/SWG:

For testing purposes, I have a VM in a subnet labeled DMZ that I’ll be using - So that will be the only VLAN that I’ll set to VPN On.

Next we’ll take a look at configuring Non-Meraki VPN Peers.

As shown in the screenshot below, we’ll need to configure the following settings:

• Name - This is locally significant, I set mine to Umbrella_SIG
• IKE Version - Set this to IKEv2
• IPSec Policies - Custom - See below
• Public IP - This is the peer Umbrella Datacenter
  • Choose the DC closest to you here
  • For this post, I’m using the New York DC
• Local ID - Set this to the Tunnel ID we got from Umbrella
• Remote ID - Leave blank
• Private Subnets - This is what destination IPs will be routed over the tunnel
  • Since this is intended as an Internet gateway, we’ll use 0.0.0.0/0
• Preshared Secret - Set this to the passphrase we configured in Umbrella
• Availability - Set this to which networks this tunnel should apply
  • Since I only have 1 network & 1 MX, I set this to All Networks

Okay - I mentioned above we’ll need to set some custom IPSec parameters. Here’s what we’ll configure as a custom policy:

Note: Turns out in the drop down menu, there is also an option for “Umbrella”, so you don’t have to create a custom policy. That being said, the Umbrella preset uses DH group 5, but Umbrella’s docs ask for DH group 14.

Lastly, we’ll be able to configure whether or not we want firewall logging (I enabled this) and also whether we want to add access-lists to permit/deny any traffic over the VPN. For now I’ll be leaving this as permit any.

Finally - we can click Save to push our configuration to the MX appliance!

Step 3: Validation & Testing

Hopefully once we get all that configuration done, we’ll be able to see our tunnel come up.

We can validate from both sides, but let’s start with Meraki. In the Dashboard menu, we’ll navigate to Security & SD-WAN > Monitor > VPN Status.

Then we’ll need to click the tab for Non-Meraki peer:

With any luck, we should see the little green circle showing that our tunnel is online.

If your tunnel doesn’t come up immediately, you may have to generate some traffic from your VPN subnet to the internet. This will force the MX to try and connect to Umbrella.

It’s also worth noting - sometimes it may take the Meraki dashboard to show a successful connection. A better indicator is by checking the Umbrella dashboard, or checking manually with a device you are expecting to route over the VPN.

If for any reason we have trouble, we can also check our VPN negotiation logs by going to Network-Wide > Monitor > Event Log.

On this screen, make sure you’ve selected for security appliances at the top. If needed, we can also filter events by All Non-Meraki VPN / Client VPN.

Since my tunnel came up with no issues, the output above shows a successful tunnel negotiation.

Let’s jump over to the Umbrella side, and see what the Umbrella dashboard shows.

Back on the Network Tunnels page - we should see that our tunnel now shows as Active.

This screen will also so the public IP that our MX is using to connect, as well as which Umbrella datacenter we’ve connected to.

Okay! That’s it for now. To try and keep this blog post from getting too long, I’ve decided to split this into two parts.

If you’re interested in seeing how to configure Umbrella’s cloud firewall & web filtering policies - please check out my YouTube Video above!

I have a local Cisco SD-WAN lab environment running at home, which was built in EVE-NG. It’s what I use whenever I need to test something for a customer, or just play around with the templates or APIs.

I’m planning on spending some hands-on time with new features soon, along with working on some automation projects - so it’s well past time to upgrade my lab.

Currently I’m running on version 18.4.302, but my intent is to upgrade to 20.3.2 - which is the latest version available today.

In this blog post, we’ll walk through how to upgrade a Cisco SD-WAN / Viptela network, including:

• Control Plane:
  • vManage
  • vBond
  • vSmart(s)
• Data Plane:
  • vEdge / vEdge Cloud

While my lab is using the on-prem controllers, these steps will work just the same if you’re using Cisco’s cloud-hosted controllers.

Downloading the Software Images

First thing’s first - in order to upgrade our environment, we need the correct software images!

Head over to Cisco Software Downloads, and search for SD-WAN (or click here!).

Once on this page, I just want to call out that we will need to click on the SD-WAN Software Update link. This may seem simple enough, but the other links for vManage Software or vSmart Software only contain images for a new install.

After that - just select the image version that you would like to upgrade to, and download both images.

In my case, since I’m downloading version 20.3.2, I’ll be using the following images:

• vSmart, vEdge Cloud, vEdge 5000, ISR1100 series and vBond upgrade image
  • File name: viptela-20.3.2-x86_64.tar.gz
• vManage upgrade image
  • File name: vmanage-20.3.2-x86_64.tar.gz

Adding Images to the Software Repository

The process for upgrading a Cisco SD-WAN environment is pretty straightforward.

The control plane can be upgraded independently of the edge devices, so long as everything stays within the bounds of the compatability matrix. In my case, I’ll be upgrading to 20.3.2 - and the controllers could still support edge devices as far back as 17.2. So no issues here!

Alright - let’s get started!

First, we’ll need to upload our images into our local vManage software repository. This is the file storage for all upgrade images, and it’s where the controllers & edge devices will go to pull their images from.

In the vManage dashboard, we’ll go to the Maintenance tab and select Software Repository.

Then, click on Add New Software.

In here we’ll see a few options: vManage and Remote Server / Remote Server - vManage.

We’ll use vManage if we want to upload & distribute images from the local vManage server we’re logged into currently.

Alternatively, we could use a remote file storage server by using the Remote Server option. If you choose to go this route, don’t forget to ensure that ALL controllers & WAN edge devices have access to this storage location.

After you select an option, it’s an easy drag & drop to upload the software images.

Planning the Upgrade

So before we get into actually applying our image upgrades - let’s address the questions of “What order do I upgrade things in?” and “What’s the impact?”.

Since this is a lab environment for me, I’ll be upgrading everything all at once - since uptime / outages aren’t a factor here

If you’re doing this in a production environment, I highly recommend performing these upgrades in an outage / maintenance window - or at least an off-peak time.

Yes, you can upgrade the controllers at any time without causing any issue. Yes, you can upgrade a redundant pair of vEdge devices and keep a branch online. However, I would advise you to try these out off-hours first - and get your own understanding of how this works & what to expect before doing it in production.

As for the upgrade order, we’re going to start at the top of the food chain and work our way down:

• Upgrade vManage first
• Then vBond
• Upgrade ONE vSmart controller & wait for it to come online / re-establish control connections
• Then upgrade the second / redundant vSmart controller
• After the control plane is upgraded & stable - move onto the edge devices

These steps can also be found under the Best Practices section of the Cisco SD-WAN Getting Started Guide. Cisco’s official recommendation is to wait 24 hours in between a few of those steps, to ensure platform stability - but for my lab that won’t be necessary

Upgrading vManage

Once we have uploaded our images & we have our upgrade plan - we can move forward with actually performing the image upgrades.

Starting with vManage - we’ll go to Maintenance > Software Upgrades > vManage.

Then we’ll click on Upgrade and select our version - in my case 20.3.2. After that, just click Upgrade

Now, what this does in the background is just pre-stage the vManage image for an upgrade. The actual software upgrade is not occurring just yet.

Think of this step like you might prepare an IOS/IOS-XE router: Copying the image to the device flash. The image is there and ready - but we haven’t booted to it yet.

Once that’s all done, we’ll go back to the vManage upgrade page and click Activate, select our image again, then click Activate.

Now this step is where vManage will reboot, apply the upgrade, and come back online with the new image.

Again, back to the IOS/IOS-XE analogy: this is the equivalent of setting out boot system flash:<image name> to the new image, then rebooting our router.

vManage may take a short while to complete & reinitialize. In my lab, about 10-15 minutes.

Upgrading vBond & vSmart

After vManage is done, it’s time to work on the real heart of our control plane: vBond & vSmart.

Similar to vManage, we’ll start by going to Maintenance > Software Upgrades > Controller.

Here we will need to select the devices we want to upgrade. As I mentioned earlier, you may want to do these one at a time & in a phased approach. However, in my lab I’ll select all of them to upgrade at once.

Now in this case, vManage will still follow the proper order (vBond, then vSmart), and even perform a rolling upgrade one device at a time. This may be suitable for you in production, but again I would urge you to test it for yourself first!

The other big difference here, as you can see in the screenshot above, is the presence of the Activate & Reboot checkbox.

This does exactly as you would anticipate. Instead of doing the two-step process with vManage where we staged the image, then performed the activation/reboot - this checkbox will do all of that in one step.

In my lab environment, I did check this box & allowed everything to reboot automatically.

Why is there a separation between uploading the image & rebooting / activating it? To allow better granularity over the process.

For example, maybe you have a poor internet connection at a branch site & the image upload may take a long time. This separation of tasks allows you to stage all of the images independently of applying them. If you have a short outage window, this could help you save time by pre-staging the images ahead of time.

Back to the upgrade - just like vManage we’ll select the version we’re applying then click Upgrade

Again - Depending on the resources available to your controllers, the image upload / activation process may take a short while…

Upgrading the WAN Edge Appliances

In my lab, I’m currently using a handful of vEdge Cloud VMs as branch office routers. The upgrade process here should apply to other edge devices as well.

After we’re confident our controller upgrades have been successful & all control connections have been re-established - we can move to upgrading our edge devices.

It’s also worth mentioning that my lab currently has no redundant deployments of WAN edge appliances. All of my test ‘branch offices’ are single-homed to one vEdge Cloud - so an outage will be required to apply the images.

If you’re using a redundant configuration at a remote site, ideally you would upgrade ONE edge device first. Then only upgrade the second after control connections & routing adjacencies had been re-established on the first device. This should allow for an upgrade with minimal downtime.

The process for upgrading the edge devices mirrors what we saw for vBond / vSmart.

We’ll go to Maintenance > Software Upgrade > WAN Edge, then select the edge devices we want to upgrade.

Click Upgrade, select the target version from the drop-down, then click Upgrade. Again, in my case, I also selected the Activate & Reboot checkbox

NOTE: It’s worth mentioning that for the WAN Edge upgrade, vManage will push the upgrade to all devices simultaneously. If you would like to perform a rolling upgrade here, you’ll have to manage it yourself. In the case of a site where a redundant vEdge is deployed & they share the same site ID, vManage automatically will handle upgrading only one at a time to maintain uptime (Thanks Tim McConnaughy for clarifying this!).

Once again, we’ll give the edge devices a few minutes to download & apply their software upgrades. Then we’ll check to ensure all of the control connections re-established & traffic is flowing.

Wrap up

Once your edge devices are back online, it’s all done! The network has been upgraded to the new version.

There are a handful of ways to check this, but one easy way is via the device monitor page: Monitor > Network

This page will list a summary of everything in the network, including the current software version & number of established control connections. For me, it’s an easy way to get a one-page summary of the network.

From the screenshot above - we can see that all of my lab devices are back online & running software version 20.3.2.

That’s it! Hope this post was helpful to you.

Thanks for reading!

So if you’ve read some of my recent posts - you may have seen that I purchased a NetGear LB 1121 LTE Cellular modem to use for home internet backup.

Well - I decided to upgrade!

After using the NetGear modem for a while, I started having some issues where it would disconnect from the cellular connection intermittently. Since it’s not necessarily intended for the purpose I’m using it for, there wasn’t any good way to set up monitoring for it either.

So I opted to upgrade to a Meraki MG21. This is one of the latest additions to the Meraki family of network devices & is available with internal (MG21) and external (MG21E) antenna.

I was pretty excited to get one, since Meraki tends to have decent analytics & configuration - and they make everything so easy!

We’ll walk through the setup of the MG below - but if you’re interested in seeing what the device looks like as well, definitely check out the video above!

MG Setup - Changing the APN

Okay - so after I got the SIM card inserted into the MG, the first configuration step is making sure we have the correct APN configured. As a reminder, I’m using Google Fi as my cellular provider.

This change will need to be done on the local web management interface - not from Meraki Dashboard.

When the MG powers on, by default it will hand out DHCP addresses to any device connected to port 1. At the time of writing, these addresses were in the 192.168.5.x range.

Connecting a PC directly to the MG port, we should be able to reach the local management web page - either by typing in the IP address into our web browser, or using mg.meraki.com:

In my case, I could see that the MG had auto-detected my Carrier as Google Fi. However, it still had the incorrect APN.

Using the Configure tab, we can change that setting. You’ll be prompted for a username & password. By default, the username will be the serial number of the device (including dashes) with a blank password.

As shown in the screenshot above, we have a handful of options - though we’ll only care about APN.

First - change the Cellular Override option to Override SIM Settings. Then type in your APN. In my case, it’s h2g2 for Google Fi. Lastly, hit save at the bottom (just outside the view in the screenshot above).

With any luck, the modem will connect and you’ll see something like this:

The modem connects, gets an IP from the provider, and is able to validate connectivity to both the internet & Meraki Cloud.

When I originally set up my MG, I did run into some issues with this. My MG connected to the internet successfully, but said it couldn’t reach the Meraki Cloud. Not sure what caused it, but it shortly resolved itself within a few minutes. Just gotta be patient sometimes, I suppose!

Configuring the MG in Meraki Dashboard

Note: I won’t get into how to claim your device in dashboard or how to attach it to a network. If you need help, please check out the video above where I did show how to accomplish these steps

Okay! Now that our MG is configured for the correct cell network, we can log into the Meraki Dashboard and begin configuring it.

After we’ve added the MG to our network, we’ll see a new Cellular Gateway menu:

We’ll start first by going over to Configure > Settings

First section we’ll see is for Addressing & NAT:

As of today, the MG doesn’t support any form of direct internet pass-through. Instead, our only option is routed mode - where the MG will hold the IP provided by our Carrier & NAT any requests from the devices behind it.

We can change the DHCP subnet configuration here, which will affect what IP addresses are handed to clients behind the MG. In my case, I’m connecting this directly to a firewall as a secondary internet uplink - so the addressing & subnet doesn’t matter as much. By default, the MG will always consume the first available address as it’s own.

Next, we have a section for DHCP & subnets:

Here we can change our DHCP lease time, and what DNS servers are provided to our clients. The DNS setting does have pre-defined options for Umbrella DNS, Google DNS, or using whatever the upstream carrier provides. You’re also welcome to manually specify which DNS servers to use.

We can also configure reserved & fixed IP addresses here.

Reserved IP ranges are IP addresses that we don’t want the MG to provide via DHCP. So if we had any statically configured IP addresses, we could reserve them here.

Fixed IP addresses are for any client that needs a DHCP address, but we want that IP assignment to be permanent. We’ll enter the client name & MAC Address here, as well as the IP we want assigned to that device. In my case, I went ahead and inserted my firewall MAC address - and I’ll just allow the firewall to get its IP via DHCP from the MG.

By default, the MG will block all inbound traffic from the cellular network. If we need to allow any traffic inbound, we can change the Port Forwarding settings:

This allows for a light configuration of an inbound NAT. Right now, I probably won’t be using this. However, I may permit VPN access into my network via the MG at a later date.

If I needed to add anything here, the MG allows us to translate an external / public IP & port to any internal IP / port combination. It appears we can even add a IP filter to permit only trusted source addresses.

Lastly - We can configure settings for Traffic Shaping:

In this section, we can throttle our cellular throughput & configure uplink monitoring.

By default, the cell bandwidth is set to unlimited - but we can drop this down if needed. In my case, I am not using an unlimited cell data plan - so I will throttle cellular speeds to preserve data & reduce charges.

In addition, we can configure one or more IP addresses to check uplink connectivity. These addresses will be used to collect loss & latency data via the cellular connection. The MG monitoring dashboard will collect & graph this data for easy insight into the performance metrics.

Note: As a word of warning, these uplink monitors are constantly sending ICMP/ping requests. If you have a limited amount of cellular data, this may consume more data than you would like. In my testing, using only one IP for uplink monitoring consumed about 70-100M per day. More on this below…

Monitoring the MG

Now we get to the good stuff! The primary reason I opted to buy an MG was for monitoring & analytics.

Back on the dashboard, if we use the lefthand menu - we’ll go over to Cellular Gateway > Monitor > Cellular Gatways. Then select our MG out of the list.

The primary summary page isn’t too exciting:

The MG does have two gigabit ethernet ports - and we’ll see the status here.

We’ll also see the connectivity history to the Meraki cloud - which in my case is nearly 100%. Seems like one very minor blip just after 4am.

We can also see the current network utilization on the MG. This is great to have - though my current utilization is pretty low… (I am using this as a backup modem, after all).

On the left side of the page, we’ll see some of the usual info we expect from a Meraki device. Current IP, location, Serial number, and IMEI. Just below the view of the screenshot, there is also an indicator for firmware version.

Onto the Uplink tab! Let’s see what we have:

First we’ll see the Configuration section. This just gives us a quick view into what settings the MG currently has.

We’ll see the current IP info provided by our carrier, and also some statistics on our cellular connection.

Just below that info, we’ll see our cellular graphs:

This is what I wanted! It’s great to see a quick view into what our active uplink traffic is - as well as look back historically at what our LTE signal quality has been.

Not pictured here - but there is also a section of graphs on this page for our uplink monitor. This is where we can see our current & historical loss & latency stats for the cellular connection. After a few days of use - I disabled the uplink monitor due to the amount of data the feature consumes.

Finally, we also have the DHCP tab:

This will show our current DHCP subnet & any clients that have been provided an address. In my case, there isn’t any current leases here - because my firewall has a fixed IP.

Performance && Considerations

I’ve had the MG running for about a week now, and wanted to provide some things to think about.

First - How does the modem perform? Well, the first day I had it set up - I was able to get ~150M download speeds using the MG’s built-in speed test utility:

That being said - I’m lucky if I get 30-50M on an average day. I might have just gotten lucky that day with some light cellular utilization in my area. Overall though, I’m pleased with the speeds I get - they’ll certainly fit my needs.

For the few days that I had the uplink monitoring running, I saw good results. Usually 0% packet loss, with a rare spike of 5-10%. Latency was a little less reliable, but usually bounced between 50-150ms. This was also much less than the NetGear modem I had been using, which averaged 200-250ms.

Speaking of uplink tests! Let’s talk about data usage….

By default, the MG communicates intermittently with the Meraki cloud - which consumes some data. By my measurement, this is usually less than 10Mb/day. No problem here.

The uplink tests, on the other hand, do consume a bit of data. I’m not sure what frequency these run on, but it’s fairly often. Even with one uplink monitor to 8.8.8.8 configured, I was seeing data usage of 70-100Mb a day.

While seeing those metrics is valuable to me, it’s also not worth the data charges. If I was using a SIM card with an unlimited data plan - no doubt I would keep this feature enabled. However, since I am paying for the cell data used - I opted to disable this feature.

The MG does still perform it’s check-ins to the Meraki Cloud - so you’ll have availability statistics & monitoring… But disabling the uplink monitor means you’ll lose the granular data on loss & latency.

Lastly, and another word of warning, when you’re actively viewing the MG monitoring page - this also consumes additional data. To demonstrate - I’ll post a snippet of the screenshot from earlier:

If you notice, all the way on the far left there was barely any activity. However, once I loaded the MG monitoring page - you begin to see minor spikes in data usage as the Meraki Cloud starts actively polling the MG for data.

In my experience so far, this isn’t a ton of data. I’ve checked in to see how the MG has been performing a few times this week, and each time has totaled around 10-15Mb of data usage.

To sum up - I’m cheap and want to avoid excess data usage. Just wanted to provide some of that info as something to be aware of.

Final Thoughts

I’m only a week in, but pretty pleased with the MG’s performance. It’s maintained a very solid & stable connection compared to the NetGear modem it replaced. The device is intended to provide LTE connectivity or backup service for business networks, so I would certainly hope it would meet my home needs :)

Outside of that, I do wish there was a little better documentation & clarity from the Meraki team on data usage. Right now their documentation only mentions the 6-8Mb of usage due to backend data to/from the Meraki Cloud:

I would be happy to see additional settings on the uplink monitor to allow me to choose the polling frequency. I feel like throttling down the amount of requests could reduce data to a point where I would be comfortable re-enabling that feature.

Oh - and currently there are no native email alerts for the MG. So if the MG goes offline, etc… there is no alerting from the Meraki Dashboard. This kinda sucks. I’m sure this is coming soon, but for the time being I’m inclined to write my own monitor using the Dashboard APIs. (see note below!)

Overall, I’m happy with the device. Definitely looking forward to future feature & firmware updates to see where the Meraki team takes this platform!

Update 08/28/2020 - Looks like in the week since I posted this, Meraki added alerting for the MG! Now you can be notified if the cell gateway goes offline:

Hey there! First thing’s first - Hope all is well with everyone. The past few months have seen all sorts of craziness going on in the world. I’ve been lucky in the sense that my current job role was already well set up to make an easy transition to working from home all the time. That being said, I know it hasn’t been easy for everyone - and I know quite a few people who have lost their jobs, etc.

Second thing! As I’m sure you may have already seen above - I opted to spend some of my new-found free time trying out a new format. I’ve had a couple of ideas in the past for making short videos, but never forced myself to sit down and give it a shot. So here we are - after about a month of on-and-off work - I finally have something to share. Please give it a look if you’re interested, and I would appreciate any comments & feedback. I still have a few other ideas - so if this one goes well I may pursue producing a few more of these.

Okay - Now onto the real content!

A few months ago I had the chance to pick up a pair of Cisco Catalyst 9100 series access points from work. My home network has been running on Ubiquiti APs for years - but they’re getting old and I desperately needed to replace them. So along comes the Catalyst 9100 APs, which are capable of supporting 802.11ax clients - and why not take the time to upgrade & future proof? I needed some practice anyways, as I have a few customers at work that I think will be interested in these in the near future. I ended up with two 9120AXI APs.

So in the process of trying to dive on the opportunity to play with something new - I completely missed the fact that the 9100 APs have two different product SKUs. One which ships the AP pre-configured with the standard lightweight AP code that you might use if you had an external wireless LAN controller (WLC). And a second SKU that ships the AP with the Catalyst 9800 Embedded WLC (eWLC) software loaded. Of course, I grabbed the lightweight APs without checking.

Product SKUs (Example, using 9120AXI):
 - Standard Lightweight AP: C9120AXI
 - Pre-load with Embedded WLC: C9120AXI-EWC

I wrote in a previous blog about getting MAC address filtering set up on the Catalyst 9800 WLC. When I wrote that blog I was actually using the C9800-CL, which is a virtual machine version of the Catalyst 9800 controller software. I was originally excited to run the controller as a VM and not need a hardware appliance - but then got to thinking that maybe I should try and save VM resources as well. Which led me to looking at the 9800 eWLC.

Finding the Software Image

In order to convert an existing lightweight access point to one running the embedded WLC software - we first need to grab a copy of the software images from Cisco.com. Search for the model of your 9100 access point (in my case, the 9120AXI).

You’ll see two options for Software Type - and it may not immediately be obvious - but we’ll need to look under IOS XE Software to find the eWLC images:

Then we’ll grab the EWC AP image bundle. In my case, I was waiting on a specific feature set that wasn’t available until 17.x - so I downloaded the 17.2.1 software:

Okay - after we’ve done that - let’s drop everything onto a server/laptop/etc which has console connectivity to our AP and a running TFTP server.

Once you un-zip the contents of that AP image bundle, you’ll notice there are quite a number of files. We’ll only need two of them - one image to load onto the AP itself, and one that will get loaded for the WLC software container. Within the image bundle, there will be a readme.txt file that will tell you which image to use with your AP:

ap1g5 : AP1815, AP154x
ap1g4 : AP180x, AP183x, AP185x
ap1g7 : C9115, C9120
ap1g6 : C9117
ap1g6a : C9130
ap3g3 : AP380x, AP280x, AP156x

So in my case, I would use the file named ap1g7 since I had the 9120 APs. In addition, you should also see a file named C9800-AP-iosxe-wlc.binwhich we’ll need to load the controller.

What’s with that second image? Well - the Catalyst 9100 APs include a feature called Application Hosting (also found on some of the Catalyst 9000 series switches). This is equivalent to running a Linux container or Docker container directly on the AP hardware. At time of writing, this is only available for the embedded WLC software. However, there will be a future software update that will allow you to provision other software containers as well (according to the data sheet).

Converting the Access Point

Now we can get started on the actual conversion process. If it’s a new AP, we can just go ahead and boot it up with a console cable connected. If it’s already running something, you’ll likely want to factory reset the AP first. You can find detailed instructions here, but a quick summary - power off the AP, hold the mode button while plugging the AP back in, and keep the mode button held for at least 20-30 seconds. If you’re logged into the console during this time - you’ll actually see the AP counting how long you’ve held the button for. Once that counter shows at least 20 seconds, release the modebutton and allow the AP to reboot.

Once the AP is booted. The default login is Cisco / Cisco. The enable password is also Cisco. You may also want to quickly issue a show version, and take note of the current software version running on the AP (hint: we’ll need that later).

Next - it’s important to note that after we convert the AP - our default CLI will actually be the WLC, not the AP (even when connected via console). So we should configure a hostname and IP address for the AP now:

0xAP# capwap ap hostname <hostname>
0xAP# capwap ap ip <ip-addr> <netmask> <gateway>

Then we can load the new AP image and WLC image. Depending on which software version you’re running today, there is a different command to load the image:

0xAP# ! If you're running software 8.9 or lower, use the following:
0xAP# ap-type mobility-express <TFTP path to AP image> <TFTP path to WLC image>
0xAP# ! If you're running anything above 8.9, use the following:
0xAP# ap-type ewc-ap <TFTP path to AP image> <TFTP path to WLC image>

Once those commands are submitted, the AP will begin copying the software from the TFTP server. The AP will automatically reboot after the image load is completed.

Quick note: For one of my APs, the WLC software didn’t load correctly - and when my AP rebooted I was left with an error that said “EWC-AP in Recovery Mode”. Re-copying the WLC image fixed it without much trouble. If this happens, the AP will print out the command to re-image the WLC software, but I’ll also put it here for reference:

0xAP# ! If you end up in "EWC-AP Recovery Mode", run the following:
0xAP# archive download-sw ewc-ap <TFTP path to WLC image>

Okay - Once our AP has come back from loading up all the new software, we can get on with a bit of minimal config required to access the WLC web UI.

First, we’ll configure the WLC hostname & our local user account. This can be switched over later to your choice of directory service:

0xC9800eWLC# config t
0xC9800eWLC(config)# hostname <hostname>
0xC9800eWLC(config)# user-name <admin username>
0xC9800eWLC(config-user-name)# password <admin password>
0xC9800eWLC(config-user-name)# privilege 15

Next we’ll also provide the WLC with the administrative credentials used to manage the connected APs:

0xC9800eWLC(config)# ap profile <profile-name>
0xC9800eWLC(config-ap-profile)# mgmtuser username <AP admin user> password 0 <AP admin password> secret 0 <AP admin secret>

After that, we’ll go ahead and configure our basic network settings. This will be the IP address & gateway info that will be used to connect to the WLC web UI and SSH:

0xC9800eWLC(config)# interface gigabit 0 
0xC9800eWLC(config-if)# ip address <managemnt IP> <Network mask>
0xC9800eWLC(config-if)# no shut
0xC9800eWLC(config)# ip default-gateway <Gateway IP address>

Lastly - we’ll need to actually enable the web server. In my case, I opted to not enable the standard HTTP server. I only enabled the encrypted SSL-enabled web server:

0xC9800eWLC(config)# ! I only enabled the HTTPS server:
0xC9800eWLC(config)# ip http secure-server
0xC9800eWLC(config)# ! If you wanted to enable the plain-text / unencrypted web server:
0xC9800eWLC(config)# ip http server
0xC9800eWLC(config)# 
0xC9800eWLC(config)# ! Finally - Save the config
0xC9800eWLC(config)# exit
0xC9800eWLC# wr mem

When we save our config for this first time, the AP will verify that the initial configuration pieces have been completed. There is a bit of background cleanup that is performed - which removes any factory config that assisted with provisioning. Once that’s all done - we’re free to log into the web UI!

Embedded WLC Web Interface

I won’t spend a lot of time here, as there are far too many things to cover with the new 9800 series controllers. However, I did want to point out a few things that stand out regarding the Embedded WLC.

When we log into our WLC web UI - we’ll be able to use the username & password combo that we configured previously. Then we’ll be dropped into our WLC dashboard.

*Note: Screenshot taken a bit after I did my initial config. This has been running a bit now…

If we click on the number under the Access Points heading, we’ll be taken to a quick monitoring view of the current connected APs:

Here we can see the APs we have configured, along with their IP address & status info. We will also see what our Current Active WLC controller is, and what our Current Standby / Preferred Active if we have either configured.

By default, if we boot up a second AP running the embedded WLC software, it will automatically join the WLC cluster as the secondary node. Any WLC config/settings will be copied, then it’s ready in case the primary controller fails. In the event of a failure, the secondary WLC will take over - and be accessible using the same IP & login info that we used on the primary.

In my case, I’ve also configured one of my APs as a preferred primary controller. For me, this AP is connected to a switch that has a short battery backup - so it’s less likely to experience failure. This option can be configured on the individual AP itself. In the left-hand menu, drop into the Configuration section, then click on Access Points under the Wireless header. We’ll see a very similar screen to the AP monitoring screen from above. Click on the AP that we want to make our preferred primary, then jump over to the Advanced tab. There will be a checkbox for Preferred Controller:

WLC Image Repository

Last thing - and this one is important. Normally the WLC will have dedicated storage to keep AP software images, which it distributes to new APs when they come online. Unfortunately, while we gain the flexibility and limited footprint of the embedded WLC - we also lose that dedicated storage space. As you might imagine, storage on the AP is limited.

So we’ll need to make sure that the WLC has an external image repository that it can use. This configuration can be done via teh web UI and CLI. I’ll cover both very quickly here.

From the web UI - We’ll go to Administration > Software Management:

Here we’ll see our options for where to get images, as well as an inventory of current APs & their images. In the Mode drop-down, we’ll have the option of tftp, sftp, CCO, and Desktop. TFTP / SFTP are what we’re used to. With Desktop, we would just upload images directly from our laptop / PC. Interestingly though, CCO allows us to provide our Cisco.com credentials - and the controller can pull images directly from Cisco. We’ll even have the option to enable automatic software downloads - and specify whether we want to auto-download the latest software release, or only the latest recommended release.

On the CLI side of things - we can accomplish the same using the following commands on our WLC CLI:

0xC9800eWLC(config)# wireless profile image-download <profile name>
0xC9800eWLC(config-wireless-image-download-profile)# image-download mode tftp
0xC9800eWLC(config-wireless-image-download-profile-tftp)# tftp-image-server <IP address>
0xC9800eWLC(config-wireless-image-download-profile-tftp)# tftp-image-path <path to image files>

With that, any new APs that join our WLC should be able to auto-load the software necessary.

Thanks for reading!

Update 2020 / 05 / 19 - I’ve added a video above that walks through the steps detailed in this blog post.

If you’re using the ‘Basic’ Wireless setup, you may see an error when trying to apply the policy: “switch 1 dbm wireless Use of default ACL preauth v4 is not permitted”

If you come across this error, it’s a known bug (CSCvt18875) specific to only the ‘Basic’ setup wizard (which is what I used in this post below). If so, check out the video above which walks through the ‘Advanced’ setup and bypasses this error.

I’ve been spending a bit of time over the past few weeks building up a wireless lab. Trying to get a good understanding of how the new Catalyst 9800 wireless controller works, and how it differs from some of the previous iterations.

In order to play around with the new controller, I decided to try to build a new configuration that mimics my current home wireless. Today I am using Ubiquiti APs, which come with their own free controller software. Most of my current config is fairly straightforward - a few SSIDs, two APs, and a guest network with captive portal. One of my SSIDs is dedicated to any IoT devices and is more restrictive than the other networks. This network uses both a pre-shared key for authentication as well as MAC-based filtering.

In this post - we’ll walk through how to set up a new SSID with client MAC filtering.

Note: This was written using Catalyst 9800-CL version 16.12.1s. APs are configured in flexconnect with local authentication (no AAA, ISE, etc)

While this post is not focused on in-depth WLAN config, we will start by quickly setting up a new network.

Once you get logged into the controller - we’ll click on the Wireless Setup icon in the upper right-hand side. Then drop down to the Basic option:

This takes us through a pretty quick and easy wizard to set up our new location & wireless networks. At first, we will have no locations configured - so we will click Add:

We’ll start off building our location by giving it a name and description. These are used for some naming of policy objects within the WLC, so make sure to use a name that makes sense.

In my case, I’m also going with a flexconnect deployment - so we’ll select that option and also provide the AP native VLAN.

Next, we’ll click over to the Wireless Networks tab. This is where we will create our WLAN and apply the initial configuration. We don’t have any networks yet, so click Add

The two primary things we need to address are highlighted in red below. We need to create a WLAN and assign it to a VLAN. Let’s start with the WLAN by clicking Define new.

On the General tab, we’ll give this WLAN a profile name and a SSID.

Next, we’ll hop on over to the Security tab, and focus on the Layer 2 sub-tab. The first thing I want to point out - is that at this point, we will not be enabling the MAC Filtering checkbox. We’ll need some additional config first, then come back to this later.

Scroll down to the bottom of the window and there will be some settings for your authentication. By default, 802.1x will be enabled. This post won’t cover how to setup/configure that. Instead, we’ll be deselecting 802.1x and checking the box for PSK. Then a text field will appear for us to enter the Pre-Shared Key.

Once we click Apply to Device, we’ll finish up by assigning our VLAN or VLAN Group. In this case, I have already created a VLAN named IoT. If you haven’t created a VLAN yet, you can do so by going to Configuration > Layer 2 > VLAN. Then add a new VLAN under the VLANtab.

Click Add, then we’ll be back to the wizard and see the new WLAN we just created.

In order to finish up with the wizard, we just need to assign our Access Points. Click on the AP Provisioning tab. If you have already configured APs to join to this controller, you will see them on the left side under Available APs. Check which ones to apply this WLAN to, then click the arrow to move them to APs on this location.

Click Apply, and that will finalize all of the configuration we just did - then drop us back to the Wireless Setup page.

Okay - Now that we have that completed, we can move onto creating our MAC filtering policies.

Back in the menu - Let’s go to Configuration > Security > AAA

In this section - we first need to create an Authorization policy. Select the AAA Method List tab, then Authorization, then Add to create the new policy.

In here we’ll specify a name, then select Type: network, and Group Type: local. Then go ahead and Apply to Device

Once we have that, let’s go over to the AAA Advanced tab, and click Add in the Attribute List Name section.

Here we need to provide the SSID we want our MAC policy to apply to. Under Attribute Type, select SSID. Then under Attribute Value, select the target SSID that our policy will be tied to.

Don’t forget to click Save on the attribute before clicking Apply to Device!

Time to input our list of device MAC addresses! Drop into the Device Authentication section, and click Add - or upload a CSV file if you have one.

Input the device MAC Addrees and select the Attribute List Name that we configured just a minute ago. Then Apply to Device

After that - we should have our completed list of MAC addresses which will be permitted to join our wireless network. All we need to do is go back to our WLAN and enable MAC filtering.

Let’s go to Configuration > Tags & Profiles > WLANs

This should give us the list of any configured WLANS - including the one we created earlier. Go ahead and click on it to edit.

Next, we’ll jump straight to the Layer 2 section under the Security tab. Check the box for MAC Filtering and select the Authorization List we created from the drop down.

And we’re done! Clients that want to join our newly created SSID will need the pre-shared key we configured, but they will also need to be manually added to our MAC address filter as well. While this isn’t a perfect security measure since MAC addresses can be easily spoofed - it does add an extra layer of protection to keep unauthorized devices from inadvertently being able to join this specific WLAN.

In the event that a client is NOT on the authorized list, you may see the following logs in a client debug:

2019/10/25 22:26:12.327 {wncd_x_R0-0}{1}: [client-orch-sm] [22573]: (note): MAC: aaaa.aaaa.aaaa  Association received. BSSID aaaa.aaaa.b34d, old BSSID 0000.0000.0000, WLAN SuperSecret, Slot 1 AP aaaa.aaaa.b340, AP_3802-1F01
2019/10/25 22:26:12.327 {wncd_x_R0-0}{1}: [client-orch-state] [22573]: (note): MAC: aaaa.aaaa.aaaa  Client state transition: S_CO_INIT -> S_CO_ASSOCIATING
2019/10/25 22:26:12.328 {wncd_x_R0-0}{1}: [client-orch-state] [22573]: (note): MAC: aaaa.aaaa.aaaa  Client state transition: S_CO_ASSOCIATING -> S_CO_MACAUTH_IN_PROGRESS
2019/10/25 22:26:12.328 {wncd_x_R0-0}{1}: [client-auth] [22573]: (note): MAC: aaaa.aaaa.aaaa  MAB Authentication initiated. Policy VLAN 0, AAA override = 0, NAC = 0
2019/10/25 22:26:12.329 {wncd_x_R0-0}{1}: [ewlc-infra-evq] [22573]: (note): Authentication Success. Resolved Policy bitmap:11 for client aaaa.aaaa.aaaa 
2019/10/25 22:26:12.329 {wncd_x_R0-0}{1}: [ewlc-infra-evq] [22573]: (ERR): SANET_AUTHC_FAILURE - AAA Server Down username ac37434a673a, audit session id 000000000000006B3E9D2A55, 
2019/10/25 22:26:12.331 {wncd_x_R0-0}{1}: [client-orch-state] [22573]: (note): MAC: aaaa.aaaa.aaaa  Client state transition: S_CO_MACAUTH_IN_PROGRESS -> S_CO_ASSOCIATING
2019/10/25 22:26:12.331 {wncd_x_R0-0}{1}: [dot11] [22573]: (ERR): MAC: aaaa.aaaa.aaaa  Failed to assoc failure tr state entry. Incorrect validation status value :1
2019/10/25 22:26:12.331 {wncd_x_R0-0}{1}: [dot11] [22573]: (ERR): MAC: aaaa.aaaa.aaaa  Dot11 update co assoc fail. Sent assoc failure to CO. delete reason: 9, CO_CLIENT_DELETE_REASON_MAB_FAILED
2019/10/25 22:26:12.331 {wncd_x_R0-0}{1}: [client-orch-sm] [22573]: (note): MAC: aaaa.aaaa.aaaa  Client delete initiated. Reason: CO_CLIENT_DELETE_REASON_MAB_FAILED, fsm-state transition

On the opposite side, when a client is successfully able to pass MAC authentication - the logs will show the following:

2019/10/25 21:25:53.148 {wncd_x_R0-0}{1}: [client-auth] [22573]: (note): MAC: aaaa.aaaa.aaaa  MAB Authentication initiated. Policy VLAN 0, AAA override = 0, NAC = 0
2019/10/25 21:25:53.150 {wncd_x_R0-0}{1}: [ewlc-infra-evq] [22573]: (note): Authentication Success. Resolved Policy bitmap:11 for client aaaa.aaaa.aaaa 
2019/10/25 21:25:53.151 {wncd_x_R0-0}{1}: [client-auth] [22573]: (note): MAC: aaaa.aaaa.aaaa  MAB Authentication success.
2019/10/25 21:25:53.151 {wncd_x_R0-0}{1}: [client-orch-state] [22573]: (note): MAC: aaaa.aaaa.aaaa  Client state transition: S_CO_MACAUTH_IN_PROGRESS -> S_CO_ASSOCIATING

2019/10/25 21:30:08.626 {wncd_x_R0-0}{1}: [client-orch-sm] [22573]: (debug): MAC: aaaa.aaaa.aaaa  Received Dot11 association request. Processing started,SSID: SuperSecret2, Policy profile: Home_WLANID_3, AP Name: AP_3802-3F01, Ap Mac Address: aaaa.aaaa.c9a0 BSSID MAC aaaa.aaaa.b34d wlan ID: 3RSSI: 0, SNR: 32
2019/10/25 21:30:08.626 {wncd_x_R0-0}{1}: [client-orch-state] [22573]: (note): MAC: aaaa.aaaa.aaaa  Client state transition: S_CO_L2_AUTH_IN_PROGRESS -> S_CO_L2_AUTH_IN_PROGRESS
2019/10/25 21:30:08.626 {wncd_x_R0-0}{1}: [dot11] [22573]: (info): MAC: aaaa.aaaa.aaaa  DOT11 state transition: S_DOT11_ASSOCIATED -> S_DOT11_MAB_PENDING
2019/10/25 21:30:08.626 {wncd_x_R0-0}{1}: [client-orch-state] [22573]: (note): MAC: aaaa.aaaa.aaaa  Client state transition: S_CO_L2_AUTH_IN_PROGRESS -> S_CO_MACAUTH_IN_PROGRESS
2019/10/25 21:30:08.626 {wncd_x_R0-0}{1}: [client-auth] [22573]: (info): MAC: aaaa.aaaa.aaaa  Client auth-interface state transition: S_AUTHIF_ADD_MOBILE_ACK_WAIT_KM -> S_AUTHIF_MAB_AUTH_DONE
2019/10/25 21:30:08.626 {wncd_x_R0-0}{1}: [client-orch-sm] [22573]: (debug): MAC: aaaa.aaaa.aaaa  Processing MAB authentication result status: 0, CO_AUTH_STATUS_SUCCESS
2019/10/25 21:30:08.626 {wncd_x_R0-0}{1}: [client-orch-state] [22573]: (note): MAC: aaaa.aaaa.aaaa  Client state transition: S_CO_MACAUTH_IN_PROGRESS -> S_CO_ASSOCIATING
2019/10/25 21:30:08.626 {wncd_x_R0-0}{1}: [dot11] [22573]: (debug): MAC: aaaa.aaaa.aaaa  dot11 send association response. Sending association response with resp_status_code: 0 
2019/10/25 21:30:08.627 {wncd_x_R0-0}{1}: [dot11] [22573]: (info): MAC: aaaa.aaaa.aaaa  dot11 send association response. Sending assoc response of length: 137 with resp_status_code: 0, DOT11_STATUS: DOT11_STATUS_SUCCESS
2019/10/25 21:30:08.627 {wncd_x_R0-0}{1}: [dot11] [22573]: (note): MAC: aaaa.aaaa.aaaa  Association success. AID 1, Roaming = True, WGB = False, 11r = False, 11w = False
2019/10/25 21:30:08.627 {wncd_x_R0-0}{1}: [dot11] [22573]: (info): MAC: aaaa.aaaa.aaaa  DOT11 state transition: S_DOT11_MAB_PENDING -> S_DOT11_ASSOCIATED

What is NFVIS?

NFVIS is an operating system developed by Cisco which is intended to be deployed at branch office locations - and allow for quick deployment of network services in lightweight VMs. For example, we might want to reduce cost and hardware footprint by deploying a single ENCS machine, then deploy our typical branch services on top of that (DNS, Firewalls, SDWAN, etc). Under the hood, NFVIS is built on top of CentOS and KVM.

In the image below, we have an ENCS unit that is running ISRv, FTDv, and a vEdge Cloud. NFVIS has the ability to build out traffic flows for service chaining. In this particular setup, we could have all branch traffic receive a default route up to our ISRv. The ISRv forwards traffic to a Firepower VM (FTDv) which performs some traffic inspection before passing everything up to the vEdge Cloud.

We’ll be coming back to this diagram later to see how we can build out this flow of services. For now, let’s dive into how we can get NFVIS up and running.

Installing NFVIS

Lucky for us - the installation of NFVIS is fairly straightforward!

1. Create a bootable USB - or mount the installation ISO via CIMC
2. Upon boot, select “Install Cisco NFV Infrastructure Software”:

3. Wait. A while. Install time can vary depending on your hardware.
4. Once completed, log into the CLI: Default login = admin/Admin123#
5. You’ll be prompted to change the default admin password immediately:

Install completed! Now let’s look at some of our base configuration..

Initial Configuration

By default the NFVIS install will have a LAN and WAN bridge (lan-br and wan-br, respectively). The LAN config will be set up with a static IP of 192.168.1.1/24, and the WAN will be set for DHCP. We can check the current network settings by running the show system settingscommand:

In this case, my WAN interface is able to get an IP via DHCP. We’re likely going to want to change this to a static IP address - which we can do from the CLI or web interface. Let’s start by trying this from the CLI:

config t
system settings wan ip address <ip addr> <netmask>
system settings default-gw <gateway addr>
system settings hostname <hostname>

Changes can then be applied using the commit command

We can verify these settings by repeating the show system settings command we used earlier.

Let’s go ahead and log into the web interface to see what the network configuration looks like there:

1. If we know our WAN or LAN IP from earlier, we can just pop that in our web browser.
2. Go ahead and log in using the new admin credentials we just created:

3. We’ll be taken to the primary NFVIS dashboard, which will currently show no active VMs deployed:

4. In the left-hand menu, expand Hostthen click on Settings:

Here we can see that we already configured our IP Addressing/hostname - but we could use the Edit button at the bottom to change any of these values. For example - I’m going to go ahead and select Static for both the Management (LAN) and WAN IP addresses.

Another quick tip - if we need to modify which physical network adapters are tied to an internal network bridge, we can find that under VM Life Cycle > Networking:

That’s all for this time. In the next post, we’ll take a look at how to package VM images and deploy our service chain.