In this post, we’ll walk through how to connect an OPNsense firewall to Mullvad’s wireguard VPN. This can be used in various deployments to help protect your home network traffic.

As a quick note, this post is not sponsored by Mullvad - I just happen to really like their service & appreciate their approach to privacy and security. In fact, they currently don’t offer any incentives or paid promotions of their product. More info on their current policy can be found here.

Creating an Account

Mullvad is unique in that they don’t really require any sign up to get started with their service. Instead, when you sign up for new service, they randomly generate an account number. That’s it. They don’t require any additional information from you.

So first we can hit their new account page here - then click to generate a new account number.

Keep this account number somewhere safe. Mullvad has very limited ability to help you recover the account number.

Once we have that, we can click the button to Add time to our account.

Mullvad doesn’t support any recurring subscriptions, so that they are able to keep less data about their customers. Instead, we just add time increments in months for how long we would like to use the service. This can be anywhere from a single month up to a year.

In terms of payments, they do offer a number of options, including the usual PayPal/credit cards - or if you’re truly concerned about privacy, they accept a few cryptocurrencies or you can actually mail them a cash payment as well.

After a payment has been made & time added to the account, we can quickly jump into the configuration side of things.

Config: Mullvad Side

First we’ll take a quick look at the configuration required on the Mullvad VPN side of things.

On the left side of the account page, we’ll click on WireGuard Configuration.

First we’ll select Linux as our platform (In this context, what is selected here isn’t really important) - and then click the button to generate WireGuard keys:

Note: Alternatively, we could have generated our WireGuard keys on our OPNsense firewall - then applied them here. It’s up to you on which method you prefer!

Next, we’ll take a look at which server(s) we would like to connect to.

When selecting a server, we have the option to pick our desired country & location - as well as picking a specific server to connect to if we choose. There is also an option to select all servers - in which case the config generator will create a WireGuard configuration file for each server.

For the purpose of this walk through, we’ll keep things simple & only select a single server. So in the screenshot above, I’ve selected us-qas-wg-004.

We’ll also take a quick look at the advanced options, which give us a little flexibility if we need it. For most people, these additional options will not be necessary to change or modify.

We can enable Multihop functionality, so long as we only selected a single server to connect to. So if you selected the All Servers option, this won’t be available. This allows us to specify an entry & exit server for our VPN. In other words, our device would directly connect to the entry server we select - then Mullvad would tunnel our traffic across their network to the exit server, where our traffic would be decrypted & forwarded out to the internet. Depending on your privacy & security desires, this is a really nice option to have the ability to enable.

Next, we have options on what type of connection we would like & which traffic to forward.

Server connection protocol will specify whether we are using IPv4 or IPv6 between our device & our VPN server. Most likely you’ll want to leave this at IPv4, unless you have an IPv6-only internet connection (or if you just would prefer to use IPv6 anyways!).

Tunnel traffic is how we can specify whether we would like IPv4 or IPv6 client-side traffic to be forwarded over the VPN. So this would depend on whether the clients on our network have IPv4 vs IPv6 connectivity, or both - and whether we prefer to only forward certain types of traffic over the VPN. I’ll be leaving this setting as the default: Both.

Next we can specify a custom port if we would like. By default, wireguard will use UDP port 51820 - and we probably won’t need to change this unless the port is being blocked upstream.

Lastly, we also have the option to enable content blocking across the VPN. Mullvad accomplishes content filtering through DNS-level blocking - and when we finish generating a configuration file, the file will include DNS servers to use. This is okay to use if you were connecting a single client to their VPN service. However, if you’re using a router or device like OPNsense, you would need to update the DNS on all clients on your network to make this work. This is possible by updating DHCP on our router with the new DNS server address - or configuring a DNS rewrite. We won’t get into either of those in this post - so for now I will leave the content blocking options unchecked.

Once we’re good with our configuration - we can click the Download File button. We’ll get a standard WireGuard config file, that looks like this:

At this point, we’re good to move onto the next part!

Config: OPNsense Side

Okay, so now that we have everything ready to go on the Mullvad side - we can configure our OPNsense device.

Make sure you already have WireGuard installed on OPNsense. This can be done by navigating to System > Firmware > Plugins then searching for wireguard & clicking the install button.

Next, we’ll enable Wireguard by navigating to VPN > Wireguard and checking the box to Enable WireGuard, then Apply.

Then we’ll hop over to the Endpoints tab & configure our Mullvad VPN peer.

For this part of the configuration, we’ll just copy our public key, allowed IPs, endpoint address, and endpoint port from our Mullvad config file. In the screenshot below, I also named my endpoint with the specific Mullvad server I’ll be connecting to:

Then we can click Save and Apply.

If you wanted multiple Mullvad servers configured, just create a new endpoint for each one. Then, make sure that you select all of the Mullvad peers on the next step below.

After that, we can move over to the Local tab to define our OPNsense tunnel configuration.

Click the button to add a new peer, then we’ll fill in our private key and tunnel address(es) from the Mullvad config file. Under Peers, we’ll also select our Mullvad VPN peer that we configured just a moment ago:

UPDATE: Looks like with a recent OPNsense update, they now require you to enter both the WireGuard private & public key into the local config (shown above). In the the screenshot, I only show entering the private key - since this was all that was required at the time. Two ways to get your public key:

1. Log into Mullvad & check the “Devices” tab under “Account Management”. This will show your device public key (They don’t keep your private key after generating it for you, only the public).
2. If you have wireguard installed somewhere else, you can use the “wg pubkey” command to derive a public key from your private key. Command: echo <private_key> | wg pubkey

Note: By default with the configuration we’ve applied so far, this VPN will forward ALL traffic on our network to Mullvad. If we would prefer to selectively choose which traffic to send over the VPN, we can check the box for Disable Routes - then use policy routing to forward specific things to Mullvad. We’ll take a look at how to do this later in the post - but for now just be aware that our current configuration will forward ALL traffic.

Okay, with that all done we can click Apply and Save here as well.

With any luck, we can check the Status tab & see that there is data being transmitted & successful WireGuard handshakes:

However, before our clients traffic can be forwarded over the VPN, we’ll need to create a firewall rule to permit traffic & a NAT rule to translate our client addresses to our Mullvad IP.

We’ll navigate to Firewall > Rules > WireGuard (Group). Then we’ll click to create a new rule.

Within this new rule, I’ll update Direction to Out and change TCP/IP Version to IPv4+IPv6. I’ll leave the source as Any:

We can also leave the Destination as Any, but I’ll update the rule to enable logging:

This rule will allow any clients behind our OPNsense firewall to reach anything on the internet.

Next, we’ll have to create a NAT rule. This ensures that our client addresses on our network get appropriately translated to the tunnel IP address that Mullvad has assigned us.

We’ll navigate to Firewall > NAT > Outbound. By default, OPNsense will be set to Automatic outbound NAT rule generation. We’ll need to update this to Hybrid outbound NAT rule generation to allow custom NAT rules. Then we can click Save and Apply.

Next, we’ll create a new Manual NAT rule, where we’ll update our Interface to WireGuard (Group):

Then we’ll make sure our Translation / target is set to Interface Address - and again, I’ll enable logging:

After we click save, we should have a NAT rule that looks like this:

At this point, we should be good to test our clients!

Testing

Of course, it’s easy enough to use one of our clients to check that we still have internet access - but how can we be sure that they’re using the VPN?

The easiest way might be to check the mullvad.net, where they do have a quick validation at the top of the page:

So according to Mullvad, it looks like we’re connected & they even show which server we’re connecting from.

We can also double check using a traceroute or tracepath command:

In this case, we can see that our traffic to Google hits our OPNsense gateway, then the Mullvad VPN gateway followed by another external address owned by Mullvad.

So based on some quick testing, it looks like we’re all good!

Policy Routing

So in the above walkthrough, we configured a Mullvad VPN from our OPNsense firewall - but it is forwarding ALL of our network clients over the VPN. What about if we only wanted certain clients to use the VPN? Or all clients to use it, but only for certain destinations?

We can accomplish this through policy routing.

So the first thing we’ll do is go back to our WireGuard config, then under the Local tab. We’ll edit our configuration here, and check the box for Disable Routes.

By default, OPNsense / WireGuard will install routes for any IPs listed in the AllowedIPs field for each peer. In our set up, we configured 0.0.0.0/0 - which matches all traffic. By checking the box for Disable Routes, we prevent OPNsense from installing that default route - and instead we can manually specify our own.

If you’re curious to double check this, you can try hitting Mullvad’s website after changing this setting - and it should show that you’re no longer connected.

Then, we’ll need to set up our WireGuard configuration to use a dedicated, named interface so that we can create a static gateway.

We’ll head to Interfaces > Assignments - and create a new interface. From the drop-down, we’ll select our WireGuard interface - in my case this was wg1. Then we can assign it a name:

Then click the + icon to add, and Save.

Then we can navigate to the interface name under the Interfaces menu - and enable the new interface:

Next we can create a gateway to route traffic through. Navigate to System > Gateways > Single.

Create a new gateway & give it a name. Then we’ll enter our Mullvad VPN gateway IP address, which in my case was 10.64.0.1. How did we find this? Well earlier when we tested our VPN connectivity - we performed a tracepath. In the output of this tracepath, our second hop (the one right after our OPNsense firewall) would be the Mullvad VPN gateway. So this is the address we’ll use for our gateway here:

Then clic Save and Apply.

Note: You may get an error here, like “The gateway address “10.64.0.1” does not lie within one of the chosen interface’s IPv4 subnets.” To resolve this, we’ll temporarily change our interface address. Navigate to the WireGuard local config, and re-enter your tunnel address excluding the “/32”. For example, if your tunnel IP was 10.10.10.10/32, change this to just 10.10.10.10 You should be able to set the gateway address now. Be sure to change your tunnel IP back afterwards!

Next, we’ll create a firewall rule for each set of sources or destinations we would like to manipulate.

So we can navigate to Firewall > Rules > Floating (or select a specific interface for clients, like LAN).

In here, we’ll set whichever parameters we would like to match. So for this example, let’s say I have a client PC at 10.100.100.10 and I only want traffic to 8.8.8.8 to use Mullvad. All other traffic can use the normal internet connection & not use the VPN.

In that case, I’ll set my source address to 10.100.100.10/32 and my destination to 8.8.8.8/32:

Then all we need to do is update our Gateway to use the Mullvad gateway we just created:

Now, if we hop over to our client PC - Mullvad’s website will say that we’re not connected to the VPN. However, we can check that traffic to 8.8.8.8 is actually being sent through Mullvad using tracepath again:

In the screenshot above, I also included a tracepath to 8.8.4.4, just to show that it is going through my normal internet connection & not Mullvad.

This was just one example, but you could easily create multiple firewall rules for different sources and/or destinations to control where traffic is sent. Aliases can also be used to group sources or destinations, so that multiple can be added to a single firewall rule.

Okay - I think that’s about all I wanted to cover in this post.

Hope it is helpful! Feel free to leave a comment below - or follow me on YouTube 😊

So in my last post, I picked up a Qotom Mini-PC to run OPNsense on. After a few months, the device has been running well & I’m very happy with it.

One of the new things I got to try out with OPNsense was Wireguard VPN. I had previously been using something else for VPN connectivity back to my home network - but I heard good things about Wireguard & wanted to give it a try.

So far my experience has been good! I’ve been pleasantly suprised with how easy it is to configure & get running. In addition, the performance & overall experience has been very positive. The VPN connects quicker than anything I’ve used in the past, and just simply works without issue.

All that being said - I wanted to put together a quick guide on how to configure Wireguard on OPNsense. Specifically, this configuraion will be for remote-access VPN - where clients will connect to a VPN headend. We’ll walk through the OPNsense configuration & a few clients as well. So let’s dig in!

Topology

For the purpose of this blog post, we’ll be using the lab topology below:

I will be using the reserved IP range 203.0.113.0/24 for the WAN-side addressing. I’ll have one Windows & one Android client that we’ll walk through & connect to the VPN.

Installing the Wireguard Plugin

To get started, first thing we will want to do is install the Wireguard plugin for OPNsense. By default, OPNsense will have standard IPSec & OpenVPN already available - but other VPN options can be enabled easily.

So in OPNsense, we’ll navigate down to System > Firmware > Plugins, then search for wireguard and click the plus icon.

This should pull down the package & install pretty quickly. No reboot required here!

Once installed, you may have to refresh the page or navigate to a new page so that the menu bar has a chance to reload. Then we’ll have a new option under VPN:

Wireguard Tunnel Configuration

Next we’ll begin configuring Wireguard on the OPNsense side.

There is a little bit of a chicken & egg scenario here since everything is based on cryptographic keys. We’ll need to generate keys on the firewall, which we need to enter on the client - but we also need the client keys to enter on the firewall. A bit of bouncing between the two - but for now we’ll try to complete as much as we can on the firewall side.

We’ll enable Wireguard by dropping down to VPN > WireGuard then clicking Enable and Apply

Next we’ll set up the Wireguard tunnel interface on OPNsense. This will be a virtual tunnel interface that will be created as interface wg<instance number>.

To do this, we’ll navigate to the Local tab, and click the plus icon to add a new tunnel.

In the above screenshot, I’ve filled in just a few details.

For Name, I’ve entered our virtual interface name wg1. Since OPNsense shows the Instance as 1 - it will create a wg interface with that instance number.

We’ll leave Public Key & Private Key blank for now. OPNsense will auto-generate these keys once we save this config.

For Listen Port, I’ve set this to 51820 which is the default for Wireguard. It’s not stated here, but this is a UDP tunnel.

For Tunnel Address - this is where we add the IP address of the virtual tunnel interface. This will be the gateway for our remote clients. In my lab I’ll be using 10.50.50.1/24 here.

We have no peers configured yet - so we can’t select any here. We’ll leave this blank for now, but come back later.

We will leave Disable Routes unchecked. By default, OPNsense will add static/connected routes for any client via the tunnel interface. You might not want this behavior if you wanted to do custom routing - for example, in a site-to-site VPN connection - but we’ll leave this enabled.

Once we’re done, we’ll click Save.

Like I mentioned before, OPNsense will now auto-generate our crypto keys for the tunnels. So if we edit our tunnel, we’ll now see those fields populated:

We’ll want to copy the Public Key & save it for later. This will need to be imported onto our clients, so that they can communicate securely with our firewall.

Wireguard Interface Assignment

Now that we have our headend tunnel interface defined, we can map our wg1 interface to an OPNsense interface. The OPNsense documentation suggests this is optional, but I would recommend it since it will allow us to create firewall rules to permit/deny access to clients.

We’ll navigate to Interfaces > Assignments, and we should see a New interface available: our wg1 tunnel.

We can assign this a name, then click the plus icon & Save.

Next we’ll enable the interface by navigating to Interfaces > WG1. Here we’ll only need to click Enable & save the change - nothing else is necessary.

Of course, we’ll be prompted to apply the changes - which we will do:

By default, all traffic through our WG1 firewall interface will be blocked - so please make sure to configure a firewall rule to permit traffic from the Wireguard clients.

Firewall Rules: Allow Inbound Wireguard Traffic

Next we need to permit the Wireguard traffic into our firewall. By default the WAN interface will block all traffic that isn’t explicitly allowed - including our Wireguard traffic.

For this, we’ll navigate to Firewall > Rules > WAN. Then click the plus icon to add a new rule.

The screenshot above shows what our firewall rule will look like.

Here’s the summary:

• Action: Pass
• Interface: WAN
• Direction: In
• TCP/IP Version: IPv4
  • You can enable IPv6 as well, if you have IPv6 connectivity (this is a lab box, which does not)
• Protocol: UDP
• Source: Any
  • This will allow anyone on the internet to reach our VPN. We could restrict source IP addresses, if our clients had permanent, static IPs.
• Destination: WAN Address
  • The firewall itself is the destination for this traffic
• Destination Port Range: (other) / 51820

I also enabled logging & added a quick description. After we have all this configured, we can click Save - then Apply Changes.

Client Setup - Windows

So first we’ll start with an easy configuration on a Windows client. Wireguard client software can be found on the Wireguard site here.

For the sake of the walkthrough, we will manually configure each client. However, this can be a difficult task if there is a large number of clients. Wireguard does support importing configurations, and there are a number of free tools available to help automate generating config files for clients - including some which will generate QR codes for easy import on mobile clients.

So on my lab Windows machine, we’ll open up the Wireguard client & click Add Empty Tunnel:

Then we’ll be given a blank config file, with only the devices public & private key pair generated for us.

We’re going to fill in details similar to the below screenshot:

The configuration under [Interface] is the local, client-side configuration. I’ve added the client’s tunnel address - which will be 10.50.50.15/32. I’ve also configured DNS servers which the client can reach via the VPN.

Then we’ll add the [Peer] section, which contains info about our VPN headend. Here’s where we’ll need the public key from our OPNsense firewall. We’ll also specify the Endpoint address, which is the IP or hostname of our VPN headend & the port (which by default is 51820).

We’ve also configured AllowedIPs as 0.0.0.0/0. This will force all client traffic over the VPN tunnel - including general internet traffic. However, we could limit this to specific subnets. For example, let’s say your network only used 172.16.90.0/24 & 10.1.1.0/16 subnets & we only wanted the user to be able to access those. We would configure the following: AllowedIPs = 172.16.90.0/24, 10.1.1.0/16. In this case, only traffic for those subnets would be routed over the VPN - any other traffic would use the devices default internet connection.

Now, we’ll save this - and again need to copy the device’s Public Key, which we’ll need to enter on the OPNsense firewall.

Client Setup - Adding Clients to OPNsense

In order for the Windows machine to connect to OPNsense, we’ll also need to configure a client profile on the firewall.

In OPNsense, we’ll navigate back to VPN > WireGuard, then click on the Endpoints tab.

Here we’ll configure a name for our client & paste in the client’s Public Key.

We’ll also set AllowedIPs to the client’s IP address, which we have configured as 10.50.50.15/32. This controls what IP addresses are reachable via this endpoint.

We do have some additional fields available, which we will leave blank. For example - Endpoint Address & Endpoint Port would be used to define a public IP that we expect our client to connect from. Since a remote-access client could connect from any IP, we leave those fields blank to allow this.

Once we’re done, we’ll click Save then Apply.

Next we’ll jump back to the Local tab, and edit our headend tunnel configuration.

We should now see our windows client in the Peers dropdown. We’ll select that client, so that Wireguard will permit that client to connect via this tunnel interface.

Then, as always, click Save & Apply

Last but not least - we should restart our Wireguard server on OPNsense. This can be done by either disabling & re-enabling Wireguard - or by navigating back to the OPNsense dashboard & clicking the restart icon next to the wireguard-go service.

Testing The Connection

Okay - now that we have all that completed, it’s finally time to test connectivity from our client.

On my Windows client, I’ll just click the Activate button for the tunnel:

And we’ll see that the Status shows Active, and we’ll start to see updates to the Last Handshake & Transfer fields - indicating we are connected & sending data.

To further validate, we can check the Log tab:

The key things to look for here, are the following messages:

• Receiving handshake response which indicates our firewall responded to our request to connect
• Keypair 1 created which indicates that our connection is healthy to our peer
• Receiving keepalive packet from peer - we should see these periodically to maintain our connection. If keepalives stop flowing, then we may have a break in connectivity between client & peer.

We should also be able to reach out wg1 tunnel address from the Windows client:

On the OPNsense side of things, we can check what client is connected via the List Configuration tab, under VPN > Wireguard:

On this screen, we can see we have 1 peer connected - which matches the IP & public key of our Windows client. We’ll see similar output like the Windows client, where we can see the latest handshake & data transfer.

Additionally, for easier access we can add the Wireguard widget to the OPNsense dashboard.

If we navigate to our dashboard, then click Add widget - we can add the Wireguard widget.

Here’s what that looks like:

Additional Client Setup - Android

Let’s also take a quick look at a mobile client. I’ll get through this pretty quickly, since the configuration will be very similar to our other client - just a different UI.

On my Android device - if I open up the Wireguard app, I have a few options for creating or importing a tunnel:

In this case, we’ll again walk through creating a tunnel manually. Again, it’s worth mentioning that there are 3rd party apps available to auto-generate config imports or QR codes to make this easier.

First we’ll have a handful of fields to fill in about the Android-side configuration. This includes a name for the tunnel, an address, and DNS servers.

We can click on the refresh icon in the Private Key field to auto-generate our key pairs:

One of the interesting parts of the mobile app is the ability to permit/exclude individual apps from traversing the VPN tunnel.

Here’s a quick screenshot, where we can pick either to allow only certain apps to use the VPN - or permit all except a few we choose to exclude:

In my case, I’ll keep this set to allow all applications.

Next we’ll work on the peer config:

After that, all we need to do is save our VPN configuration - then we can toggle the tunnel on or off:

And similar to the Windows client, we can click on the tunnel itself to see current status / data transfer:

So it looks like we should be connected & can try accessing VPN resources!

We can also use the same methods as earlier to check connectivity from the OPNsense side.

Okay - that’s all I wanted to share today. I’ve been quite pleased with how easy to use WireGuard has been - and how well it performs! I hope this blog post was helpful if you’re interested in trying it out.

Thanks!!

^{Note: I may receive commissions for purchases made through links in this post. This is to help support my blog and does not have any impact on my recommendations.}

One day I would love to have symmetric internet speeds, but for the meantime I have to work with what I have - which is dismal upload speeds that don’t come anywhere close to the download speed.

So I’m currently at 200Mb/s down & 10Mb/s up - and I would prefer to have higher upload speeds. However, even just 20Mb/s upload requires me to purchase a 500Mb/s download plan. 🙄

I’ve been hesitant to upgrade, since my current Meraki MX64 tops out at ~250Mb/s throughput. I would have to upgrade my firewall or half of my new download speed would be unusable.

While Meraki does have faster firewalls, they’re a bit expensive & at the moment somewhat difficult to obtain. So instead I searched around for alternatives, and came across the Qotom Q750G5 - which is a barebones mini-PC that I could load pfSense or OPNsense onto.

So in this blog post - I wanted to talk about the device & some of the performance testing I did.

Qotom Q750G5 - Hardware

First let’s take a quick look at the hardware. What got my attention quickly was the five 2.5GbE ports, but this small device offers quite a lot for how inexpensive it was.

A quick look at the specs:

• Intel Celeron J4125 Quad Core 2.0Ghz (Burst 2.7Ghz)
• Five Intel I225-V 2.5Gb Ethernet ports
• Optional WiFi slot
• Optional 3G Cellular & SIM card slots
• 1 RAM slot
• 1 mSATA slot
• 1 2.5in SATA HDD slot
• 2x USB 2.0 & 3x USB 3.0 ports

By default it seems this is a barebones kit, however I did opt to buy a model that included 16GB RAM & a 256GB mSATA SSD. It arrived with a single 16GB TeamGroup DDR4 3200Mhz module & 256GB Hoodisk mSATA SSD.

Are the specs a bit overkill for an embedded firewall? Yeah definitely. But the whole kit was only around $230 USD - which seemed crazy to me.

The big thing that pulled me toward this model (besides price) was the 2.5GbE ports. Since a lot more places have gigabit residential internet these days, and some lucky places are starting to see 2Gb/s - I was hoping that this little box would offer me a bit of future-proofing.

Of course, I was a little curious about what performance this device could realistically achieve - but more on that later!

I wanted to provide a few photos of the unit as well. The casing itself is pretty minimal, but the box is definitely heavier than you would expect!

Front Photo:

On the front there isn’t a whole lot to see. Power button, reset button, status LEDs, and USB ports. Oh - and there is an HDMI port as well, which comes in handy when installing an operating system (or if you’re using this as an actual PC).

Back Photo:

The back shows off the five 2.5Gb Ethernet ports, as well as the power plug. While I didn’t get a WiFi-enabled unit, the back faceplate still has cutouts for wireless antenna. I could opt to add wireless to this later, but I do wish the faceplate came with plugs or something to cover the holes in the meantine.

Inside Photo:

Now here’s where things get interesting! This box surprisingly packs a lot in it’s tiny size.

In the upper right, there is a PCI slot for WiFi. There’s also a WiFi/3G PCI slot in the lower left.

Just above the 3G slot is the SSD mSATA slot. When a drive is installed, it does cover the SIM card slot - meaning that the SSD would need to be removed to change SIM cards.

Also on the left, mostly out of the photo, is a connector for a 2.5in HDD. While I didn’t snap a photo of the bottom of the unit, the drive would screw into the bottom plate.

Interestingly enough, the CPU is on the other side of the motherboard. While you might think that the overall unit kinda looks like a heatsink - I was surprised to see that’s exactly what it is. The CPU is located right under the four screws with black washers and has thermal paste pre-applied. It rests up against the top of the case & uses the top as a heatsink.

As a last note about the CPU - it does get quite toasty at times. During one of my performance tests, the CPU reached ~90°C - and the top of the unit was very hot to the touch. It may not act as the world’s best heatsink, so I would avoid placing anything on top of the unit - or resting your hand there for too long 🙂.

Performance Testing

As a quick note before we get to the good stuff: All of my tests were performed using iPerf3. This may not show us the best real-world throughput tests, but I had quite a difficult time getting dpdk drivers to compile correctly for the Intel 2.5GbE ports (so I could use something like TRex). I may attempt to go back to this later, but the iPerf data is what I’ll provide for now.

The Testbed

For the performance testing, I used the two Intel NUC11 PCs that I purchased recently for my VMware lab. These already come with a 2.5GbE port, which I am currently using for vMotion between the two. I disconnected the vMotion port temporarily, and instead enabled PCI-passthrough to connect the ports directly to a VM.

I built a VM on each NUC with the following specs:

• Debian 11
• 8x vCPU
• 16GB RAM
• PCI-passthrough to 2.5GbE adapter

iPerf 3.9 was used for all tests. The iPerf server was enabled with the iperf3 -s command, and the client tests were run with iperf3 -c <client ip> -P 8 -t 600. Each test was run for 10 minutes.

The devices were connected & configured with the following topology:

Note: In the below paragraphs, I will talk about the configuration I used for each test. If you’re interested in more detail, please check out the video at the top of this blog post. In the video, I walk through the configuration & setup for most of the tests below.

Test 1: No Firewall

Avg: 2.35Gb/s

Okay, so expecting that I likely wouldn’t get the full 2.5Gb speeds once I added the firewall - I wanted to perform a baseline test with the VM’s directly connected via the 2.5GbE passthrough port.

In each of these tests, I was able to reach an average speed of: 2.35Gb/s

Test 2: Routing, NAT, Simple Firewall rules

Avg: 2.35Gb/s

In this test case, OPNsense was configured to match the diagram above. The iPerf server was located on the LAN segment, with an IP of 10.2.2.1 & a default route toward the LAN interface of the OPNsense box (10.2.2.2). The client is connected via the WAN interface, at 203.0.113.50.

I created a proxy-ARP virtual address for the 203.0.113.25 IP address, then created a NAT rule to forward traffic sent to that address to 10.2.2.1.

Next, there was a single firewall rule created to permit TCP port 5201 inbound from the WAN. This is the default port that iPerf will use.

During these tests, I averaged about 2.35Gb/s and relatively low CPU usage around 10-20%.

As an interesting side note to this: Originally I didn’t use the default LAN & WAN ports (ports 1 & 2), but instead used ports 3 & 4. During testing between those ports, I could only reach ~1.7Gb/s. Once I switched to ports 1 & 2, I could easily hit 2.35Gb/s. I’m curious to dig in later & see if this is a hardware issue, or possibly OPNsense is prioritizing the LAN/WAN traffic differently

Test 3: Large Firewall Ruleset

Avg: 2.35Gb/s

While my home network will likely have a somewhat minimal firewall ruleset, I was curious to see how well the device would perform with a large ruleset.

So I wrote a script to auto-generate ~1,200 firewall rules with random IP addresses, ports, and a mix of permit/block actions. I applied the ruleset inbound on both the LAN & WAN ports.

Under this test, I was still able to reach the 2.35Gb/s speeds - but now the CPU was creeping up to 30-40%.

So what next? Well I decided to see if I could stress the box a little - and increased the auto-generated ruleset to just over ~15,000 rules.

The increased ruleset took about 5-10 minutes to load into the system, and CPU was pegged at 100% the whole time. In fact, the CPU never really came back down. Even after the rules had loaded, the CPU was flat at 100% - and it took anywhere from 2-4 minutes to navigate between pages in the web GUI. (This is the part where my CPU temps went from <50°C to over 90°C 🙃)

I ended up having to re-image the box.

Test 4: Suricata IPS

Avg: 2.35Gb/s (But sometimes much, much less)

Next I loaded up the built-in IDS/IPS, which uses suricata. I downloaded all available free rulesets, and enabled all of them. Ideally, you wouldn’t necessarily enable everything - but I wanted to start off with a full load test.

My experience with these tests was highly variable. Most times, I was surprised to see that I could still push 2.35Gb/s through without any issue. During these tests, the CPU usually bounced between 50-80% usage.

Strangely, every so often I would test and get significantly less than that. Most times when this happened, I would instead only see somewhere between 400-700Mb/s - but on one test the box slowed to just over 200Mb/s. Each time this happened, the performance tests would remain degraded until I restarted the suricata service - then I would be able to reach the 2.35Gb/s again.

My best guess is that something is getting stuck. I know some older IPS systems I’ve worked with in the past had issues with CPU-pinning, and you might get garbage performance if your traffic hit the wrong CPU core. I didn’t spend a lot of time digging into this, but it feels a bit similar.

Test 5: Wireguard VPN

Avg: 800Mb/s, 650-700Mb/s with IPS also enabled

I also wanted to try VPN performance. There are a lot of options for VPN services within OPNsense, including standard IPSec and OpenVPN. However, I opted to try out wireguard - since it’s really easy to set up & get running.

In this case, I definitely hit a limitation with CPU-pinning. With my initial tests, I could only reach about ~450-500Mb/s - but the Qotom CPU was only spiking up to 70%.

Seems like the wireguard client (at least for same source/destination/port traffic) does pin everything to a single CPU. So my client CPU, between encrypting the traffic & generating it, was actually maxing out long before the Qotom was.

I ended up spinning up a second VM on the client side (which required disabling PCI passthough & adding both VMs to a shared vSwitch). With both of these running, I was able to max out the Qotom CPU at 100% - and hit a fairly consistent 800Mb/s VPN throughput.

As a final test, I enabled the suricata IPS on top of the VPN as well. Now the traffic would need to be decrypted & inspected before reaching the server. With a single client, I averaged ~400-450Mb/s - and with both clients it was around ~650-700Mb/s

Overall I’m really pleased with how well this thing performs, especially for how relatively inexpensive it was. I also expected OPNsense to have a steep learning curve, but it was surprisingly easy to get up and running very quickly.

Next I’ll be working on getting this firewall up & running at home. I may be tempted to write up some more on OPNsense - so if there are any questions or specific configurations you might like to see, please leave a comment!