Note: I may receive commissions for purchases made through links in this post. This is to help support my blog and does not have any impact on my recommendations.
One day I would love to have symmetric internet speeds, but for the meantime I have to work with what I have - which is dismal upload speeds that don't come anywhere close to the download speed.
So I'm currently at 200Mb/s down & 10Mb/s up - and I would prefer to have higher upload speeds. However, even just 20Mb/s upload requires me to purchase a 500Mb/s download plan. 🙄
I've been hesitant to upgrade, since my current Meraki MX64 tops out at ~250Mb/s throughput. I would have to upgrade my firewall or half of my new download speed would be unusable.
While Meraki does have faster firewalls, they're a bit expensive & at the moment somewhat difficult to obtain. So instead I searched around for alternatives, and came across the Qotom Q750G5 - which is a barebones mini-PC that I could load pfSense or OPNsense onto.
So in this blog post - I wanted to talk about the device & some of the performance testing I did.
Qotom Q750G5 - Hardware
First let's take a quick look at the hardware. What got my attention quickly was the five 2.5GbE ports, but this small device offers quite a lot for how inexpensive it was.
A quick look at the specs:
- Intel Celeron J4125 Quad Core 2.0Ghz (Burst 2.7Ghz)
- Five Intel I225-V 2.5Gb Ethernet ports
- Optional WiFi slot
- Optional 3G Cellular & SIM card slots
- 1 RAM slot
- 1 mSATA slot
- 1 2.5in SATA HDD slot
- 2x USB 2.0 & 3x USB 3.0 ports
By default it seems this is a barebones kit, however I did opt to buy a model that included 16GB RAM & a 256GB mSATA SSD. It arrived with a single 16GB TeamGroup DDR4 3200Mhz module & 256GB Hoodisk mSATA SSD.
Are the specs a bit overkill for an embedded firewall? Yeah definitely. But the whole kit was only around $230 USD - which seemed crazy to me.
The big thing that pulled me toward this model (besides price) was the 2.5GbE ports. Since a lot more places have gigabit residential internet these days, and some lucky places are starting to see 2Gb/s - I was hoping that this little box would offer me a bit of future-proofing.
Of course, I was a little curious about what performance this device could realistically achieve - but more on that later!
I wanted to provide a few photos of the unit as well. The casing itself is pretty minimal, but the box is definitely heavier than you would expect!
Front Photo:
On the front there isn't a whole lot to see. Power button, reset button, status LEDs, and USB ports. Oh - and there is an HDMI port as well, which comes in handy when installing an operating system (or if you're using this as an actual PC).
Back Photo:
The back shows off the five 2.5Gb Ethernet ports, as well as the power plug. While I didn't get a WiFi-enabled unit, the back faceplate still has cutouts for wireless antenna. I could opt to add wireless to this later, but I do wish the faceplate came with plugs or something to cover the holes in the meantine.
Inside Photo:
Now here's where things get interesting! This box surprisingly packs a lot in it's tiny size.
In the upper right, there is a PCI slot for WiFi. There's also a WiFi/3G PCI slot in the lower left.
Just above the 3G slot is the SSD mSATA slot. When a drive is installed, it does cover the SIM card slot - meaning that the SSD would need to be removed to change SIM cards.
Also on the left, mostly out of the photo, is a connector for a 2.5in HDD. While I didn't snap a photo of the bottom of the unit, the drive would screw into the bottom plate.
Interestingly enough, the CPU is on the other side of the motherboard. While you might think that the overall unit kinda looks like a heatsink - I was surprised to see that's exactly what it is. The CPU is located right under the four screws with black washers and has thermal paste pre-applied. It rests up against the top of the case & uses the top as a heatsink.
As a last note about the CPU - it does get quite toasty at times. During one of my performance tests, the CPU reached ~90°C - and the top of the unit was very hot to the touch. It may not act as the world's best heatsink, so I would avoid placing anything on top of the unit - or resting your hand there for too long 🙂.
Performance Testing
As a quick note before we get to the good stuff: All of my tests were performed using iPerf3. This may not show us the best real-world throughput tests, but I had quite a difficult time getting dpdk drivers to compile correctly for the Intel 2.5GbE ports (so I could use something like TRex). I may attempt to go back to this later, but the iPerf data is what I'll provide for now.
The Testbed
For the performance testing, I used the two Intel NUC11 PCs that I purchased recently for my VMware lab. These already come with a 2.5GbE port, which I am currently using for vMotion between the two. I disconnected the vMotion port temporarily, and instead enabled PCI-passthrough to connect the ports directly to a VM.
I built a VM on each NUC with the following specs:
- Debian 11
- 8x vCPU
- 16GB RAM
- PCI-passthrough to 2.5GbE adapter
iPerf 3.9 was used for all tests. The iPerf server was enabled with the iperf3 -s command, and the client tests were run with iperf3 -c <client ip> -P 8 -t 600. Each test was run for 10 minutes.
The devices were connected & configured with the following topology:
Note: In the below paragraphs, I will talk about the configuration I used for each test. If you're interested in more detail, please check out the video at the top of this blog post. In the video, I walk through the configuration & setup for most of the tests below.
Test 1: No Firewall
Avg: 2.35Gb/s
Okay, so expecting that I likely wouldn't get the full 2.5Gb speeds once I added the firewall - I wanted to perform a baseline test with the VM's directly connected via the 2.5GbE passthrough port.
In each of these tests, I was able to reach an average speed of: 2.35Gb/s
Test 2: Routing, NAT, Simple Firewall rules
Avg: 2.35Gb/s
In this test case, OPNsense was configured to match the diagram above. The iPerf server was located on the LAN segment, with an IP of 10.2.2.1 & a default route toward the LAN interface of the OPNsense box (10.2.2.2). The client is connected via the WAN interface, at 203.0.113.50.
I created a proxy-ARP virtual address for the 203.0.113.25 IP address, then created a NAT rule to forward traffic sent to that address to 10.2.2.1.
Next, there was a single firewall rule created to permit TCP port 5201 inbound from the WAN. This is the default port that iPerf will use.
During these tests, I averaged about 2.35Gb/s and relatively low CPU usage around 10-20%.
As an interesting side note to this: Originally I didn't use the default LAN & WAN ports (ports 1 & 2), but instead used ports 3 & 4. During testing between those ports, I could only reach ~1.7Gb/s. Once I switched to ports 1 & 2, I could easily hit 2.35Gb/s. I'm curious to dig in later & see if this is a hardware issue, or possibly OPNsense is prioritizing the LAN/WAN traffic differently
Test 3: Large Firewall Ruleset
Avg: 2.35Gb/s
While my home network will likely have a somewhat minimal firewall ruleset, I was curious to see how well the device would perform with a large ruleset.
So I wrote a script to auto-generate ~1,200 firewall rules with random IP addresses, ports, and a mix of permit/block actions. I applied the ruleset inbound on both the LAN & WAN ports.
Under this test, I was still able to reach the 2.35Gb/s speeds - but now the CPU was creeping up to 30-40%.
So what next? Well I decided to see if I could stress the box a little - and increased the auto-generated ruleset to just over ~15,000 rules.
The increased ruleset took about 5-10 minutes to load into the system, and CPU was pegged at 100% the whole time. In fact, the CPU never really came back down. Even after the rules had loaded, the CPU was flat at 100% - and it took anywhere from 2-4 minutes to navigate between pages in the web GUI. (This is the part where my CPU temps went from <50°C to over 90°C 🙃)
I ended up having to re-image the box.
Test 4: Suricata IPS
Avg: 2.35Gb/s (But sometimes much, much less)
Next I loaded up the built-in IDS/IPS, which uses suricata. I downloaded all available free rulesets, and enabled all of them. Ideally, you wouldn't necessarily enable everything - but I wanted to start off with a full load test.
My experience with these tests was highly variable. Most times, I was surprised to see that I could still push 2.35Gb/s through without any issue. During these tests, the CPU usually bounced between 50-80% usage.
Strangely, every so often I would test and get significantly less than that. Most times when this happened, I would instead only see somewhere between 400-700Mb/s - but on one test the box slowed to just over 200Mb/s. Each time this happened, the performance tests would remain degraded until I restarted the suricata service - then I would be able to reach the 2.35Gb/s again.
My best guess is that something is getting stuck. I know some older IPS systems I've worked with in the past had issues with CPU-pinning, and you might get garbage performance if your traffic hit the wrong CPU core. I didn't spend a lot of time digging into this, but it feels a bit similar.
Test 5: Wireguard VPN
Avg: 800Mb/s, 650-700Mb/s with IPS also enabled
I also wanted to try VPN performance. There are a lot of options for VPN services within OPNsense, including standard IPSec and OpenVPN. However, I opted to try out wireguard - since it's really easy to set up & get running.
In this case, I definitely hit a limitation with CPU-pinning. With my initial tests, I could only reach about ~450-500Mb/s - but the Qotom CPU was only spiking up to 70%.
Seems like the wireguard client (at least for same source/destination/port traffic) does pin everything to a single CPU. So my client CPU, between encrypting the traffic & generating it, was actually maxing out long before the Qotom was.
I ended up spinning up a second VM on the client side (which required disabling PCI passthough & adding both VMs to a shared vSwitch). With both of these running, I was able to max out the Qotom CPU at 100% - and hit a fairly consistent 800Mb/s VPN throughput.
As a final test, I enabled the suricata IPS on top of the VPN as well. Now the traffic would need to be decrypted & inspected before reaching the server. With a single client, I averaged ~400-450Mb/s - and with both clients it was around ~650-700Mb/s
Overall I'm really pleased with how well this thing performs, especially for how relatively inexpensive it was. I also expected OPNsense to have a steep learning curve, but it was surprisingly easy to get up and running very quickly.
Next I'll be working on getting this firewall up & running at home. I may be tempted to write up some more on OPNsense - so if there are any questions or specific configurations you might like to see, please leave a comment!