With a lot of the recent nonsense going on after VMware was acquired last year, I’m seeing people start to look for ESX alternatives - especially for home labs. In fact, I also just migrated my home lab from ESX to Proxmox.

One of the things I needed to keep running was Cisco Modeling Labs (CML). I often use CML to build quick virtual topologies for testing things, so it was important for me that it worked on Proxmox.

It’s worth noting that Proxmox isn’t an officially supported platform. The docs state that only bare metal & VMware platforms are supported.

That being said, I’m happy to say that Proxmox has worked just fine for me - it just needed a few tweaks. Though just be warned that you may still run into issues and it is technically an unsupported configuration.

In the post below, I’ll share the steps that I used to get CML up & running on Proxmox.

Prerequisites

To start with, we’ll need a few things:

• A server running Proxmox
  • I’m currently using version 8.1.3
• A CML install ISO, found here
  • We’ll be using version 2.6.1 for this guide
• CML’s reference platform ISO, also found at the link above

Our Proxmox machine will still need enough resources to build a VM with the minimum requirements listed here.

The CML & refplat ISOs should be placed into a Proxmox ISO storage location.

Creating the VM

Next we can get to the fun part: Setting up the VM in Proxmox.

We’ll first locate a node in Proxmox to place our VM. Then right-click that node & select Create VM:

Then, we’ll give our VM a name & ID. I’ve named mine CML:

On the next page, we can select our CML ISO from the correct storage device. We can leave the Guest OS as Linux and 6.x - 2.6 Kernel:

Next, we’ll need to make some minor adjustments on the System tab.

The default BIOS will likely be set to Default (SeaBIOS). We’ll change that to OVMF (UEFI), which will also give us a few additional options.

We’ll keep Add EFI Disk checked, and select a storage location for that EFI disk.

On the Disks tab, feel free to increase the size of the VM disk. While 32GB is the minimum required, you’ll likely want more than that. I’ve increased mine to 50G for now. Keep the image format as QEMU.

We’ll also want to update the Async IO setting to native. This is hidden under the Advanced settings:

On the CPU tab, we’ll update the core count to a minimum of 4 per the requirements. Of course, you’re welcome to increase this as needed.

More importantly however, we’ll need to set the CPU type to host. This allows the VM to use the underlying nested virtualization features:

Next we can assign memory to our VM.

The requirements state a minimum of 8GB, however this will likely only accommodate simpler labs. Some individual CML nodes require more than that to start.

In my case, I’ll start with 32GB - and we can always increase this later:

Lastly, on the Network tab, no changes are necessary. Of course, you can assign a VLAN if needed.

After that, we can head over to the Confirm tab & finish up the wizard.

Add Refplat ISO

Next, we’ll need to make sure our reference platform ISO is connected to the VM.

We’ll head over to the VM’s hardware tab, then click Add and select CD/DVD Drive:

Then we’ll select the appropriate ISO storage, and pick our refplat ISO file:

Install CML

Once that’s done, we can go ahead and power on our VM & pop open the console!

If you’ve installed CML previously on another platform, this process is just the same.

At boot, we’ll select Install CML.

After a moment & a few screens, we’ll start the system configuration.

First, we’ll input a system hostname:

Then, we’ll input credentials for the system management user.

Note: This user isn’t used to log into the CML software. Rather, it’s used for system administration tasks via the Cockpit UI, such as: Installing updates, checking logs, joining a domain, or powering off / rebooting the system.

After that, we can configure our CML admin user:

Next we have our network config. By default, CML will prompt to use DHCP, but we’ll likely want to change this to a static IP assignment:

After that, CML will prompt us to confirm our settings - then it will begin the installation. Since a lot of the VM images come from the reference platform ISO, this process may take a while as each of those images are copied over to the system.

When CML is ready, we’ll see something similar on the console telling us how to reach the web UI:

Again for reference, the main UI is reachable at https://<CML IP> and the Cockpit management UI is at https://<CML IP>:9090.

At this point, we can log in & start building labs!

The process for setting up CML on Proxmox isn’t super crazy, but there are just a few tweaks that need to be made during setup. I wrote this hoping that if anyone else out there is trying to set this up, maybe it would help.

As always, Thanks for reading!

Okay, so this post is ever-so-slightly off-topic from what I usually write about here - but I’m allowed to have fun sometimes, right?

With the holidays coming up, I’m getting ready to take some time off & relax with my favorite video games.

And just in time, Satisfactory just released their new Update 5 content - which finally adds support for dedicated servers. So I’ll be able to more easily play with friends & family who are in different time zones.

This will be a quick guide, mostly derived from the Satisfactory Wiki with some additional steps/context.

Disclaimer: The dedicated server software is pretty new & still a work in progress, so I imagine there is a good chance some of these steps may change. I’ll do my best to keep it updated!

A Quick Note

Before getting started, just a quick note about what is used for this setup & what assumptions are made.

This guide will walk through building a new dedicated server on Debian 11. There is additional information regarding other Linux distributions & Windows in the Satisfactory Wiki page linked above.

This guide assumes that you have some fairly basic level of Linux knowledge. You should be comfortable with connecting to a Linux server via SSH and editing files. If you need additional guidance, please consider checking out the video above where I walk through the whole process!

This guide will not look at installing Linux, or setting up port forwarding, etc. We will only be focusing on getting the dedicated server software installed & running.

Setting up Static Networking

By default my Debian 11 install was configured with a dynamic / DHCP address. While this might work for other use cases, we’ll want to set a static IP address for our game server.

So first we’ll open up our network config file, using the following command: sudo nano /etc/network/interfaces

Then, we’ll update our network config to look like the one below:

auto ens192
iface ens192 inet static
 address 192.168.1.20
 netmask 255.255.255.0
 gateway 192.168.1.1
 dns-nameservers 8.8.8.8

Please note, the values listed above are for example only. You’ll need to assign the correct addressing from your own local network. If you’re unsure about the interface name (mine is ens192), you can use the command ip address to list current network interfaces & find yours.

Once we’ve edited our network configuration, we’ll need to restart the network process for the changes to take effect: sudo systemctl restart networking

Note: Assuming the static IP you configure is different than the current dynamically-assigned IP, you will lose connection with your server & need to reconnect using the new IP address.

Install Steamcmd

Next, we’ll need to install the Steam commandline utility so we can download & manage our server.

On Debian, the default repositories don’t include this package. So we’ll need to add a third-party package repository.

We’ll edit our active/configured repositories with the following command: sudo nano /etc/apt/sources.list

Then add the following at the end of the file:

deb http://mirrors.linode.com/debian bullseye main non-free
deb-src http://mirrors.linode.com/debian bullseye main non-free

We’ll also need to enable multiarchitecture support: sudo dpkg --add-architecture i386

Once that’s done, we’ll update our local cache to include the newly available list of packages: sudo apt-get update

Then we can install steamcmd: sudo apt-get install steamcmd

We will likely want to create a dedicated user to run our game server. We’ll create a user called steam using the following command: sudo useradd -m steam

Then we can switch to our steam user account: su steam

Optionally, we can create a shortcut to the steamcmd executable within the steam user’s home directory: ln -s /usr/games/steamcmd steamcmd

Download / Install Satisfactory Dedicated Server

Now we can pull down a copy of the server software.

Assuming you’ve created the steamcmd shortcut as shown above, we can use the following command to pull down the server software: ./steamcmd +force_install_dir ~/satisfactory +login anonymous +app_update 1690800 validate +quit

This command does a few things:

• +force_install_dir - Tells steamcmd which directory to place the game files in, in this case ~/satisfactory
• +login anonymous - Skip logging in with a steam user account, since we just need to download an update
• +app_update 1690800 - This is the Satisfactory server application ID, so steamcmd knows which software to install
• validate - Runs a validation to ensure that the current files match what’s hosted on Steam
• +quit - Asks steamcmd to log off of Steam when everything is finished.

If we wanted to use the experimental branch, we would also add the flag: -beta experimental

Once that command finishes running, the Satisfactory software should be downloaded!

We can try to test this out by running the software manually first. Using the following command to move into the game server folder, and execute the software: cd ~/satisfactory && ./FactoryServer.sh

Assuming it runs, we can press Ctrl-C at any point to stop the game server.

Registering a Background Service

Okay, so we were able to run the server software manually for a quick test - but we’ll want to configure it to run as a background service that will start & stop with our Linux system. We’ll use the info located on the Satisfactory Wiki.

So we’ll create a new service definition using the following command: sudo nano /etc/systemd/system/satisfactory.service

Then paste the following into that file:

[Unit]
Description=Satisfactory dedicated server
Wants=network-online.target
After=syslog.target network.target nss-lookup.target network-online.target

[Service]
Environment="LD_LIBRARY_PATH=./linux64"
ExecStartPre=/usr/games/steamcmd +force_install_dir "/home/steam/SatisfactoryDedicatedServer" +login anonymous +app_update 1690800 validate +quit
ExecStart=/home/steam/SatisfactoryDedicatedServer/FactoryServer.sh
User=steam
Group=steam
StandardOutput=journal
Restart=on-failure
WorkingDirectory=/home/steam/SatisfactoryDedicatedServer

[Install]
WantedBy=multi-user.target

Please note, if you used a different user account name (instead of steam), or downloaded the game in a different directory - you will need to modify those values in the above text.

Once that’s done, we can enable our new service to start & stop with the machine: sudo systemctl enable satisfactory.service

Then we can start the game server: sudo systemctl start satisfactory.service

After a few seconds, we can check the service status here: sudo systemctl status satisfactory.service

If everything looks good, you should see output similar to below:

● satisfactory.service - Satisfactory dedicated server
     Loaded: loaded (/etc/systemd/system/satisfactory.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2021-12-01 11:45:31 EST; 18s ago
    Process: 3679 ExecStartPre=/usr/games/steamcmd +force_install_dir /home/steam/SatisfactoryDedicatedServer +login anonymous +app_update 1690800 validate +quit (code=exited, status=0/SUCCESS)
   Main PID: 3749 (FactoryServer.s)
      Tasks: 21 (limit: 7087)
     Memory: 1.0G
        CPU: 18.526s
     CGroup: /system.slice/satisfactory.service
             ├─3749 /bin/sh /home/steam/SatisfactoryDedicatedServer/FactoryServer.sh
             └─3756 /home/steam/SatisfactoryDedicatedServer/Engine/Binaries/Linux/UE4Server-Linux-Shipping FactoryGame

Dec 01 11:45:33 Satisfactory FactoryServer.sh[3756]: [2021.12.01-16.45.33:178][  0]LogAIModule: Creating AISystem for world DedicatedserverEntry
Dec 01 11:45:33 Satisfactory FactoryServer.sh[3756]: [2021.12.01-16.45.33:178][  0]LogLoad: Game class is 'BP_GameModeMenu_C'
Dec 01 11:45:33 Satisfactory FactoryServer.sh[3756]: [2021.12.01-16.45.33:179][  0]LogReplicationGraph: Display: SetActorDiscoveryBudget set to 20 kBps (5333 bits per network tick).
Dec 01 11:45:33 Satisfactory FactoryServer.sh[3756]: [2021.12.01-16.45.33:205][  0]LogNetCore: DDoS detection status: detection enabled: 0 analytics enabled: 0
Dec 01 11:45:33 Satisfactory FactoryServer.sh[3756]: [2021.12.01-16.45.33:205][  0]LogInit: BSD IPv4/6: Socket queue. Rx: 262144 (config 131072) Tx: 262144 (config 131072)
Dec 01 11:45:33 Satisfactory FactoryServer.sh[3756]: [2021.12.01-16.45.33:205][  0]LogNet: Created socket for bind address: :: on port 7777
Dec 01 11:45:33 Satisfactory FactoryServer.sh[3756]: [2021.12.01-16.45.33:205][  0]PacketHandlerLog: Loaded PacketHandler component: DTLSHandlerComponent ()
Dec 01 11:45:33 Satisfactory FactoryServer.sh[3756]: [2021.12.01-16.45.33:205][  0]PacketHandlerLog: Loaded PacketHandler component: Engine.EngineHandlerComponentFactory (StatelessConnectHandlerComponent)
Dec 01 11:45:33 Satisfactory FactoryServer.sh[3756]: [2021.12.01-16.45.33:205][  0]LogNet: GameNetDriver EOSNetDriver_2147482561 IpNetDriver listening on port 7777
Dec 01 11:45:33 Satisfactory FactoryServer.sh[3756]: [2021.12.01-16.45.33:205][  0]LogWorld: Bringing World /Game/FactoryGame/Map/DedicatedserverEntry.DedicatedserverEntry up for play (max tick rate 30) at 2021.12.01-11.45.33

Joining the Server

We’re almost there!

After the game server has been set up & running, we’ll need to perform the initial setup within the game itself.

So it’s time to launch Satisfactory & setup our game server!

In the screenshot above, the title screen now has an option for Server Manager. We’ll click that. You should then see an option to add server, which will display the following dialog:

Here’s where we put in the IP or hostname of our local dedicated server.

As a reminder: If you have someone outside of your local network, they’ll need a different address to connect. Again, this post won’t dive into port forwarding or allowing access through your home router - there are plenty of good tutorials out there already!

After we successfully connect to our server, we’ll be prompted to claim it. Since it’s newly created, this is where we can set our admin login & the server name.

First we’ll be asked to provide a name to claim the server:

Then an administrator password - don’t forget to save this somewhere safe!

Okay. So now we’ll have a few options. First we’ll check out the settings page:

Here we can modify our server name or update our admin password if we need to.

We can also set a Player password - This will keep people from joining your server unless they know the password.

Interestingly enough, we can opt to have the game pause when no players are connected - or uncheck the box to let your factory continue building even when no one is playing.

And lastly an option to autosave whenever someone disconnects from the server.

If we’re good with those settings - we can start a new game session!

We’ll head over to the Create Game tab:

This should look pretty similar to the normal game. Select the starting area & give the session a name.

We’ll also have a checkbox to automatically enter the game after it’s done being created. If you don’t check this box, you can enter the server from the Server Status page:

Here we can get a quick view of the current server status - which tier we’re on, the current milestone, and whether or not anyone is currently connected.

Down in the bottom left, we’ll have our Join Game button.

Time to play!

Okay - and that about wraps it up. I hope this post was helpful for anyone else looking to build a dedicated server. The process was surprisingly easy, and I greatly appreciate the effort that the Coffee Stain devs have put into this!!

Happy building!

What is NFVIS?

NFVIS is an operating system developed by Cisco which is intended to be deployed at branch office locations - and allow for quick deployment of network services in lightweight VMs. For example, we might want to reduce cost and hardware footprint by deploying a single ENCS machine, then deploy our typical branch services on top of that (DNS, Firewalls, SDWAN, etc). Under the hood, NFVIS is built on top of CentOS and KVM.

In the image below, we have an ENCS unit that is running ISRv, FTDv, and a vEdge Cloud. NFVIS has the ability to build out traffic flows for service chaining. In this particular setup, we could have all branch traffic receive a default route up to our ISRv. The ISRv forwards traffic to a Firepower VM (FTDv) which performs some traffic inspection before passing everything up to the vEdge Cloud.

We’ll be coming back to this diagram later to see how we can build out this flow of services. For now, let’s dive into how we can get NFVIS up and running.

Installing NFVIS

Lucky for us - the installation of NFVIS is fairly straightforward!

1. Create a bootable USB - or mount the installation ISO via CIMC
2. Upon boot, select “Install Cisco NFV Infrastructure Software”:

3. Wait. A while. Install time can vary depending on your hardware.
4. Once completed, log into the CLI: Default login = admin/Admin123#
5. You’ll be prompted to change the default admin password immediately:

Install completed! Now let’s look at some of our base configuration..

Initial Configuration

By default the NFVIS install will have a LAN and WAN bridge (lan-br and wan-br, respectively). The LAN config will be set up with a static IP of 192.168.1.1/24, and the WAN will be set for DHCP. We can check the current network settings by running the show system settingscommand:

In this case, my WAN interface is able to get an IP via DHCP. We’re likely going to want to change this to a static IP address - which we can do from the CLI or web interface. Let’s start by trying this from the CLI:

config t
system settings wan ip address <ip addr> <netmask>
system settings default-gw <gateway addr>
system settings hostname <hostname>

Changes can then be applied using the commit command

We can verify these settings by repeating the show system settings command we used earlier.

Let’s go ahead and log into the web interface to see what the network configuration looks like there:

1. If we know our WAN or LAN IP from earlier, we can just pop that in our web browser.
2. Go ahead and log in using the new admin credentials we just created:

3. We’ll be taken to the primary NFVIS dashboard, which will currently show no active VMs deployed:

4. In the left-hand menu, expand Hostthen click on Settings:

Here we can see that we already configured our IP Addressing/hostname - but we could use the Edit button at the bottom to change any of these values. For example - I’m going to go ahead and select Static for both the Management (LAN) and WAN IP addresses.

Another quick tip - if we need to modify which physical network adapters are tied to an internal network bridge, we can find that under VM Life Cycle > Networking:

That’s all for this time. In the next post, we’ll take a look at how to package VM images and deploy our service chain.

The config I used is based on this blog post. It’s a great write-up on how to back up F5 configurations using CatTools. I don’t want to replicate what was written over there - but I did hit some issues that were specific to my use-case that I wanted to share.

While I was happy to find the article linked above, the immediate results didn’t work so smooth for me. This may be due to some key configuration differences that I face in my network:

• All the F5’s are LDAP integrated - so there isn’t an easy way to provide LDAP users with direct bash access
• All of my F5’s are remote appliances, where the backup configuration is being copied across the WAN

Getting around the first problem was my biggest challenge. CatTools is a very command/response-oriented application. Any remote LDAP authenticated users are immediately dropped into F5’s shell: tmsh. To get from there to their ‘advanced shell’ is as simple as typing bash. However, When the terminal prompt changes, it often throws CatTools into a state of “I didn’t receive the response prompt I expected, therefore kill the job - something went wrong”. I spent a bit more time on this than I wanted to - but the underlying problem was that the “F5.BigIP” device type was specifically looking for the tmsh shell and couldn’t handle the prompt change. The fix? Switch the device type to “Linux.RedHat.Bash”, then add the bash command to the first line of your backup script.

The next problem was using TFTP to copy the backup archives over the WAN. Even some of the new F5’s with minimal configuration still generate a 10Mb file. Doesn’t seem like much, but when you’re copying that over a WAN between two datacenters, that turns into a ~5 minute file transfer. CatTools by default will only wait 30 seconds after executing a command before it expects a response. So every time I tried to run the job, CatTools would kill it only 30-seconds into the file transfer. Luckily enough, they support a utility command that can alter the normal timeout:

%ctUM: Timeout 600
tftp -m binary 192.168.1.10 -c put $filename
%ctUM: Timeout 0

The command %ctUM: Timeout 600 changes the timeout value to 600 seconds, or 10 minutes. The TFTP file transfer command is next, which is now permitted up to 10 minutes to finish. The last command resets the timeout back to the default (30 seconds).

I also realized that the original script doesn’t purge the backup archive afterwards. For my use case, I would much rather automatically clean up the backup files once they’ve been transferred to a central location.

So after all that, here is the version of that script that I’m using:

export date=`date +"%y%m%d"​`
export filename=$HOSTNAME.$date.ucs
tmsh save /sys ucs /var/local/ucs/$filename
cd /var/local/ucs
%ctUM: Timeout 600
tftp -m binary 192.168.1.10 -c put $filename
%ctUM: Timeout 0
rm -f $filename

Thanks again to the original blog post for getting me on the right track with this! I hope that my ramblings here are helpful to anyone with a similar deployment scenario.

I’ve been considering the idea of posting some short reviews of products or services I use. Not at all meant to make this a review site, but just a bit of “here is what I use and what I think of it”. This is going to be the first shot at that idea - so let me know what you think!

When I was first looking to start up my own personal blog again, I did quite a bit of digging around to try and find where to host it. I considered hosting it myself at home, but overall I didn’t really want to manage that overhead. I also looked at a number of hosting providers, but I never really found anything that I was quite happy with. So many of them would offer you an instance of a web platform like WordPress or Drupal, but not much control beyond that. I really wanted something where I could have full control over my own VM, like a VPS, but most of these services cost more than I wanted to pay.

Right in the middle of my research for a hosting provider last year, Amazon announced a new AWS service offering: LightSail. I had already briefly considered picking up a small AWS VM for what I wanted to do, but I honestly didn’t want to try and manage usage when every single little thing is an additional cost (storage, bandwidth, etc). However, LightSail really caught my attention because of the stupid-simple pricing structure - the smallest allocation is only $5 a month and includes 512MB of RAM, 1 core CPU, 20GB SSD, and 1TB of free data transfer. This was probably more than enough for me to get started with a site, and the pricing was very tempting. As if they wanted to provoke me to try it even further, Amazon actually offers your first month free (for the $5/mo instance). Additionally, Amazon offers up to three free public IP addresses and free DNS management along with every plan - which I thought made this a fantastic offer.

With all that LightSail seemed to offer, I decided to jump on the $5/mo plan and use the free trial period to give it a shot. The setup of my first VM was extremely simple and only a few clicks: Pick a size, give it a name, and select which applications you want pre-deployed (optional). I went ahead and selected my options, and within seconds my VM was already powered on, pre-configured, and ready to use. Unfortunately, it seemed like my VM deployment didn’t quite go as smoothly as I would have liked. I wasn’t able to log into the admin console for my pre-deployed web application, and the deployment guide didn’t seem to match up with what I was seeing on my VM. I spent about ten minutes or so trying to troubleshoot this before it hit me: I could just delete and re-deploy this VM in seconds. So I did exactly that, and within 2 minutes I already had another VM instance deployed - and this one worked perfectly.

Next I went ahead and applied a static IP to my instance. You don’t necessarily have to do this, and if you host your DNS within LightSail then it will automatically update your DNS any time your instance IP changes. However, since they offer up to three static IP addresses for free, I don’t really see a reason why you wouldn’t use it - unless the VM was temporary or not being published publicly. This process is also extremely straightforward: Click the button to create a new static IP, give it a name, and select which VM to assign it to. Easy enough, right? I also tried out the DNS hosting for only a very brief period of time. I ultimately ended up opting to move my DNS hosting out to CloudFlare, which I would like to cover in another post.

Once the LightSail VM is up and running, it’s easy enough to connect to it. They offer a web-based SSH console, and by default allow access to ports 443 and 22 for remote connections. Remote SSH connections are handled by public/private key pairs only - no username/passwords permitted for login. I actually prefer this, and there is an extremely simplistic interface for generating new or additional SSH keys and automatically configuring them within your VM. The management console also offers an easy-to-use firewall system, where you can open ports for common services via a drop-down menu. You’re also able to enter custom port numbers, or remove any/all open ports entirely. As a quick note: LightSail offers no traditional console access to your VM - so if you close port 22, then you won’t even be able to manage the VM from the web console since it uses SSH. For me, I would rather take the additional security step to only enable that port when I absolutely need to access the VM via SSH.

So it’s been almost a year since I started using Lightsail and overall I am extremely pleased. I’ve run one primary VM since I opened my account, and I’ve spun up a few additional VMs here and there for different testing. It’s easy enough to just turn up a new VM, try it out, and then just purge it if/when you don’t need it any more. The billing reports are very straightforward too - and I’ve so far never come close to using all of my 1TB free data transfer. The VM itself is extremely snappy for only a single core with 512MB of RAM.

Overall I would highly recommend giving LightSail a try, even if you only use it for the 1 month free trial. I’ve been very happy with the service so far, and I’m looking forward to any new features/functionality that might be added in the future.

As a quick summary:

Benefits:

• Extremely easy to use (Simplistic interface)
• Three free static IPs
• Free DNS management
• Up to 1TB of free data tranfer
• First month of a $5 instance is free
• Pre-deployment/configuration of multiple different web applications

Drawbacks:

• You can’t move which zone your VM is deployed in
• You can’t rename your VM instance
• In order to upgrade plans, you have to take a snapshot and restore it to a new VM - and this is currently only available via the AWS APIs, so no way to do this in the web portal
• Snapshots take forever, and you get charged for how long they are stored for (I’ve had a snapshot of a 20G VM take >30 minutes in the past)

My wishlist:

• An integrated load balancer - even one with very basic options available
• Native support for IPv6 addresses - Amazon already offers a free IPv4 address, so why not v6 too?
• More storage options - I am careful about how many images i upload to my site, because I only have 20GB of space available. I don’t want to pay for S3 if I don’t have to.

UPDATE 12/19/2017- As of the end of November, Amazon has added both Load Balancing and additional storage to LightSail. Currently the load balancing functionality costs  $18 per month, and additional disk space is $0.10 per GB per month.

Have you used LightSail before? What are your thoughts? Comment below!

• Part 1 - Initial Thoughts
• Part 2 - Leaning Django
• Part 3 - Creating the Dashboard

Alright - We’re finally at the last step to this project. We already have Django running with a somewhat decent-looking web dashboard. Now the only thing that remains is polling our SRX firewalls for VPN connections, then populating this information into the database.

For this to happen, I built a custom management command for Django - the cool thing here is that we can easily use this within a cron job to automatically schedule updates at regular intervals. So within our application (the vpn folder), we’re going to create a management folder then a commands folder within that. In here you can name your script whatever you want, so I went with cronpoller.py.

The script that I wrote is extremely simplistic and could probably use quite a number of improvements - which I will slowly make over time. But for now, it is what it is - and it does exactly what I need it to.

First thing we need to do is import a few things we will need:

from django.core.management.base import BaseCommand, CommandError
from vpn.models import Firewall, Datacenter
from jnpr.junos import Device
from lxml import etree

We have some Django modules for treating this as a management command - and we’re also importing our existing models into this script. This part is really cool to me, because it means that this cron script can just reference the existing objects that we created to get their attributes. Next, we import the JunOS stuff from their pyEZ packages (don’t forget to install pyEZ!). The last one is going to be used to parse the responses from our SRX into something we can use.

Alright - so in order to build a management command, we have to create a class called Command, then define our functions within that. Within the Command class, Django will look for a function called handle to execute. In my final script, I have that function plus one called getVPNStatus. Let’s start with putting together getVPNStatus:

# This function will poll the device for status 
def getVPNStatus(self, fw, dc):
    connectedlist = []
    # So we generate a JunOS pyEZ device connection using the information about the 
    # firewall that we gather from the database object
    device = Device(fw.firewall_manageip,
                    user=fw.firewall_user,
                    password=fw.firewall_pass)
    # Try to open a connection out to the target device
    try:
        dev.open()
    except:
        # If for some reason this doesn't work, just return UNREACHABLE - which
        # we'll assume means the device is down
        return "UNREACHABLE"
 
    # Here is where we poll the SRX for a list of all IPSec Security Associations.
    # The equivalent of the 'show security ipsec sa' command
    response = etree.tostring(dev.rpc.get_security_associations_information())
    # The SRX returns a response in XML, which we'll need to dig through
    # Credit to the guys over at <a href="http://packetpushers.net/parsing-junos-xml-python/" target="_blank" rel="noopener">Packet Pushers</a> for a great post explaining how to
    # parse these responses
    with open(response) as a:
        xmldoc = etree.parse(a)
        docroot = xmldoc.getroot()
        rootchildren = docroot.iter()
        for child in rootchildren:
            # For each IPSec SA returned, we need to find the remote gateway IP, which 
            # we use to tie the connection back to the connected datacenter
            if child.tag == "sa-remote-gateway":
                connectedlist.append(child.text)
   
    # Once we've built our list, send it back!
    return connectedlist

That already is major part of our script. It will connect out to each device, grab a list of every connected IPSec tunnel, then return back a list of connected gateways. Now we just need to write our handle function, which will do all of the remaining work.

def handle(self, *args, **options):
    # Let's quickly create two lists - based on our firewall and datacenter models
    # This actually polls the database for anything that's been created
    dclist = Datacenter.objects.order_by('datacenter_code')
    fwlist = Firewall.objects.order_by('firewall_name')
 
    # This will loop through each firewall, then call the getVPNStatus function
    for fw in Firewall.objects.order_by('firewall_name'):
        statuslist = self.getVPNStatus(fw, dclist)


    # Once we have our response, we're going to check the returned list of connected
    # gateways against our list of datacenters from the database
    for dc in dclist:
        for remoteFW in fwlist:
            # If we find another datacenter in the connected gateway list, add 
            # that datacenter to the vpnstatus list as "VPNUP", otherwise assume it's down
            if remoteFW.firewall_vpnip in statuslist:
                vpnstatus[dc.datacenter_code] = "VPNUP"
                break
            else:
                vpnstatus[dc.datacenter_code] = "VPNDOWN"
 
    # After all that is done - we just save the new vpnstatus list to the firewall_vpnstatus
    # field in the database
    fw.firewall_vpnstatus = vpnstatus
    fw.save(update_fields=["firewall_vpnstatus"])

I’ve said this before, but I think it’s worth noting again - This probably isn’t the best or most efficient/reliable way of doing this. I think that over time I would like to revisit this and refine it a bit - but this is what I’ve put together so far. In fact, I think it’s worth stating that this will probably break in a fantastically horrific manner. This isn’t a finished product, but pretty much just version 0.1 - the base functionality works, but it’s in desperate need of refinement.

Alright - so now we just have to add a cron job in our system to call this script and run it. Depending on how many firewalls you have and how the latency is, you might need a different interval than I am using - but I went ahead and set mine to run once every 10 minutes. I would like to get this down to 5 minutes or less, but I might need to figure out a way to potentially multi-thread the cronpoller.py script. So here is what I did in /etc/cron.d/vpnstatuspoller:

*/10  * * * * * python /root/junos-dashboard/manage.py cronpoller

After letting the script poll my firewalls - I ended up with a dashboard that looks like this:

Future improvements

Of course, I still have a few things I would like to add or improve on. I’ve been keeping a list of some ideas that I’ve had, which I’ll share here:

• Add a timestamp to each firewall that contains the last poll time (in case something gets missed or hasn’t updated yet)
• Possibly add ability to click a ‘down’ VPN cell to force clear SA
• Set up email alerts from the cronpoller to automatically notify me of a failure
• Ability to click on any VPN cell and get additional info - like VPN uptime, subnets routed across it, or maybe the IKE/IPSec negotiation parameters
• Ability to click on any firewall name and get additional info - like uptime, JunOS version, CPU/Memory

I finally got around to getting myself a GitHub account, so I might put the final code up there once I’m done. I also have a number of other JunOS scripts that are probably worth posting up there as well.

Well, I hope you enjoyed reading about this project - because I certainly enjoyed working on it. This was one of my first real projects using the JunOS pyEZ libraries, and I got to pick up and learn how to use Django as well. The experience of building this has given me ideas for other JunOS automation projects - so look out for some of those in the future!

• Part 1 - Initial Thoughts
• Part 2 - Leaning Django

In the last post, we got Django working well enough to start serving up web requests. That’s a great start - So now it’s on to actually building the project. We’re going to cover this in two parts: the front-end dashboard, and the back-end scripts. In this post, I’m going to show you how I began putting everything together to assemble the front-end HTML dashboard.

As a quick refresher note, we created a project in the last post called junos-dashboard. Within that project, we created our app named vpn. This folder is primarily where we’ll be spending our time today. There are two files in our app folder that will need a few edits:

views.py - This is where we originally created our basic index page, which currently just spits out a message. So we’ll need to add code in here to render our dashboard instead.

models.py - In here we will define our objects and the characteristics about those objects that we want to store/retrieve in the sqlite database.

Let’s go ahead and open up our models.py file. For my dashboard, I have two objects I want to track - Firewalls (the SRX hosting the VPN) and Datacenter (as a location tracking). So in order to create those objects, we just have to add two classes into the models.py file:

from __future__ import unicode_literals
from django.db import models

class Datacenter(models.Model):
    def __str__(self):
        return self.datacenter_code

class Firewall(models.Model):
    def __str__(self):
        return self.firewall_name

Easy enough, right? So here is the really cool thing about Django (or probably any web framework really) - These model objects are now something that we can reference in our code to change or query attributes about them. Even cooler is that Django automatically builds an admin page to our site, where we can create new instances of each model right there (which I’ll cover later). So this means that all I have to do is define these two models and their attributes, then anyone on my team can log into the dashboard admin page and add new firewalls or datacenters.

Okay, so in order to actually make these models useful, we need to add our attributes. For the datacenter object, I only care about two things - What is the datacenter code (identifier), and is the datacenter an active location. So here is where we add those options under our datacenter class:

class Datacenter(models.Model):
    datacenter_code = models.CharField('Datacenter Code', max_length=10)
    datacenter_active = models.BooleanField('Datacenter Active?', default=True)

The firewall object gets a little more complicated because I want to be able to define quite a few things. The obvious attributes are: the firewall name, which datacenter it belongs to, and whether or not it is an active firewall. I’ll also need to collect this firewall’s VPN peer IP, so that I can compare it against other devices to check the connection. But wait - how am I actually checking that VPN status? Oh, I guess I’ll also need to collect a username/password for the SRX API, as well as a valid management IP to reach the API.

Let’s take a moment though - because I stopped here and realized that I’ll need to collect user/password data in a form, then it’s going to be stored in a sqlite database. Oh, and of course it will be unencrypted. I didn’t really like that idea - so I did some research on how to encrypt one of the object attributes. Turns out, it’s actually pretty easy since someone has already created a module to do exactly that.

So let’s grab that module real quick:

[root@0x2142-Centos vpn]# pip install django-encrypted-fields

Once we have the module, it requires us to generate some keys which will be used for encrypting our data. The following is based on the steps listed on the GitHub page for django-encrypted-fields:

[root@0x2142-Centos junos-dashboard]# mkdir fieldkeys
[root@0x2142-Centos junos-dashboard]# keyczart create --location=fieldkeys --purpose=crypt
[root@0x2142-Centos junos-dashboard]# keyczart addkey --location=fieldkeys --status=primary --size=256

Next, we just need to add this keyfile into our settings.py:

ENCRYPTED_FIELDS_KEYDIR = '/root/junos-dashboard/fieldkeys'

Alright - back to actually creating the attributes we need for our firewalls. Since we’re using this new module, we’ll need to add an additional import statement - otherwise, we are just adding the fields I covered earlier:

from encrypted_fields import EncryptedCharField

class Firewall(models.Model):
    firewall_name = models.CharField('Firewall Name', max_length=50)
    firewall_location = models.ForeignKey('Datacenter', on_delete=models.CASCADE)
    firewall_active = models.BooleanField('Firewall Active?', default=True)
    firewall_manageip = models.GenericIPAddressField('Management IP')
    firewall_vpnip = models.GenericIPAddressField('VPN Interface IP')
    firewall_user = models.CharField('API User', max_length=50, blank=True)
    firewall_pass = EncryptedCharField('API Pass', max_length=50, blank=True)
    firewall_vpnstatus = models.TextField(editable=False, blank=True)

A few things to note here - The firewall_location attribute will actually query the table of datacenter objects - so we’re not creating any data twice. You might also notice that I added a firewall_vpnstatus text field, which is going to be hidden in the administrative console. Because I’m a terrible programmer, I’ll be storing all of the VPN status info in this text field in the database. Yes, there is probably a much more elegant way of handling this - but again, I’m a network admin not a professional coder. For what I need, this gets the job done. If you have a better way of accomplishing this - let me know! I would be very interested in another method.

I also found out later that just because I encrypted the field doesn’t mean that Django knows the field is a password. I wanted the field to be treated like a password input, so the data wasn’t visible to anyone who just logged in. I found out that you can create a forms.py file within the vpn directory, and pretty much override the field type to account for this.

So here is what I threw into the forms.py file to handle password input:

from django.forms import ModelForm, PasswordInput
from .models import Firewall

class FirewallForm(ModelForm):
    firewall_pass = CharField(widget=forms.PasswordInput)
    class Meta:
        model = Firewall
        fields = ['firewall_pass']

That was actually way easier than I had anticipated.

Awesome, now our work on the models is completely done - why don’t we log into the administrative  web interface and take a look at how it all turned out?

Here is a view of the datacenter page - where we can add a new location:

And now for the firewall objects - with all the fields we need to get the job done:

Looks pretty good, huh? And we didn’t even need to do any work to get the free admin functionality! This is honestly my favorite part of Django. Without posting a ton of additional screenshots, the admin page also provides audit logs - so we can see who changed what objects. Very helpful.

Alright - now let’s start work on our views.py file, so we can actually start putting all this together in our dashboard!

Within our views.py file, we already had our index function, which just returned a string of text. We’ll be adding to that in a moment - but first I created a separate function called buildrows, which does exactly what you think. This function will pull each firewall and it’s status, and build our little HTML table. I’m not going to go into great detail on this - but check out the comments for more information on what’s going on here:

# This is our function to build the HTML table - We're going to be expecting this function
# to be fed a list of firewalls and datacenters
def buildrows(firewall_list, datacenter_list):
    firewallstatus = []
    # We will iterate through each firewall and compile each row for our table
    for fw in firewall_list:
        # The first cell in our row is going to be the firewall name and it's own VPN peer IP
        onerow = "<td><b>%s</b><i>%s</i></td>" % (fw.firewall_name, fw.firewall_vpnip)
        # Now we iterate through every datacenter, in order to see which ones we have a
        # connection to from this firewall
        for dc in datacenter_list:
            # First thing is first - If this firewall contains the name of the datacenter
            # then we'll print N/A, since we wouldn't expect a VPN to itself 
            if str(dc.datacenter_code) in str(fw.firewall_name):
                onerow += ("<td>N/A</td>")
                continue
            # Also, if there is no VPN status in the database, then just print 'No Status'
            if not fw.firewall_vpnstatus:
                onerow += ("<td>No Status</td>")
                continue
     
            # This portion is pretty self-explanatory - We parse the vpnstatus field in the
            # database. If the state of the connection is 'VPNUP', then we print a cell for
            # that datacenter with the text 'UP'. If the state is 'VPNDOWN', then we print
            # 'DOWN'. And if nothing matches, print 'No Status'
            statuslist = ast.literal_eval(fw.firewall_vpnstatus)
            try:
                status = statuslist[dc.datacenter_code]
                if status == "VPNUP":
                    onerow += ("<td class=\"vpnup\">")
                    onerow += ("UP")
                elif status == "VPNDOWN":
                    onerow += ("<td class=\"vpndown\">")
                    onerow += ("DOWN")
                else:
                    onerow += ("<td>")
                    onerow += ("No Status")
                    onerow += ("</td>")
            except KeyError:
                onerow += ("<td>")
                onerow += ("No Status")
                onerow += ("</td>")
        # Once complete, we append this row to the status list
        firewallstatus.append(onerow)
    
    # All done! Return our list of statuses!
    return firewallstatus

That block might not make a ton of sense at the moment - but it will shortly. Essentially I have a script that is connecting to each firewall via the SRX API and polling the VPN status for each peer. The script is then writing all of this information to a field in the database called vpnstatus in one long string, which kinda looks like this:

{u'US-1': 'VPNUP', u'US-2': 'VPNUP', u'EU-1': 'VPNUP', u'EU-2': 'VPNUP', u'CN-1': 'VPNDOWN'}

The buildrows function is then just grabbing each firewall and then parsing this data to figure out what it has connections to - then generates a row in our table.

With that done, our index function within views.py needs a little work to start displaying our table. You’ll also notice we’re importing the models we created, as well as a template loader - which I’ll get to in a moment. So here is what that function will look like when we’re done:

from .models import Firewall, Datacenter
from django.template import loader
import sys, ast

def index(request):
    firewall_list = []
    datacenter_list = []
    firewall_status = []
    # This will grab each datacenter object (if they're marked active) from the 
    # database and add them to a list
    for each in Datacenter.objects.order_by('datacenter_code'):
        if each.datacenter_active == True: datacenter_list.append(each)
    # Same thing here - grab our firewalls and append to a list
    for each in Firewall.objects.order_by('firewall_name'):
        if each.firewall_active == True: firewall_list.append(each)
    # Now we pass those lists to the buildrows function to assemble our HTML table
    firewall_status = buildrows(firewall_list, datacenter_list)
    # We're also going to apply a template to make the page look fancy
    template = loader.get_template('web/index.html')
    # This is taking all of our lists, and passing them into our HTML template
    # so that it can build the page semi-automatically
    context = {
        'datacenter_list':datacenter_list,
        'firewall_list': firewall_list,
        'firewall_status':firewall_status,
    }
    return HttpResponse(template.render(context,request))

Alright - our views.py file is now complete. I wanted to keep the index function kind of simple, so we just have it pull data from the database, pass it to another function for processing, then return it to the browser for display.

So in the code above, you might have noticed that I was using an HTML template to make the page look fancier. I did this by going out to Google and finding a free CSS template for HTML tables. I won’t post the CSS here, but there are plenty of free templates out there - just find whatever suits your taste. Once you have that CSS file, create a directory (within our app folder) called static and place the CSS file in there.

Then, it’s one simple template HTML file to load our CSS and render our table. Within the app directory, I created a templates folder, then a folder called web within that. I added a new file called index.html - which looks like this:

<title>SRX VPN Dashboard</title>
<div align="center">
    {% load static %}
    <!-- Load the CSS template here -->
    <link rel="stylesheet" type="text/css" href="{% static 'style.css' %}" />
    <!-- Add a header to the table -->
    <div class="table-title">
        <h3>SRX VPN Dashboard</h3>
    </div>

    <!-- Create our first row, which will be a list of each data center location -->
    {% if firewall_list %}
        <table class="container">
        <tr>
        <th></th>
        {% for datacenter in datacenter_list %}
            <th>{{ datacenter.datacenter_code }}</th>
        {% endfor %}
        </tr>

        <!-- Then we add a row for each firewall, followed by its status 
             for each datacenter -->
        {% for firewall in firewall_status %}
            <tr>
            {{firewall|safe}}
            </tr>
        {%endfor%}

        </table>
        <!-- If something goes wrong, we just print an error -->
    {% else %}
        No firewalls were found.
    {% endif %}
</div>

Guess what? The dashboard is complete! Now we can run our Django server using ‘./manage.py runserver’ and take a look at what we have created.

Pretty simplistic - but we’re getting very close to what I wanted to accomplish. The dashboard loads and generates the table, but it doesn’t have any data yet.

In the next post - We’ll take a look at building the backend script to poll all of our SRX firewalls for VPN connections. I hope you enjoyed reading this - let me know what you think in the comments below!

This is a multi-part series - Check out the other posts:

• Part 1 - Initial Thoughts
• Part 2 - Leaning Django
• Part 4 - Polling the SRX

• Part 1 - Initial Thoughts

Once I figured out the overall idea behind what I wanted to do - the next big part was figuring out where to start. As I mentioned in part 1, I had never used Django before starting this project. So I figured that learning how to to use a new web framework might be the best place to begin.

The great thing, is that Django already has some great tutorials on how to get the packages installed and begin working on your first project. Their tutorials can be found here:

• Installing Django
• Writing your first app - Credit to this page for some parts of the examples below

I used both of these tutorials when I wrote my dashboard, and pretty much just modified what they were doing to fit my needs. After I got a baseline down, it was pretty easy to keep going and complete what I wanted to do.

Note: Just as a pure warning - I’m a network admin, not a professional programmer. Python, Django, and the Junos PyEZ libraries make it easy enough for me to make my ideas into a reality. However, that doesn’t mean I write perfect or even great code. (I learned Python from Learn Python The Hard Way - I highly recommend this site, as it was a great resource!)

Alright, with that out of the way - Let’s get started!

Step One - Installing Django

First thing is first, I learned Python on the 2.7.x chain, and I’ve been terrible about forcing myself to switch to 3.x. So for the purposes of this tutorial, everything I’ve written was intended for Python 2.7.5. Installing the actual Django package is a super simple task (if you use pip):

[root@0x2142-Centos ~]# pip install django
Collecting django
 Downloading Django-1.11.2-py2.py3-none-any.whl (6.9MB)
 100% |████████████████████████████████| 7.0MB 140kB/s
Collecting pytz (from django)
 Downloading pytz-2017.2-py2.py3-none-any.whl (484kB)
 100% |████████████████████████████████| 491kB 1.9MB/s
Installing collected packages: pytz, django
Successfully installed django-1.11.2 pytz-2017.2

Step Two - Get JunOS pyEZ libraries

I already wrote about this earlier in Getting Started with JunOS PyEZ. If you don’t already have the JunOS packages installed, check out that page for the instructions.

Step Three - Create the Project

Django is pretty wonderful from my brief experiences with it. They package a number of tools that make starting a new project very simple.

So to start off with our project, we’re going to use the django-admin tool:

[root@0x2142-Centos ~]# django-admin startproject junos-dashboard

This will create a base directory structure for us to begin our project. We’ll dive into what these files and folders are as we touch them - but for now, you’ll need to make one minor edit to ~/junos-dashboard/junos-dashboard/settings.py. Open that file up in your favorite text editor, and find the ALLOWED_HOSTS setting. Add in the IP address of your linux VM, like this:

ALLOWED_HOSTS = ['10.2.32.90']

Once that’s done, go to the root of your project (~/junos-dashboard/) and test out the web server:

[root@0x2142-Centos junos-dashboard]# ./manage.py runserver 10.2.32.90:8000

Django includes it’s own web server for development and testing, which is extremely helpful. Once you run that command, try to hit your page in a browser and just make sure it loads.

Once that’s done, make sure you’re in the root of your project - and we will create our dashboard application:

[root@0x2142-Centos junos-dashboard]# django-admin startapp vpn

This command just creates another directory structure within your project. I think this is helpful though, because then I don’t have to worry about which files I need to create or how they should be laid out - Django just handles that for us.

Step Four - Getting our web server to return our app page

Okay - So next we need to make a couple of minor edits in a few files. Within our app directory, there are two files (views.py and urls.py) that handle the main parts of our web interface. Note - urls.py doesn’t exist by default and you will have to create it!

Within views.py, we’re going to create our dashboard homepage with a quick test response:

# Import stuff
from django.shortcuts import render
from django.http import HttpResponse
from django.template import loader
import sys, ast

# This is our index page
def index(request):
    # Return some plain text - just so we know it worked!
    return HttpResponse("This page will eventually be a magical dashboard!")

This file is where we will eventually be putting in all the code for our dashboard page. When we hit our webserver, this file is executed to generate the view that we’ll see.

Next, we need to create a urls.py file within the same folder as views.py. This file is just a way to direct traffic. If we receive traffic to a regex of ^$, which is no path, then we are going to redirect it to the index function in views.py:

# Import stuff
from django.conf.urls import url
from . import views

#Redirect to index function in views.py
urlpatterns = [
 url(r'^$', views.index, name='index'),
]

Finally, we’re going to grab the urls.py file in our project folder - not the one we just edited within the app. So this should be ~/junos-dashboard/junos-dashboard/urls.py. In this file, we just need to tell the web server to include the urls.py file we created when evaluating traffic. This file should already exist and have content in it for routing traffic to the admin page, so just add an entry under urlpatterns - should look like this:

urlpatterns = [
 url(r'^$', include('vpn.urls')),
 url(r'^admin/', admin.site.urls),
]

I just want to point out that in this case, we’re catching traffic again with the ^$ regex. This is so that any traffic to the main page (in this case http://10.2.32.90:8000) is automatically sent to our vpn app. However, if you had multiple apps running, you might want to give each one it’s own directory.

Go ahead and execute the runserver tool again, and validate that you’re able to successfully hit the page and retrieve the test text we put in.

Okay, I think we’re going to stop here for this post. At this point we have Django installed and our project/app created. We also have the beginnings of our dashboard page. In the next post, we’ll actually start building out the dashboard interface!

This is a multi-part series - Check out the other posts:

• Part 1 - Initial Thoughts
• Part 3 - Creating the Dashboard
• Part 4 - Polling the SRX

One thing my current job has never had is a good way to view VPN tunnel status between our firewalls. Our Check Point firewalls don’t really provide a good high-level view, which has unfortunately caused some confusion around whether or not a site-to-site VPN tunnel is currently established. Luckily, over the past year I’ve had the opportunity to install over 20 new Juniper SRX firewalls - mostly made up of SRX 1500 and SRX 345 models. I’ve written a bit before about the JunOS APIs and pyEZ (their Python library), but I’m always really excited at a new use case for network automation.

So recently I decided that it would be great to build a web-based dashboard, which would query all of our SRX firewalls for currently connected VPN tunnels. Approximately 90% of the current tunnels are site-to-site between our own data center locations, with the other 10% being external to customers. My idea was that I would have a simple HTML table, which would show each data center along the top and bottom and whether or not each was connected to each other, kinda like this:

I wasn’t really concerned about making it look fancy - just dynamically updatable by whatever backend mechanism I used. Speaking of which, I had also assumed I would just be writing some simple Python script to pull VPN info from each device, then just write it to an HTML file. I haven’t really done much web stuff with Python in the past, so I set out to learn a bit and figure out what the best approach might be.

I ended up settling on trying out Django to write the frontend web stuff. I had never used it before, but I’ve been interested in trying - and what’s a better time to learn, than when you have something that needs to be accomplished? One thing that really pushed me towards Django for this project was the built-in administration page. This was huge for me, because it meant that I could easily have a way for other people to update the web dashboard. Whenever a new location came online, I wouldn’t have to go update the script directly - anyone on my team could log into the admin page and make the changes.

In this post I just wanted to get through what my ideas were behind this project. In the next few weeks, I’ll begin explaining how I built the dashboard using Django and use pyEZ to scrape VPN status from each firewall.

Thoughts? Drop a comment below!

This is a multi-part series - Check out the other posts:

• Part 2 - Leaning Django
• Part 3 - Creating the Dashboard
• Part 4 - Polling the SRX

As part of my home lab, I have an older Synology DS411 that I picked up in early 2012. I’ve been using the device since then with 4x 3TB drives as an iSCSI backend to my ESX host. Of course, I’ve also been using the NAS for general file storage (photos, videos, documents, etc). So in late 2014 I decided that I needed to find a good backup solution for it.

I started using CrashPlan, because they offer unlimited cloud storage for only $5.99 a month. This was amazing to me, because I had around 1.2TB of data at the time. I found a great community package here, which allowed me to install the backup client directly on the NAS. This ran great until late 2016 when CrashPlan updated to 4.8.0 and ended support for ARM processors (which is what the DS411 uses). For the past 6-7 months, I’ve been unable to back up my primary storage device at home. Since my DS411 is reaching the end of it’s life anyways, I figured I would just wait until I replaced it with one of the new Synology devices that use an Intel processor.

Well a few days ago, I received an email from CrashPlan threatening to delete my backups since my NAS hadn’t connected in over 6 months. It makes sense why they do that - but I didn’t want to lose my backups before I could replace the device. Re-seeding my backups whenever I picked up a replacement NAS would take forever (I think I have ~2TB backed up currently). So I finally forced myself to sit down and find an alternate solution.

Alright - so as of today I now have a dedicated CentOS VM running on my ESX host, which is connected to the Synology via NFS. This VM is running the CrashPlan client, and my backups have resumed! Here is how I got this all set up:

Synology Configuration

1. Enable NFS on the Synology

   • Open up the Control Panel and go to File Services
   • Scroll down to the NFS section, and check the box for Enable NFS
   • Click Apply

2. Apply NFS permissions to each share

   • Still under Control Panel, visit Shared Folders
   • For each folder you need to back up via CrashPlan:
     • Select the folder and click Edit
     • Click the tab for NFS Permissions, then click Create
     • Enter the IP address for your CentOS VM (or other linux system)
     • Set privilege to Read-Only (you could probably leave this read-write, but CrashPlan only needs read permissions to back up data)
     • Click OK

Linux VM Setup

1. Build a CentOS VM - This can be on an ESX host like I used, or just a standalone PC

   • Don’t forget to use the same IP address that we entered in the Synology for NFS

2. Install packages

   • Make sure you get the latest package updates first: yum -y update
   • Install NFS tools: yum -y installnfs-utils nfs-utils-lib

3. Test NFS

   • If you have the packages installed and NFS set up correctly on the Synology, then you should be able to validate the configuration by using showmount. For example, here are the directories that I was using CrashPlan to backup:

   [root@SynologyBackupVM~]# showmount -e 10.12.32.2
   Export list for 10.12.32.2:
   /volume1/backups   10.12.32.209
   /volume1/documents 10.12.32.209
   /volume1/homes     10.12.32.209
   /volume1/photos    10.12.32.209
   

   • Make your local Linux directory structure, which shouldmatch what the Synology structure is. So in my case, I made new directories on my local Linux VM for /volume1/backups, /volume1/documents, etc
   • Edit /etc/fstab to auto-mount your NFS shares on boot. In my case, I added the following lines:

   # NFS Share location          Local Folder
   10.12.32.2:/volume1/backups   /volume1/backups   nfs defaults 0 0
   10.12.32.2:/volume1/documents /volume1/documents nfs defaults 0 0
   10.12.32.2:/volume1/homes     /volume1/homes     nfs defaults 0 0
   10.12.32.2:/volume1/photo     /volume1/photo     nfs defaults 0 0
   

   • Reboot, and check to make sure each NFS share was actually mounted. If you do a ls /volume1/backups, do yousee all the files on the NAS in that folder?
   • If everything works, then grab the CrashPlan Client installer (4.8.2 was the latest at the time)
     • wget https://download.code42.com/installs/linux/install/CrashPlan/CrashPlan_4.8.2_Linux.tgz
   • Extract the files: tar -xzvf CrashPlan_4.8.2_Linux.tgz
   • Run the installer: cdcrashplan-install/ && ./install.sh
     • I just let CrashPlan use all the defaults, so I didn’t change any options during install
   • Set CrashPlan to start automatically: chkconfig crashplan on
   • Make sure the service is already running: /etc/init.d/crashplan status

At this point we should be able to run CrashPlan’s backup client on our VM, which will pull the data across the network from the NAS. The last step is to set up our local PC for remote administration of the Linux VM. Unfortunately, this process has become more and more painful as CrashPlan keeps updating their clients. They provide their own documentation on how to do this piece, but I’ll summarize what I did here:

Connecting the CrashPlan UI to the Linux VM

1. Download and install the CrashPlan Client on your PC (In this case, I’m using a Windows 10 laptop)

2. On your linux VM, run the following command to findyour authentication token: cat /var/lib/crashplan/.ui_info

3. Back on the Windows side, place that token in the following file: C:\ProgramData\CrashPlan.ui_info

   • You will replace the existing token in the file
   • Also change the port from 4243 to 4200
   • Should look something like this when you’re done: 4200,3da4v903-7q38-4r52-e67e-79aecxf760c4,127.0.0.1

4. Download PuTTY, which will be used to create a SSH tunnel to our linux box

5. Open PuTTY and set the following configurations:

   • On the left side, go to Connection > SSH > Tunnels
   • Enter thesource port: 4200
   • Enter the destination: localhost:4243
   • Click Add
   • On the left side, go back up to Session
   • Enter the IP address of your Linux VM
   • (optional) Under Saved Sessions, put in a name and clickSave

6. Once that’s done, go ahead and clickOpen to connect to your Linux VM

7. Log into the VM and just leave the window open.

   • Note: This SSH session will need to remain open any time you need to connect to your Linux VM and administer CrashPlan

8. Open the CrashPlan client locally on your Windows machine

9. Log into CrashPlan using your account (Should be the same one you were previously using to back up your Synology with)

10. CrashPlan may give you a warning about migrating to a new PC, and ask if you want to adopt the backups - You want to acceptthis prompt, and let CrashPlan know that your new Linux VM is a replacement PC for your Synology.

As long as everything went according to plan, the CrashPlan client should start scanning the NFS shares on your Linux VM and comparing them to what’s already backed up. Once it completes its synchronization, it will initiate the backup processes again.

I was extremely happy that this worked, because I was able to start backing up my data again (which had apparently almost doubled since the last time CrashPlan was connected). In my case, moving to a Linux VM provided me with much better backup performance as well, since the Synology DS411 only has a 1.6Ghz single-core processor and 512MB of RAM.

I hope this helps out anyone else out there who may have been trapped in a similar scenario. If you have any questions, please feel free to leave me a comment below!

When I started working for a cloud service provider a few years ago, the first thing to start coming up extremely often is network latency and performance issues. These are things I never had to worry too much about previously, as most of my jobs had been with enterprise environments where everyone is on the same LAN (or at least within one state). However, when you get into hosting a Software-as-a-Service cloud on a global scale, then slight performance issues begin to mean big slowdowns for your customers.

I was amazed at the current network infrastructure monitoring that was in place when I began working for the SaaS provider:  A few bare-bones Cacti instances, completely unmanaged by anyone, and not configured to monitor any relevant ports or data. Today that situation is vastly different - I have installed a few different applications that allow us to get alerted on network variances and quickly determine exactly where the issue is. One of the tools that has helped us get to this point is called SmokePing, which I would like to talk about today.

Setup and Installation

I won’t get into the details of installing SmokePing, as there are already a number of good tutorials out there (like this one or this one). If you have a decent familiarity with Linux, then the process should be fairly straightforward. Keep in mind that your SmokePing graphs will show latency and packet loss between the machine you have SmokePing installed on and the targets you define. So make sure that you plan out where you deploy your SmokePing machine(s) to provide beneficial information.

Once you have SmokePing installed and setup, it’s time to start defining targets to monitor. We have over a dozen points of presence globally, so I’ve installed SmokePing on a single machine in each location. Each instance has ping targets defined for every network segment within it’s own datacenter, network segments in every other datacenter, and some public IP space of every datacenter. So we accomplish latency and packet loss monitoring within the datacenter, across the site-to-site VPNs between each datacenter, and the general internet connections between each datacenter. For certain customers, particularly those who have dedicated MPLS circuits to us, we are also monitoring latency/packet loss to customer endpoints.

SmokePing also supports deployment in a controller/worker configuration, where you have a single primary configuration/management point and several workers to perform testing. I really want to test this out for our environment, but I haven’t quite had the time to dedicate to it. If you’re interested though, you can find the details on that here.

Interpreting the graphs

The graphs created by Smokeping might not seem clear the first time you see them. For example, take a look at this:

This graph is the result of a standard latency test - 20 pings every 5 minutes. So for every step on the graph, SmokePing draws out the range of responses in those 20 pings - shown by the gray ‘smoke’. The darker the gray area, the more pings came back with that response time - and similarly the lighter areas mean that fewer pings had that response time. The solid colored part of the line marks the average response across all 20 pings, and also gives an indication of percentage of packets lost.

So the first thing I would notice about this graph is that the average response time is varying quite significantly between about 15ms and 200ms. In a normal healthy network, you should not expect to see such a drastic change in response times like that - some variation is normal, but not to this extreme. Two other things to note from this graph: The time of each latency jump seems to line up almost every 30 minutes, and towards the end we begin seeing some slight packet loss.

After being informed that there was a performance issue between a few different systems, I opened up SmokePing immediately to start looking for anything that jumped out - like the graph above. In this case, this was a 200Mb dedicated MPLS circuit used only for replication traffic between data centers. Every 30 minutes, a replication job was kicking off and saturating the line for a few minutes - which in turn was causing excessive jumps in latency and some minor packet loss.

As another example:

The first thing you probably notice about the graph above is the sudden stabilization of latency. This graph monitors traffic between two data centers over an IPsec VPN tunnel - and we happened to be suspecting that one of the two peer firewalls was having performance issues. We swapped out to new hardware on one side of the connection, and the latency immediately started flat-lining. A consistent 85ms is way better than averaging anywhere from 90-180ms. (And if you happened to notice the slight packet loss after the new device was implemented - that was actually due to an unrelated upstream provider issue). My point with this graph is really just to show how helpful it is to have the historical data available. It would have been extremely difficult to prove that  the one firewall was the root cause of our problems if I didn’t have a way to track the issue.

So that’s a bit about SmokePing and how I’ve deployed it within a cloud provider’s environment. It’s only been up and running for a few months, but I’ve already found it to be extremely helpful in troubleshooting performance and latency issues. SmokePing is also extensible via scripting, which can help to collect additional data at the time of an issue. I’ve written a few quick scripts to run extended traceroutes during packet loss events, which I might post up here in the future.

Have you installed SmokePing in your environment? How do you use it? Has it helped you with performance issues?

Comment below!