Okay, so this post is ever-so-slightly off-topic from what I usually write about here - but I’m allowed to have fun sometimes, right?

With the holidays coming up, I’m getting ready to take some time off & relax with my favorite video games.

And just in time, Satisfactory just released their new Update 5 content - which finally adds support for dedicated servers. So I’ll be able to more easily play with friends & family who are in different time zones.

This will be a quick guide, mostly derived from the Satisfactory Wiki with some additional steps/context.

Disclaimer: The dedicated server software is pretty new & still a work in progress, so I imagine there is a good chance some of these steps may change. I’ll do my best to keep it updated!

A Quick Note

Before getting started, just a quick note about what is used for this setup & what assumptions are made.

This guide will walk through building a new dedicated server on Debian 11. There is additional information regarding other Linux distributions & Windows in the Satisfactory Wiki page linked above.

This guide assumes that you have some fairly basic level of Linux knowledge. You should be comfortable with connecting to a Linux server via SSH and editing files. If you need additional guidance, please consider checking out the video above where I walk through the whole process!

This guide will not look at installing Linux, or setting up port forwarding, etc. We will only be focusing on getting the dedicated server software installed & running.

Setting up Static Networking

By default my Debian 11 install was configured with a dynamic / DHCP address. While this might work for other use cases, we’ll want to set a static IP address for our game server.

So first we’ll open up our network config file, using the following command: sudo nano /etc/network/interfaces

Then, we’ll update our network config to look like the one below:

auto ens192
iface ens192 inet static
 address 192.168.1.20
 netmask 255.255.255.0
 gateway 192.168.1.1
 dns-nameservers 8.8.8.8

Please note, the values listed above are for example only. You’ll need to assign the correct addressing from your own local network. If you’re unsure about the interface name (mine is ens192), you can use the command ip address to list current network interfaces & find yours.

Once we’ve edited our network configuration, we’ll need to restart the network process for the changes to take effect: sudo systemctl restart networking

Note: Assuming the static IP you configure is different than the current dynamically-assigned IP, you will lose connection with your server & need to reconnect using the new IP address.

Install Steamcmd

Next, we’ll need to install the Steam commandline utility so we can download & manage our server.

On Debian, the default repositories don’t include this package. So we’ll need to add a third-party package repository.

We’ll edit our active/configured repositories with the following command: sudo nano /etc/apt/sources.list

Then add the following at the end of the file:

deb http://mirrors.linode.com/debian bullseye main non-free
deb-src http://mirrors.linode.com/debian bullseye main non-free

We’ll also need to enable multiarchitecture support: sudo dpkg --add-architecture i386

Once that’s done, we’ll update our local cache to include the newly available list of packages: sudo apt-get update

Then we can install steamcmd: sudo apt-get install steamcmd

We will likely want to create a dedicated user to run our game server. We’ll create a user called steam using the following command: sudo useradd -m steam

Then we can switch to our steam user account: su steam

Optionally, we can create a shortcut to the steamcmd executable within the steam user’s home directory: ln -s /usr/games/steamcmd steamcmd

Download / Install Satisfactory Dedicated Server

Now we can pull down a copy of the server software.

Assuming you’ve created the steamcmd shortcut as shown above, we can use the following command to pull down the server software: ./steamcmd +force_install_dir ~/satisfactory +login anonymous +app_update 1690800 validate +quit

This command does a few things:

• +force_install_dir - Tells steamcmd which directory to place the game files in, in this case ~/satisfactory
• +login anonymous - Skip logging in with a steam user account, since we just need to download an update
• +app_update 1690800 - This is the Satisfactory server application ID, so steamcmd knows which software to install
• validate - Runs a validation to ensure that the current files match what’s hosted on Steam
• +quit - Asks steamcmd to log off of Steam when everything is finished.

If we wanted to use the experimental branch, we would also add the flag: -beta experimental

Once that command finishes running, the Satisfactory software should be downloaded!

We can try to test this out by running the software manually first. Using the following command to move into the game server folder, and execute the software: cd ~/satisfactory && ./FactoryServer.sh

Assuming it runs, we can press Ctrl-C at any point to stop the game server.

Registering a Background Service

Okay, so we were able to run the server software manually for a quick test - but we’ll want to configure it to run as a background service that will start & stop with our Linux system. We’ll use the info located on the Satisfactory Wiki.

So we’ll create a new service definition using the following command: sudo nano /etc/systemd/system/satisfactory.service

Then paste the following into that file:

[Unit]
Description=Satisfactory dedicated server
Wants=network-online.target
After=syslog.target network.target nss-lookup.target network-online.target

[Service]
Environment="LD_LIBRARY_PATH=./linux64"
ExecStartPre=/usr/games/steamcmd +force_install_dir "/home/steam/SatisfactoryDedicatedServer" +login anonymous +app_update 1690800 validate +quit
ExecStart=/home/steam/SatisfactoryDedicatedServer/FactoryServer.sh
User=steam
Group=steam
StandardOutput=journal
Restart=on-failure
WorkingDirectory=/home/steam/SatisfactoryDedicatedServer

[Install]
WantedBy=multi-user.target

Please note, if you used a different user account name (instead of steam), or downloaded the game in a different directory - you will need to modify those values in the above text.

Once that’s done, we can enable our new service to start & stop with the machine: sudo systemctl enable satisfactory.service

Then we can start the game server: sudo systemctl start satisfactory.service

After a few seconds, we can check the service status here: sudo systemctl status satisfactory.service

If everything looks good, you should see output similar to below:

● satisfactory.service - Satisfactory dedicated server
     Loaded: loaded (/etc/systemd/system/satisfactory.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2021-12-01 11:45:31 EST; 18s ago
    Process: 3679 ExecStartPre=/usr/games/steamcmd +force_install_dir /home/steam/SatisfactoryDedicatedServer +login anonymous +app_update 1690800 validate +quit (code=exited, status=0/SUCCESS)
   Main PID: 3749 (FactoryServer.s)
      Tasks: 21 (limit: 7087)
     Memory: 1.0G
        CPU: 18.526s
     CGroup: /system.slice/satisfactory.service
             ├─3749 /bin/sh /home/steam/SatisfactoryDedicatedServer/FactoryServer.sh
             └─3756 /home/steam/SatisfactoryDedicatedServer/Engine/Binaries/Linux/UE4Server-Linux-Shipping FactoryGame

Dec 01 11:45:33 Satisfactory FactoryServer.sh[3756]: [2021.12.01-16.45.33:178][  0]LogAIModule: Creating AISystem for world DedicatedserverEntry
Dec 01 11:45:33 Satisfactory FactoryServer.sh[3756]: [2021.12.01-16.45.33:178][  0]LogLoad: Game class is 'BP_GameModeMenu_C'
Dec 01 11:45:33 Satisfactory FactoryServer.sh[3756]: [2021.12.01-16.45.33:179][  0]LogReplicationGraph: Display: SetActorDiscoveryBudget set to 20 kBps (5333 bits per network tick).
Dec 01 11:45:33 Satisfactory FactoryServer.sh[3756]: [2021.12.01-16.45.33:205][  0]LogNetCore: DDoS detection status: detection enabled: 0 analytics enabled: 0
Dec 01 11:45:33 Satisfactory FactoryServer.sh[3756]: [2021.12.01-16.45.33:205][  0]LogInit: BSD IPv4/6: Socket queue. Rx: 262144 (config 131072) Tx: 262144 (config 131072)
Dec 01 11:45:33 Satisfactory FactoryServer.sh[3756]: [2021.12.01-16.45.33:205][  0]LogNet: Created socket for bind address: :: on port 7777
Dec 01 11:45:33 Satisfactory FactoryServer.sh[3756]: [2021.12.01-16.45.33:205][  0]PacketHandlerLog: Loaded PacketHandler component: DTLSHandlerComponent ()
Dec 01 11:45:33 Satisfactory FactoryServer.sh[3756]: [2021.12.01-16.45.33:205][  0]PacketHandlerLog: Loaded PacketHandler component: Engine.EngineHandlerComponentFactory (StatelessConnectHandlerComponent)
Dec 01 11:45:33 Satisfactory FactoryServer.sh[3756]: [2021.12.01-16.45.33:205][  0]LogNet: GameNetDriver EOSNetDriver_2147482561 IpNetDriver listening on port 7777
Dec 01 11:45:33 Satisfactory FactoryServer.sh[3756]: [2021.12.01-16.45.33:205][  0]LogWorld: Bringing World /Game/FactoryGame/Map/DedicatedserverEntry.DedicatedserverEntry up for play (max tick rate 30) at 2021.12.01-11.45.33

Joining the Server

We’re almost there!

After the game server has been set up & running, we’ll need to perform the initial setup within the game itself.

So it’s time to launch Satisfactory & setup our game server!

In the screenshot above, the title screen now has an option for Server Manager. We’ll click that. You should then see an option to add server, which will display the following dialog:

Here’s where we put in the IP or hostname of our local dedicated server.

As a reminder: If you have someone outside of your local network, they’ll need a different address to connect. Again, this post won’t dive into port forwarding or allowing access through your home router - there are plenty of good tutorials out there already!

After we successfully connect to our server, we’ll be prompted to claim it. Since it’s newly created, this is where we can set our admin login & the server name.

First we’ll be asked to provide a name to claim the server:

Then an administrator password - don’t forget to save this somewhere safe!

Okay. So now we’ll have a few options. First we’ll check out the settings page:

Here we can modify our server name or update our admin password if we need to.

We can also set a Player password - This will keep people from joining your server unless they know the password.

Interestingly enough, we can opt to have the game pause when no players are connected - or uncheck the box to let your factory continue building even when no one is playing.

And lastly an option to autosave whenever someone disconnects from the server.

If we’re good with those settings - we can start a new game session!

We’ll head over to the Create Game tab:

This should look pretty similar to the normal game. Select the starting area & give the session a name.

We’ll also have a checkbox to automatically enter the game after it’s done being created. If you don’t check this box, you can enter the server from the Server Status page:

Here we can get a quick view of the current server status - which tier we’re on, the current milestone, and whether or not anyone is currently connected.

Down in the bottom left, we’ll have our Join Game button.

Time to play!

Okay - and that about wraps it up. I hope this post was helpful for anyone else looking to build a dedicated server. The process was surprisingly easy, and I greatly appreciate the effort that the Coffee Stain devs have put into this!!

Happy building!

Hey folks. So, we got a new Comcast xFi Gateway 3rd Generation at my parent’s house, and all the devices were able to discover our SSIDs & connect to the internet at both 2.4Gz & 5Ghz …except my personal computer. My personal computer was listing networks not associated with our house, yet it wasn’t showing OUR networks. What gives?!

First off, System Information about my computer…

• OS: Windows 10 Home
• Manufacturer & Model: Dell Inspiron 5558
• Wireless adapter: Intel(R) Dual Band Wireless-AC 3160
• xFi Gateway models numbers: CGM4331COM or TG4482A

Back to the goods…

I restarted my computer twice. Turned on & off the wi-fi on my computer. Played around with network settings on my computer. Still no luck.

So of course I headed to Twitter and asked for assistance. You can read the tweet and read the replies here. I got many different tips from people: update the wi-fi drivers, buy a USB Wi-fi Dongle, etc. But my friend Pat ( @Battle_Nerd_1 ) messaged me and suggested that I look into what wi-fi standards (802.11 b,a,g,n,ac) are associated with my wi-fi adapter.

Let’s walk through how to do that, why don’t we?

1. Look at the Advanced properties of the wifi adapter. (I have a Windows 10 device)

Go to Start -> Device Manager -> Network Adapters -> Right click the adapter name -> click Properties -> then Advanced

The Advanced network properties of the computer’s wi-fi adapter is what we need to review. Notice in the “Property” field; the wi-fi standard displays 802.1b/g for my computer. This is an important detail.

2. Compare & Change the settings

(I use modem and router interchangeably. That’s probably not good practice, but oh well lol)

Login to your router (*** I’ll add instructions below how to do this) & start clicking around until you find the Wi-fi Mode settings. Here’s the settings for our router at 2.4GHz.

From the same page, here are the Mode options of our router:

3. Compare & Change the settings

Remember, my wi-fi standard for my computer’s wi-fi adapter is 802.1b/g. Initially the modem’s Wi-fi Mode was set to 802.11 g/n/ax, then Pat told me to change it to 802.11 g/n because the new ax isn’t supported by a lot of wireless drivers; it’s Wi-Fi 6. Once I changed the Mode to 802.11 g/n, IMMEDIATELY my computer recognized the 2.4GHz network and connected!! Yay Pat!!

How to Log in to your Router/Modem

(These instructions are for Windows devices )

1. Find your Gateway/Router/Modem’s IP address

• Go to Start and type cmd for Command Prompt.
• Type ipconfig, then hit Enter
• Scroll down until the Wi-fi section, & take note of the default gateway IP address.
• This number will most likely look like 10.x.x.x or 192.168.x.x

2. Log into your Router/Modem

Next open up a web browser and type in the aforementioned IP address, then hit Enter.

Once prompted for a Username and password; try Admin for username & Password for password. Also, It’s good practice to change the default password, so definitely do that :)

So now you know how to find the IP address, log into, & change the password of your Gateway/Router/Modem. Yay you! Now you can go back to Step 2 and learn how to view your Router/Modem’s wi-fi settings.

So thanks to my Twitter friend, Pat, for walking me through how to find info about my computer’s wi-fi adapter and how to change the modem/router’s wi-fi settings; also he saved me from having to spend money on a new computer! In the 5GHz Mode settings, there is no 802.11 b/g option, therefore my computer can only connect at the 2.4GHz frequency, which is not an issue for me. I can now connect to a home network & have internet access on my personal computer.

About ARIN

If you haven’t already worked with them - ARIN is the American Registry for Internet Numbers. They are a non-profit organization and their purpose is to assign/manage Internet number resources for all of North America. This includes IPv4/IPv6 addresses and BGP Autonomous System Numbers (ASNs). ARIN is one of five Regional Internet Registries (RIRs) - each managing Internet resources for it’s own individual region. All of these report back to a top-level organization, the Internet Assigned Numbers Authority (IANA).

What I didn’t know: ARIN actually used to manage resources for all of South America and Africa as well. LACNIC formed and took ownership of South America in 2001, and AFRINIC took Africa in late 2004. ARIN itself has only been around since 1997, and will be celebrating it’s 20th anniversary this December.

Outside of assigning/managing number resources - ARIN manages a huge manual of numbering policies and standards (The Number Resource Policy Manual). A good note here is that these policies are heavily influenced by the community - so if any individual or group of network operators want to change/modify or add new policies, then they can submit proposals to do so.

IPv4 Depletion

I was very interested to hear about what’s going on with IPv4/IPv6 - mostly because I’ve been trying to push for IPv6 in many of the places I have worked. The ARIN group spent a little bit of time talking about how the depletion of IPv4 addresses has affected their workload. Overall, it seems like their work has remained about the same - but it has transitioned from mostly IPv4 allocations to more IPv4 transfer requests.

An interesting note from this discussion was that ARIN only performs the backend registration changes for IPv4 block transfers. They play no part in the actual negotiations between two organizations. However, they do perform their own investigations during transfers to ensure that the source organization legitimately owns the IP block, and the destination organization can justify the use of the space.

I had heard previously that ARIN kept a block of IPv4 addresses for transition to IPv6 - but I never researched it further. This was a topic ARIN touched on during the event. Essentially, they have kept ownership of a /10 block of addresses, which is split up into individual /24 blocks for assignment. Any organization can request one of the /24s when they request a block of IPv6 addresses. The organization must fill out a justification form, in which they demonstrate how the IPv4 blocks will be used to help transition to IPv6. Organizations can request one of these blocks every 6 months, provided they can still justify the need for them. This is all documented in NRPM section 4.10.

The somewhat surprising thing here is that ARIN was actively encouraging people to take advantage of this. Probably because they need to push IPv6 adoption in any way they can. As of the date of the event, ARIN stated that only ~60 /24 blocks had been assigned so far.

IPv6 Adoption

This part of the event wasn’t quite everything I wanted it to be. Overall ARIN touched on statistics from Google and other organizations that show the trending uptake in IPv6 network access. They also spoke briefly about how the structure of IPv6 addresses makes life easier - because the last 64 bits can always be used for host-based MAC autoconfig, then network operators only worry about subnetting above that.

Interestingly enough, ARIN was advocating for the method of ‘assign way more addresses than you’ll ever need’ mentality for IPv6. Another attendee asked the question ‘Won’t we run into the same thing as IPv4, if we just throw out v6 blocks like candy’? This actually led to hearing something I wasn’t aware of - IANA has currently only made 1/8th of IPv6 blocks public available for use. The current numbering scheme/standard will be used for this first block of addresses. If we run through them too quickly, then we can step back and re-evaluate best practices before handing out the next 1/8th block of addresses.

DNSSec

Initially I was a bit confused that DNSSec was on the topic list - but I figured maybe ARIN was just trying to push this for the betterment of the Internet. While they spoke a bit about DNSSec for forward DNS, their primary topic was how DNSSec for reverse DNS isn’t something people are normally thinking about. As it turns out, ARIN offers reverse-lookup DNSSec for any IP blocks that they assign out. This is good to know, since reverse DNS can be important for things like email security - and its certainly something I’ve never really considered in the past.

If you have purchased IPv4/v6 blocks directly from ARIN - I would recommend that you check this out.

RPKI

Resource Public Key Infrastructure (RPKI) is a way of cryptographically validating ownership of IP address space or routing objects. Since BGP is primarily a trust-based protocol between organizations, RPKI allows network operators to implement additional security by providing a certificate-based system of trust. The majority of this discussion was around how bad BGP security is, and that overall North America is far behind on implementing RPKI.

ARIN has a service available where they will act as your Certificate Authority (CA) for RPKI - so it only requires network operators to sign records then implement a few device changes.

My Thoughts

Overall the event was fairly informative! It wasn’t quite everything I wanted it to be, but I did walk away with additional knowledge that I didn’t have before. I was really hoping to learn more about how other organizations are implementing IPv6, or even how other people are convincing their employers to take IPv6 adoption seriously. When I spoke with some other attendees, it seemed like not many people had IPv6 running in a production environment yet - only a few of them had even started testing. Surprisingly, even the ARIN reps were repeatedly asking people to contact them if they had an IPv6 success story to share.

One thing I found really interesting was surrounding DNSSec/RPKI. A few attendees asked about how many people are actually validating signed resources. It’s one thing to implement signing, but it won’t matter if no one validates the resources, right? Surprisingly, ARIN had no statistics about this - and stated the point that they cannot enforce adoption of these standards. It certainly makes sense, but it’s not something I gave much thought to previously. Since they’re just a registry, they can only make these services available - not enforce their usage. This is why they put on events such as this to raise awareness and provide education.

ARIN pushed the fact that all of their policies are community driven. There were quite a few examples throughout the event of how individual members of the community could impact changes to their policies. My primary concern is that it seemed like a majority of the individuals in attendance represented government or educational organizations - and not a lot who worked in similar network environments to what I manage. They raised their own concerns and questions, which were certainly valid for the types of infrastructure and designs that they maintain. However, a number of these things don’t really apply to my infrastructure in quite the same ways.

If I have to make one point here: If you’re a network operator, go subscribe to ARINs mailing lists and get involved. Maybe you don’t have any ideas for policy changes, but you never know what might come up that you could provide meaningful input on. The ARIN reps provided an example or two of when a smaller group of people suggested policy changes which drastically affected bigger companies - and almost no one opposed it until it took effect. Only you have the ability to voice your opinion and concerns about how a proposed policy could affect your network. If not, the next time you try to request a block of IP addresses or a BGP ASN, you could potentially run into roadblocks because of a policy change proposed by someone with very different needs.

The staff at ARIN don’t live and work in the networks that we do. They try to work with network operators to understand use cases and the possible ramifications of policy changes - but ultimately they are a small non-profit. They can’t think of everything, nor can they force network operators to contribute their opinions. Get involved and make a difference.

As a final note, ARIN has a Fellowship Program where you can apply to attend one of their Public Policy meetings for free. Fill out an application and if you’re chosen they’ll provide a ticket, hotel room, and travel expenses. It’s a great opportunity to experience one of these meetings, especially if you might not have the financial means to otherwise.

The slide deck from the event is publicly available on ARIN’s website: here.

How does this actually protect user privacy? Best described in the intro to RFC 7858, DNS over TLS “eliminates opportunities for eavesdropping and on-path tampering with DNS queries”. Using the new standard, DNS queries of a client can be encrypted using TLS and tunneled directly to the DNS server. The intent is that monitoring of user activity becomes more difficult, since their DNS requests are no longer sent across the network in plain text.

It seems that DNS security has become a point of focus recently. Over the past year or so, we have seen two new big companies join the public DNS space: Quad9 (backed by IBM) and CloudFlare. Both companies advertise that they are focused on security and privacy of the individual user. They offer features like: blocking of malicious domains, DNS over TLS support, and limited historical logging of user requests. CloudFlare has even already posted steps for enabling their DNS over TLS service with the new version of Android. For the end user, this is great - free services that offer better security and increased privacy. However, this can impact how businesses protect their users.

A lot of organizations use web filtering as an important step toward securing client traffic - which can be accomplished through DNS filtering and/or web proxies. These methods might be used for compliance reasons, malware/threat detection, or even just managers who want to monitor productivity. However, there are a few big changes happening in user security/privacy right now that begin to make that more difficult: Increased focus on DNS privacy and the recent release of TLS 1.3.

Years ago I used to work for a local government organization where security was a high priority for IT operations. Web access was extremely limited and configured to whitelist only for known sites that were business related. If you wanted general internet access, you had to log into one of two segmented computers that were dedicated to that purpose. Even still, our IT security manager had concerns on the potential for threats and/or data exfiltration via HTTPS-enabled websites. One of my projects was to tackle this problem by implementing SSL decryption on our web filtering platform. At the time, implementing a web proxy as a man-in-the-middle wasn’t very complicated. Generate a CA cert for the proxy, distribute that cert to clients, then enable SSL decryption. Sure, there were bumps and hurdles - but the overall process achieved the result we were looking for.

The problem today is that newer standards and features (like TLS1.3, HSTS, or certificate pinning) are adding enhanced privacy features - which make that user data more difficult to reliably decrypt. That’s good right? Well, maybe - but only for the user. As an organization trying to secure your systems and private information, these measures are making life a lot more difficult. Even as a network admin, it’s become more difficult to troubleshoot web applications via packet captures. The use of ephemeral keys for TLS connections means that you can’t decrypt the session even if you have the server-side private keys.

Now the addition of DNS over TLS give us more of the same problem. Even if you couldn’t look into an encrypted web session, maybe you could gather what the destination is by reading the DNS requests. Instead, users now have the ability to encrypt their DNS queries, encrypt their web sessions, and make the life of a security admin a nightmare.

From the perspective of a corporate security team, the future is going to be more challenging (and probably for a lot more reasons than just this). However, I think that at some point we will have to reach a better balance of user privacy while still providing adequate security coverage. For some that might mean installing feature-heavy agents on every client device to inspect everything before it leaves the device. Or it might mean trying to rely on statistical, analytical, and behavioral data to detect patterns and anomalies (something like what ETA is trying to accomplish) rather than just tearing apart the individual user sessions.

Whatever the solution ends up being, it should be an interesting area to watch develop. In the meantime, let’s just agree not downgrade user security or privacy just to shove our security tools in. We’re all users of some service or another, and we all have some expectation of privacy. Is it worth reducing security and increasing risk just to see if your employees are using their web access responsibly? Probably not. Instead it’s time to look for new ways to solve this problem, and find ways to stay secure while still ensuring user privacy.

The future is APIs! SD-EVERYTHING! Automation! Orchestration! Artificial Intelligence and Machine Learning! Sound familiar? It’s all part of the messaging going around in just about everything IT-related. With as much as you keep hearing about it, you might think that it’s all anyone is doing anymore. Yet it still just seems like not a whole lot of people are really getting into it in my area. Every vendor event I’ve gone to this year has asked attendees the same questions: “How many of you are leveraging the APIs in your network hardware/software?”. And every time the same answer - maybe two or three people in a room of 40 raise their hands.

So where is the problem? Is all of this just marketing fluff or am I just talking to the wrong people?

Let’s think about this from a typical network admin’s perspective. Shifting from traditional CLI to automation and APIs can seem difficult or overwhelming. Let’s say I want to automate a new VLAN deployment. Oh, you’re telling me I need to stop and learn vendor APIs… but before that I need to understand how to write scripts. But I’ve never even programmed something before. There are dozens of languages - how do I pick one? How much fundamental programming knowledge do I really need to have before starting? I don’t want to be a developer!

Okay, okay - just stop there for a second. No one is asking you to drop networking and write code for a living. The end goal of all this programmability stuff isn’t to turn networkers into developers - It’s to enable network/systems admins to be more efficient at their jobs. Why  copy/paste the same config change to 100+ devices, if you can mass-deploy the change via an API? That’s a lot of time savings that could be used toward educating yourself on new products, planning other projects, or thinking about your ideal network design.

I’ve heard a lot of the same things over the past few years:

“Programming is difficult” or “I don’t know where to start”

Try learning Python. It’s simple to get started and you can build from there.

“I don’t know what an API is or how to use it”

Don’t worry about that yet - start with learning the basics and APIs will make sense later.

“I’m not a developer”

No one is asking you to be one! But learning the basics of scripting and automation gives you a whole new toolset to solve problems.

For me personally - I would never want to be a developer. I can’t stand the thought of coming into work every day and just writing code. Some people might enjoy that, but for me it doesn’t sound like fun. However - I enjoy writing scripts to solve problems, especially when it ends up making my job easier. I think that’s the part where some people tend to get stuck though. A lot of automation sounds like I need to be able to develop a huge 10,000+ line application to pull data from 15 sources and aggregate it to make intelligent network changes. Ehhh… Nope, not really. But what about just a quick script that runs every 5 minutes to check an interface statistic, and email you when a particular threshold is exceeded? Realistically that could be done in less than 50-100 lines of a script and maybe 30 minutes worth of work.

Still not interested? That’s okay too. Traditional networking isn’t going away any time soon, and over time the vendors will write all of that automation for you. They will package it up in a pretty GUI and sell it off to companies that want it. In fact, this has already happening and has been for quite some time. This isn’t a bad thing - vendors need to make money, and not all companies will have the time or skilled resources to automate all the things. However, a network admin who can write their own scripts/automation won’t be exclusively tied to a vendor to help them - and instead they will be empowered to solve more problems themselves.

Where do you get started? I already wrote a bit earlier this year on a few resources for learning Python - which you can find here. I also wanted to point out some other great resources that are a bit more specific to using those skills for network automation:

• Python For Network Engineers - Don’t know anything about Python yet? Start here! This is a free course provided by Kirk Byers for anyone who is interested in using Python for network automation. Once a week you’ll get an email with all the great free content, but it will be up to you to spend time going through it. Go sign up, and set aside an hour or two each week to practice.

• Cisco DevNet - There is a ton of great content here. While DevNet does offer some tutorials on basic Python fundamentals,  the real value here is examples on how to use some network APIs (NX-OS, Meraki, UCS, etc). Also - one of the best parts about DevNet is the sandboxes they offer. Want to write scripts against the FirePower Management Center, but you don’t have one to test with? Well with DevNet you can get access to one!  Get familiar with your Python basics, then come here to see where you can start using those skills with your existing infrastructure.

• Network Programmability and Automation - This is a fantastic book. Not free, but it is well worth the ~$30. Once you have a good handle on how to write some basic network automation with Python, I highly recommend picking this up. While Python is covered here, the book does a great job of introducing you to all of the other toolsets available. Curious about how Linux or Ansible fit into network automation? You can find out here - and learn about APIs and source control systems too!

So - What are you waiting for? Go get started, and see what you can accomplish. Learn the basics - and keep an open mind for opportunities to use those skills.

Have suggestions on where else to learn? Comment below!

On the other side of things, I’ve worked with people over the years who haven’t had quite the same experiences as I have. Some of them don’t typically call vendors for support - or maybe they’ve just never been lucky enough to be the first responder on a Severity-1/Production-down case. This has occasionally resulted in a number of vendor support cases being closed with highly unsatisfactory results. In fact, there have been times where this has been bad enough that it gives management the impression that we received bad support from the vendor - but what if it was just our own inability to work effectively with that vendor? Maybe we didn’t push hard enough on an issue or stress the importance.

I feel that in a majority of cases, it shouldn’t be difficult to get the results you want out of a vendor support case. So I’ve put together a list of my own personal tips and guidelines for working with technical support.

The Vendor is your partner - Not the enemy

I’ve seen a lot of people call vendor support for help, then treat the support engineer as the bad guy. If you want to get a quick resolution, then you need to look at them as your partner. They’re here to help you figure out your issue and get everything fixed. When you open a new case, give them a concise summary of your issue with any relevant details you think are important. Don’t hide information, as this will just prolong finding a resolution - Give them everything they need. If you know your vendor always asks for the same diagnostics information every time you open a case, then have that information ready before you contact them.

Always remember how difficult the job of a technical support rep can be. They’re likely sitting in a call center, just waiting for the next case. They may have in-depth knowledge of their product, but they’re walking into your network completely blind. They won’t know your traffic flows, or that some systems are redirected through a proxy, or that one band-aid fix that Joe put in a year ago and never documented. Your support rep is going to do the best job they can to get a handle on what your network looks like, but be prepared to guide them. How would you like to troubleshoot a completely different and unknown network every time your phone rings?

Have realistic expectations

Especially when you’re new to a career in IT, it can be hard to gauge what you can and can’t ask of your tech support rep. One of my former jobs had a policy that for every ticket with AT&T we opened, we would be required to call back every hour for a status update. If there wasn’t one, then we were supposed to demand escalation to a manager. That policy might make sense if it’s a critical issue, but what about something that’s not? Have realistic expectations about what your vendor can do.

Troubleshooting takes time. If your support engineer grabs a bunch of logs and says they’ll need to get back to you - then you’ll need to give them the time they need. Feel free to ask for a time estimate - but if they say they’ll have something to you by the following day, don’t start bugging them every hour.

Remember that your support engineer’s job is to help you. If you don’t feel like that’s happening, you have a few options. You can ask for a case escalation or ask for the case to be reassigned. You never know the skillset of the person receiving your case, and you might get someone who isn’t super familiar with the problem you’ve raised. As long as there is no urgency, I will usually give the person time to work the issue - but be prepared to request a case transfer if it becomes apparent that they’re not getting anywhere. For example, I once had a case for a web-based firewall management system. The engineer I got was very good with the GUI side of things, but wasn’t very knowledgeable when troubleshooting took a turn toward the underlying linux system. A quick request to transfer the case to someone more experienced in this side of the system and we had the case solved within an hour. If an escalation or case transfer doesn’t help, you can also usually reach out to your local account representative and ask them to help push the issue for you.

It’s also very helpful to have an idea of your vendor’s support policies. Have a question about how to set up a new feature? Some vendors don’t permit you to open a case for a new configuration, and will refer you to their professional services team. On the other hand, some vendors are perfectly okay with helping you figure out how to set up something. Even better, some support teams are willing to stand by during migrations and upgrades, just in case you need their help. In my experience, if you’re not 100% confident in your changes, then it’s better to open a proactive case beforehand.

Be clear about the impact

If your entire datacenter is offline because of an issue, make sure that you immediately stress the importance. Again, your support engineer is jumping into your environment blind. Does this firewall performance issue impact twenty people, where it is just a minor inconvenience? Or is this issue prohibiting 50,000 customers from using the services you provide? The last thing you want is a misunderstanding of impact when it’s a high priority issue for your business.

Usually when I open a high severity case, I’ll let the engineer know: “Just so we’re on the same page, this issue impacts a large datacenter impacting 600+ customers. We need to get this back into a stable state as soon as possible”. High severity cases can be stressful for both sides, and I try to be clear about the impact without making that worse.

On the other side of things, if there is a low severity issue - don’t blow it out of proportions. I’ve worked with too many engineers who open up a Sev 1/Prod-Down case for every issue, even if the issue is just a minor inconvenience. Categorize your issues appropriately when you open them - and do your best to be realistic. A slow download for three users probably doesn’t warrant getting half a dozen TAC engineers on a conference bridge.

In case of emergency

Emergency situations are a completely different subject - so I want to spend a bit of time covering them separately. It’s really important to know what constitutes a true emergency in your environment. Is it when an office (or datacenter) goes offline? Or maybe even a single extremely critical business application? Things break - so have a plan and be prepared.

Step one - Always call into your vendors support line. You don’t want to open a web/email case and wait around for a technician. This might seem obvious, but I’ve known a lot of people who complain about the vendor’s response on a critical issue when they opened a case via a support portal. Find the vendor’s support number (or have it saved somewhere) and call them.

Step two - Ask for a warm handoff. In most cases, the person answering the support line is just creating a case and routing it to the appropriate team/ticket queue. They may just give you a ticket number and tell you to expect a call back shorty. If the issue is truly critical, ask them for a warm handoff to a technician. Most vendors I have worked with have had no problem doing this, and it helps you get to troubleshooting faster.

Step three - Clarify your issue and set expectations. You may be in a rush to get the issue fixed, but take a minute to explain your issue thoroughly and clearly. The more information you give to your support technician, the more easily they can dive into troubleshooting. And as I had mentioned earlier, be sure to set expectations and be extremely clear about the impact of your issue.

Step four - Keep troubleshooting on track. As I’ve stated before, you know your environment/network better than your vendor does. If they start looking at something you don’t believe is related, you need to guide them back to the main problem.

In addition, if you feel after a bit that the troubleshooting isn’t making progress - then request an escalation or a second set of eyes. There is no harm in asking for more eyes on the problem. I’ve even had situations before where the technician said “Well, I think we need to do X to fix it”, and I’ve just asked them to see what their peers think. You would rather be sure about a change, than make the issue worse.

Step five - See the issue through to resolution. Make sure you get your environment back to a stable state before ending the call. If the technician wants to drop off and call you back after reviewing logs, let them know you’re willing to just wait on hold. Once they’re off the phone, it’s easy for your technician to get dragged into another issue.

If the call ends with everything in a temporary state - then take follow ups on your next steps and make sure you accomplish them! Maybe you were able to restore connectivity, but need to wait for a maintenance window to make a change that is more service impacting than the original issue. Or maybe your support technician needs you to gather additional logs that they can forward to development. Whatever it ends up being, make sure you take note of it and follow through.

These are just some of my own personal tips that have worked for me. Support calls with vendors don’t always need to be a massive pain to deal with. Sure, sometimes you might have bad luck and get an inexperienced technician - but I find most issues with vendors can be solved easily enough once you know how much you can push them, and what you can and cannot ask for.

I hope these are useful - Let me know in the comments if you have any additional tips!

Earlier today, Cisco released their 2018 Annual Cyber Security Report. I’ve spent some time digging through the report and thinking about what they’ve written. It’s interesting to read through the trends and survey results, and try to get an idea of where security efforts should be focused for the coming year. This post is going to cover just a subset of what’s in the complete report. I’ll be covering the topics that I found particularly interesting, and give my own thoughts/views on them.

The Encrypted Web is Great… For Attackers

Unsurprisingly, Cisco reports a growing trend in attacks and exploits that are taking advantage of encrypted transport. As a lot of large companies and Internet bodies are pushing for a 100% encrypted web, should we be surprised? Nah, it’s the logical next step. Users want encryption because it means privacy - but that privacy also brings a method of concealing attacks.

New exploits and malware are heavily leveraging encrypted transport to bypass all of the security we put in place to detect them. Typical defense technologies like intrusion prevention systems (IPS) are fantastic, but only when they can actually read the data. If a user download’s malware through an HTTPS call, IPS won’t usually catch it. And when that malware can now take advantage of SSL to reach back out to a command & control server? Yeah, IPS might not help us there either.

There are technologies out there that allow enterprises to see this traffic - but maybe not enough of us are adopting it yet. A forward proxy for outbound web filtering is great. One that implements SSL decryption and inspection is even better. If your company isn’t already decrypting outbound web traffic, then this needs to be a priority.

Inbound web traffic can be just as dangerous. New IIS vulnerability? Sure, let’s grab the latest IPS signatures and … Oh wait, our IPS runs on our edge firewall, which sits in front of those web servers - which are all using SSL for encryption. That means any malicious traffic is going to slide right through our IPS undetected and land on our unpatched web servers. Get a Web Application Firewall (WAF), and let it front-end your SSL traffic. These things can be expensive and a nightmare to configure and tune properly, but right now they are one of your best options for inspecting web traffic.

Old Attacks Aren’t Going Away - They’re Just Getting an Upgrade

This year’s report highlighted that much of the older attacks are still here, and they’re not giving up yet. Attacks via email are still present and doing more damage than you would hope. We’re certainly getting better about implementing spam filters with reputation filtering, but attackers aren’t giving up yet.

Email attacks are relying more on social engineering and targeted phishing. These messages are also utilizing SSL to encrypt the malicious links within the emails. Infected attachments are surprisingly still a big issue, with Microsoft Office and PDF files still being the worst offenders. Just because attackers are finding new and exciting ways to hit us doesn’t mean they’re giving up on the tried and true methods. We still need to focus on all the standard attack vectors, like email. Implementing intelligent email/spam filters and providing user awareness training are the primary methods we have to combat this.

The Cloud is Secure! …We Think

This one I found particularly fun. Out of all the companies surveyed by Cisco for this report, 57% of them said they believe the cloud offers better security. Wait - Did I misread that? More than halfof respondents think that the cloud offers better security than their own infrastructure! This makes me wonder…

From my perspective, a cloud service provider is just another company. In most cases, they run just another network and hit a lot of the same challenges that non-cloud companies are facing. And we can only assume that cloud providers are prioritizing security and not just trying to turn a quick profit. Cloud companies have the advantage of being able to hire a dedicated security team that their customers can leverage. However, enterprises are complaining about lack of skilled security engineers, and I’ll bet it’s not because cloud providers are picking them all up.

Cloud definitely offers benefits - but this needs to be a well-calculated risk. For a smaller company without dedicated IT staff, a cloud solution would most likely offer security improvements over their own infrastructure. As companies scale, however, their security requirements do too. We need to make sure that the cloud providers we choose are also capable of adhering to those standards. Before you move to the cloud: ask questions about their security practices, get answers, and demand more information on the parts that are important to your business.

Another fun note from Cisco - some of the damage done by cloud providers is a simple mis-understanding of ownership. If you subscribe to a complete Software as a Service (SaaS) provider, chances are good that the provider worries about all of the critical security configurations. However, if you’re going to a cloud provider just for infrastructure (like AWS), then you are likely responsible. In the case of AWS, you’re being provided a server - and that’s where Amazon’s responsibilities end. It’s up to the enterprise to still make sure that those servers are patched, hardened, and audited. Treat the cloud as an extension of your own infrastructure and polices, not a separate entity.

Diversifying Risks? Maybe not

It used to be a somewhat well-established security practice to use multiple vendors. Have a need for two sets of firewalls? Make sure you use two vendors, so that a vulnerability in one doesn’t affect the other. Seems like sounds logic - until you have to train staff to be experts on multiple platforms, and keep up to date on all the latest patches from each vendor.

Cisco is finding that the more vendors a company has in their environment, the more problem we have maintaining everything. From my own experiences, I can say this is certainly a problem. In environments where I’ve had up to four vendors for firewalls and switching, it becomes difficult to work with. It’s hard on IT staff to maintain knowledge of configuration and best practices for each different vendor - and when a new vulnerability comes out, we end up spending way more time trying to track down each vendor’s responses and patches.

It makes sense that companies who have a more tightly integrated infrastructure might have an easier time managing it. Cisco might want you to buy 100% into their ecosystem (of course), but I do think there is value in consolidating your infrastructure. One or two vendors will be much easier to establish relationships with than half a dozen of them. Your IT staff can dedicate their focus to mastering only a couple of technologies, rather than spreading themselves over a dozen different platforms. And when that new vulnerability is released? It should be much more straightforward to patch all of your systems quickly.

There’s That Automation Thing Again

I think we’re finally beginning to reach a point where automation is really showing it’s value in the security realm. A typical company is going to have so many different systems and alerts that it doesn’t make sense for someone to manually review and act upon every one. This is where automation really begins to shine.

Cisco’s report shows that more companies are relying heavily on automation. This can be used for alert response, reporting, and behavioral analytics. Especially when I keep hearing that there is a skills shortage in security, we need to take advantage of what automation can offer. This doesn’t always have to be home-grown scripts either - there are a number of offerings already available.

Take a second look this year. Try to see where automation can fit into your infrastructure to help improve both operations and security.

Thanks for reading! Just as a friendly reminder - All of the opinions stated in this post (and all others here) are 100% my own, and do not represent any vendor or employer. Since security has become more of an important part of my job, reports like this are always very interesting to read. I’ve only covered a handful of what was in the report - just what was particularly interesting to me. If you’re interested in reading more, check out the full report here.

Risk acceptance is something that I’ve seen misunderstood by engineers over and over again. Especially when you’re earlier in your career, you get the feeling that you know what you’re doing - and therefore management should listen to you. I get it - I’ve been there too. It can be hard to propose an idea, then have that idea immediately shot down by your peers or management. That’s not to say that they aren’t listening, but maybe you might not have a great view of everything behind their decision.

Let’s take an example. As an engineer (server/network/security/etc), you’ve been alerted by your vendor that there is a huge vulnerability in a critical business application. That vendor has been extremely proactive, and has promptly provided you with detailed KB articles and download links to the upgrade package. The vulnerability that’s been disclosed certainly seems bad enough - it allows for remote code execution by an unauthenticated user. What do we do with this? Well we immediately meet with management to tell them that we need to patch this application immediately.

Their answer?

No.

Well that seems just impractical, doesn’t it? They don’t really want to leave the company at a massive risk to attack, do they? No - they certainly don’t. However, it may be that management has chosen to accept the risk for one reason or another. Let’s dig into a few reasons why they might:

Expense (Money) - It could be that the cost to remediate the issue is significant. Maybe the software is a bit out-dated and the company hasn’t paid for support. Renewing that support contract might be tens of thousands of dollars or more. If there are enough mitigating factors, that cost might not be worth it.

Expense (Time) - People also cost money, and time spent on fixing this issue also means lost productivity on other tasks. For some enterprises, an upgrade of a critical business application could take months of work. What happens when this new version breaks compatibility with another business application? Well now we’ve added on extra time so we can upgrade both. If there is no other value to the upgrade, then the fix might be held until there is.

Mitigating Factors - There may be enough other security controls in the network to reduce risk. Maybe in this case only the core servers for this application are affected - and they reside on their own network segment behind a firewall. Maybe that firewall also runs an intrusion prevention system, which has already been updated with signatures to stop this particular attack. There could be a number of things going on which have given management the comfort that the system is still safe.

Any of these might lead to the final decision: we’re going to live with this risk. By accepting it, we’ve evaluated it, we’re aware of it, and we are consciously choosing not to fix it. We may have limited the scope of this risk via other mitigations - but ultimately we are not completely ridding ourselves of this risk.

Does risk acceptance only apply to security? Nope. It applies to just about anything really. What if I recommend that we need to replace older hardware or risk network instability? That risk could be acceptable if the hardware isn’t critical. Same thing goes if we decide to forego renewing support on a pair of firewalls. Maybe it’s even suggesting an implementation of a new technology. Proposing to a big cloud provider that they should prepare for IPv6? There is a good chance they may still accept the risk of IPv4 depletion 🙂

As an IT engineer, it’s not just our job to keep things running and get projects done - it’s also our responsibility to keep the best interests of the business in mind when it comes to design, implementation, and maintenance. We need to evaluate all our options and always recommend what we think the best path is. However, we also need to respect that our recommendations may not always be followed. There may always be something else going on that we’re not aware of.

One last piece of advice I can give - If there is ever something you feel very strong about, get the final decision in writing. If you truly feel that the risk is too great, and you’ve made your pitch that has still been rejected - get the decision in writing (whether email, chat log, etc). This may seem silly at the time, but it’s better to be safe. People forget, whether intentionally or not - and if the risk becomes a reality, you don’t want to give them any reason to put the blame on you.

Many of the places that I have worked at in the past have been typical 8a-5p/M-F shops. Outside of normal business hours, no one really cared if the network was available. Sure, we might have people who worked late - but a few hours notice via email was always enough. However, the company I work for currently has much tighter restrictions on when work can be performed. We have worldwide customers in over a dozen datacenters and some fairly strict uptime SLAs. What this comes down to is a once-a-month allowance for scheduled maintenance - where the timeframe is limited anywhere from 15 minutes to 4 hours.

Some of the immediate impacts of these differing maintenance window schedules are somewhat obvious. Network maintenance can be practically open to all nights and weekends with a lot of typical 8-5 businesses. This means changes can happen much more frequently - especially changes involving a full network outage. For example, at one of my previous jobs I needed to upgrade each floor of the building from individual Cisco Catalyst 3548 switches to new 2960X stacks. This required moving the cables for  up to 200 ports per floor (while also trying to clean up cable management). I was able to complete the work by just coming into the office earlier every day to move the connections before anyone else arrived.

On the other hand, a cloud service provider can’t just decide one day to take a few hour outage to swap out network equipment. Instead, changes have to be carefully planned, scheduled, then executed within a short window. Customers have come to expect 100% uptime - and rightfully so. However, we still need some amount of time dedicated to performing upgrades, changes, or other maintenance activities. The simple switch migration from the last example suddenly becomes a multi-month ordeal in an environment such as this. You might be ready to jump on the work, but you need to wait for the next regularly scheduled window - and even then you may have only a handful of time to complete your task. If you don’t complete all of it in the time allocated? Well now your project gets pushed back another month.

So you might ask - as a business scales, does it always end up creating this maintenance monolith? It might - but it certainly doesn’t have to end that way. The effects of higher uptime requirements and shorter maintenance periods might seem like nothing but bad news. However, the change in mindset that comes along with that does bring some unique benefits.

The first major benefit comes in the form of planning. When you have 15 or 30 minutes to complete an entire migration or upgrade, it becomes extremely beneficial to plan out a complete play-by-play of every activity. The limited window means that simple mistakes can cost you valuable time. Of course, the tendency for maintenance windows to be scheduled for late nights also compounds the problem since you may be tired or less alert. For critical maintenance tasks that I need to accomplish, I take the time to create a step-by-step checklist of every command that must be run, every system that must be tested, and  every step needed to roll back. Sufficient planning means less mistakes, which in turn increases chances of success during a tight work period.

Automation and efficiency start to become a necessity when you have only a few minutes to perform a task. Sure, I might create a very detailed checklist of what must be accomplished - but what happens if it’s simply too much for the time allocated? You can’t complete a 20-minute task in a 15-minute outage window, right? Sometimes we can schedule extended maintenance periods, but this certainly isn’t feasible every month. This is where we begin to try and identify inefficiencies and tasks that would benefit from automation. Over the years I have written a handful of scripts and utilities that allow for normal maintenance tasks to be completed quickly. These are things that might have otherwise continued to be done manually (and error-prone) without the timing restrictions.

A short maintenance period also encourages more careful network design. If you’re only permitted a half-hour of downtime, then you start looking for ways to minimize the impact. Could the network be designed in a way that allows for a no-downtime switch upgrade or replacement? If not, then how do we get there? In many smaller business networks you might plan for redundancy but never test it - but in a high-uptime environment you begin to relyon it. If you want to get to a point where work can be accomplished with minimal downtime (or even during normal hours), then you must be confident that your network can seamlessly absorb the impact.

I certainly wish some days that I could go back to a life where downtime is acceptable any time during off-hours. However, I’m sure that the desire for higher uptime and greater reliability are likely here to stay - and I believe that I’ve learned some valuable lessons in trying to meet those requirements. An extremely short maintenance period certainly complicates things, but it also forces us to look for process and design improvements. I believe that the end result is a better network for both the business and it’s customers.

What are your maintenance practices like? Do you have hours or minutes? Comment below!

I’m very excited to join the wonderful group of existing Cisco Champions, a few of which who I’ve bugged on Twitter already. I’ve heard a lot of great things about the program, but since it’s my first year I’m interested to experience it for myself. Overall I’m happy for the opportunity and I’m looking forward to what it brings!

Thanks for being here - It’s going to be an exciting 2018!

A lot of people take the beginning of the year to make new resolutions and goals for the coming months. So this year, I’m urging you to add one more to your list: Try and automate something that will make your life easier.

Where to Start

What you choose to automate doesn’t need to be extremely complex or elaborate, just anything that will save you a little bit of time. Never used a scripting language? I can’t recommend enough using Learn Python The Hard Way to start learning. This site is what I used about five years ago to get into scripting. Another great resource is CodeAcademy, where they have web-based interactive tutorials (also check out their specific module on Python and APIs).

Once you get a good handle on the basics, start thinking about repeatable tasks that are great candidates for automation. Start with something simple - maybe a script that prompts the user for information, then generates the command-line entries to configure new switch ports. Then someone can easily copy and paste the commands from the script output to achieve their desired configuration. Something like this might not immediately seem like a huge time savings, but it gives you a place to start and get familiar with what is possible. Once you get something like that working, it’s not too difficult to extend the script later and actually include calls to the switch APIs to automate the changes.

Is Python/scripting your only option? Not at all. There are also automation toolsets like Ansible, which can abstract the code layer a bit. For quite a number of systems that I deploy to an average datacenter, I already have Ansible playbooks written to handle that work. My actual time involved in deploying standard network monitoring applications and tools to a new datacenter went from hours to less than five minutes. For the purposes of this post, I’ll be speaking more to the Python/Scripting side. However, the important point is not necessarily which toolset you choose - it’s the fact that you try to use any one of these tools or others to automate something.

Stick with it

Learning a scripting language at first might seem like a very unnecessary and time consuming task. However, this is something that will pay off in the long run. When I started learning Python, all I wanted to do was parse data from several CSV files and combine the necessary data into one large file. Stupid simple script, but it saved me half an hour each day for a previously manual task.

I’m not at all a fantastic developer by any means, nor would I want to write code for a profession. I just really enjoy problem solving, and sometimes the best way to solve a problem is with a bit of custom scripting. What gets me excited is the process of finding something that wouldn’t normally be possible and knowing that I have the skills and ability to make it happen. Over the past five years, my basic level Python abilities have enabled me to work my way through a number of problems - or write various scripts to make my job easier.

You’ll need to dedicate some time and put in the effort up front to learn a new skill, but trust me it will be worth it.

Look for New Opportunities

Once you have learned the basics, start looking for ways to use your new skills. It’s a different way of thinking in some cases, and will likely take a bit of adjustment. Whats that? Your load balancer doesn’t have built-in reporting functionality to tell you how many server pools you have (and how many are actually fully functioning vs degraded)? Yep, they probably have an API which would be easy enough to pull that data from.

Over the years, I’ve built scripts to automate load balancer configurations, generate reports, alert on BGP peering changes, auto-remediate IPSec VPN disconnects, and even a full IPSec VPN dashboard (since the vendor doesn’t supply one). As a network administrator, having the automation skills in Python has allowed me to accomplish many tasks that my co-workers have stated aren’t possible (solely based on the functionality not being native to a product). Sure, I spend a bit of time up front writing and testing out scripts - but it not only saves me time/effort, but also my peers who I share the scripts with. For example, my team used to have a maintenance task that would take a full hour to complete on a monthly basis. About a week worth of my own effort to write a script, and all of that work is now automated into a 30 second process.

Think about how automation can help not just you, but your whole team.

The Future of Networking

If you follow practically any news source for computer networking, I’m sure you’ve heard this already. Over the next few years the role of a traditional network administrator can and will change. Businesses are evolving more rapidly to meet customer demands, and we need to ensure that our networks can keep up. The only way this is going to be possible is through automation, or hiring an ridiculous number of people.

Practically all major networking vendors are integrating APIs into the new iterations of their device platforms. Some are fantastic, and some are less than ideal - but they’re all working on it. In some way or another these new APIs will become a part of your job - whether you’re writing the code to perform tasks or just using a script written by someone else. Does this mean we have an end to our careers in the future? No - the CLI will take a while to completely disappear. Even if it does go away completely, the role of a network admin will not be replaced, but evolve into something a bit different from what we know today. Even today, having the skills to automate tasks out of your daily job can allow you to spend your time on more important things (like a new network design, or that big project you haven’t had time to look at yet).

You can probably get away with not learning scripting and automation for quite some time yet - but don’t you want to make your life easier and be prepared for the future of your career? I know I do.

One of the things I’ve had problems with in the past is keeping a blog updated. Usually I would start, write an entry or two, then completely forget about it. I never thought I had good enough content to warrant sharing, or I was trying to keep to too narrow a topic. So when I started this blog, I said that I was going to focus on networking but leave it a bit more open-ended. I also wanted to try sharing some more generalized IT experience and career advice. I started off with a list of topics that I wanted to write about, and even began pre-writing a few of them so that I had a bit of content lined up ahead of time.

Even though I told myself originally that I was only going to post something whenever I had something good to share, I still ended up setting myself a goal of writing one thing a week. For a while this actually worked out, because I was forcing myself to think about it more often - but eventually I ran out of immediate ideas. I had to remind myself that it was more important for me to write/post content that was actually worth reading, not just having something available on a weekly basis. Even so, I’ve managed to post 44 items since I started, 40 of which were in 2017 - Much better than I had actually anticipated.

So here is to 2018 - I’m not going to try and set any strict goals for myself in terms of posting content (or at least I’ll tell myself that now). However, I’m also going to try and work on getting better at putting up content. I spend too much time waiting for that ‘great thing’ to write about, and not enough time on just writing something that might not be particularly fantastic - even though it might still benefit someone. I feel like I have a lot to share, and not everyone is an expert. Continuing to think that much of my content ‘isn’t good enough to post’ is just holding me back. I’m going to try and be better this year about this - and not keep waiting for only the ‘great things’ to share.

The other big thing I’ll be focusing on this year is studying for the CCIE R&S, which I wrote about in October. I bought a few books and found some training videos, which I’ve been slowly working though… and when I say slowly, I mean probably much slower than I should be. Now that the holidays are over and it’s a new year, I’ll be pushing myself a more to actually make progress. My current tentative goal for attempting the written exam is June - so I’m hopeful that I’ll be able to make it work.

The blog has been fun so far, and I’ve done a bit more than I thought I would with it. However, there was one thing over the past year that I wasn’t really expecting at all - getting to talk with a bunch of other people who are interested in networking/IT. I’ve mostly been on Twitter, and more recently on Reddit’s /r/networking and /r/cisco. There have been a ton of people I’ve gotten to talk to, get opinions from, or even a few people that I’ve been able to help out with some of their problems. A large portion of my career has been limited to working with just a small team of people, few which actually have much interest in networking. I’ve really enjoyed the experiences over the past year, and I’m really looking forward to what else might come. If you’re one of the people I’ve interacted with over the past year, thank you!

A new year comes with new challenges, problems, and complaints - but it also comes with new accomplishments and new things to look forward to. I hope that all of you reading this are able to set new goals for the year and pass your expectations!

In a lot of jobs, the initial training you receive is fairly straightforward. You are usually taught how to respond to a task by following a series of steps to get an intended result. Training like this is great - It helps to achieve consistency and efficiency. You bring in any new person and give them the exact same troubleshooting steps, implementation steps, and/or validation steps - and you’re likely to get a similar result every time.

This is the point where I have seen far too many people stop though. They are happy with doing their job, and don’t necessarily want to progress their career or maybe they don’t know how. These engineers will continue to produce decent work at the quality that they were taught at. Even for those who try and progress further (maybe through certifications or otherwise), there is a difference between learning new technologies/concepts and truly understanding them. For some people out there, having this basic level of skill is all they really want - and if that’s their goal in life, then this type task-based knowledge is perfect. But if you really want to master the domain of technology that you are interested in, then you need to put forth the time and effort into gaining that understanding.

For me, a true understanding of a technology means that you’re able to speak confidently about how something works, abstract concepts to apply to similar products, and mentally walk though how the technology might handle a given situation.

Let me provide an example or two that might help to frame this a bit better. Given a particular network, an engineer might know that for traffic to get from point A to point B, it travels through two firewalls. Every time there is a new request to permit a new traffic flow, that engineer knows that they must make a configuration change to one or both of those firewalls to allow that traffic through. However, to this engineer, the inner workings of that firewall are a complete mystery. The firewalls are complete black boxes which take in traffic through one interface and spit it out another. So when there is a technical issue within the firewall appliance, they may be extremely limited in their troubleshooting abilities - and they may have no choice but to call the vendor for support.

Another engineer who has a deeper understanding of how firewalls work might see the problem differently. This engineer knows that for every packet received, the firewall follows a specific flow of processing. That flow could include any number of things, including NAT, routing, firewalling, IPS, VPN, etc. This engineer knows which order those things get processed and what effects those processes can have on the traffic. So when we have a technical issue with this firewall appliance, this engineer may be able to mentally walk though the packet flow/processing and determine where the problem may be - sometimes without even looking further than general log files.

Another example is something that I see quite often. An engineer is asked to implement something - let’s say a new port configuration for a server. They follow their known process for implementing this change, but something doesn’t quite work right. So they change settings or maybe delete the entire port configuration and start over - but eventually they get it working. They don’t know why it didn’t work the first time, or what caused it to work the second time - but it works now, so they aren’t concerned with it. However, it’s possible this engineer ends up running into this same problem more than once. The ideal step here would be to step back and look at what was different between the original configuration and the working configuration. Maybe there is an additional command in the original configuration which seems suspect - a quick search of the internet could turn up an explanation behind why that command was preventing the port from working as expected. After that research, that engineer would not only know why their configuration didn’t work - but now they know what that command actually does, which could be beneficial in a future scenario.

As I briefly mentioned earlier, not all IT admins or engineers are concerned with gaining a significant level of understanding. There are those who want to come to work, get their job done, then go home to their families - and there is absolutely nothing wrong with that. For me personally, I can’t handle running into a problem and not knowing exactly what the cause was. An issue that “fixes itself” is never an acceptable answer to me, because if something caused the problem once then it can certainly happen again. I don’t enjoy having to blindly configure an option on a system without knowing what’s going on in the background. Some people might call me crazy, but this seems to be a skill/trait shared by many higher-level engineers I have worked with. So how do you get to a point where you really understand a system? For me, it’s been a lot of playing in labs, reading vendor documentation, and not settling until I feel like I can speak confidently to how something works. I never feel truly comfortable in a new company until I can mentally walk through every device a packet touches from source to destination - and know which devices configs/routes may have an impact on that flow. Any time there is a problem with something, I spend time digging into it until I know what caused it - even if the problem is only momentary and goes away. Not only do I then understand why the problem happened, but I also learn how to quickly identify similar issues again.

Especially if you’re still in the beginning stages of your career, I can’t stress enough how important it is to understand the technologies you’re responsible for. Take the extra time and study it, play with it, break it and fix it again. Know how things work and what their behaviors are under different conditions. Don’t settle for “It just works because it does”. One of the key skills I’ve seen in engineers who truly understand their domain of technology, is the ability to abstract concepts to apply to other systems. Someone who has a deep understanding of routing and switching technologies might prefer to work with a certain vendor, but given any router/switch they can make it work.

Have you worked with anyone who you think has a great understanding of what they do? What other skills or traits do they display that makes them successful? Comment below!

Anyways - after I cut over to my new firewall, I’ve been digging through logs to make sure that I didn’t miss anything. I have all of my device/lab logs going into an instance of Splunk Light (their free product). It makes it easy to collect and search through logs, and it’s extremely easy to set up and use. A few quick queries and I came across one or two minor things that needed to be tweaked on my firewall - but I also saw some traffic that I wasn’t sure about.

So that brings me to my question of the day: Do you know what’s going out of your network?

A lot of people I know only use firewalls to block inbound access, both in homes and businesses. For homes it’s more understandable since most average people aren’t network admins. However, it still surprises me how many businesses are willing to add a ‘permit any any’ out to the internet. Yes, I block all traffic by default through my home firewall, both inbound and outbound. Yes, it’s a bit of a pain sometimes when something isn’t quite working right - but it’s usually a quick ACL change, and overall I would rather take the minor inconvenience for the security gains.

When I originally built the firewall policy for my network, I started off simple. I know we need DNS, HTTP, and HTTPS outbound - easy enough, right? Then I started watching logs for blocked traffic and trying to decipher what else was trying to communicate outbound using another port. Some things were very easy to determine - TCP 5228 out to a Google owned IP? Yep that’s actually a known thing - a lot of Google services, like Chrome, will use this. Some other things were harder to figure out - like game consoles which use a very wide range of non-standard ports. Many of these weren’t really documented well by the console manufacturer, and meant that I spent a while between browsing forums and some trial and error.

This really gets interesting when you start digging past the stuff you know about. What about a PC in my home network that is trying (and getting blocked) to reach a few random IPs in Korea and Russia over a bunch of non-standard TCP ports? Yeah that doesn’t make me feel comfortable. Could it be a legitimate application, or is it malware? A few quick searches on the internet don’t turn up anything immediately helpful. For the time being, I’ll keep stuff like this blocked until I have time to spin up some packet captures to see what this traffic is actually doing.

For a business I feel like this type of thing is even more important than just what I’m doing at home. You certainly don’t want end users (or servers) possibly running strange applications, which might be transferring data to some unknown external party. It seems like larger companies seem to have a better handle on restricting outbound access than most smaller companies, who likely don’t have the time or see the value. However, I’ve also worked with a few larger organizations who still permit all user and server traffic out to the internet with no filtering in place.

If you’re not already blocking outbound traffic - Get some good logging in place. Use something like Splunk Light and start collecting firewall logs for everything going out of your environment. Start with the basics - create a list of the software/ports you know you’ll need to open. After a few weeks, start digging through the logs to figure out what else might need to be added to your list. Once you feel comfortable that you’ve compiled a sufficient base ruleset, schedule a time to make the change and put it in place. Start blocking the unknown traffic - and only permit when necessary.

How do you have your firewalls configured today? Do you permit everything or are you very restrictive? Comment below - I’m curious to see what other people are doing.

I’ve been considering the idea of posting some short reviews of products or services I use. Not at all meant to make this a review site, but just a bit of “here is what I use and what I think of it”. This is going to be the first shot at that idea - so let me know what you think!

When I was first looking to start up my own personal blog again, I did quite a bit of digging around to try and find where to host it. I considered hosting it myself at home, but overall I didn’t really want to manage that overhead. I also looked at a number of hosting providers, but I never really found anything that I was quite happy with. So many of them would offer you an instance of a web platform like WordPress or Drupal, but not much control beyond that. I really wanted something where I could have full control over my own VM, like a VPS, but most of these services cost more than I wanted to pay.

Right in the middle of my research for a hosting provider last year, Amazon announced a new AWS service offering: LightSail. I had already briefly considered picking up a small AWS VM for what I wanted to do, but I honestly didn’t want to try and manage usage when every single little thing is an additional cost (storage, bandwidth, etc). However, LightSail really caught my attention because of the stupid-simple pricing structure - the smallest allocation is only $5 a month and includes 512MB of RAM, 1 core CPU, 20GB SSD, and 1TB of free data transfer. This was probably more than enough for me to get started with a site, and the pricing was very tempting. As if they wanted to provoke me to try it even further, Amazon actually offers your first month free (for the $5/mo instance). Additionally, Amazon offers up to three free public IP addresses and free DNS management along with every plan - which I thought made this a fantastic offer.

With all that LightSail seemed to offer, I decided to jump on the $5/mo plan and use the free trial period to give it a shot. The setup of my first VM was extremely simple and only a few clicks: Pick a size, give it a name, and select which applications you want pre-deployed (optional). I went ahead and selected my options, and within seconds my VM was already powered on, pre-configured, and ready to use. Unfortunately, it seemed like my VM deployment didn’t quite go as smoothly as I would have liked. I wasn’t able to log into the admin console for my pre-deployed web application, and the deployment guide didn’t seem to match up with what I was seeing on my VM. I spent about ten minutes or so trying to troubleshoot this before it hit me: I could just delete and re-deploy this VM in seconds. So I did exactly that, and within 2 minutes I already had another VM instance deployed - and this one worked perfectly.

Next I went ahead and applied a static IP to my instance. You don’t necessarily have to do this, and if you host your DNS within LightSail then it will automatically update your DNS any time your instance IP changes. However, since they offer up to three static IP addresses for free, I don’t really see a reason why you wouldn’t use it - unless the VM was temporary or not being published publicly. This process is also extremely straightforward: Click the button to create a new static IP, give it a name, and select which VM to assign it to. Easy enough, right? I also tried out the DNS hosting for only a very brief period of time. I ultimately ended up opting to move my DNS hosting out to CloudFlare, which I would like to cover in another post.

Once the LightSail VM is up and running, it’s easy enough to connect to it. They offer a web-based SSH console, and by default allow access to ports 443 and 22 for remote connections. Remote SSH connections are handled by public/private key pairs only - no username/passwords permitted for login. I actually prefer this, and there is an extremely simplistic interface for generating new or additional SSH keys and automatically configuring them within your VM. The management console also offers an easy-to-use firewall system, where you can open ports for common services via a drop-down menu. You’re also able to enter custom port numbers, or remove any/all open ports entirely. As a quick note: LightSail offers no traditional console access to your VM - so if you close port 22, then you won’t even be able to manage the VM from the web console since it uses SSH. For me, I would rather take the additional security step to only enable that port when I absolutely need to access the VM via SSH.

So it’s been almost a year since I started using Lightsail and overall I am extremely pleased. I’ve run one primary VM since I opened my account, and I’ve spun up a few additional VMs here and there for different testing. It’s easy enough to just turn up a new VM, try it out, and then just purge it if/when you don’t need it any more. The billing reports are very straightforward too - and I’ve so far never come close to using all of my 1TB free data transfer. The VM itself is extremely snappy for only a single core with 512MB of RAM.

Overall I would highly recommend giving LightSail a try, even if you only use it for the 1 month free trial. I’ve been very happy with the service so far, and I’m looking forward to any new features/functionality that might be added in the future.

As a quick summary:

Benefits:

• Extremely easy to use (Simplistic interface)
• Three free static IPs
• Free DNS management
• Up to 1TB of free data tranfer
• First month of a $5 instance is free
• Pre-deployment/configuration of multiple different web applications

Drawbacks:

• You can’t move which zone your VM is deployed in
• You can’t rename your VM instance
• In order to upgrade plans, you have to take a snapshot and restore it to a new VM - and this is currently only available via the AWS APIs, so no way to do this in the web portal
• Snapshots take forever, and you get charged for how long they are stored for (I’ve had a snapshot of a 20G VM take >30 minutes in the past)

My wishlist:

• An integrated load balancer - even one with very basic options available
• Native support for IPv6 addresses - Amazon already offers a free IPv4 address, so why not v6 too?
• More storage options - I am careful about how many images i upload to my site, because I only have 20GB of space available. I don’t want to pay for S3 if I don’t have to.

UPDATE 12/19/2017- As of the end of November, Amazon has added both Load Balancing and additional storage to LightSail. Currently the load balancing functionality costs  $18 per month, and additional disk space is $0.10 per GB per month.

Have you used LightSail before? What are your thoughts? Comment below!

Over the past few weeks, we have been lightly troubleshooting an issue between a VMware vCenter server and the ESX hosts that it manages. ESX hosts were randomly showing up as disconnected for a brief moment, then would reconnect. It was nothing extremely impacting, but a mild annoyance for the server guys. A couple of people on my team had taken a quick look on the network side, and turned up empty handed. Due to some upcoming maintenance work the server team needed to perform, I was asked to spend some time trying to isolate the root cause of this issue.

First thing was digging through the logs from the two different sets of firewalls between these systems. The first firewall set showed that traffic was passing normally as I would expect. However, I started seeing some unexpected logs for the second firewall set, a CheckPoint cluster. The logs showed that vCenter was opening connections out to the ESX hosts for a short while, then the CheckPoint would log a “TCP Packet out of state” error.  The details of this log would show that vCenter sent a non-SYN packet to the ESX host (usually a PSH ACK).

Seeing an error like that indicates that something is killing the TCP connection before vCenter is finished using it. vCenter still believes that the connection is open, which is why it sends packets with incorrect flags. Since we were already aware that this particular CheckPoint cluster has some issues, we began examining this cluster first. Sure enough, the IPS logs on the device showed that the cluster was often reaching >80% of it’s maximum concurrent connections and then enabling the “Aggressive Aging” feature.

Aggressive Aging is a CheckPoint protection which prevents the cluster from running out of memory and potentially crashing. By default, this is set to take effect whenever the cluster exceeds 80% of it’s available memory or concurrent connections. This protection will continue to be enabled until the cluster drops below another threshold, which is below 78% by default. Seems like a helpful feature to have, right? Yeah - but there are some considerations with how this protection works. When Aggressive Aging is activated, the cluster significantly reduces all of the normal TCP timeout values. For example, CheckPoint’s documentation shows that new TCP sessions are given only 5 seconds to establish, instead of the normal 25 seconds. This also changes how long a TCP session can be open from 1 hour to 10 minutes. In order to help drop below the 78% threshold, Aggressive Aging will evaluate and terminate 10 connections for every individual new connection that is established.

As I stated previously, this cluster was already pretty busy - often hitting CPU limits mostly. However, through the brief research I completed, it looks like increasing the concurrent connections table mostly affects RAM utilization more than anything else. This system has over 20G of RAM and is typically only using around 4GB. I was still concerned that an increase in total concurrent connections could mean more CPU usage, because that means more connections for the IPS to process. Unfortunately, CheckPoint has no publicly available utilities to help calculate what to set your max concurrent connection limit to. In fact, when I opened a support ticket with them, I was told to “just keep increasing it, until you hit a point where the cluster is no longer triggering Aggressive Aging. Then add about 10-20k above that to set the new maximum concurrent connection limit”. That’s not really an acceptable answer to me, but I wasn’t able to get anything more out of them.

So in order to change the maximum concurrent connections (Using R77.xx), you need to open SmartDashboard and open the cluster object. Then find Optimizations in the left-hand menu. Here you can set a new manually-defined limit, or allow the cluster to automatically scale the maximum connections. If this cluster was significantly less busy, I might be tempted to enable the automatic limit for a bit and try to get a baseline. However, I would rather not open myself up to the chance of crashing the cluster - so I manually increased the limit from 25,000 to 50,000. Install the policy for the configuration to take effect. You can see the current concurrent connections by either looking at the Overview page in SmartDashboard, or logging into the cluster CLI and using the cpview utility.

In my case - the new connections almost immediately started ramping up to ~35,000. Within a day we started encountering the Aggressive Aging protection again, but it was happening significantly less often than before. This also resolved our ESX host disconnection problem, which proved my theory that the Aggressive Aging feature was causing our problem. I’ve been slowly monitoring and increasing the concurrent connections limit since, and I think we have finally stabilized around 90,000. Just think of how many connections were denied or terminated early because this limit was in place!

Moral of the story here: Understand the systems that you own. This firewall cluster had been in place years before I was hired, and all of the settings were left at their defaults. Default settings probably work for most cases, but they also come with their own problems. This setting had likely been the cause of multiple problems in the past, however no one truly understood they system enough to find out what was happening. Ever have a scenario where a default setting caused problems? Share it in the comments!

When I got my CCNA certification, I was only two months out of high school. I had just finished two years of the Cisco Networking Academy coursework, and I had no idea what that would actually mean for me. I went and took the certification exam mostly because I felt that like that was the only way to validate what I had learned during the two year class. I failed it once or twice, which nearly discouraged me enough to not try again. However, I ended up passing the test and becoming Cisco certified on August 27th, 2007.

Obtaining that certification didn’t immediately make me valuable to anyone. However, it definitely helped to get my resume in front of a number of people who probably wouldn’t have taken a look otherwise. At the time, I had no college degree and absolutely no real-world networking experience. I owe that CCNA cert for helping me get my first job - but after that it was up to me to prove my worth.

It’s amazing to sit back and realize that ten years has passed already. So much has happened, so much has changed. I spent the first three or four years of my career studying hard to additional Cisco certs, which I used as motivation to learn more about networking. Certifications can be great for validating what you know, but it’s the real-world skill that really pays off in the end. Even with my original intent to become a network admin, I’ve ended up wearing a lot of hats and picking up more of a variety of skills than I ever thought I would. It’s definitely been a good thing though, since it has allowed me to get a better understanding of other systems - which in turn helps me to better support them as a network admin.

Even though today I don’t really manage much in the way of Cisco equipment, the original skills that I learned in the Cisco Networking Academy program have given me a great base knowledge to work with. All of the fundamental networking skills I learned have translated quite well to other vendors and products. I’ve spent the past few years working with Brocade, Juniper, Check Point, and a number of other vendors - and I feel like I have had a much easier time picking up the new skills than I might have had otherwise.

Today I still hold and maintain my Cisco certifications - and I plan to continue doing so for the foreseeable future. Someday I would like to achieve a CCIE/CCDE-level certification, but for now I am happy with what I have and what these certifications have helped me to achieve in my career.

Thanks for reading - here is to hoping for the next ten years to be just as good as the last.

A few years into my first IT job, and I worked with a great guy who took the time to teach me some critical resume skills. The things he taught me seem like common sense now, but at the time they seemed to do magical things for me. I actually had a few recruiters comment on how much they liked my resume - which seemed pretty amazing.

So what I want to share today are some of those tips that I was given, including some things I personally look for when I’m interviewing people.

1. Formatting, spelling, and grammar

I have to put this first because it is absolutely critical. For me personally, I am very disappointed when I see a resume with bad formatting, misspelled words, or cringe-worthy grammar. Your resume doesn’t have to look like a work of art, but don’t just include massive blobs of text. Break it up and create a nice simple layout. Don’t forget to use spell check - it’s already included in most word processing applications.

Grammar can be more difficult - again, I don’t expect perfection, but have someone else read through to ensure everything makes sense. I’ve literally received resumes before where they didn’t use any punctuation. Everything on your resume needs to maintain the same tense, and I prefer it to be past tense. Every point on your resume should reflect work that you have done - so you want to speak about it as a past experience.

2. Use clear, concise bullet points to describe your work history

I’ve come across a lot of people who type an entire paragraph under each job. It’s harder to read through quickly, and it just doesn’t look good. Remember when I mentioned formatting? Use bullet points.

Each bullet point should be kept to a maximum of one line. The occasional two-lined bullets are acceptable, but try to keep them to a minimum. Usually, if you need more than one line, then your point may be too wordy and you may need to re-phrase your statement. Each bullet should be a contained statement of your work experience. For example: “Managed Cisco ASA firewalls across five business locations”. From that single statement, I should be able to get an idea of what that means.

3. Keep it under two pages - but keep the relevant details

Maybe this is again personal preference, but I don’t really want to look over a resume that’s longer than two pages. When you only have a single job to put on your resume, it’s important that you put a lot of detail about what you’ve done. In this case, it’s acceptable to fill most or all of a page with your sole point of job experience. However, as you start adding more job history, the detail listed for your older positions should be stripped down to the most important points.

Always keep your most recent experience first!

4. Don’t re-write your job description - focus on your individual achievements

I’ve read too many resumes where the job history read like a list of job postings. Mentioning some of your job responsibilities is fine, but it shouldn’t be the entire thing. As someone reading your resume, I don’t want to read about what you’re supposed to do in your job role - I want to read about the things you’ve accomplished.

As an example to this, someone might put on their resume “Monitored security logs for anomalous events”. That’s great - but I’m not getting a sense of this person being self-motivated. Their job was monitoring logs for events, so they did exactly that. But what if this person was someone who really made something of that job duty? Then it would be better if they listed an actual individual accomplishment, like “Mitigated major threat to the company by identifying early indicators of security breach” or even something like “Refined security log review processes and mentored new employees on performing thorough review”. These statements give talking points - both something for me to ask about, and something for you to show your skills and pride of your work.

5. Tailor your resume to the job you want

When I started my career, I worked for an IT consulting company. As a result, I had an extremely wide range of experiences. Everything including help desk, network admin, storage admin, Windows sysadmin, antivirus admin, VoIP admin, etc. My resume from my first job was a mess - mostly because of the amount of roles I had to fill. If anyone looked at that resume, it would be impossible for them to tell that I really wanted to become a full-time network engineer.

My point is that you should revise your resume to fit what you’re looking for. If you have a strong drive to specialize in server hardware and Windows administration, then your resume shouldn’t have a ton of detail about your experiences with database administration. You don’t have to remove that experience completely from your resume, but your resume should carry a theme surrounding the job you want.

If you want to take this a step further - revise your resume for the job you’re applying for. If they’re looking for significant experience in vulnerability scanning and analysis, then make it a point to highlight your experiences that match that. Maybe you have a bit of experience with Nessus, but it’s not typically something you call out much on your resume. If the job posts that the position requires a expert knowledge of Nessus, then you definitely want to make sure your knowledge is immediately visible to them. Remove some less important bullet points, and add in a few more that pertain to the job role.

6. If you put something on your resume, be prepared to defend it

I love this from both the standpoint of an interviewer and an interviewee. Resumes are essentially a quick summary of your experiences, and you should be expected to be called out on anything listed there. Never put something on your resume that you can’t speak to.

As an interviewer, I like to find things on someones resume that they’ve claimed experience or expertise with and ask them about it. Especially people who put a list of technologies they’re familiar with. I don’t expect you to be able to answer extremely in-depth questions about every single thing on your resume. But more often than not, I ask “Okay, so I see you’ve listed experience with BGP and MPLS on your resume” and the response I get is either “Yeah, I know what those are” or “We used it at my last job (but I had no direct experience with it)”.. You don’t want to find yourself in this situation.

As an interviewee, I’ve had this same thing happen to me multiple times - except that these turn into ways for me to speak about things I’ve done with those technologies. It makes a good impression if you’re able to quickly recount knowledge or experiences of any random thing listed on your resume.

7. Keep it updated

IT is fast paced. Things change - and you would be surprised how much you accomplish in six months. You probably won’t be surprised at how quickly you forget everything that you did in that six months though. For that reason, you need to make it a point to keep your resume updated - even if you’re not looking for a new job in the immediate future. That way, you make sure that you remember your recent accomplishments to add onto your resume - and you can also take a moment to review and remove older/less relevant items. I would recommend doing this every six to eight months.

You never know when you’re going to see a job you want, or get contacted by a recruiter for a perfect opportunity. Don’t miss your opportunity because you need time to update your resume. Keep it updated. I’ve known a lot of people to say “Oh yeah I’ll apply for that, I just need to update my resume first” - but then they never do.

8. About objectives or mission statements

Alright, one last thing. Some people put objectives or personal mission statements on the top of their resumes. Personally, I don’t like them at all. I’m okay with them if you are a young professional who is looking for their first or second job. After that - I don’t want to see it.

Also - If you feel the need to put an objective at the top of your resume, please do me a favor: do not put “I’m working to become a ”. I would say that about 80% of the resumes I see with objectives look exactly like that. I understand that you want this job, otherwise you wouldn’t have applied for it, right? So if you absolutely have to put an objective, make use of the space - tell me where you see yourself in five or ten years, or what your ideal position is. It means a lot more to me if you put something like “Working to gain knowledge and expertise as a network admin, with the goal of becoming a senior network architect”. Then I see that you have goals, and you’re working toward them.

Okay! That’s all I’ve got for now on this - but I hope that it helps you if you’re reading through this. I owe a lot of my resume-writing knowledge to the guy who originally helped me seven years ago, but I make it a point to pass along the help whenever I can.

If you have any other tips, feel free to share them in the comments below!

So now that I’ve had a bit to sit back and take a break, I’m starting to begin itching toward certification studies again. I really enjoy certifications because they give me a goal to work towards, and I can study the materials at my own pace.

For reference, I currently hold the following active certifications:

Cisco: CCNA, CCNA Security, CCNA Voice (retired), CCDA, CCNP, CCDP

I’ve been looking a bit at the Cisco Cloud and Data Center certification tracks, since I’m dealing a lot more with the Nexus switching line and data center technologies overall - but after reviewing the cert syllabus, I’m not really feeling very strongly toward those. I’m also hesitant because it would mean starting back over at the CCNA-level for the new tracks and working my way back up to the CCNP-level. I’ve also previously considered getting my CCNP Security, but I’m not actively working in Cisco ASA firewalls much any more.

The only next choice in the Cisco world would be going for the CCIE R&S or the CCDE. I’ve been considering for a long time that I would eventually like to get there, but those certifications also require a significant investment of time and money. I definitely think the information and skills I would learn along the way would be worth it, and I’m beginning to really consider this an option in the near future. I’ve spent a bit of time reviewing the exam topics listed on Cisco’s site, and debating which of the two would be a better first choice.

My other option is pursuing the Juniper side of things. Most of the data centers I manage now are shifting toward Cisco for switching and Juniper for firewalls - so it would certainly benefit me to educate myself further on the Juniper equipment. Until this point, I’ve been just learning on the job by buying Juniper SRX firewalls and figuring it out as I go. My only real hesitation on this would be maintaining two separate lines of certifications. Both Juniper and Cisco enforce a 3-year expiration on their certifications, so I would need to keep on top of both - which isn’t necessarily a bad thing.

So at this point, I really don’t have a clear idea of what I want to do. Those are my current thoughts and options, but I’m having a hard time settling on what would be the best option for me at this time. I definitely want to start studying for something (and potentially take the exam) before the end of this year though, so I would like to figure it out rather soon.

If you have any suggestions or thoughts on the certifications I’ve mentioned, leave me a comment below!

Starting off as a less experienced network engineer, this can easily get overwhelming. A ton of other teams blaming the network infrastructure for problems that might be entirely unrelated. However, it seems like a good part of your job will likely be dedicated to proving that the root cause of the problem isn’t the network.

I threw together a few tips  that should help you get a better start to defending yourself against the angry sysadmins out there:

Get a good handle on the problem - Above all else, it’s extremely difficult to troubleshoot something without a good description of the problem - so get that first. Which systems are having the problem? Get host names or IP addresses. When did the problem start? Did it ever work? What behavior is being seen? Can it easily be replicated, so that you can watch logs in real-time? If not, get the last date/time that the issue occurred.

Know your infrastructure - Being a good network admin means understanding traffic flows and routing through your infrastructure. When someone says two systems are having a problem communicating, you should already have a good idea of what network components reside in the space between them. Are these on the same network segment? Is there a firewall (or multiple) in between? Does this traffic utilize a proxy or load balancer?

Check logs - Once you know which systems are in the way, check through logs for those systems. Particularly with firewalls, do your best to filter out logs to see the traffic calls between each system. Seeing ports blocked or traffic being dropped? A lot of firewall platforms will include enough detail in the traffic logs to quickly identify the issue, if it is in fact a network problem.

The basics are still important: Check TCP flags - This one has honestly saved me more often than not. Two systems aren’t establishing a connection, and the sysadmin says they are just receiving a “connection timeout” error. Check through the firewall logs - Yeah, we see the typical TCP handshake - but then the remote system sends back a TCP RST to the client. In most cases, this means the connection is actually succeeding from a network perspective. However, the target system is getting something that it doesn’t like from the client, so the application kills the session. Same thing goes for a client system sending the RST.

Wireshark - A lot of people see this as the nuclear option. All else has failed, so we have to resort to a packet capture. I used to think this way too until about a year or two ago. The vast amount of information within a full packet capture can easily be overwhelming - but once you get a handle on how to read it, it can also be incredibly useful. Raw packet captures don’t lie - and all the information you need is within those details. Start a capture, reproduce the issue, then analyze the results.

Be patient, and explain your defense - Not everyone is a network admin, and a lot of IT professionals don’t necessarily have a good grasp of how networking truly works. So once you’ve gathered your defense, be ready to explain it in a way that the other party will understand clearly. There is a huge difference from saying “I see TCP RST packets” to trying “Looks like the connection is succeeding, but the server-side system is resetting the connection”. Some people won’t want to admit that the problem actually exists with their system either, so be patient and work with them while they figure it out.

Bonus: Know the application - In some of my previous jobs, I was responsible for all systems and applications in the environment - even through I was primarily focused on networking. This experience has helped a ton, because even today I can still speak to how some applications work. I have installed and configured VMware ESX, Windows Server, backup and replication products, and much more. So when an application administrator says they are seeing a particular issue with something, I am more easily able to troubleshoot since I have a basic understanding of the applications they’re working with and how those applications communicate. This certainly isn’t a required skill - but it does help speed up troubleshooting efforts and minimize confusion around what’s going on with the application.

Have any other tips you would like to share? Throw them in the comments below!

Growing in any profession means making mistakes. It’s inevitable, and everyone makes them. The critical point is realizing when you’ve done something wrong, fixing it, and learning from it. Ever meet someone who seems to just be great at everything they do? Maybe it seems like they never screw up? That’s because of years of experience and learning from every mistake they’ve made.

So a few things I want to get out there:

1. You’re allowed to make mistakes - Like I mentioned above, everyone does it. Perfection can be a goal, but it’s not realistic. Stuff happens - sometimes something may get messed up because of unknown circumstances, or maybe your own carelessness.

2. Admit fault - One of the most important things about making a mistake is admitting that you did it. There is nothing worse than people who try to hide their mistakes or blame other people - there is just no sense in it. There is a lot more respect for people who are able to admit their mistakes to others, yet this seems to be a fairly rare quality in my experience.

3. Figure out what went wrong - Whenever you make a mistake, you need to own it. Take responsibility for hunting down exactly what went wrong and determining how to fix it. It’s possible that the issue was inevitable, and something you couldn’t plan for. However, in many cases a lot of simple mistakes are preventable by just exerting extra care, planning, or taking the time to do better research.

4. Fix it - This goes along with what I said in #3 - but take ownership for your mistakes and fix them. Don’t pass them off to other people. Don’t pretend you don’t know whats going on. Just say “I screwed up, but I’m going to fix it”.

5. Learn from it - After everything is done, take a step back and look at everything that happened. What can you learn? Everyone hates that sinking feeling that something just went wrong, so why not try to prevent yourself from having to experience that again? Take note of what could have prevented the issue. Make sure you never make that mistake again.

6. Share the knowledge - The only thing better than learning from your own mistakes is being able to learn from others, so that you never make the same mistakes. If you’ve done something that is worth sharing, then do so. Take a few minutes to sit down with your team and explain the scenario - what went wrong, how it could have been prevented, and what lessons you’ve taken away from the incident. You have the power to help less-experienced people to learn how to be better - make the right impression and show them that it’s okay to admit fault.

A lot of this may seem like it should go without saying - that it all should be common sense, right? However, in my experience it seems like a lot of IT admins are more than willing to try and hide their own mistakes, because they think it will make them look bad. In my opinion, hiding your mistakes makes you look far worse than having the ability to admit fault.

In one example, I recently worked with an individual who immediately began blaming the network team for an issue he was experiencing. They claimed that the firewall was blocking their communication between two systems where they were trying to install an application. The individual started complaining to management that the network team was holding up their progress because of an incorrect firewall configuration. The network team did everything they could to help troubleshoot - but ultimately it seemed like an application issue. The next day, the admin just came in and said “Oh, I don’t know what you guys changed but it started working today - so thanks!”  Did the network team change anything? No. And it was found out later by another admin that the application was mis-configured the whole time. This guy lost a lot of respect from both management and his peers - just because he couldn’t admit fault.

As another example, I had a previous co-worker once attempt to remove software from a production database cluster in the middle of the day. Of course, this happened to be our busiest database cluster, which ran the backend for a vast majority of our customers. When the software was uninstalling, it dropped all network communication to the database cluster - which forced all customers in the data center offline. The second he realized that something was wrong, the IT admin informed his manager of what he had done. He took ownership and led the issue until it was resolved and customers were back online. The next morning, he made it a point to give a quick summary to the entire IT staff of the issue - and admitted that he screwed up. While it doesn’t change the fact that he made a huge mistake, this approach led to a much quicker issue resolution. He would have been in far more trouble if he had tried to hide the fact that he caused this outage event.

What’s the biggest mistake you’ve made in your IT career? How did you handle it? Share in the comments below!

I talked about this briefly in my initial background story posts, but I went straight from Cisco Networking Academy in high school out to working a full time job at a local IT consulting company. By the time I finished high school, I had already passed the Cisco Certified Network Associate (CCNA) exams and become certified. Having that certification is what got me in the door for a number of interviews, and eventually got me the job at the consulting company. At that point, I really didn’t have much else going for me - I didn’t have a college education nor any real-world experience. In my time working at this company, I spent a significant amount of time doing self-study and labs for my certification goals. When I got my CCNP certification, I used it along with the experience I had gathered to get my next job. This new employer was heavily focused on their IT staff needing to have a college education - so they pressured me for a while to go back until I eventually gave in.

I spent a while reviewing many colleges in the area and online, trying to figure out what would meet my needs. I ended up picking out a four-year degree in network security, and opted to go the online-only route because it benefited my schedule better. I packed my classes up to a full-time schedule, because I didn’t want a four-year degree to take any longer than four years. At this point, I also had the benefit that my employer was willing to reimburse 100% of the costs - which certainly helped convince me to go back.

Over the course of the past four years, I have taken many classes that include general IT, development, networking, and security (not including the normal required materials). I found that a significant portion of these classes didn’t directly benefit me. A lot of the material was much more focused toward beginners who haven’t already been working in the field for six years - which is completely understandable. The most I really got out of this was improving my abilities to push myself through work that I didn’t want to do. I did have a few interesting classes, like an Android development course, which I found to be extremely fun even if I probably won’t use the knowledge much.

Four years later and I’m done - did I benefit from it? On some level yes, I think I did. At the time of my degree completion, I have now been Cisco certified for ten years and I’ve been working in networking nearly the same amount of time. I’m already further in my career than I thought I would be at this point, and I’m happy with my position and pay (the degree isn’t going to change either of these things). At this point in time, finishing the degree is not much more than an accomplishment that I can add to my resume. Sure, having the degree on my resume may get me past HR screening for new jobs and opportunities - but it likely won’t actually play much into a company’s decision to hire me.

In the end I think that both certifications and college education are useful - they can both be great indications to an employer that you’ve been trained on certain technologies or fields. However, I think that the actual on-the-job experience is what really matters - and I experienced a direct benefit from getting in the field early and working while all of my friends were still in college. I would not be as far in my career as I am today if I had waited four more years to start working.  Unfortunately, I think that we place a little too much importance on completing a formalized degree program, when equivalent experience and certifications may benefit a company more.

I understand that I had a bit of a unique situation, but I figured it would be worth sharing my experiences and how they have affected my view of college education. I’m still happy that I went through with it and completed the degree, but you won’t see me throwing a big celebration - except that I’m just super glad it’s all finished. At this point, I will take a few months to relax and spend time on hobbies - but I do plan on going back to certification studies (Juniper stuff and likely begin working on a CCIE).

Any thoughts? Comment below with your experiences - I’m interested to see if there are many people who have had similar experiences to me, or possibly even the complete opposite.

1. Identify the problem

This can be the both the easy part and the hard part. For example, let’s take a recent example at my job: Poor coordination across teams for project work. Awesome, we have identified the problem, right? Well, not quite - that may be a high-level summary of the problem, but why is there poor coordination? Maybe the teams aren’t meeting often enough with each other, or maybe those meetings aren’t structured in an effective way.

2. Identify the solution

So it’s easy for anyone to say “I FOUND A PROBLEM”, yet it’s more difficult to come up with a reasonable and effective solution to that problem. Sit back and evaluate the problem, but make sure you consider the perceptive of both sides. Maybe Team A works better if they have all of the information up front, so they can design a proper solution - yet Team B likes to work as they go, and tackle things as they come up. In this case, we probably won’t get very far in asking Team B to schedule architectural/design meetings before starting a project, will we?

3. Propose the solution

This part is important, because we don’t want to start making changes without anyone understanding what or why we are doing it. If changes come out of no where, people are more likely to reject them. So maybe we sit down with Team A and Team B and explain our solution: We will hold a quick, high-level design meeting at the beginning of a project - but Team B will be responsible for trying to notify Team A  as soon as they identify a new requirement, and Team A will be responsible for identifying when those new requirements warrant a bigger meeting to gather requirements/details.

4. Make the change

This is probably going to be the most difficult step. If you want the change to happen, you cannot stand back and hope that someone else does it. You have to lead the change. For example, if you are on Team A and you propose this idea, then you must hold Team B accountable to sitting down for a requirements-gathering meeting when you think one is needed. Give it a good effort, because if other teammates see you working hard to make working-life better for everyone then they will be more likely to join in.

5. Re-evaluate and refine

No one is perfect, and no idea will ever be absolutely perfect on the first try. So give it a while, then make sure you sit down and take a look at how this change has impacted everything. Has Team A been more productive, since they can get requirements earlier in the process? Has Team B become less productive due to the increase of meetings? You might get lucky and have a fairly smooth transition into a better working environment - but chances are good that the overall change might still need some tweaks. Don’t let yourself fall into the ‘set it and forget it’ mentality - getting better means constant improvement.

This might seem like a lot of work just to make a simple change, but it doesn’t really have to be. As a real-life example, I recently worked with another team who was starting a new project to install an entirely new application. They began by submitting individual tickets to the network team with bits and pieces of what network changes they believed their project would need. Once I realized this was happening, I asked the person leading the project if they had 30-minutes to sit down and talk that afternoon. It was a very quick meeting where I asked about the application they were installing, what it did, how it was intended to be used, and what other applications/systems they believed it would need access to. I also provided a little insight into why this mattered to me from a network design perspective. From that I had enough of an understanding of their project to put together an effective design from the network side, which makes both of our lives easier - because we don’t have to piecemeal it together now then realize later that it wasn’t the ideal configuration. After that meeting, the project lead said “Man, I always wished we had a lot more meetings like this - this was really helpful”.

That is the difference between wanting change and driving change. The bottom line is: If you want to inspire positive change - You have to be the catalyst.

I’m sure we have all identified areas of improvement with our workplace. Have you ever been the one to drive change? Leave a comment below, and tell me your story!

I learned this from the great boss I had a few years ago. He taught me a lot about how to improve as a professional, which has given me a bunch of habits that I still continue today. Earlier in my career, I would occasionally be given a task where I would put it off a little longer than I should have. For example, I might be asked to add a few new subnets to a site-to-site VPN tunnel between our main office and a remote office. Simple enough to do in theory, yet  I might put it off for a day or two longer than needed (especially since it wasn’t a high priority task).

Every day I would need to provide an update on the tasks I had been assigned, so after a day I would get the question “Well, you’re hesitating for some reason - So what’s going on?”. It’s a reasonable question and one I think that needs to be asked more often. When I sit back and think about it, maybe I am hesitating because of fear. Maybe I’m not confident in the change - Will the change drop the VPN tunnel? What if I lose access to the remote site? Usually this resulted in me realizing that either I just needed to do a little more research what I was doing or I just needed to push myself to get it done.

We are all human, and sometimes we let ourselves get held back by simple things. Even at my current job, I have watched people say that they will send an email to a vendor - but then take two days to send it. Could this be helped by just asking them why they are hesitating? Maybe they don’t feel comfortable communicating with that particular vendor (for any number of reasons). I’ve learned my lesson, and now every time I catch myself thinking “I’ll get to that later” I just ask “But why am I waiting”? It might seem ridiculous, but it helps. Figuring out what your big hesitation is will help you to stop using it as an excuse.

Push yourself through it. Be productive. Get things accomplished. Next time you catch yourself holding back, just ask yourself “why?” - and when you answer that question, hold yourself to addressing it.

I am often an advocate of taking the time to do things the right way the first time. It’s extremely time-consuming to go back and fix things a year or two later only because something else requires it. Every time a problem comes up, I race through several different variations of a solution. Usually, there is a “No way/Only if we absolutely must” idea, a “This would work, but not be ideal” idea, and a “perfect world” solution. Often the perfect world idea will take much more time and effort than we currently have to complete - which means that we are too likely to go with the non-ideal solution.

At some level, you have to be able to accept that not everything can be done perfectly. We don’t live in a perfect world with infinite time and resources. Corners have to be cut in some way or another, but it’s up to us to determine where and how. That being said, we shouldn’t blindly accept imperfection either. Use the multiple options to fight for a solution that edges toward the perfect world idea. Maybe we accept the temporary solution, only as long as we are allocated time within the near future to make it more ideal. Or it’s possible that we accept the temporary solution, but with some mitigating factors.

For example, maybe we are told that there is a new product being installed on our network. Servers are already being built and the application will need to be running quickly. However, the application serves a very different purpose than most of the remaining network and it introduces new risks. In the ideal situation, we might create an entirely new virtual network segment to isolate this type of traffic. However, maybe because of the time crunch, we only have enough time to create a new VLAN and put the application behind it’s own SVI. Not really a true separation, but it’s a start.

In this scenario, we could propose two better solutions. First, we may want to suggest that we create the new VLAN/SVI to get the application running immediately, but we allocate time over the next week or two to create VRFs/firewall contexts/etc and completely isolate the application. This would allow us to meet the requirement in the meantime, but work toward our ideal long-term state. Our other solution could be to just extend the VLAN up to a firewall and apply basic filtering - not ideal but this still gets us to a much better state than just allowing the application into our existing environment.

When you look back, it’s always easier to see how things could have been done better. Usually you just have to work with the resources you were given at the time, which is often less than perfect. However, we shouldn’t settle on just ‘getting it done’ or insist that it must be perfect every time - but instead push for a middle-ground. Whenever you see yourself in a situation like this, take a moment and outline a few options. Talk to your direct supervisor about them - in most cases they will listen. If we can provide an option that is better than “good enough” but also doesn’t require a significant additional time investment, then they are much more likely to give in. Just be sure to stress the potential risks of taking the easy way out.

So I have a few posts that I plan on scattering here and there, which will cover some tips on how to become a networking magician. I will aim to provide detail behind some of the expert intuition and skills which can amaze and confuse others. Let’s get started with Magic Tip #1:

Monitor your network

No, really, just monitor your network. I don’t mean “Oh, there is a ping alert for that switch” - I mean pay attention. How will you ever know what an anomaly looks like, if you don’t have an internal baseline of what your network should look like? This tip really takes time, but I’ve found that it pays off in the long run.

I use a couple of open-source tools and applications, like Observium and SmokePing, to track metrics on my networks. I spend a quick 5-10 minutes each morning quickly skimming through the pretty graphs to get an idea of how we are performing today. About once every or every other week, I will spend a bit more time for a deeper dive into the metrics. However, the important thing here is not the time spent, but the fact that I look at these. In the back of my mind, I keep a mental note of the general averages for bandwidth, latency, packet loss, etc.

Once in a while, I might look at a graph and notice that something is a bit off. Defining the word ‘off’ in this sense is difficult. Maybe a router interface that averages 20Mb/s spiked to over 40Mb/s through the night. Maybe traffic was actually far lower than the average. Sometimes I might see a slight increase or drop in latency between a pair of data centers. Some of these things could mean absolutely nothing - but in many cases they are an indicator of something else.

As an example to this - A few weeks ago, I noticed that the average latency between two data centers had increased slightly, and SmokePing was reporting occasional packet loss of up to 5%. I also track historical traceroute tests - so when I reviewed those, I found that the upstream carrier’s route had changed about 2-3 hops out. No big issues - but I made a note of these findings. A few days later, we began experiencing a spike in packet loss between those two data centers. Rather than being caught completely off-guard, I already had all of the information I needed to work with the upstream carrier. Issue resolved - quickly, simply, and without wasting time during a network degradation event.

Let me just reiterate that I don’t expect everyone out there to stare at bandwidth graphs all day long - that’s not going to get you anywhere. However, we do need to spend a little bit of time giving our network the attention that it deserves, even if its just a quick check-up every day. Once you have a good idea of how things typically operate, it can be much simpler to pinpoint issues and get ahead of them - which means being resolved without wasting time.

Ever had someone claim you’ve performed magic? Tell me about your experiences in the comments!

The problem here stemmed from the fact that there were no documented standards in place. An engineer was given a device to configure, and it was configured depending on who did it and what they felt needed configuring. In a few cases, this actually led to unnecessary security risks being introduced into the environment because something was left enabled. In one instance, this included open root SSH logins via the Internet to a production firewall. Scary, huh?

So how do we go about changing this? Here is a quick little guide I threw together on my method for tackling the situation:

1. Define a standard

Begin creating a baseline document, whether it be a spreadsheet, word doc, or a wiki page. Start small and choose a single system, like your external firewalls for example.

2. Research best practices

Check out the vendor’s website to see what they recommend. There are also some amazing free resources out there like the Center for Internet Security’s configuration benchmarks, Do your research - there is plenty available to help you.

3. Figure out what’s best for your network

Not all of the best practices or security hardening guides will be a perfect fit for your environment. So it will take a little manual review to see what actually fits. For example, many of these guides recommend disabling local authentication in exchange for something centralized like TACACS+ or RADIUS. But if you don’t have that available, then you’re going to stick with local authentication. This can still be a great time to find room for future improvement projects though.

4. Test

If you have a development or test environment available, then run a device or two through your checklist and make sure there are no big issues. If you don’t have a dedicated test area, then try and choose a low-impact device - where not much will be impacted if the changes go wrong.

5. Roll out the changes

Make sure you have a list of every device that needs to be touched, so that you have a way to validate. Then make the configuration changes to get each device into compliance with your new standards. Have a validation/testing checklist ready, so that you can quickly ensure that no production traffic was impacted

6. Train your peers

Configuration standards only work well as long as everyonefollows them. It only takes one person to ignore the checklist and potentially expose a vulnerability. So take an afternoon, schedule a training session with your team. Help them understand the importance of maintaining these standards, and train them on how to apply the changes (if necessary).

7. Automate

This part is optional, but highly recommended. If nothing else, spend the time to automate verification of the standards - which will make it easy to locate a device that falls out of compliance. If you or your team have the skill set, then automate the entire process from initial deployment to continuous validation. Why is this the last step, instead of being included with the roll out? I am a firm believer that you should completely understand how your device functions and reacts to changes before automating those changes.

So that’s more or less how I worked to implement a standardized configuration at my current job. I began with a completely new device platform that we were integrating into our environment, then began to go back to older device platforms. It might be a lot of upfront work, but it certainly helps me sleep better at night not having to wonder if there might be one device out there that’s misconfigured (and will cause an issue later, due to that misconfiguration).

So let me know in the comments below - have you ever implemented something like this? If so, what did you do differently? If not, then let me know if you give this a try!

Back when I worked at a local IT consulting company, they hired a network admin who had worked as several large service providers in the past. He was very experienced and intelligent, and was able to walk into the organization and immediately begin making positive changes. Exactly the type of person that you would want to hire, right?

Well after a few months in, he began checking through some of the equipment we had in our spare store-room. A bunch of Cisco routers and switches, some older than others. After a week, he began complaining about how the devices had sat on the shelves too long. It seemed as though the flash memory was degraded, which caused the devices to not retain their configuration settings. Almost every device he checked through seemed to be experiencing this issue. What else can you do at this point but throw out the bad hardware?

So I decided to pick up one of the devices to see what he was talking about. After all, I was still very early in my career - so if I could stand to learn something from how the devices were behaving, I wanted to see it. So I boot up an old Cisco 2610 router and make a few configuration changes. Save, reboot, and sure enough my changes were gone. However, I had also just been studying how to password reset these devices - since I had a pile of them that needed to be reset. Part of resetting the devices was booting into rommon mode and changing the configuration register value to a hex value of 0x2142.

So what is 0x2142? It’s a hex value that tells the router upon boot to ignore any saved configuration. Of course that easily explained the “degraded flash” issue that the experienced network admin had seen. So I changed the configuration register back to 0x2102, made a few more configuration changes, then rebooted. Sure enough, everything was still there. So I went and told the network admin what I had found. “Oh, checking up on me, huh?”

This story has been a bit of a running joke for a while. But really the importance is that even when you’re extremely intelligent and experienced, you can still overlook simple things. He had been password resetting the devices, but never reverting the configuration register values back to the defaults. Even when you think you might know everything, you should still keep an open mind - because even someone with no experience might have a different view on something. Sure, this wasn’t really a big “save the day” moment, but it helped to show that guy that I had some idea of what I was talking about. From then on, he actually began to work with me on understanding more networking concepts and started asking me to help out with some more of the work he was doing.

What was the most ridiculous simple mistake you’ve made? And how did you find out about it? Share in the comments!

So once I had that magical CCNP certification, I finally felt like I needed to move on. I had gained as much experience from that first job as I thought I would, which meant that I needed to start looking. I got some help from a co-worker of mine at the time, who gave me some wonderful resume tips (which I will share in a future post). Two months and a handful of interviews later, and I found myself jumping on a contract-to-hire position for a local government organization.

The three and a half years spent with this organization taught me so much. I had a great boss, to whom I owe many personal improvements that helped me get where I am today. I walked into the place in a role that was technically supposed to be a Junior Systems Administrator, but the position was much more widely focused than that. I did everything and anything, including managing an Avaya phone system, desktop support, networking, Windows administration, and even a bit of VMware ESX. Obviously, I began to lean more and more toward the networking side of the house, as the team was relatively well split in terms of specializations. One guy loved virtualization and storage, another loved application support, and I owned all things networking.

Another thing this job brought me was the push I needed to go back to school. The organization didn’t like to hire people without a college degree, but I managed to make it in under a very rare set of circumstances. Unfortunately, that meant that I was constantly told that I really need to go back to school and get a degree. After a short while, I gave in and picked up a four-year online degree program in Network Security.

This place was my first real experience in actually owning a network. Having complete control and being able to call it my own. I spent the first couple of months doing exploratory research - what did we have running and how was it configured. Then I built a list of recommendations for things I thought needed to be improved. After a few years, I had replaced almost every device (many were end of life) and made the network significantly more secure and resilient. I had many great learning opportunities in managing my own time and building project plans. I designed network upgrades and made detailed plans to make it all work - and it did, surprisingly.

While that job was an absolutely amazing experience for me in terms of personal and career growth, I eventually reached a point where those things slowed down. Soon the negative aspects of the job were starting to outweigh the positives, and so I began my job search once more. A friend of mine, who I had previously worked with at the consulting company, ended up referring me to a position with a company he worked for. The position was a Network Administrator for a local cloud Software as a Service provider.

I didn’t know it when I took the job, but I ended up walking into an environment where I had the most experience on the team. For having several datacenters around the world, the network architecture left much to be desired - A lot of designs built upon the need of the moment and not the future. At the time of this writing, I’m still with this company - and I’ve already gained quite a different set of skill and experiences: Being the senior team member, designing scalable network architecture, and learning the ability to lead others.

I’m going to stop here with my story for now - but hopefully this provides a bit of context around where my experiences and insight have come from. I have a lot of future post ideas which will build upon everything that I have learned over the past ten years. Thanks for reading!

In the beginning - I started off doing some minor PC repair for family and friends. Really quite minor stuff, like replacing power supplies, reinstalling the operating system, or troubleshooting application issues. The technical work really was fun for me, but at that point I had never considered the possibility of it becoming a career. It just seemed like a fun hobby that was great to do in my spare time.

After I completed my second year of high school, I found out that I would have to change schools. Luckily, I found out that my new high school offered this fun program called the Cisco Networking Academy. The program was three hours a day for two years, and taught all of the networking fundamentals necessary to pass the Cisco Certified Network Associate (CCNA) exam. I quickly found that this is something that I truly enjoy doing and I was actually good at it. We had quite a few networking professionals come into the class over those two years and tell stories of how successful a career in computer networking could be. That was the point where I realized that this might actually be a career option - so I went with it.

Within two months of finishing high school, I took and passed the CCNA exam. Cisco certified at the age of eighteen, and now left wondering how to find a job. My next stroke of luck came in the form of a family member who had actively been working in IT for about 10-15 years already. She sat down with me and helped me build my first resume, then showed me where to post it online. Within a few weeks, I began receiving calls from recruiters in the area about a variety of positions. “Level 1 Help desk? No, I want to be a Network Engineer making ALL THE MONIES”. Of course at the time, I had no idea that jumping directly into a network engineer position was very unlikely - especially given that I had no real world experience yet.

A couple interviews and a few months later, and I happened upon a local IT consulting company. I remember interviewing with the manager at the time and mentioning how difficult it was to find a job, since everyone wants you to have experience but no one wants to help you get it. Well, he decided that he was willing to help out and offered me a job as a Level 1 Network Operations Center Engineer.

I spent nearly four long years at that job. I was new to the field so I took advantage of every opportunity they offered me. Certification training? Yes. Networking projects? Yes. Consulting for a variety of businesses? Yep! The company culture was heavily focused on making money quickly, which meant that they didn’t always take care of the employees very well - but there is something to be said about the amount of varied experience I gained, especially for my first real tech job. While I was working here, I also added onto my collection of Cisco certifications: CCNA Voice, CCNA Security, CCDA. I finally finished up by achieving one of my goals of becoming CCNP certified.

So this has been part one of my history, and to make this a bit more readable I’m going to split it into two postings. Continue the story in the next post!

So here I am again, giving this another shot. This time I won’t be backing myself into a corner from the start. This blog is intended to be networking oriented but with a bit of a wider focus. I’ve already come up with quite a few ideas for content I would like to write here, so I’m more prepared. That being said - I’m not committing to regular updates or always exciting content. When I have something I feel is worth sharing, I will share it.

So to provide a general overview, here is the outline of topic ideas I have for this blog:

Education/Certification Studies- Every network admin has a blog to document their road to CCIE certification, right? This has certainly been one of my goals over the years, so I’ll be writing about the awesome things that I learn. This is also meant to include general networking education topics, since you can never stop learning.

Career - I’ve been Cisco certified and working in networking for nearly ten years, but that doesn’t mean everyone has. I’ve finally reached the point in my career where I’m meeting a lot of people new to the field and I’m able to help guide them. So I would like to share some of the career advice that I have, both from my own experiences and advice that I have received from others.

Network Design/Architecture- This stuff is really important, as I’ve run into more than enough situations where a network wasn’t originally designed for the type of workload it handles today. I want to cover both network design topics, as well as why it is important. I have some stories to share on how bad network architectures can have significant consequences.

A Little Bit of Magic - This is probably my favorite topic. You ever work with an IT professional who just somehow knows how to fix everything? The person who can pick up almost any technology and become an overnight expert in it? Well, I would like to share some of my insight into how this type of thing is accomplished, and why it just seems so magical.

This likely won’t be everything I cover here, but these are the primary topics. I’m going to give this site my best shot over the next few months - so let’s see how it goes.

Feel free to bug me in the comments with any questions!