So in my last post, I picked up a Qotom Mini-PC to run OPNsense on. After a few months, the device has been running well & I’m very happy with it.

One of the new things I got to try out with OPNsense was Wireguard VPN. I had previously been using something else for VPN connectivity back to my home network - but I heard good things about Wireguard & wanted to give it a try.

So far my experience has been good! I’ve been pleasantly suprised with how easy it is to configure & get running. In addition, the performance & overall experience has been very positive. The VPN connects quicker than anything I’ve used in the past, and just simply works without issue.

All that being said - I wanted to put together a quick guide on how to configure Wireguard on OPNsense. Specifically, this configuraion will be for remote-access VPN - where clients will connect to a VPN headend. We’ll walk through the OPNsense configuration & a few clients as well. So let’s dig in!

Topology

For the purpose of this blog post, we’ll be using the lab topology below:

I will be using the reserved IP range 203.0.113.0/24 for the WAN-side addressing. I’ll have one Windows & one Android client that we’ll walk through & connect to the VPN.

Installing the Wireguard Plugin

To get started, first thing we will want to do is install the Wireguard plugin for OPNsense. By default, OPNsense will have standard IPSec & OpenVPN already available - but other VPN options can be enabled easily.

So in OPNsense, we’ll navigate down to System > Firmware > Plugins, then search for wireguard and click the plus icon.

This should pull down the package & install pretty quickly. No reboot required here!

Once installed, you may have to refresh the page or navigate to a new page so that the menu bar has a chance to reload. Then we’ll have a new option under VPN:

Wireguard Tunnel Configuration

Next we’ll begin configuring Wireguard on the OPNsense side.

There is a little bit of a chicken & egg scenario here since everything is based on cryptographic keys. We’ll need to generate keys on the firewall, which we need to enter on the client - but we also need the client keys to enter on the firewall. A bit of bouncing between the two - but for now we’ll try to complete as much as we can on the firewall side.

We’ll enable Wireguard by dropping down to VPN > WireGuard then clicking Enable and Apply

Next we’ll set up the Wireguard tunnel interface on OPNsense. This will be a virtual tunnel interface that will be created as interface wg<instance number>.

To do this, we’ll navigate to the Local tab, and click the plus icon to add a new tunnel.

In the above screenshot, I’ve filled in just a few details.

For Name, I’ve entered our virtual interface name wg1. Since OPNsense shows the Instance as 1 - it will create a wg interface with that instance number.

We’ll leave Public Key & Private Key blank for now. OPNsense will auto-generate these keys once we save this config.

For Listen Port, I’ve set this to 51820 which is the default for Wireguard. It’s not stated here, but this is a UDP tunnel.

For Tunnel Address - this is where we add the IP address of the virtual tunnel interface. This will be the gateway for our remote clients. In my lab I’ll be using 10.50.50.1/24 here.

We have no peers configured yet - so we can’t select any here. We’ll leave this blank for now, but come back later.

We will leave Disable Routes unchecked. By default, OPNsense will add static/connected routes for any client via the tunnel interface. You might not want this behavior if you wanted to do custom routing - for example, in a site-to-site VPN connection - but we’ll leave this enabled.

Once we’re done, we’ll click Save.

Like I mentioned before, OPNsense will now auto-generate our crypto keys for the tunnels. So if we edit our tunnel, we’ll now see those fields populated:

We’ll want to copy the Public Key & save it for later. This will need to be imported onto our clients, so that they can communicate securely with our firewall.

Wireguard Interface Assignment

Now that we have our headend tunnel interface defined, we can map our wg1 interface to an OPNsense interface. The OPNsense documentation suggests this is optional, but I would recommend it since it will allow us to create firewall rules to permit/deny access to clients.

We’ll navigate to Interfaces > Assignments, and we should see a New interface available: our wg1 tunnel.

We can assign this a name, then click the plus icon & Save.

Next we’ll enable the interface by navigating to Interfaces > WG1. Here we’ll only need to click Enable & save the change - nothing else is necessary.

Of course, we’ll be prompted to apply the changes - which we will do:

By default, all traffic through our WG1 firewall interface will be blocked - so please make sure to configure a firewall rule to permit traffic from the Wireguard clients.

Firewall Rules: Allow Inbound Wireguard Traffic

Next we need to permit the Wireguard traffic into our firewall. By default the WAN interface will block all traffic that isn’t explicitly allowed - including our Wireguard traffic.

For this, we’ll navigate to Firewall > Rules > WAN. Then click the plus icon to add a new rule.

The screenshot above shows what our firewall rule will look like.

Here’s the summary:

• Action: Pass
• Interface: WAN
• Direction: In
• TCP/IP Version: IPv4
  • You can enable IPv6 as well, if you have IPv6 connectivity (this is a lab box, which does not)
• Protocol: UDP
• Source: Any
  • This will allow anyone on the internet to reach our VPN. We could restrict source IP addresses, if our clients had permanent, static IPs.
• Destination: WAN Address
  • The firewall itself is the destination for this traffic
• Destination Port Range: (other) / 51820

I also enabled logging & added a quick description. After we have all this configured, we can click Save - then Apply Changes.

Client Setup - Windows

So first we’ll start with an easy configuration on a Windows client. Wireguard client software can be found on the Wireguard site here.

For the sake of the walkthrough, we will manually configure each client. However, this can be a difficult task if there is a large number of clients. Wireguard does support importing configurations, and there are a number of free tools available to help automate generating config files for clients - including some which will generate QR codes for easy import on mobile clients.

So on my lab Windows machine, we’ll open up the Wireguard client & click Add Empty Tunnel:

Then we’ll be given a blank config file, with only the devices public & private key pair generated for us.

We’re going to fill in details similar to the below screenshot:

The configuration under [Interface] is the local, client-side configuration. I’ve added the client’s tunnel address - which will be 10.50.50.15/32. I’ve also configured DNS servers which the client can reach via the VPN.

Then we’ll add the [Peer] section, which contains info about our VPN headend. Here’s where we’ll need the public key from our OPNsense firewall. We’ll also specify the Endpoint address, which is the IP or hostname of our VPN headend & the port (which by default is 51820).

We’ve also configured AllowedIPs as 0.0.0.0/0. This will force all client traffic over the VPN tunnel - including general internet traffic. However, we could limit this to specific subnets. For example, let’s say your network only used 172.16.90.0/24 & 10.1.1.0/16 subnets & we only wanted the user to be able to access those. We would configure the following: AllowedIPs = 172.16.90.0/24, 10.1.1.0/16. In this case, only traffic for those subnets would be routed over the VPN - any other traffic would use the devices default internet connection.

Now, we’ll save this - and again need to copy the device’s Public Key, which we’ll need to enter on the OPNsense firewall.

Client Setup - Adding Clients to OPNsense

In order for the Windows machine to connect to OPNsense, we’ll also need to configure a client profile on the firewall.

In OPNsense, we’ll navigate back to VPN > WireGuard, then click on the Endpoints tab.

Here we’ll configure a name for our client & paste in the client’s Public Key.

We’ll also set AllowedIPs to the client’s IP address, which we have configured as 10.50.50.15/32. This controls what IP addresses are reachable via this endpoint.

We do have some additional fields available, which we will leave blank. For example - Endpoint Address & Endpoint Port would be used to define a public IP that we expect our client to connect from. Since a remote-access client could connect from any IP, we leave those fields blank to allow this.

Once we’re done, we’ll click Save then Apply.

Next we’ll jump back to the Local tab, and edit our headend tunnel configuration.

We should now see our windows client in the Peers dropdown. We’ll select that client, so that Wireguard will permit that client to connect via this tunnel interface.

Then, as always, click Save & Apply

Last but not least - we should restart our Wireguard server on OPNsense. This can be done by either disabling & re-enabling Wireguard - or by navigating back to the OPNsense dashboard & clicking the restart icon next to the wireguard-go service.

Testing The Connection

Okay - now that we have all that completed, it’s finally time to test connectivity from our client.

On my Windows client, I’ll just click the Activate button for the tunnel:

And we’ll see that the Status shows Active, and we’ll start to see updates to the Last Handshake & Transfer fields - indicating we are connected & sending data.

To further validate, we can check the Log tab:

The key things to look for here, are the following messages:

• Receiving handshake response which indicates our firewall responded to our request to connect
• Keypair 1 created which indicates that our connection is healthy to our peer
• Receiving keepalive packet from peer - we should see these periodically to maintain our connection. If keepalives stop flowing, then we may have a break in connectivity between client & peer.

We should also be able to reach out wg1 tunnel address from the Windows client:

On the OPNsense side of things, we can check what client is connected via the List Configuration tab, under VPN > Wireguard:

On this screen, we can see we have 1 peer connected - which matches the IP & public key of our Windows client. We’ll see similar output like the Windows client, where we can see the latest handshake & data transfer.

Additionally, for easier access we can add the Wireguard widget to the OPNsense dashboard.

If we navigate to our dashboard, then click Add widget - we can add the Wireguard widget.

Here’s what that looks like:

Additional Client Setup - Android

Let’s also take a quick look at a mobile client. I’ll get through this pretty quickly, since the configuration will be very similar to our other client - just a different UI.

On my Android device - if I open up the Wireguard app, I have a few options for creating or importing a tunnel:

In this case, we’ll again walk through creating a tunnel manually. Again, it’s worth mentioning that there are 3rd party apps available to auto-generate config imports or QR codes to make this easier.

First we’ll have a handful of fields to fill in about the Android-side configuration. This includes a name for the tunnel, an address, and DNS servers.

We can click on the refresh icon in the Private Key field to auto-generate our key pairs:

One of the interesting parts of the mobile app is the ability to permit/exclude individual apps from traversing the VPN tunnel.

Here’s a quick screenshot, where we can pick either to allow only certain apps to use the VPN - or permit all except a few we choose to exclude:

In my case, I’ll keep this set to allow all applications.

Next we’ll work on the peer config:

After that, all we need to do is save our VPN configuration - then we can toggle the tunnel on or off:

And similar to the Windows client, we can click on the tunnel itself to see current status / data transfer:

So it looks like we should be connected & can try accessing VPN resources!

We can also use the same methods as earlier to check connectivity from the OPNsense side.

Okay - that’s all I wanted to share today. I’ve been quite pleased with how easy to use WireGuard has been - and how well it performs! I hope this blog post was helpful if you’re interested in trying it out.

Thanks!!

In this post, we’ll walk through configuring a Cisco Viptela SD-WAN network to integrate with Cisco Umbrella’s Secure Internet Gateway (SIG) & Secure Web Gateway (SWG).

If you’re interested in reading more about what SIG & SWG are, please check out my last post where I walked through this integration with a Meraki MX firewall.

For additional reading, there’s also a good Umbrella blog post on all the components of Cisco’s SASE architecture, which you can find here.

Okay, with that being said - let’s get started!

Step 1: Umbrella API Keys

Okay, so we’ll start on the Umbrella side. We’ll need to generate a set of API keys, which we’ll push out to our WAN edge devices. These API keys will allow our remote devices to reach out to Umbrella & auto-configure their SIG tunnels.

We’ll log into our Umbrella dashboard at https://login.umbrella.com/

Once we log in, we’ll hop over to Admin > API Keys. Then up in the upper-right corner, click Create.

For this integration, we’ll need to select Umbrella Management to make sure our API keys have the access they need. Then click Create. Easy enough, right?

Once we do that, the Umbrella dashboard will display our API keys - which we’ll need to copy over to our Viptela SD-WAN environment.

We’ll also need to collect our Umbrella Organization ID. This is actually just embedded within the Umbrella Dashboard URL, and should be a seven-digit number. For example, if we look at our current URL on the API keys page:

https://dashboard.umbrella.com/o/<ORG ID>/#/admin/apikeys

And that’s all we need for now from the Umbrella side. So let’s hop over to Viptela.

Step 2: Viptela Templates & Config

For this post, I’ll be using my SD-WAN lab in EVE-NG. Currently I just upgraded the lab to run Viptela version 20.4.1, though the SIG integration has been supported since 20.1.

Just for reference, here is the lap topology that I’m working with:

This lab includes a bridged connection to allow direct internet access from the lab network.

To configure our SIG automatic tunnels, we’ll need to create / update a few templates:

• Create a SIG Credentials feature template
• Create a SIG feature template
• Assign SIG templates to device templates
• Edit Service-side VPN Template to inject a service route

2a: Creating a SIG Credentials Template

First we’ll create a template which will store our Umbrella API credentials.

In the vManage dashboard, we’ll go to Configuration > Templates, then drop into the Feature tab, and click on Add Template.

Once we get to the next screen, we’ll select our device type. In my case, I’m using a vEdge Cloud.

A list of templates will pop up - we’ll drop down to SIG Credentials, which will be near the bottom under the Other Templates header.

We’ll then be taken to the page where we create our template.

We’ll fill in our template name & description, then paste in our Umbrella details (Org ID, API Key, API Secret).

Once we’re done - We’ll hit Save.

Note: At time of writing, the “Get Keys” button only functions if you’ve purchased your SD-WAN subscription with DNA Premier licensing. This license level includes Umbrella SIG, and allows this 1-click integration that pulls your Umbrella API keys automatically.

2b: Creating a SIG Tunnel Template

Okay, next we’ll need to create another feature template to specify our IPSec tunnel configuration.

Just like before, we’ll head back over to Configuration > Templates > Feature > Add Template.

This time, after selecting our device type, we’ll choose Secure Internet Gateway (SIG) / WAN - which is located under the VPN header.

In the template configuration, we’ll give the template a name & description as always. Then we’ll jump down to the tunnel config.

We’ll select Umbrella for SIG Provider, then click Add Tunnel.

Within the tunnel config, we’ll specify an interface name - I’ll name mine ipsec1 (and I’ll create an ipsec2 shortly). We’ll also specify a Tunnel Source, which in my lab is ge0/0 for the biz-internet VPN 0 interface.

We’ll keep Data-Center as Primary, then click Add. Then - we’ll add a second tunnel configuration, but using Data-Center as Secondary and the interface name as ipsec2.

When all that is done, we should have the following:

If we scroll down to the bottom of the template config, we’ll have some settings for High Availability. Here, we’ll specify what our active & backup IPSec tunnels are, and their weights. I’ll specify ipsec1 as active & ipsec2 as backup. Then click Save to finish our template.

Note: Weight settings are only available starting with 20.4.1 & allow for ECMP routing to SIG. Not shown above, you can create up to four HA tunnel pairs. Depending on weighting, you can equal-cost or unequal-cost load balance between those pairs.

Why would you want to load balance across multiple tunnels? At time of writing, each IPSec tunnel is limited to a maximum 250Mb/s throughput. By creating multiple tunnels & load balancing, we can overcome this limitation if we need higher bandwidth.

2c: Attaching SIG to Device Templates

After we’ve built out our two SIG templates, we can now attach them to our device templates.

For me, I currently only have a single device template which is applied to all of my remote WAN edge devices.

So we’ll go back to Configuration > Templates and find whichever device template we want to use - then click the ellipsis on the far right, end select Edit.

In the device template, we’ll scroll down and look for our VPN 0 configuration under Transport & Management VPN. We’ll attach our SIG tunnel template to VPN 0, since that’s where those IPSec tunnels are being sourced from.

On the right side, under Additional VPN 0 Templates, we’ll click Secure Internet Gateway to add our template. Then from the drop-down, we can select the feature template we just created:

We’ll also include our SIG credentials template, which we can find at the bottom of the page, under Additional Templates:

Once we’re done, we can click Save at the bottom. This will take us through the process of pushing out these changes to our remote WAN edge devices. When this configuration is deployed, each vEdge will now reach out to Umbrella & auto-configure an IPSec tunnel.

Note: While the IPSec tunnels will be established when this configuration is pushed - no traffic will flow over them yet. We’ll need to add a service route for that, which we’ll do next!

2d: Injecting a Service Route

Now we have our IPSec tunnels deployed - all we need to do is start routing traffic out to Umbrella. We’ll accomplish this using a service route on our LAN-side VPN.

So we’ll go over to Configuration > Templates > Feature - and we’ll scroll down & find whichever template we’re currently using on our LAN side. For me, I want VPN 10 to route over the tunnels, so I’ll be editing my VPN 10 template.

Within the template, we’ll scroll down to the Service Route section, click New Service Route, and we’ll enter our desired prefix. This will be what traffic will be routed over the IPSec tunnel to Umbrella. Since this is intended to be an internet gateway, I’ll enter 0.0.0.0/0 to inject a default route to Umbrella.

Service should be pre-selected as SIG, so we’ll click Add, then Update & deploy our changes to the remote edge devices.

Step 3: Validation & Troubleshooting

Awesome! Now we should have everything configured & working. So let’s jump through some initial testing and validation we can do.

Within vManage, we can check the status of our IPSec tunnels. We’ll go over to Monitor > Network then select one of our WAN edge devices.

We’ll click on the Interfaces tab on the left side - and we should be able to see a list of all interfaces on our device. This should include an ipsec1 & ipsec2 interface.

I’ve also selected the Real Time monitor on mine, and filtered to just show traffic on the ipsec1 interface:

You might recall from my topology above, that I have a linux VM running behind each of the two vEdge Cloud appliances. Using these, I can also test web access & see what my external IP is:

And sure enough, I am getting a 146.112.x.x address - which belongs to the Umbrella datacenter.

If we log into one of our vEdge devices, we can check the routing table with show ip route vpn 10 (or whichever VPN you’re using for the LAN-side). We should see our default 0.0.0.0/0 route via ipsec1, with a next-hop-VPN of VPN0:

vEdge-01# show ip route vpn 10

     ADDRESS               PATH             PROTOCOL          NEXTHOP  NEXTHOP                         NEXTHOP
VPN  FAMILY   PREFIX       ID    PROTOCOL   SUB TYPE  METRIC  IFNAME   ADDR     TLOC IP  COLOR  ENCAP  VPN      STATUS
------------------------------------------------------------------------------------------------------------------------
10   ipv4     0.0.0.0/0    0     std-ipsec  -         0       ipsec1   -        -        -      -      0        F,S
10   ipv4     10.2.2.0/24  0     connected  -         0       ge0/2    -        -        -      -      -        F,S

We can also check specifically our SIG tunnel status using the command show secure-internet-gateway tunnels:

vEdge-01# show secure-internet-gateway tunnels

TUNNEL                                                        API   LAST
IF                                                            HTTP  SUCCESSFUL     TUNNEL
NAME    TUNNEL ID  TUNNEL NAME                     FSM STATE  CODE  REQ            STATE
-------------------------------------------------------------------------------------------
ipsec1  530233543  SITE200SYS10x10x10x235IFipsec1  st-tun-up  200   create-tunnel  -
ipsec2  530233542  SITE200SYS10x10x10x235IFipsec2  st-tun-up  200   create-tunnel  -

In that output above, we’ll see that we have both ipsec1 & ipsec2 tunnels shown. The last API queries to Umbrella have a HTTP 200, which is good. We’ll also see our tunnel names, which we can use to find our device tunnel configuration in the Umbrella dashboard.

The tunnel name might look like a mess at first, but it’s a unique identifier to represent each tunnel that was created. So if we break it down for this vEdge:

• This vEdge is at Site ID 200
  • Shown as “SITE200”
• This vEdge has a system IP of 10.10.10.235
  • Shown as “SYS10x10x10x235”
• This vEdge has two IPSec interfaces, ipsec1 & ipsec2
  • Shown as “IFipec1” and “IFipsec2”

If we jump over to the Umbrella dashboard, we’ll see the same. Within the Umbrella dashboard, we can jump to Deployments > Network Tunnels.

On this page, we should see a total of four tunnels listed (two from each vEdge appliance):

And with everything configured & validated - now we can move onto configuring firewall and web filtering policies within Umbrella!

If you’re interested in seeing how to configure Umbrella’s cloud firewall & web filtering policies - please check out my YouTube Video above!

Twitter has come out fairly quickly to say that the incident was a result of social engineering. Someone convinced an employee at Twitter to hijack accounts via a backdoor administrative system.

What I wanted to focus on today is this: Why do backdoor systems like this exist - and how do we protect ourselves?

We detected what we believe to be a coordinated social engineering attack by people who successfully targeted some of our employees with access to internal systems and tools.

• Twitter Support (@TwitterSupport)

Everything Has a Backdoor

If you think there isn’t a backdoor way into a system, you probably just haven’t found it yet.

From years as working in IT - one thing is certain: Someone controls the data. And that person(s) can do with it what they wish.

Are backdoors always a bad thing? Not inherently, no. We all want the convenience of being able to regain access to a locked account right? Then someone has to have the keys to override the system to make that change.

We all take our corporate email and domain logins for granted. Sure, we have a kind helpdesk employee that may help to reset a password - but how much access does that person have to our account? Even so, maybe we expect that in this case.

For example, there was a company I worked for a long while ago. HR/Payroll wanted to switch to using internal email to distribute paystubs. In order to do so, they required that every employee manually opt-in. None of the systems admins in the organization opted-in. We all knew how easy it was for any of us to read each others email. In fact, our internal security policy required any email containing sensitive information going in or out of the organization to be manually reviewed by someone. You know what qualifies as sensitive info? Payroll data.

Social media platforms (or any major website really) are no different. Someone needs to manage the backend systems. Someone has to be able to reset passwords & assist those who use the platform. In this particular case, someone at Twitter had the ability to hijack accounts by changing email addresses & resetting the passwords.

What About MFA?

MFA is like anything else in security - it’s another layer, but it doesn’t solve all problems.

MFA can protect you from someone who has your login credentials (username/password). If they don’t have your token, phone, or whatever else you’re using - they can’t successfully authenticate.

That being said, if you already control the keys to the backend systems - bypassing or disabling MFA is just another click on your journey to taking over accounts.

How Do We Prevent This?

That’s a good question. And a hard one to answer.

The general idea is to limit the amount of privilege that a single individual has. For example, if the average Twitter help desk employee is only responsible for resetting passwords - they shouldn’t even have the option to disable MFA or change email addresses.

That being said - that’s not always easy to accomplish. Systems need to be built around security and implement strict access control where necessary. Unfortunately we still live in a world where security is often an afterthought - or at least not designed into the solution on Day 1. That’s not even considering that often security is still under-funded, and/or under-prioritized….

Depending on the size of the organization, this control comes in many flavors. On the more strict side, we could have an administrative panel that is only accessible by high-level, trusted employees. Even then, we could implement a multi-step process by which a single employee could not make changes to sensitive accounts without secondary approval (or more). Going a step further - proper logging & alerting may be triggered to alert someone that a change has happened, which prompts review to see if this was legitimate.

More commonly, what I see in organizations is just a simple policy. Yes, we acknowledge that you have unrestricted access to view/modify/delete customer data. But we have a policy that tells you not to.

The problem with that? One, it expects that most people read (and care about) the policies. I’ve seen quite a number of organizations that hand you a huge pile of policies & paperwork to review in your first week of employment. How many people truly take the time to read, review, and consider the content of those?

For two, a policy does nothing to prevent someone from taking action. Sure, fear of repercussion or losing a job may help. But if we’re talking about a potentially malicious individual, those things likely do not concern them.

As another example, I’ve previously worked at an organization with such policies. “Don’t look at customer data unless you have explicit approval from the customer”. Okay, but the customer opened a support ticket because of some issue they’re having. This issue sounds familiar and I can fix it real quick, if I just log into their tenant and make a change. OH. Whoops. I just saw sensitive data.

I wish I could say things like that never happened…

What Now?

Well, now we wait.

Twitter has been fairly forthcoming with information so far. Yes, it’s been vague - but that’s to be expected while they pick up the pieces of what happened. The important thing is that they’ve been steadily communicating what they do know so far.

Rumors are that an internal employee may have been paid to make changes to the accounts. I suppose we’ll find out eventually - but if this is truly the case, there isn’t much to be done.

One would hope that access to high-profile accounts would be restricted. One would hope that the people with that access are trusted, reliable individuals. Yet here we are.

It will certainly be interesting to see how much data Twitter releases about the actual events & timeline. Preventing this from happening again will likely require further restricting access to those accounts & possibly some form of the multi-tiered change approval mentioned earlier.

In the end - it sounds very likely this was an internal job. Those can be extremely hard to prevent, and require security training, additional security controls, etc. Even then - Humans create security controls, which means we also create ways to get around them.

How does this actually protect user privacy? Best described in the intro to RFC 7858, DNS over TLS “eliminates opportunities for eavesdropping and on-path tampering with DNS queries”. Using the new standard, DNS queries of a client can be encrypted using TLS and tunneled directly to the DNS server. The intent is that monitoring of user activity becomes more difficult, since their DNS requests are no longer sent across the network in plain text.

It seems that DNS security has become a point of focus recently. Over the past year or so, we have seen two new big companies join the public DNS space: Quad9 (backed by IBM) and CloudFlare. Both companies advertise that they are focused on security and privacy of the individual user. They offer features like: blocking of malicious domains, DNS over TLS support, and limited historical logging of user requests. CloudFlare has even already posted steps for enabling their DNS over TLS service with the new version of Android. For the end user, this is great - free services that offer better security and increased privacy. However, this can impact how businesses protect their users.

A lot of organizations use web filtering as an important step toward securing client traffic - which can be accomplished through DNS filtering and/or web proxies. These methods might be used for compliance reasons, malware/threat detection, or even just managers who want to monitor productivity. However, there are a few big changes happening in user security/privacy right now that begin to make that more difficult: Increased focus on DNS privacy and the recent release of TLS 1.3.

Years ago I used to work for a local government organization where security was a high priority for IT operations. Web access was extremely limited and configured to whitelist only for known sites that were business related. If you wanted general internet access, you had to log into one of two segmented computers that were dedicated to that purpose. Even still, our IT security manager had concerns on the potential for threats and/or data exfiltration via HTTPS-enabled websites. One of my projects was to tackle this problem by implementing SSL decryption on our web filtering platform. At the time, implementing a web proxy as a man-in-the-middle wasn’t very complicated. Generate a CA cert for the proxy, distribute that cert to clients, then enable SSL decryption. Sure, there were bumps and hurdles - but the overall process achieved the result we were looking for.

The problem today is that newer standards and features (like TLS1.3, HSTS, or certificate pinning) are adding enhanced privacy features - which make that user data more difficult to reliably decrypt. That’s good right? Well, maybe - but only for the user. As an organization trying to secure your systems and private information, these measures are making life a lot more difficult. Even as a network admin, it’s become more difficult to troubleshoot web applications via packet captures. The use of ephemeral keys for TLS connections means that you can’t decrypt the session even if you have the server-side private keys.

Now the addition of DNS over TLS give us more of the same problem. Even if you couldn’t look into an encrypted web session, maybe you could gather what the destination is by reading the DNS requests. Instead, users now have the ability to encrypt their DNS queries, encrypt their web sessions, and make the life of a security admin a nightmare.

From the perspective of a corporate security team, the future is going to be more challenging (and probably for a lot more reasons than just this). However, I think that at some point we will have to reach a better balance of user privacy while still providing adequate security coverage. For some that might mean installing feature-heavy agents on every client device to inspect everything before it leaves the device. Or it might mean trying to rely on statistical, analytical, and behavioral data to detect patterns and anomalies (something like what ETA is trying to accomplish) rather than just tearing apart the individual user sessions.

Whatever the solution ends up being, it should be an interesting area to watch develop. In the meantime, let’s just agree not downgrade user security or privacy just to shove our security tools in. We’re all users of some service or another, and we all have some expectation of privacy. Is it worth reducing security and increasing risk just to see if your employees are using their web access responsibly? Probably not. Instead it’s time to look for new ways to solve this problem, and find ways to stay secure while still ensuring user privacy.

Earlier today, Cisco released their 2018 Annual Cyber Security Report. I’ve spent some time digging through the report and thinking about what they’ve written. It’s interesting to read through the trends and survey results, and try to get an idea of where security efforts should be focused for the coming year. This post is going to cover just a subset of what’s in the complete report. I’ll be covering the topics that I found particularly interesting, and give my own thoughts/views on them.

The Encrypted Web is Great… For Attackers

Unsurprisingly, Cisco reports a growing trend in attacks and exploits that are taking advantage of encrypted transport. As a lot of large companies and Internet bodies are pushing for a 100% encrypted web, should we be surprised? Nah, it’s the logical next step. Users want encryption because it means privacy - but that privacy also brings a method of concealing attacks.

New exploits and malware are heavily leveraging encrypted transport to bypass all of the security we put in place to detect them. Typical defense technologies like intrusion prevention systems (IPS) are fantastic, but only when they can actually read the data. If a user download’s malware through an HTTPS call, IPS won’t usually catch it. And when that malware can now take advantage of SSL to reach back out to a command & control server? Yeah, IPS might not help us there either.

There are technologies out there that allow enterprises to see this traffic - but maybe not enough of us are adopting it yet. A forward proxy for outbound web filtering is great. One that implements SSL decryption and inspection is even better. If your company isn’t already decrypting outbound web traffic, then this needs to be a priority.

Inbound web traffic can be just as dangerous. New IIS vulnerability? Sure, let’s grab the latest IPS signatures and … Oh wait, our IPS runs on our edge firewall, which sits in front of those web servers - which are all using SSL for encryption. That means any malicious traffic is going to slide right through our IPS undetected and land on our unpatched web servers. Get a Web Application Firewall (WAF), and let it front-end your SSL traffic. These things can be expensive and a nightmare to configure and tune properly, but right now they are one of your best options for inspecting web traffic.

Old Attacks Aren’t Going Away - They’re Just Getting an Upgrade

This year’s report highlighted that much of the older attacks are still here, and they’re not giving up yet. Attacks via email are still present and doing more damage than you would hope. We’re certainly getting better about implementing spam filters with reputation filtering, but attackers aren’t giving up yet.

Email attacks are relying more on social engineering and targeted phishing. These messages are also utilizing SSL to encrypt the malicious links within the emails. Infected attachments are surprisingly still a big issue, with Microsoft Office and PDF files still being the worst offenders. Just because attackers are finding new and exciting ways to hit us doesn’t mean they’re giving up on the tried and true methods. We still need to focus on all the standard attack vectors, like email. Implementing intelligent email/spam filters and providing user awareness training are the primary methods we have to combat this.

The Cloud is Secure! …We Think

This one I found particularly fun. Out of all the companies surveyed by Cisco for this report, 57% of them said they believe the cloud offers better security. Wait - Did I misread that? More than halfof respondents think that the cloud offers better security than their own infrastructure! This makes me wonder…

From my perspective, a cloud service provider is just another company. In most cases, they run just another network and hit a lot of the same challenges that non-cloud companies are facing. And we can only assume that cloud providers are prioritizing security and not just trying to turn a quick profit. Cloud companies have the advantage of being able to hire a dedicated security team that their customers can leverage. However, enterprises are complaining about lack of skilled security engineers, and I’ll bet it’s not because cloud providers are picking them all up.

Cloud definitely offers benefits - but this needs to be a well-calculated risk. For a smaller company without dedicated IT staff, a cloud solution would most likely offer security improvements over their own infrastructure. As companies scale, however, their security requirements do too. We need to make sure that the cloud providers we choose are also capable of adhering to those standards. Before you move to the cloud: ask questions about their security practices, get answers, and demand more information on the parts that are important to your business.

Another fun note from Cisco - some of the damage done by cloud providers is a simple mis-understanding of ownership. If you subscribe to a complete Software as a Service (SaaS) provider, chances are good that the provider worries about all of the critical security configurations. However, if you’re going to a cloud provider just for infrastructure (like AWS), then you are likely responsible. In the case of AWS, you’re being provided a server - and that’s where Amazon’s responsibilities end. It’s up to the enterprise to still make sure that those servers are patched, hardened, and audited. Treat the cloud as an extension of your own infrastructure and polices, not a separate entity.

Diversifying Risks? Maybe not

It used to be a somewhat well-established security practice to use multiple vendors. Have a need for two sets of firewalls? Make sure you use two vendors, so that a vulnerability in one doesn’t affect the other. Seems like sounds logic - until you have to train staff to be experts on multiple platforms, and keep up to date on all the latest patches from each vendor.

Cisco is finding that the more vendors a company has in their environment, the more problem we have maintaining everything. From my own experiences, I can say this is certainly a problem. In environments where I’ve had up to four vendors for firewalls and switching, it becomes difficult to work with. It’s hard on IT staff to maintain knowledge of configuration and best practices for each different vendor - and when a new vulnerability comes out, we end up spending way more time trying to track down each vendor’s responses and patches.

It makes sense that companies who have a more tightly integrated infrastructure might have an easier time managing it. Cisco might want you to buy 100% into their ecosystem (of course), but I do think there is value in consolidating your infrastructure. One or two vendors will be much easier to establish relationships with than half a dozen of them. Your IT staff can dedicate their focus to mastering only a couple of technologies, rather than spreading themselves over a dozen different platforms. And when that new vulnerability is released? It should be much more straightforward to patch all of your systems quickly.

There’s That Automation Thing Again

I think we’re finally beginning to reach a point where automation is really showing it’s value in the security realm. A typical company is going to have so many different systems and alerts that it doesn’t make sense for someone to manually review and act upon every one. This is where automation really begins to shine.

Cisco’s report shows that more companies are relying heavily on automation. This can be used for alert response, reporting, and behavioral analytics. Especially when I keep hearing that there is a skills shortage in security, we need to take advantage of what automation can offer. This doesn’t always have to be home-grown scripts either - there are a number of offerings already available.

Take a second look this year. Try to see where automation can fit into your infrastructure to help improve both operations and security.

Thanks for reading! Just as a friendly reminder - All of the opinions stated in this post (and all others here) are 100% my own, and do not represent any vendor or employer. Since security has become more of an important part of my job, reports like this are always very interesting to read. I’ve only covered a handful of what was in the report - just what was particularly interesting to me. If you’re interested in reading more, check out the full report here.

Risk acceptance is something that I’ve seen misunderstood by engineers over and over again. Especially when you’re earlier in your career, you get the feeling that you know what you’re doing - and therefore management should listen to you. I get it - I’ve been there too. It can be hard to propose an idea, then have that idea immediately shot down by your peers or management. That’s not to say that they aren’t listening, but maybe you might not have a great view of everything behind their decision.

Let’s take an example. As an engineer (server/network/security/etc), you’ve been alerted by your vendor that there is a huge vulnerability in a critical business application. That vendor has been extremely proactive, and has promptly provided you with detailed KB articles and download links to the upgrade package. The vulnerability that’s been disclosed certainly seems bad enough - it allows for remote code execution by an unauthenticated user. What do we do with this? Well we immediately meet with management to tell them that we need to patch this application immediately.

Their answer?

No.

Well that seems just impractical, doesn’t it? They don’t really want to leave the company at a massive risk to attack, do they? No - they certainly don’t. However, it may be that management has chosen to accept the risk for one reason or another. Let’s dig into a few reasons why they might:

Expense (Money) - It could be that the cost to remediate the issue is significant. Maybe the software is a bit out-dated and the company hasn’t paid for support. Renewing that support contract might be tens of thousands of dollars or more. If there are enough mitigating factors, that cost might not be worth it.

Expense (Time) - People also cost money, and time spent on fixing this issue also means lost productivity on other tasks. For some enterprises, an upgrade of a critical business application could take months of work. What happens when this new version breaks compatibility with another business application? Well now we’ve added on extra time so we can upgrade both. If there is no other value to the upgrade, then the fix might be held until there is.

Mitigating Factors - There may be enough other security controls in the network to reduce risk. Maybe in this case only the core servers for this application are affected - and they reside on their own network segment behind a firewall. Maybe that firewall also runs an intrusion prevention system, which has already been updated with signatures to stop this particular attack. There could be a number of things going on which have given management the comfort that the system is still safe.

Any of these might lead to the final decision: we’re going to live with this risk. By accepting it, we’ve evaluated it, we’re aware of it, and we are consciously choosing not to fix it. We may have limited the scope of this risk via other mitigations - but ultimately we are not completely ridding ourselves of this risk.

Does risk acceptance only apply to security? Nope. It applies to just about anything really. What if I recommend that we need to replace older hardware or risk network instability? That risk could be acceptable if the hardware isn’t critical. Same thing goes if we decide to forego renewing support on a pair of firewalls. Maybe it’s even suggesting an implementation of a new technology. Proposing to a big cloud provider that they should prepare for IPv6? There is a good chance they may still accept the risk of IPv4 depletion 🙂

As an IT engineer, it’s not just our job to keep things running and get projects done - it’s also our responsibility to keep the best interests of the business in mind when it comes to design, implementation, and maintenance. We need to evaluate all our options and always recommend what we think the best path is. However, we also need to respect that our recommendations may not always be followed. There may always be something else going on that we’re not aware of.

One last piece of advice I can give - If there is ever something you feel very strong about, get the final decision in writing. If you truly feel that the risk is too great, and you’ve made your pitch that has still been rejected - get the decision in writing (whether email, chat log, etc). This may seem silly at the time, but it’s better to be safe. People forget, whether intentionally or not - and if the risk becomes a reality, you don’t want to give them any reason to put the blame on you.

Given a brand new switch, can you provide me the commands you would use to configure the first four ports for hosts in VLAN 15?

This question is always interesting because I get such a wide variety of responses. You can certainly filter out people quickly who have never touched a switch. Some people will start with conf t, while others just jump straight into setting the VLAN tag. Some people specify that they’ll use an interface range command, while others get confused and want to configure the ports as a trunk. In fact, during this year’s interviews I had quite a significant number of people who completed the scenario by providing the commands to configure a trunk port, instead of static access. One thing that struck me as noteworthy is that the people who did this also provided the commands they needed to change the default native VLAN.

For a vast majority of networking devices (maybe all of them), the default/native VLAN for a trunk port is VLAN 1. This is not the best configuration for reasons we’ll get into a bit later - but unfortunately it needs to be manually changed. In every interview where the candidate suggested this be changed, I followed up with asking why. I asked so that I can find out two things: How well the individual is at explaining concepts, and whether or not this is something they just do because they were taught or if they actually understand the logic behind it. Surprisingly, a vast majority of the candidates could only provide the reasoning that “Well, VLAN 1 is bad” - but they couldn’t elaborate on why.

So why is VLAN 1 bad to use?

Technically, VLAN 1 itself isn’t the problem. The concept of a default VLAN allows for someone to attack a network by taking advantage of how switches use a default VLAN. Since VLAN 1 is typically set as the default for most vendors, then it becomes a well-known configuration for attackers to abuse. If every vendor had set the default native VLAN to 52, then we would still run into the exact same problem except the ‘bad’ VLAN would be 52. So let’s back up just a little here and explain a bit of background. The concept of a VLAN is a segmented logical network. Rather than requiring a different piece of physical hardware to keep hosts separate, we can assign them into different virtual LAN segments. This is accomplished by ’tagging’ each Ethernet packet with a VLAN ID. Internally, the switch code does not allow packets that contain one VLAN ID tag to be sent to hosts configured with a different VLAN ID tag.

Let’s look at a few practical examples of how this works. When you configure a host port, you would configure the switch with the appropriate VLAN ID tag that the host should be assigned to. The actual server may know nothing about the VLAN it is in, but the switch knows to inject that VLAN tag into the Ethernet headers of every packet received from that server. For the rest of that packet’s life on the network, every switch reads that VLAN tag to assist in forwarding decisions. Trunk ports are commonly used between switches, and these ports are enabled to carry multiple VLAN ID tags. The problem here is that each switch is expecting that the packets it receives already contain a VLAN tag in the Ethernet header.

So whats the problem with the default native VLAN? This is a special type of VLAN in which the switch never attaches a VLAN tag to the headers. Whenever we connect two switches via a trunk port, we are likely configuring multiple VLANs that need to cross that link. However, the switches themselves still need to communicate directly with eachother (for protocol negotiations/spanning-tree), and each switch isn’t necessarily going to know what VLAN the other wants to use for this management traffic. This is where the concept of a native VLAN comes in. For every configured trunk link, each switch has a default native VLAN for which it expects to receive packets with no VLAN tag headers. Even though normal network traffic crossing a trunk link is going to require a VLAN tag in the headers, the switch-to-switch control-plane communication is sent with no header present.

This is where VLAN 1 becomes a problem because of the native VLANs are processed/interpreted by the switch. Whenever a switch receives a packet which contains the VLAN ID set to the native VLAN, it knows that packet doesn’t need to contain a VLAN header - so it removes the VLAN ID header.

How could this be exploited? Well for example, let’s say that we had a network where every developer PC was using a trunk port and permitted to use VLAN tags. This might be so they could run local VMs for testing that connect to both the DEV and QA networks. If we leave the default native VLAN as 1, then a malicious developer could exploit this to gain access to another segment. This is accomplished by using a software package to double-tag an Ethernet packet with two separate VLAN ID headers. The first VLAN tag header will be set to the native VLAN (VLAN 1), and the second header will be set to the target VLAN - let’s say we use VLAN 20 for the accounting network. When the switch receives the packet across the trunk link, it will read the Ethernet headers. When it processes the first VLAN tag and sees that it matches the default native VLAN, the switch strips this header - which then leaves the second VLAN tag header for VLAN 20. When the switch goes to forward this traffic to it’s destination, the remaining VLAN header will allow the packet to bypass any security measures and be directly forwarded on VLAN 20.

Well what if we don’t provide end users with a trunk port? Could they still execute this type of attack? Remember that the default switchport configuration allows for dynamic negotiation - where the connecting computer can tell the switch whether or not it needs an access port or a trunk port.

If VLAN 1 is well-known as the default, what should the native VLAN be set to?

Anything you like! The key thing is that it should be a VLAN which has no access to any network resources - so it should be a VLAN with no hosts and gateway. I usually don’t even create the VLAN on the switch itself, therefore immediately black-holing all traffic sent to that VLAN.

How else can this be prevented?

Always statically set your end user ports to switchport mode access, and enable switchport nonegotiate. This will prevent the switch from allowing port negotiations, which prevents a user from tricking the switch into assigning a trunk port.

Newer Cisco switches also support a global configuration command: vlan dot1q tag native. This command will force the switch to require VLAN tags to be present on packets in the native VLAN. See more detail here. This will need to be configured on all switches in your network to be effective.

Do you change the native VLAN in your network? Or is it not a big security concern for you? Comment below!

In a lot of jobs, the initial training you receive is fairly straightforward. You are usually taught how to respond to a task by following a series of steps to get an intended result. Training like this is great - It helps to achieve consistency and efficiency. You bring in any new person and give them the exact same troubleshooting steps, implementation steps, and/or validation steps - and you’re likely to get a similar result every time.

This is the point where I have seen far too many people stop though. They are happy with doing their job, and don’t necessarily want to progress their career or maybe they don’t know how. These engineers will continue to produce decent work at the quality that they were taught at. Even for those who try and progress further (maybe through certifications or otherwise), there is a difference between learning new technologies/concepts and truly understanding them. For some people out there, having this basic level of skill is all they really want - and if that’s their goal in life, then this type task-based knowledge is perfect. But if you really want to master the domain of technology that you are interested in, then you need to put forth the time and effort into gaining that understanding.

For me, a true understanding of a technology means that you’re able to speak confidently about how something works, abstract concepts to apply to similar products, and mentally walk though how the technology might handle a given situation.

Let me provide an example or two that might help to frame this a bit better. Given a particular network, an engineer might know that for traffic to get from point A to point B, it travels through two firewalls. Every time there is a new request to permit a new traffic flow, that engineer knows that they must make a configuration change to one or both of those firewalls to allow that traffic through. However, to this engineer, the inner workings of that firewall are a complete mystery. The firewalls are complete black boxes which take in traffic through one interface and spit it out another. So when there is a technical issue within the firewall appliance, they may be extremely limited in their troubleshooting abilities - and they may have no choice but to call the vendor for support.

Another engineer who has a deeper understanding of how firewalls work might see the problem differently. This engineer knows that for every packet received, the firewall follows a specific flow of processing. That flow could include any number of things, including NAT, routing, firewalling, IPS, VPN, etc. This engineer knows which order those things get processed and what effects those processes can have on the traffic. So when we have a technical issue with this firewall appliance, this engineer may be able to mentally walk though the packet flow/processing and determine where the problem may be - sometimes without even looking further than general log files.

Another example is something that I see quite often. An engineer is asked to implement something - let’s say a new port configuration for a server. They follow their known process for implementing this change, but something doesn’t quite work right. So they change settings or maybe delete the entire port configuration and start over - but eventually they get it working. They don’t know why it didn’t work the first time, or what caused it to work the second time - but it works now, so they aren’t concerned with it. However, it’s possible this engineer ends up running into this same problem more than once. The ideal step here would be to step back and look at what was different between the original configuration and the working configuration. Maybe there is an additional command in the original configuration which seems suspect - a quick search of the internet could turn up an explanation behind why that command was preventing the port from working as expected. After that research, that engineer would not only know why their configuration didn’t work - but now they know what that command actually does, which could be beneficial in a future scenario.

As I briefly mentioned earlier, not all IT admins or engineers are concerned with gaining a significant level of understanding. There are those who want to come to work, get their job done, then go home to their families - and there is absolutely nothing wrong with that. For me personally, I can’t handle running into a problem and not knowing exactly what the cause was. An issue that “fixes itself” is never an acceptable answer to me, because if something caused the problem once then it can certainly happen again. I don’t enjoy having to blindly configure an option on a system without knowing what’s going on in the background. Some people might call me crazy, but this seems to be a skill/trait shared by many higher-level engineers I have worked with. So how do you get to a point where you really understand a system? For me, it’s been a lot of playing in labs, reading vendor documentation, and not settling until I feel like I can speak confidently to how something works. I never feel truly comfortable in a new company until I can mentally walk through every device a packet touches from source to destination - and know which devices configs/routes may have an impact on that flow. Any time there is a problem with something, I spend time digging into it until I know what caused it - even if the problem is only momentary and goes away. Not only do I then understand why the problem happened, but I also learn how to quickly identify similar issues again.

Especially if you’re still in the beginning stages of your career, I can’t stress enough how important it is to understand the technologies you’re responsible for. Take the extra time and study it, play with it, break it and fix it again. Know how things work and what their behaviors are under different conditions. Don’t settle for “It just works because it does”. One of the key skills I’ve seen in engineers who truly understand their domain of technology, is the ability to abstract concepts to apply to other systems. Someone who has a deep understanding of routing and switching technologies might prefer to work with a certain vendor, but given any router/switch they can make it work.

Have you worked with anyone who you think has a great understanding of what they do? What other skills or traits do they display that makes them successful? Comment below!

Anyways - after I cut over to my new firewall, I’ve been digging through logs to make sure that I didn’t miss anything. I have all of my device/lab logs going into an instance of Splunk Light (their free product). It makes it easy to collect and search through logs, and it’s extremely easy to set up and use. A few quick queries and I came across one or two minor things that needed to be tweaked on my firewall - but I also saw some traffic that I wasn’t sure about.

So that brings me to my question of the day: Do you know what’s going out of your network?

A lot of people I know only use firewalls to block inbound access, both in homes and businesses. For homes it’s more understandable since most average people aren’t network admins. However, it still surprises me how many businesses are willing to add a ‘permit any any’ out to the internet. Yes, I block all traffic by default through my home firewall, both inbound and outbound. Yes, it’s a bit of a pain sometimes when something isn’t quite working right - but it’s usually a quick ACL change, and overall I would rather take the minor inconvenience for the security gains.

When I originally built the firewall policy for my network, I started off simple. I know we need DNS, HTTP, and HTTPS outbound - easy enough, right? Then I started watching logs for blocked traffic and trying to decipher what else was trying to communicate outbound using another port. Some things were very easy to determine - TCP 5228 out to a Google owned IP? Yep that’s actually a known thing - a lot of Google services, like Chrome, will use this. Some other things were harder to figure out - like game consoles which use a very wide range of non-standard ports. Many of these weren’t really documented well by the console manufacturer, and meant that I spent a while between browsing forums and some trial and error.

This really gets interesting when you start digging past the stuff you know about. What about a PC in my home network that is trying (and getting blocked) to reach a few random IPs in Korea and Russia over a bunch of non-standard TCP ports? Yeah that doesn’t make me feel comfortable. Could it be a legitimate application, or is it malware? A few quick searches on the internet don’t turn up anything immediately helpful. For the time being, I’ll keep stuff like this blocked until I have time to spin up some packet captures to see what this traffic is actually doing.

For a business I feel like this type of thing is even more important than just what I’m doing at home. You certainly don’t want end users (or servers) possibly running strange applications, which might be transferring data to some unknown external party. It seems like larger companies seem to have a better handle on restricting outbound access than most smaller companies, who likely don’t have the time or see the value. However, I’ve also worked with a few larger organizations who still permit all user and server traffic out to the internet with no filtering in place.

If you’re not already blocking outbound traffic - Get some good logging in place. Use something like Splunk Light and start collecting firewall logs for everything going out of your environment. Start with the basics - create a list of the software/ports you know you’ll need to open. After a few weeks, start digging through the logs to figure out what else might need to be added to your list. Once you feel comfortable that you’ve compiled a sufficient base ruleset, schedule a time to make the change and put it in place. Start blocking the unknown traffic - and only permit when necessary.

How do you have your firewalls configured today? Do you permit everything or are you very restrictive? Comment below - I’m curious to see what other people are doing.

Securing your BGP peering (Know who you’re connecting to)

BGP is a little different from most other routing protocols, since it uses a single unicast TCP connection between peers to exchange routing updates. Lucky for us, that means that we can easily filter traffic from only known peers. Once you have direct connectivity up between your edge router/firewall and your direct peer, lock down that connection with an ACL. Permit TCP port 179 traffic ONLY from your directly connected peer IP - no one else.

While you’re at it, let’s take it another step further: Request that your ISP set up BGP authentication. Sure, a majority of BGP implementations today still require use of MD5 for auth (which is terrible) - but some authentication is still better than none. This can usually be arranged at the time of turning up peering. Both sides configure the same authentication password and with any luck the peering still establishes.

BGP by nature is unfortunately not the most secure protocol - but a few simple steps like this will help ensure you’re only connecting out to authorized peers.

Route filtering (Don’t trust anyone)

Usually when you’re filling out the BGP peering paperwork for your service provider, they will ask you what kinds of routes you want. In most cases, you should be able to request one of the following:

Default only - Exactly what it sounds like. Your provider will only advertise a route for 0.0.0.0/0. In many cases, this is probably what you’re going to want. With this type of advertisement, each upstream provider will just give us the same default route to the internet. From there we can weight which one we want to use, and traffic will automatically fail-over to the secondary connection should the primary fail.

Partial - If for any reason you want to weight routes to certain destinations differently, then we might request this. In this case, you’re probably going to still receive 0.0.0.0/0 plus any specific routes you ask for. A good example of this is if we wanted to specifically manipulate routes for a remote office we have. Maybe we want to weight Internet traffic for one uplink, and VPN traffic to a remote office on the other uplink.

Full - In 99% of typical business cases, this won’t be required. This option means the upstream providers will be dumping the entire Internet routing table on you. While this offers you a ton of control over path manipulation, it also requires significant memory resources on your routers in order to maintain that routing table.

After we figure this out, the next step is to make sure we are filtering the routes we accept from the upstream provider. Wait - didn’t we just tell them exactly what routes to send us? Why do we need to filter them? Well you can never be too safe here - and we would rather perform an unnecessary filtering than have an ISP accidentally misconfigure route advertisements. So if you’re only expecting a route for 0.0.0.0/0, then filter your inbound route advertisements so you only accept that route.

Same thing goes for outbound route advertisement  - if we own a /24 of public IP space, then we only want that range to be advertised out. Some providers may already filter this on their end, but again it doesn’t hurt here to be extra cautious. If we are accepting anything other than a default route from our provider, then we run the risk of leaking those additional routes between the two providers - which would lead to inadvertently becoming a transit AS. Chances are pretty good that you don’t want that, so make sure you configure filtering for all outbound route advertisements.

Minimum Advertisement (Oh no, we have to re-address everything)

I mentioned this in the original post - but typically when you are peering with two separate upstream providers, you need to advertise no less than a /24. We ran into this at my last job, where we had been provided a /25 by AT&T but we needed to bring in a second carrier via BGP. The reasoning behind this is to keep global routing tables as small as possible, by not allowing them to end up flooded with a ton of routes for smaller subnets. It makes sense, but on the other hand I feel like requiring a /24 in all cases can be a bit wasteful. My last job only required maybe 30 publicly addressable hosts - which meant that the remaining addresses went unused.

At any rate - should you find yourself in this scenario then you’re going to have to face the inevitable: Renumbering into a new IP space. Any time you have to do this, it’s going to be a bit of a pain - but for external addressing like this it might be easier. So in our case, the entire /25 space was hosted on our external firewall then NAT’ed into DMZ servers.

Here is the quick steps that I used to do a side-by-side migration without taking any significant downtime:

• Get the new subnet up and running - assign the interface addresses on your firewall and BGP up and running
• Assign new IP addresses to all of your existing services
• Configure NAT rules for the new external IP addresses to the DMZ hosts - while leaving the existing NAT rules for the old subnet (Also make sure your firewall rules permit the same traffic to either IP)
• Migrate DNS entries externally to point to the new IP space
• Once traffic stops flowing to the old IP, remove the old NAT

As a side note - if you procure redundant internet connections through the same upstream provider, then you might be able to work out something else. They may be able to provide you a private ASN to use, and they will likely accept any minimum advertisement - since they will be summarizing upstream within their network anyways.

I had a few more things I originally intended to cover here - but it seems that these topics are filling way more space than I thought they would. Specifically, I’m thinking about a dedicated post to BGP path manipulation - which is probably something you’re going to want to implement after peering is established. Hopefully these tips help! If you have any questions, throw them in the comments below.

So the concept of port security is fairly simple - We want to secure each individual switch port to a physical layer 2 MAC address, or at least limit how many unique MAC addresses might be learned on an individual port. The technology could be used to just limit the number of simultaneous devices on a port - by just setting a MAC threshold. Or we can also take it to the extreme and lock down each port to a hard-coded MAC address - which will never allow another device to connect. You might be thinking that the second method is absolutely ridiculous, but it really depends on the business needs.

First, let’s take brief look at the typical port security configuration and some of the options available.

SecureSwitch(config)# interface x/x  ! Whichever interface we want to lock down
SecureSwitch(config-if)# switchport port-security max xx  ! Max number of MAC addresses that can be learned
SecureSwitch(config-if)# switchport port-security violation xxxxx  ! Choose to either restrict or shutdown the port (description below)
SecureSwitch(config-if)# switchport port-security  ! This actually enables the port security config

Fairly straight-forward, right? We choose a port (or you could do a range) and set a few options. The default number of MAC addresses able to be learned on a port is 1, so it’s likely you’re going to want to change this - unless 1 is all you need. Port security can only be enabled on access ports, so 1 MAC address works in most cases  - except where you have a PC daisy-chained off of an IP phone (in which case this will need to be set to 2 or 3).

Next we set our preferred violation action. This step is pretty important because it defines what happens when the port exceeds it’s MAC count. Restrict is the passive approach. If we have two PC’s plugged into a single access port (maybe using an unmanaged switch), then the second PC will just never be able to work as long as our max MAC limit is 1. The first PC to connect will be fine, and the switch will log a message and send an SNMP trap when the second MAC is picked up. Shutdown is the more forceful approach. Once that second MAC address turns up on the port, the switch puts the port in an err-disabled state - which shuts down the port to all traffic. This event is also logged and generates an SNMP trap - however the port will not come back online until an administrator manually re-enables it.

Now that we see a basic config, let’s take a look at a few different use cases for this feature. In one of my previous jobs, I worked as a network admin for a local government organization. Port security configuration in that environment was extremely strict. Each switchport was configured to permit only one MAC address, shutdown upon violation, and the switchport port-security mac-address sticky command was also used. This command takes the first MAC address learned on the port and commits it to the running configuration, which means that this MAC is essentially hard-coded to be the only MAC permitted on the port. So in this environment, a single PC was tied to a single port - nothing else could ever be plugged into that port without either shutting down the port or administrator intervention. In a government office, this was absolutely necessary because every device on the network needed to be tracked and personal devices were not permitted to be connected. We needed to know if anything was ever plugged in that wasn’t an authorized device - so manual intervention and investigation was a requirement.

In a more typical office environment - port security configurations can just be a good security practice without going overboard with it. We never want a user to plug in a rouge switch into our network without our knowledge, right? So maybe we assume each user has an IP phone and PC, and limit the port to 2 MAC addresses. In this case, we can go ahead and just set the port to restrict. We don’t want to prevent the user from working if a port violation occurs, nor do we want to spend time resetting the port for them - but we might still want to be notified, especially if it happens often. In addition, port security is an excellent way to secure ports public areas. For example, maybe we have an IP phone or kiosk PC in our lobby. These need access to the network, but we don’t want anyone to be able to unplug that device and gain access into our network. In cases like this, it would actually make sense to have the switch only permit access from that single MAC address.

Outside of the ‘practical’ use cases, there is also the strictly security side of things. I’ve touched on a few considerations already - but there are also certain types of attacks that can be defeated by port security. One of those would be exhausting the CAM table resources. A malicious person could use publicly available tools to spoof MAC addresses in the packets they send to the switch. Tools like this force the switch to learn hundreds of thousands of MAC addresses, which eventually will overload the CAM table. When a switch CAM table becomes full, the switch begins flooding packets out all interfaces. This is because the switch can no longer assign mappings between MAC addresses and the ports they originate from - so the switch has no choice but to flood everything and hope the correct recipient receives the data. For the attacker, this means they can run a packet capture on the port and collect information they wouldn’t have otherwise needed to. This scenario could be prevented by implementing port security, which could simply restrict the number of MAC addresses learned off of any individual interface.

Port security configuration can be implemented in a few different ways depending on your use case. Overall though, it can prove to be a useful way to help implement security controls on your network.  What do you think about port security? Extremely useful or does it just get in the way? Comment below and tell me how you have implemented it!