I’ve been getting a handful of questions lately on the process of bringing a Cisco Catalyst 8000v or CSR 1000v into an SD-WAN environment. So I figured maybe I should put something together to share.

On a related note - I’ve been debating for a while doing a blog and/or video on building out a Cisco Viptela SD-WAN lab in EVE-NG. This would (tentatively) include everything from building controllers, bringing up remote sites, and template/policy configs. If this is something you might get value from, please let me know!! I’m looking for some motivation :)

Okay - all that being said, let’s go ahead and get started.

Note: This guide should work with CSR 1000v devices as well. But it will NOT be 100% accurate for physical ISR/IOS-XE routers, as there are some additional steps with certificates & the Plug and Play portal to get those running.

Topology

So to start with, figured I would share the topology that I’m working from. If you read my last post you might recognize this, but with an added location. This new location (site id 400) contains our Catalyst 8000v VM, running IOS-XE version 17.04.01a.

Controller or Autonomous Mode?

Back in the earlier days of IOS-XE SD-WAN, there used to be two separate software images to load on your network appliance - one for traditional IOS-XE, and one for SD-WAN code.

With the newer releases of IOS-XE, we’re now getting a unified image that contains both software sets. So our options now are two modes: autonomous (traditional IOS-XE) or controller (SD-WAN).

One way we can check this, is by running a show version and looking for Router operating mode:

Router# show version
<-- Output omitted -->
cisco C8000V (VXE) processor (revision VXE) with 2035355K/3075K bytes of memory.
Processor board ID XXXXXXXXXXX
Router operating mode: Autonomous
4 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
3965744K bytes of physical memory.
5234688K bytes of virtual hard disk at bootflash:.

Configuration register is 0x2102

Okay, and looks like the Catalyst 8000v image I’m using booted up in autonomous mode. No big deal, we can change modes pretty easily!

So in normal exec-mode, we’ll use the command controller-mode enable:

Router# controller-mode enable
Enabling controller mode will erase the nvram filesystem, remove all configuration files, and reload the box!
Ensure the BOOT variable points to a valid image
Continue? [confirm]

As noted in the snippet above - this command will erase the config! So you will want to take a backup snapshot of your existing configuration, if this is already being used. In my case, it’s a brand new VM - so we’ll continue on.

Once the device is back online, we’ll log in with the default login of admin/admin. Note that you will be forced to change this upon first login.

User Access Verification

Username: admin
Password:

Default admin password needs to be changed.

Enter new password:
Confirm password:
Router#

And just for fun, we’ll run a show version again to ensure we’re in controller mode:

Router# show version 
<-- Output omitted -->
cisco C8000V (VXE) processor (revision VXE) with 2035355K/3075K bytes of memory.
Processor board ID XXXXXXXXXXX
Router operating mode: Controller-Managed
4 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
3965756K bytes of physical memory.
5234688K bytes of virtual hard disk at bootflash:.

Configuration register is 0x2102

Catalyst 8000v Initial Config

So since this is a lab environment, and I’m not trying to provide any Day0 provisioning files - I’ll have to complete some manual configuration to get started.

First, we’ll start with some SD-WAN specific config (site id, org name, etc). I’ll be using the values that apply to my lab, so be sure to change these in yours!

Note: If you haven’t used IOS-XE SD-WAN previously, be aware that “conf t” doesn’t work! In SD-WAN/controller mode, you’ll use “config-transaction”. In this mode, all changes will need to be committed before they’re applied to the device.

Router# config-transaction
 ! Set hostname
Router(config)# hostname Cat8k-Site400
 ! Required SD-WAN system configs
Router(config)# system
Router(config-system)# organization-name "SDWAN-LAB"
Router(config-system)# site-id 400
Router(config-system)# vbond 192.168.99.241
Router(config-system)# system-ip 10.10.10.237
Router(config-system)# commit
Commit complete.

Cat8k-Site400# 

Once we have those basics configured, at a minimum we’ll also need to configure our internet-facing tunnel interface. This allows our Catalyst 8000v to communicate with our control plane for bring-up.

Cat8k-Site400#config-transaction
 ! Config physical interface (This is a lab, so I'm using a static IP)
Cat8k-Site400(config)# interface GigabitEthernet1
Cat8k-Site400(config-if)# ip address 192.168.99.237 255.255.255.0
Cat8k-Site400(config-if)# no shut
 ! Config tunnel interface
Cat8k-Site400(config-if)# interface Tunnel1
Cat8k-Site400(config-if)# no shut
Cat8k-Site400(config-if)# ip unnumbered GigabitEthernet1
Cat8k-Site400(config-if)# tunnel source GigabitEthernet1
Cat8k-Site400(config-if)# tunnel mode sdwan
Cat8k-Site400(config-if)# exit
 ! SD-WAN tunnel config
Cat8k-Site400(config)# sdwan
Cat8k-Site400(config-sdwan)# interface GigabitEthernet1
Cat8k-Site400(config-interface-GigabitEthernet1)# tunnel-interface
Cat8k-Site400(config-tunnel-interface)# encapsulation ipsec
Cat8k-Site400(config-tunnel-interface)# color biz-internet
Cat8k-Site400(config-tunnel-interface)# exit
 ! Default route to our internet gateway
Cat8k-Site400(config)# ip route 0.0.0.0 0.0.0.0 192.168.99.1
Cat8k-Site400(config)# commit
Commit complete.

Cat8k-Site400#

A Note on Certificates

Since this is a lab environment, I’m using self-signed local certificate authority to provision all of my certificate infrastructure. Because of this, I’ll need to install my local CA certificate on the Catalyst 8000v. If you’re using the default Cisco-provisioned certificate setup, you won’t need to do this.

Since my SD-WAN lab doesn’t have direct access to my local TFTP server - I do have an out-of-band management interface connected to my Cat 8000v. We’ll start by configuring that interface:

Cat8k-Site400# config-transaction
Cat8k-Site400(config)# vrf definition 512
Cat8k-Site400(config-vrf)# address-family ipv4
Cat8k-Site400(config-vrf)# exit
Cat8k-Site400(config)# interface GigabitEthernet 3
Cat8k-Site400(config-if)# vrf forwarding 512
Cat8k-Site400(config-if)# ip address dhcp
Cat8k-Site400(config-if)# exit
Cat8k-Site400(config)# ip tftp source-interface GigabitEthernet 3
Cat8k-Site400(config)# commit

The standard management VRF/VPN for SD-WAN is 512, so I kept that config to match when I configured this management interface. This will all be over-written anyways once we get connected to vManage & configure/push our template configs.

Once that’s done, we can go ahead and copy our CA certificate to bootflash.

Cat8k-Site400# copy tftp://10.0.0.2/cacert.pem bootflash:
Destination filename [cacert.pem]?
Accessing tftp://10.0.0.2/cacert.pem...
Loading cacert.pem from 10.0.0.2 (via GigabitEthernet3): !
[OK - 1406 bytes]

1406 bytes copied in 0.112 secs (12554 bytes/sec)

Cat8k-Site400# dir bootflash: | inc cacert.pem
16      -rw-             1406  May 14 2021 14:41:27 +00:00  cacert.pem
```text

Then we'll use the command below to install the CA certificate:

```text
Cat8k-Site400# request platform software sdwan root-cert-chain install bootflash:cacert.pem
Uploading root-ca-cert-chain via VPN 0
Copying ... /bootflash/cacert.pem via VPN 0
Updating the root certificate chain..
Successfully installed the root certificate chain

And we can validate by using the show sdwan certificate root-ca-cert command:

Cat8k-Site400 show sdwan certificate root-ca-cert
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            11:3b:0a:12:17:b0:e0:b5:4b:fa:c2:e9:2c:9c:12:84
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: DC = local, DC = 0x2142, O = SDWAN-LAB, CN = 0x2142-0XWIN1-CA-2
        Validity
            Not Before: Nov 29 20:00:11 2018 GMT
            Not After : Nov 29 20:10:11 2043 GMT
        Subject: DC = local, DC = 0x2142, O = SDWAN-LAB, CN = 0x2142-0XWIN1-CA-2
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
        <-- Output omitted -->

Activating the Catalyst 8000v

Almost there! Now that we’ve finished our pre-config & added our root CA certificate - we’re ready to join the Catalyst 8000v to the SD-WAN fabric.

We’ll start over in vManage - by going to Configuration > Devices.

Then we’ll find our target, unused Catalyst 8000v device. Click the ellipsis on the right side, then select Generate Bootstrap Configuration

This will give us a prompt to select which configuration style to generate. We’ll leave this on “Cloud-init”:

Once we hit okay - we’ll be presented with the info we need. We won’t necessarily need all of this information, but we’ll want to take note of our uuid and otp:

We’ll drag this info back over to our Catalyst 8000v, and we can now use it to activate the device & join to our SD-WAN fabric.

For the command below, chassis-number will be our uuid value - and token will be our otp.

Cat8k-Site400# request platform software sdwan vedge_cloud activate chassis-number C8K-178BXXXX-XXXX-XXXX-XXXX-XXXXXXXXBC24 token 421ecxxxxxxxxxxxxxxxxxxxxxbd53b5

Validation

After a few moments, we’ll see some log messages start to appear showing our control connections coming up. You might see these on the terminal if you’re using the console port, or you can use show log:

*May 14 15:39:06.910: %Cisco-SDWAN-Cat8k-Site400-OMPD-3-ERRO-400002: R0/0: OMPD: vSmart peer 10.10.10.242 state changed to Init
*May 14 15:39:07.980: %DMI-5-AUTH_PASSED: R0/0: dmiauthd: User 'vmanage-admin' authenticated successfully from 10.10.10.240:48140 and was authorized for netconf over ssh. External groups:
*May 14 15:39:08.798: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400002: R0/0: OMPD: vSmart peer 10.10.10.242 state changed to Handshake
*May 14 15:39:08.804: %Cisco-SDWAN-Cat8k-Site400-OMPD-5-NTCE-400002: R0/0: OMPD: vSmart peer 10.10.10.242 state changed to Up
*May 14 15:39:08.808: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400005: R0/0: OMPD: Number of vSmarts connected : 1
*May 14 15:39:09.827: %Cisco-SDWAN-Cat8k-Site400-OMPD-3-ERRO-400002: R0/0: OMPD: vSmart peer 10.10.10.243 state changed to Init
*May 14 15:39:10.584: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
*May 14 15:39:11.756: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400002: R0/0: OMPD: vSmart peer 10.10.10.243 state changed to Handshake
*May 14 15:39:11.762: %Cisco-SDWAN-Cat8k-Site400-OMPD-5-NTCE-400002: R0/0: OMPD: vSmart peer 10.10.10.243 state changed to Up
*May 14 15:39:11.762: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400005: R0/0: OMPD: Number of vSmarts connected : 2
*May 14 15:39:12.738: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400007: R0/0: OMPD: Using policy from peer 10.10.10.242
*May 14 15:39:13.570: %Cisco-SDWAN-Cat8k-Site400-FTMD-6-INFO-1000020: R0/0: FTMD: SLA class added : class 'Default' at index '1' loss = 25%, latency = 300ms, jitter = 100ms, app-probe-class = None
*May 14 15:39:14.738: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400007: R0/0: OMPD: Using policy from peer 10.10.10.242

We can also see our control connections using the command show sdwan control connections:

Cat8k-Site400# show sdwan control connections
                                                                                       PEER                                          PEER                                          CONTROLLER
PEER    PEER PEER            SITE       DOMAIN PEER                                    PRIV  PEER                                    PUB                                           GROUP
TYPE    PROT SYSTEM IP       ID         ID     PRIVATE IP                              PORT  PUBLIC IP                               PORT  ORGANIZATION            LOCAL COLOR     PROXY STATE UPTIME      ID
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
vsmart  dtls 10.10.10.242    100        1      192.168.99.242                          12446 192.168.99.242                          12446 SDWAN-LAB  biz-internet    No    up     0:00:00:51  0
vsmart  dtls 10.10.10.243    100        1      192.168.99.243                          12446 192.168.99.243                          12446 SDWAN-LAB  biz-internet    No    up     0:00:00:49  0
vmanage dtls 10.10.10.240    100        0      192.168.99.240                          12646 192.168.99.240                          12646 SDWAN-LAB  biz-internet    No    up     0:00:00:52  0

Of course, we can also check to see our device status in the vManage dashboard as well.

Over on the Monitor > Network page, we can see that our new Catalyst 8000v is now online:

Extra: How do I check the routing table on an IOS-XE SD-WAN device?

So - if you’ve only used the vEdge software devices, you may be used to using the show ip route or show ip route vpn 10 commands.

In the IOS-XE world, most SD-WAN commands are prefixed with the sdwan keyword. For example: show sdwan bfd sessions (where on vEdges, it would just be show bfd sessions)

This might lead you to believe that you can use show sdwan ip route or show sdwan ip route vrf 10 - but these won’t work! In fact, you’ll get the following message:

Cat8k-Site400# show sdwan ip route
% Error: This command is not supported

For the IOS-XE based devices, they actually just use the standard IOS-XE routing table and VRF constructs.

So on a vEdge, you would have VPN 0 as your transport VPN. On IOS-XE, this is just the default global routing table, shown with show ip route.

But what about our LAN-side service VPNs? In this case, our routes are being dumped into a VRF on the IOS-XE device.

So for example, I have VPN 10 in my lab which is used for LAN-side clients. We can use the show vrf command to see that this exists, and then show ip route vrf 10 to see the routes from our other SD-WAN locations:

Cat8k-Site400# show vrf
  Name                             Default RD            Protocols   Interfaces
  10                               <not set>             ipv4        Gi2
  512                              <not set>             ipv4        Gi3
  65528                            <not set>             ipv4        Lo65528

Cat8k-Site400# show ip route vrf 10

Routing Table: 10
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
       n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       H - NHRP, G - NHRP registered, g - NHRP registration summary
       o - ODR, P - periodic downloaded static route, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR
       & - replicated local route overrides by connected

Gateway of last resort is not set

      10.0.0.0/24 is subnetted, 2 subnets
m        10.2.2.0 [251/0] via 10.10.10.235, 00:09:02, Sdwan-system-intf
m        10.3.3.0 [251/0] via 10.10.10.236, 00:09:02, Sdwan-system-intf

Okay, that’s it! Pretty quick process overall - and now we can get into applying our device/feature templates.

Hope this was helpful!!

In this post, we’ll walk through configuring a Cisco Viptela SD-WAN network to integrate with Cisco Umbrella’s Secure Internet Gateway (SIG) & Secure Web Gateway (SWG).

If you’re interested in reading more about what SIG & SWG are, please check out my last post where I walked through this integration with a Meraki MX firewall.

For additional reading, there’s also a good Umbrella blog post on all the components of Cisco’s SASE architecture, which you can find here.

Okay, with that being said - let’s get started!

Step 1: Umbrella API Keys

Okay, so we’ll start on the Umbrella side. We’ll need to generate a set of API keys, which we’ll push out to our WAN edge devices. These API keys will allow our remote devices to reach out to Umbrella & auto-configure their SIG tunnels.

We’ll log into our Umbrella dashboard at https://login.umbrella.com/

Once we log in, we’ll hop over to Admin > API Keys. Then up in the upper-right corner, click Create.

For this integration, we’ll need to select Umbrella Management to make sure our API keys have the access they need. Then click Create. Easy enough, right?

Once we do that, the Umbrella dashboard will display our API keys - which we’ll need to copy over to our Viptela SD-WAN environment.

We’ll also need to collect our Umbrella Organization ID. This is actually just embedded within the Umbrella Dashboard URL, and should be a seven-digit number. For example, if we look at our current URL on the API keys page:

https://dashboard.umbrella.com/o/<ORG ID>/#/admin/apikeys

And that’s all we need for now from the Umbrella side. So let’s hop over to Viptela.

Step 2: Viptela Templates & Config

For this post, I’ll be using my SD-WAN lab in EVE-NG. Currently I just upgraded the lab to run Viptela version 20.4.1, though the SIG integration has been supported since 20.1.

Just for reference, here is the lap topology that I’m working with:

This lab includes a bridged connection to allow direct internet access from the lab network.

To configure our SIG automatic tunnels, we’ll need to create / update a few templates:

• Create a SIG Credentials feature template
• Create a SIG feature template
• Assign SIG templates to device templates
• Edit Service-side VPN Template to inject a service route

2a: Creating a SIG Credentials Template

First we’ll create a template which will store our Umbrella API credentials.

In the vManage dashboard, we’ll go to Configuration > Templates, then drop into the Feature tab, and click on Add Template.

Once we get to the next screen, we’ll select our device type. In my case, I’m using a vEdge Cloud.

A list of templates will pop up - we’ll drop down to SIG Credentials, which will be near the bottom under the Other Templates header.

We’ll then be taken to the page where we create our template.

We’ll fill in our template name & description, then paste in our Umbrella details (Org ID, API Key, API Secret).

Once we’re done - We’ll hit Save.

Note: At time of writing, the “Get Keys” button only functions if you’ve purchased your SD-WAN subscription with DNA Premier licensing. This license level includes Umbrella SIG, and allows this 1-click integration that pulls your Umbrella API keys automatically.

2b: Creating a SIG Tunnel Template

Okay, next we’ll need to create another feature template to specify our IPSec tunnel configuration.

Just like before, we’ll head back over to Configuration > Templates > Feature > Add Template.

This time, after selecting our device type, we’ll choose Secure Internet Gateway (SIG) / WAN - which is located under the VPN header.

In the template configuration, we’ll give the template a name & description as always. Then we’ll jump down to the tunnel config.

We’ll select Umbrella for SIG Provider, then click Add Tunnel.

Within the tunnel config, we’ll specify an interface name - I’ll name mine ipsec1 (and I’ll create an ipsec2 shortly). We’ll also specify a Tunnel Source, which in my lab is ge0/0 for the biz-internet VPN 0 interface.

We’ll keep Data-Center as Primary, then click Add. Then - we’ll add a second tunnel configuration, but using Data-Center as Secondary and the interface name as ipsec2.

When all that is done, we should have the following:

If we scroll down to the bottom of the template config, we’ll have some settings for High Availability. Here, we’ll specify what our active & backup IPSec tunnels are, and their weights. I’ll specify ipsec1 as active & ipsec2 as backup. Then click Save to finish our template.

Note: Weight settings are only available starting with 20.4.1 & allow for ECMP routing to SIG. Not shown above, you can create up to four HA tunnel pairs. Depending on weighting, you can equal-cost or unequal-cost load balance between those pairs.

Why would you want to load balance across multiple tunnels? At time of writing, each IPSec tunnel is limited to a maximum 250Mb/s throughput. By creating multiple tunnels & load balancing, we can overcome this limitation if we need higher bandwidth.

2c: Attaching SIG to Device Templates

After we’ve built out our two SIG templates, we can now attach them to our device templates.

For me, I currently only have a single device template which is applied to all of my remote WAN edge devices.

So we’ll go back to Configuration > Templates and find whichever device template we want to use - then click the ellipsis on the far right, end select Edit.

In the device template, we’ll scroll down and look for our VPN 0 configuration under Transport & Management VPN. We’ll attach our SIG tunnel template to VPN 0, since that’s where those IPSec tunnels are being sourced from.

On the right side, under Additional VPN 0 Templates, we’ll click Secure Internet Gateway to add our template. Then from the drop-down, we can select the feature template we just created:

We’ll also include our SIG credentials template, which we can find at the bottom of the page, under Additional Templates:

Once we’re done, we can click Save at the bottom. This will take us through the process of pushing out these changes to our remote WAN edge devices. When this configuration is deployed, each vEdge will now reach out to Umbrella & auto-configure an IPSec tunnel.

Note: While the IPSec tunnels will be established when this configuration is pushed - no traffic will flow over them yet. We’ll need to add a service route for that, which we’ll do next!

2d: Injecting a Service Route

Now we have our IPSec tunnels deployed - all we need to do is start routing traffic out to Umbrella. We’ll accomplish this using a service route on our LAN-side VPN.

So we’ll go over to Configuration > Templates > Feature - and we’ll scroll down & find whichever template we’re currently using on our LAN side. For me, I want VPN 10 to route over the tunnels, so I’ll be editing my VPN 10 template.

Within the template, we’ll scroll down to the Service Route section, click New Service Route, and we’ll enter our desired prefix. This will be what traffic will be routed over the IPSec tunnel to Umbrella. Since this is intended to be an internet gateway, I’ll enter 0.0.0.0/0 to inject a default route to Umbrella.

Service should be pre-selected as SIG, so we’ll click Add, then Update & deploy our changes to the remote edge devices.

Step 3: Validation & Troubleshooting

Awesome! Now we should have everything configured & working. So let’s jump through some initial testing and validation we can do.

Within vManage, we can check the status of our IPSec tunnels. We’ll go over to Monitor > Network then select one of our WAN edge devices.

We’ll click on the Interfaces tab on the left side - and we should be able to see a list of all interfaces on our device. This should include an ipsec1 & ipsec2 interface.

I’ve also selected the Real Time monitor on mine, and filtered to just show traffic on the ipsec1 interface:

You might recall from my topology above, that I have a linux VM running behind each of the two vEdge Cloud appliances. Using these, I can also test web access & see what my external IP is:

And sure enough, I am getting a 146.112.x.x address - which belongs to the Umbrella datacenter.

If we log into one of our vEdge devices, we can check the routing table with show ip route vpn 10 (or whichever VPN you’re using for the LAN-side). We should see our default 0.0.0.0/0 route via ipsec1, with a next-hop-VPN of VPN0:

vEdge-01# show ip route vpn 10

     ADDRESS               PATH             PROTOCOL          NEXTHOP  NEXTHOP                         NEXTHOP
VPN  FAMILY   PREFIX       ID    PROTOCOL   SUB TYPE  METRIC  IFNAME   ADDR     TLOC IP  COLOR  ENCAP  VPN      STATUS
------------------------------------------------------------------------------------------------------------------------
10   ipv4     0.0.0.0/0    0     std-ipsec  -         0       ipsec1   -        -        -      -      0        F,S
10   ipv4     10.2.2.0/24  0     connected  -         0       ge0/2    -        -        -      -      -        F,S

We can also check specifically our SIG tunnel status using the command show secure-internet-gateway tunnels:

vEdge-01# show secure-internet-gateway tunnels

TUNNEL                                                        API   LAST
IF                                                            HTTP  SUCCESSFUL     TUNNEL
NAME    TUNNEL ID  TUNNEL NAME                     FSM STATE  CODE  REQ            STATE
-------------------------------------------------------------------------------------------
ipsec1  530233543  SITE200SYS10x10x10x235IFipsec1  st-tun-up  200   create-tunnel  -
ipsec2  530233542  SITE200SYS10x10x10x235IFipsec2  st-tun-up  200   create-tunnel  -

In that output above, we’ll see that we have both ipsec1 & ipsec2 tunnels shown. The last API queries to Umbrella have a HTTP 200, which is good. We’ll also see our tunnel names, which we can use to find our device tunnel configuration in the Umbrella dashboard.

The tunnel name might look like a mess at first, but it’s a unique identifier to represent each tunnel that was created. So if we break it down for this vEdge:

• This vEdge is at Site ID 200
  • Shown as “SITE200”
• This vEdge has a system IP of 10.10.10.235
  • Shown as “SYS10x10x10x235”
• This vEdge has two IPSec interfaces, ipsec1 & ipsec2
  • Shown as “IFipec1” and “IFipsec2”

If we jump over to the Umbrella dashboard, we’ll see the same. Within the Umbrella dashboard, we can jump to Deployments > Network Tunnels.

On this page, we should see a total of four tunnels listed (two from each vEdge appliance):

And with everything configured & validated - now we can move onto configuring firewall and web filtering policies within Umbrella!

If you’re interested in seeing how to configure Umbrella’s cloud firewall & web filtering policies - please check out my YouTube Video above!

In today’s blog post - we’ll be checking out how to integrate a Meraki MX firewall to Cisco’s Umbrella Secure Internet Gateway (SIG) & Secure Web Gateway (SWG) services.

Interested in seeing how to do this with Cisco Viptela SD-WAN? Check out this post

Note: This integration is still rather new - so I anticipate some of the screenshots & steps below may differ as this post ages.

What are SIG & SWG?

Good question! A lot of us are probably familiar with Cisco Umbrella (formerly OpenDNS) for the DNS-layer security products. Using Cisco Umbrella, we can configure our end PCs or DNS forwarders to use the Cisco DNS servers. Then, we apply security policies to allow/deny DNS queries based on our policy.

Now applying security policy at the DNS level is great! Typical web filtering only inspects HTTP/HTTPS traffic, but inspecting DNS traffic allows us to stop threats that might not use those typical ports. While there may be some applications or malware that use hard-coded IP addresses, a good majority use a domain name that requires a DNS lookup.

But what if we wanted to still do web filtering too? Or maybe we want a centralized, cloud-hosted firewall service? That’s where Umbrella SIG & SWG come in.

Secure Web Gateway (SWG) is pretty much exactly what it sounds like. It’s a cloud-hosted web filter or web proxy, where we can set URL-based policys to permit or deny access. It supports the usual allow-lists, deny-lists, HTTPS inspection, file inspection, and policies based on user/group identities.

Secure Internet Gateway (SIG) is a large, cloud-hosted firewall service. What we can do with SIG is tunnel all of our traffic to one centrallized location to apply firewall policies (permit/deny/etc).

Why would we want either of these functions to be cloud-hosted? Well we might not have the resources at every remote site to perform firewalling & web filtering - or we don’t want to invest in beefy hardware in our corporate datacenter, and backhaul all of our remote traffic back for inspection. The cloud-hosted piece also means a central management point, so only one place to make change for all of our remote sites - rather than having to touch dozens or hundreds of remote devices…..Did someone say SASE? 🙃

With all that said - Let’s get into making this work!

Note: There are many ways to use Umbrella SIG/SWG (IPSec tunnel, PAC file, Anyconnect, etc). This particular post will only cover IPSec via Meraki MX.

Step 1: Getting our Keys

First thing we’ll need to do is on the Umbrella side. We’ll need to generate tunnel keys for our Meraki MX to use for IPSec negotiation.

We’ll log into our Umbrella dashboard at https://login.umbrella.com/

Once we’re in, we’ll navigate to Deployments > Core Identities > Network Tunnels. Shown below, we currently have no tunnels configured:

In the upper right corner, let’s click the Add Tunnel button.

We’ll now see the following screen to begin creating our IPSec tunnel:

Here, we’ll need to specify a name for our tunnel & which device type. The name can be anything we choose that helps us identify what locations are using this tunnel. I’ve specified MX64 as my tunnel name, since I only have one MX that I’ll be testing this with. We also have Meraki MX as an option in the Device Type drop down - so we’ll select that as well. Then, we’ll click Next.

Then we’ll need to configure our Tunnel ID and Passphrase:

The Tunnel ID we’ll be sending later, as part of our IPSec local ID. The passphrase will be used as our pre-shared key for the tunnel config. Once we’re done with that, click Save.

As long as our ID & passphrase are all good, we’ll be presented with a box to easily copy these parameters into the Meraki config:

Finally, we’ll be dropped back to our Network Tunnels page - where we can see that our tunnel is configured, but not yet established. Let’s fix that & hop over to the Meraki dashboard!

Step 2: Meraki MX IPSec Configuration

Okay, now onto the fun stuff.

We’ll log into our Meraki Dashboard at https://account.meraki.com/secure/login/dashboard_login

Once we’re in, you’ll need to select the network with the MX you want to connect. For me, I currently only have a single network, which contains my MX firewall.

Then we’ll navigate to Security & SD-WAN > Configure > Site-to-site VPN.

Assuming we have no other VPNs configured, you’ll have to change the VPN Type to Hub (Mesh) or Spoke. In my case, Umbrella SIG will be the only VPN I’ll have configured - so I’ll go ahead and use Hub (Mesh). Don’t mind the error regarding exit hubs, as we’ll be configuring a non-Meraki peer below.

Okay, scrolling down the page just a bit - we’ll need to specify which VLANs/internal subnets will be routed across the VPN & pushed through Umbrella for SIG/SWG:

For testing purposes, I have a VM in a subnet labeled DMZ that I’ll be using - So that will be the only VLAN that I’ll set to VPN On.

Next we’ll take a look at configuring Non-Meraki VPN Peers.

As shown in the screenshot below, we’ll need to configure the following settings:

• Name - This is locally significant, I set mine to Umbrella_SIG
• IKE Version - Set this to IKEv2
• IPSec Policies - Custom - See below
• Public IP - This is the peer Umbrella Datacenter
  • Choose the DC closest to you here
  • For this post, I’m using the New York DC
• Local ID - Set this to the Tunnel ID we got from Umbrella
• Remote ID - Leave blank
• Private Subnets - This is what destination IPs will be routed over the tunnel
  • Since this is intended as an Internet gateway, we’ll use 0.0.0.0/0
• Preshared Secret - Set this to the passphrase we configured in Umbrella
• Availability - Set this to which networks this tunnel should apply
  • Since I only have 1 network & 1 MX, I set this to All Networks

Okay - I mentioned above we’ll need to set some custom IPSec parameters. Here’s what we’ll configure as a custom policy:

Note: Turns out in the drop down menu, there is also an option for “Umbrella”, so you don’t have to create a custom policy. That being said, the Umbrella preset uses DH group 5, but Umbrella’s docs ask for DH group 14.

Lastly, we’ll be able to configure whether or not we want firewall logging (I enabled this) and also whether we want to add access-lists to permit/deny any traffic over the VPN. For now I’ll be leaving this as permit any.

Finally - we can click Save to push our configuration to the MX appliance!

Step 3: Validation & Testing

Hopefully once we get all that configuration done, we’ll be able to see our tunnel come up.

We can validate from both sides, but let’s start with Meraki. In the Dashboard menu, we’ll navigate to Security & SD-WAN > Monitor > VPN Status.

Then we’ll need to click the tab for Non-Meraki peer:

With any luck, we should see the little green circle showing that our tunnel is online.

If your tunnel doesn’t come up immediately, you may have to generate some traffic from your VPN subnet to the internet. This will force the MX to try and connect to Umbrella.

It’s also worth noting - sometimes it may take the Meraki dashboard to show a successful connection. A better indicator is by checking the Umbrella dashboard, or checking manually with a device you are expecting to route over the VPN.

If for any reason we have trouble, we can also check our VPN negotiation logs by going to Network-Wide > Monitor > Event Log.

On this screen, make sure you’ve selected for security appliances at the top. If needed, we can also filter events by All Non-Meraki VPN / Client VPN.

Since my tunnel came up with no issues, the output above shows a successful tunnel negotiation.

Let’s jump over to the Umbrella side, and see what the Umbrella dashboard shows.

Back on the Network Tunnels page - we should see that our tunnel now shows as Active.

This screen will also so the public IP that our MX is using to connect, as well as which Umbrella datacenter we’ve connected to.

Okay! That’s it for now. To try and keep this blog post from getting too long, I’ve decided to split this into two parts.

If you’re interested in seeing how to configure Umbrella’s cloud firewall & web filtering policies - please check out my YouTube Video above!

I have a local Cisco SD-WAN lab environment running at home, which was built in EVE-NG. It’s what I use whenever I need to test something for a customer, or just play around with the templates or APIs.

I’m planning on spending some hands-on time with new features soon, along with working on some automation projects - so it’s well past time to upgrade my lab.

Currently I’m running on version 18.4.302, but my intent is to upgrade to 20.3.2 - which is the latest version available today.

In this blog post, we’ll walk through how to upgrade a Cisco SD-WAN / Viptela network, including:

• Control Plane:
  • vManage
  • vBond
  • vSmart(s)
• Data Plane:
  • vEdge / vEdge Cloud

While my lab is using the on-prem controllers, these steps will work just the same if you’re using Cisco’s cloud-hosted controllers.

Downloading the Software Images

First thing’s first - in order to upgrade our environment, we need the correct software images!

Head over to Cisco Software Downloads, and search for SD-WAN (or click here!).

Once on this page, I just want to call out that we will need to click on the SD-WAN Software Update link. This may seem simple enough, but the other links for vManage Software or vSmart Software only contain images for a new install.

After that - just select the image version that you would like to upgrade to, and download both images.

In my case, since I’m downloading version 20.3.2, I’ll be using the following images:

• vSmart, vEdge Cloud, vEdge 5000, ISR1100 series and vBond upgrade image
  • File name: viptela-20.3.2-x86_64.tar.gz
• vManage upgrade image
  • File name: vmanage-20.3.2-x86_64.tar.gz

Adding Images to the Software Repository

The process for upgrading a Cisco SD-WAN environment is pretty straightforward.

The control plane can be upgraded independently of the edge devices, so long as everything stays within the bounds of the compatability matrix. In my case, I’ll be upgrading to 20.3.2 - and the controllers could still support edge devices as far back as 17.2. So no issues here!

Alright - let’s get started!

First, we’ll need to upload our images into our local vManage software repository. This is the file storage for all upgrade images, and it’s where the controllers & edge devices will go to pull their images from.

In the vManage dashboard, we’ll go to the Maintenance tab and select Software Repository.

Then, click on Add New Software.

In here we’ll see a few options: vManage and Remote Server / Remote Server - vManage.

We’ll use vManage if we want to upload & distribute images from the local vManage server we’re logged into currently.

Alternatively, we could use a remote file storage server by using the Remote Server option. If you choose to go this route, don’t forget to ensure that ALL controllers & WAN edge devices have access to this storage location.

After you select an option, it’s an easy drag & drop to upload the software images.

Planning the Upgrade

So before we get into actually applying our image upgrades - let’s address the questions of “What order do I upgrade things in?” and “What’s the impact?”.

Since this is a lab environment for me, I’ll be upgrading everything all at once - since uptime / outages aren’t a factor here

If you’re doing this in a production environment, I highly recommend performing these upgrades in an outage / maintenance window - or at least an off-peak time.

Yes, you can upgrade the controllers at any time without causing any issue. Yes, you can upgrade a redundant pair of vEdge devices and keep a branch online. However, I would advise you to try these out off-hours first - and get your own understanding of how this works & what to expect before doing it in production.

As for the upgrade order, we’re going to start at the top of the food chain and work our way down:

• Upgrade vManage first
• Then vBond
• Upgrade ONE vSmart controller & wait for it to come online / re-establish control connections
• Then upgrade the second / redundant vSmart controller
• After the control plane is upgraded & stable - move onto the edge devices

These steps can also be found under the Best Practices section of the Cisco SD-WAN Getting Started Guide. Cisco’s official recommendation is to wait 24 hours in between a few of those steps, to ensure platform stability - but for my lab that won’t be necessary

Upgrading vManage

Once we have uploaded our images & we have our upgrade plan - we can move forward with actually performing the image upgrades.

Starting with vManage - we’ll go to Maintenance > Software Upgrades > vManage.

Then we’ll click on Upgrade and select our version - in my case 20.3.2. After that, just click Upgrade

Now, what this does in the background is just pre-stage the vManage image for an upgrade. The actual software upgrade is not occurring just yet.

Think of this step like you might prepare an IOS/IOS-XE router: Copying the image to the device flash. The image is there and ready - but we haven’t booted to it yet.

Once that’s all done, we’ll go back to the vManage upgrade page and click Activate, select our image again, then click Activate.

Now this step is where vManage will reboot, apply the upgrade, and come back online with the new image.

Again, back to the IOS/IOS-XE analogy: this is the equivalent of setting out boot system flash:<image name> to the new image, then rebooting our router.

vManage may take a short while to complete & reinitialize. In my lab, about 10-15 minutes.

Upgrading vBond & vSmart

After vManage is done, it’s time to work on the real heart of our control plane: vBond & vSmart.

Similar to vManage, we’ll start by going to Maintenance > Software Upgrades > Controller.

Here we will need to select the devices we want to upgrade. As I mentioned earlier, you may want to do these one at a time & in a phased approach. However, in my lab I’ll select all of them to upgrade at once.

Now in this case, vManage will still follow the proper order (vBond, then vSmart), and even perform a rolling upgrade one device at a time. This may be suitable for you in production, but again I would urge you to test it for yourself first!

The other big difference here, as you can see in the screenshot above, is the presence of the Activate & Reboot checkbox.

This does exactly as you would anticipate. Instead of doing the two-step process with vManage where we staged the image, then performed the activation/reboot - this checkbox will do all of that in one step.

In my lab environment, I did check this box & allowed everything to reboot automatically.

Why is there a separation between uploading the image & rebooting / activating it? To allow better granularity over the process.

For example, maybe you have a poor internet connection at a branch site & the image upload may take a long time. This separation of tasks allows you to stage all of the images independently of applying them. If you have a short outage window, this could help you save time by pre-staging the images ahead of time.

Back to the upgrade - just like vManage we’ll select the version we’re applying then click Upgrade

Again - Depending on the resources available to your controllers, the image upload / activation process may take a short while…

Upgrading the WAN Edge Appliances

In my lab, I’m currently using a handful of vEdge Cloud VMs as branch office routers. The upgrade process here should apply to other edge devices as well.

After we’re confident our controller upgrades have been successful & all control connections have been re-established - we can move to upgrading our edge devices.

It’s also worth mentioning that my lab currently has no redundant deployments of WAN edge appliances. All of my test ‘branch offices’ are single-homed to one vEdge Cloud - so an outage will be required to apply the images.

If you’re using a redundant configuration at a remote site, ideally you would upgrade ONE edge device first. Then only upgrade the second after control connections & routing adjacencies had been re-established on the first device. This should allow for an upgrade with minimal downtime.

The process for upgrading the edge devices mirrors what we saw for vBond / vSmart.

We’ll go to Maintenance > Software Upgrade > WAN Edge, then select the edge devices we want to upgrade.

Click Upgrade, select the target version from the drop-down, then click Upgrade. Again, in my case, I also selected the Activate & Reboot checkbox

NOTE: It’s worth mentioning that for the WAN Edge upgrade, vManage will push the upgrade to all devices simultaneously. If you would like to perform a rolling upgrade here, you’ll have to manage it yourself. In the case of a site where a redundant vEdge is deployed & they share the same site ID, vManage automatically will handle upgrading only one at a time to maintain uptime (Thanks Tim McConnaughy for clarifying this!).

Once again, we’ll give the edge devices a few minutes to download & apply their software upgrades. Then we’ll check to ensure all of the control connections re-established & traffic is flowing.

Wrap up

Once your edge devices are back online, it’s all done! The network has been upgraded to the new version.

There are a handful of ways to check this, but one easy way is via the device monitor page: Monitor > Network

This page will list a summary of everything in the network, including the current software version & number of established control connections. For me, it’s an easy way to get a one-page summary of the network.

From the screenshot above - we can see that all of my lab devices are back online & running software version 20.3.2.

That’s it! Hope this post was helpful to you.

Thanks for reading!

What is NFVIS?

NFVIS is an operating system developed by Cisco which is intended to be deployed at branch office locations - and allow for quick deployment of network services in lightweight VMs. For example, we might want to reduce cost and hardware footprint by deploying a single ENCS machine, then deploy our typical branch services on top of that (DNS, Firewalls, SDWAN, etc). Under the hood, NFVIS is built on top of CentOS and KVM.

In the image below, we have an ENCS unit that is running ISRv, FTDv, and a vEdge Cloud. NFVIS has the ability to build out traffic flows for service chaining. In this particular setup, we could have all branch traffic receive a default route up to our ISRv. The ISRv forwards traffic to a Firepower VM (FTDv) which performs some traffic inspection before passing everything up to the vEdge Cloud.

We’ll be coming back to this diagram later to see how we can build out this flow of services. For now, let’s dive into how we can get NFVIS up and running.

Installing NFVIS

Lucky for us - the installation of NFVIS is fairly straightforward!

1. Create a bootable USB - or mount the installation ISO via CIMC
2. Upon boot, select “Install Cisco NFV Infrastructure Software”:

3. Wait. A while. Install time can vary depending on your hardware.
4. Once completed, log into the CLI: Default login = admin/Admin123#
5. You’ll be prompted to change the default admin password immediately:

Install completed! Now let’s look at some of our base configuration..

Initial Configuration

By default the NFVIS install will have a LAN and WAN bridge (lan-br and wan-br, respectively). The LAN config will be set up with a static IP of 192.168.1.1/24, and the WAN will be set for DHCP. We can check the current network settings by running the show system settingscommand:

In this case, my WAN interface is able to get an IP via DHCP. We’re likely going to want to change this to a static IP address - which we can do from the CLI or web interface. Let’s start by trying this from the CLI:

config t
system settings wan ip address <ip addr> <netmask>
system settings default-gw <gateway addr>
system settings hostname <hostname>

Changes can then be applied using the commit command

We can verify these settings by repeating the show system settings command we used earlier.

Let’s go ahead and log into the web interface to see what the network configuration looks like there:

1. If we know our WAN or LAN IP from earlier, we can just pop that in our web browser.
2. Go ahead and log in using the new admin credentials we just created:

3. We’ll be taken to the primary NFVIS dashboard, which will currently show no active VMs deployed:

4. In the left-hand menu, expand Hostthen click on Settings:

Here we can see that we already configured our IP Addressing/hostname - but we could use the Edit button at the bottom to change any of these values. For example - I’m going to go ahead and select Static for both the Management (LAN) and WAN IP addresses.

Another quick tip - if we need to modify which physical network adapters are tied to an internal network bridge, we can find that under VM Life Cycle > Networking:

That’s all for this time. In the next post, we’ll take a look at how to package VM images and deploy our service chain.