I've been getting a handful of questions lately on the process of bringing a Cisco Catalyst 8000v or CSR 1000v into an SD-WAN environment. So I figured maybe I should put something together to share.
On a related note - I've been debating for a while doing a blog and/or video on building out a Cisco Viptela SD-WAN lab in EVE-NG. This would (tentatively) include everything from building controllers, bringing up remote sites, and template/policy configs. If this is something you might get value from, please let me know!! I'm looking for some motivation :)
Okay - all that being said, let's go ahead and get started.
Note: This guide should work with CSR 1000v devices as well. But it will NOT be 100% accurate for physical ISR/IOS-XE routers, as there are some additional steps with certificates & the Plug and Play portal to get those running.
Topology
So to start with, figured I would share the topology that I'm working from. If you read my last post you might recognize this, but with an added location. This new location (site id 400) contains our Catalyst 8000v VM, running IOS-XE version 17.04.01a.
Controller or Autonomous Mode?
Back in the earlier days of IOS-XE SD-WAN, there used to be two separate software images to load on your network appliance - one for traditional IOS-XE, and one for SD-WAN code.
With the newer releases of IOS-XE, we're now getting a unified image that contains both software sets. So our options now are two modes: autonomous (traditional IOS-XE) or controller (SD-WAN).
One way we can check this, is by running a show version and looking for Router operating mode:
Router# show version
<-- Output omitted -->
cisco C8000V (VXE) processor (revision VXE) with 2035355K/3075K bytes of memory.
Processor board ID XXXXXXXXXXX
Router operating mode: Autonomous
4 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
3965744K bytes of physical memory.
5234688K bytes of virtual hard disk at bootflash:.
Configuration register is 0x2102Okay, and looks like the Catalyst 8000v image I'm using booted up in autonomous mode. No big deal, we can change modes pretty easily!
So in normal exec-mode, we'll use the command controller-mode enable:
Router# controller-mode enable
Enabling controller mode will erase the nvram filesystem, remove all configuration files, and reload the box!
Ensure the BOOT variable points to a valid image
Continue? [confirm]As noted in the snippet above - this command will erase the config! So you will want to take a backup snapshot of your existing configuration, if this is already being used. In my case, it's a brand new VM - so we'll continue on.
Once the device is back online, we'll log in with the default login of admin/admin. Note that you will be forced to change this upon first login.
User Access Verification
Username: admin
Password:
Default admin password needs to be changed.
Enter new password:
Confirm password:
Router#And just for fun, we'll run a show version again to ensure we're in controller mode:
Router# show version
<-- Output omitted -->
cisco C8000V (VXE) processor (revision VXE) with 2035355K/3075K bytes of memory.
Processor board ID XXXXXXXXXXX
Router operating mode: Controller-Managed
4 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
3965756K bytes of physical memory.
5234688K bytes of virtual hard disk at bootflash:.
Configuration register is 0x2102Catalyst 8000v Initial Config
So since this is a lab environment, and I'm not trying to provide any Day0 provisioning files - I'll have to complete some manual configuration to get started.
First, we'll start with some SD-WAN specific config (site id, org name, etc). I'll be using the values that apply to my lab, so be sure to change these in yours!
Note: If you haven't used IOS-XE SD-WAN previously, be aware that "conf t" doesn't work! In SD-WAN/controller mode, you'll use "config-transaction". In this mode, all changes will need to be committed before they're applied to the device.
Router# config-transaction
! Set hostname
Router(config)# hostname Cat8k-Site400
! Required SD-WAN system configs
Router(config)# system
Router(config-system)# organization-name "SDWAN-LAB"
Router(config-system)# site-id 400
Router(config-system)# vbond 192.168.99.241
Router(config-system)# system-ip 10.10.10.237
Router(config-system)# commit
Commit complete.
Cat8k-Site400#
Once we have those basics configured, at a minimum we'll also need to configure our internet-facing tunnel interface. This allows our Catalyst 8000v to communicate with our control plane for bring-up.
Cat8k-Site400#config-transaction
! Config physical interface (This is a lab, so I'm using a static IP)
Cat8k-Site400(config)# interface GigabitEthernet1
Cat8k-Site400(config-if)# ip address 192.168.99.237 255.255.255.0
Cat8k-Site400(config-if)# no shut
! Config tunnel interface
Cat8k-Site400(config-if)# interface Tunnel1
Cat8k-Site400(config-if)# no shut
Cat8k-Site400(config-if)# ip unnumbered GigabitEthernet1
Cat8k-Site400(config-if)# tunnel source GigabitEthernet1
Cat8k-Site400(config-if)# tunnel mode sdwan
Cat8k-Site400(config-if)# exit
! SD-WAN tunnel config
Cat8k-Site400(config)# sdwan
Cat8k-Site400(config-sdwan)# interface GigabitEthernet1
Cat8k-Site400(config-interface-GigabitEthernet1)# tunnel-interface
Cat8k-Site400(config-tunnel-interface)# encapsulation ipsec
Cat8k-Site400(config-tunnel-interface)# color biz-internet
Cat8k-Site400(config-tunnel-interface)# exit
! Default route to our internet gateway
Cat8k-Site400(config)# ip route 0.0.0.0 0.0.0.0 192.168.99.1
Cat8k-Site400(config)# commit
Commit complete.
Cat8k-Site400#A Note on Certificates
Since this is a lab environment, I'm using self-signed local certificate authority to provision all of my certificate infrastructure. Because of this, I'll need to install my local CA certificate on the Catalyst 8000v. If you're using the default Cisco-provisioned certificate setup, you won't need to do this.
Since my SD-WAN lab doesn't have direct access to my local TFTP server - I do have an out-of-band management interface connected to my Cat 8000v. We'll start by configuring that interface:
Cat8k-Site400# config-transaction
Cat8k-Site400(config)# vrf definition 512
Cat8k-Site400(config-vrf)# address-family ipv4
Cat8k-Site400(config-vrf)# exit
Cat8k-Site400(config)# interface GigabitEthernet 3
Cat8k-Site400(config-if)# vrf forwarding 512
Cat8k-Site400(config-if)# ip address dhcp
Cat8k-Site400(config-if)# exit
Cat8k-Site400(config)# ip tftp source-interface GigabitEthernet 3
Cat8k-Site400(config)# commitThe standard management VRF/VPN for SD-WAN is 512, so I kept that config to match when I configured this management interface. This will all be over-written anyways once we get connected to vManage & configure/push our template configs.
Once that's done, we can go ahead and copy our CA certificate to bootflash.
Cat8k-Site400# copy tftp://10.0.0.2/cacert.pem bootflash:
Destination filename [cacert.pem]?
Accessing tftp://10.0.0.2/cacert.pem...
Loading cacert.pem from 10.0.0.2 (via GigabitEthernet3): !
[OK - 1406 bytes]
1406 bytes copied in 0.112 secs (12554 bytes/sec)
Cat8k-Site400# dir bootflash: | inc cacert.pem
16 -rw- 1406 May 14 2021 14:41:27 +00:00 cacert.pemThen we'll use the command below to install the CA certificate:
Cat8k-Site400# request platform software sdwan root-cert-chain install bootflash:cacert.pem
Uploading root-ca-cert-chain via VPN 0
Copying ... /bootflash/cacert.pem via VPN 0
Updating the root certificate chain..
Successfully installed the root certificate chainAnd we can validate by using the show sdwan certificate root-ca-cert command:
Cat8k-Site400 show sdwan certificate root-ca-cert
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
11:3b:0a:12:17:b0:e0:b5:4b:fa:c2:e9:2c:9c:12:84
Signature Algorithm: sha1WithRSAEncryption
Issuer: DC = local, DC = 0x2142, O = SDWAN-LAB, CN = 0x2142-0XWIN1-CA-2
Validity
Not Before: Nov 29 20:00:11 2018 GMT
Not After : Nov 29 20:10:11 2043 GMT
Subject: DC = local, DC = 0x2142, O = SDWAN-LAB, CN = 0x2142-0XWIN1-CA-2
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (2048 bit)
Modulus:
<-- Output omitted -->Activating the Catalyst 8000v
Almost there! Now that we've finished our pre-config & added our root CA certificate - we're ready to join the Catalyst 8000v to the SD-WAN fabric.
We'll start over in vManage - by going to Configuration > Devices.
Then we'll find our target, unused Catalyst 8000v device. Click the ellipsis on the right side, then select Generate Bootstrap Configuration
This will give us a prompt to select which configuration style to generate. We'll leave this on "Cloud-init":
Once we hit okay - we'll be presented with the info we need. We won't necessarily need all of this information, but we'll want to take note of our uuid and otp:
We'll drag this info back over to our Catalyst 8000v, and we can now use it to activate the device & join to our SD-WAN fabric.
For the command below, chassis-number will be our uuid value - and token will be our otp.
Cat8k-Site400# request platform software sdwan vedge_cloud activate chassis-number C8K-178BXXXX-XXXX-XXXX-XXXX-XXXXXXXXBC24 token 421ecxxxxxxxxxxxxxxxxxxxxxbd53b5Validation
After a few moments, we'll see some log messages start to appear showing our control connections coming up. You might see these on the terminal if you're using the console port, or you can use show log:
*May 14 15:39:06.910: %Cisco-SDWAN-Cat8k-Site400-OMPD-3-ERRO-400002: R0/0: OMPD: vSmart peer 10.10.10.242 state changed to Init
*May 14 15:39:07.980: %DMI-5-AUTH_PASSED: R0/0: dmiauthd: User 'vmanage-admin' authenticated successfully from 10.10.10.240:48140 and was authorized for netconf over ssh. External groups:
*May 14 15:39:08.798: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400002: R0/0: OMPD: vSmart peer 10.10.10.242 state changed to Handshake
*May 14 15:39:08.804: %Cisco-SDWAN-Cat8k-Site400-OMPD-5-NTCE-400002: R0/0: OMPD: vSmart peer 10.10.10.242 state changed to Up
*May 14 15:39:08.808: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400005: R0/0: OMPD: Number of vSmarts connected : 1
*May 14 15:39:09.827: %Cisco-SDWAN-Cat8k-Site400-OMPD-3-ERRO-400002: R0/0: OMPD: vSmart peer 10.10.10.243 state changed to Init
*May 14 15:39:10.584: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is ON
*May 14 15:39:11.756: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400002: R0/0: OMPD: vSmart peer 10.10.10.243 state changed to Handshake
*May 14 15:39:11.762: %Cisco-SDWAN-Cat8k-Site400-OMPD-5-NTCE-400002: R0/0: OMPD: vSmart peer 10.10.10.243 state changed to Up
*May 14 15:39:11.762: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400005: R0/0: OMPD: Number of vSmarts connected : 2
*May 14 15:39:12.738: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400007: R0/0: OMPD: Using policy from peer 10.10.10.242
*May 14 15:39:13.570: %Cisco-SDWAN-Cat8k-Site400-FTMD-6-INFO-1000020: R0/0: FTMD: SLA class added : class 'Default' at index '1' loss = 25%, latency = 300ms, jitter = 100ms, app-probe-class = None
*May 14 15:39:14.738: %Cisco-SDWAN-Cat8k-Site400-OMPD-6-INFO-400007: R0/0: OMPD: Using policy from peer 10.10.10.242We can also see our control connections using the command show sdwan control connections:
Cat8k-Site400# show sdwan control connections
PEER PEER CONTROLLER
PEER PEER PEER SITE DOMAIN PEER PRIV PEER PUB GROUP
TYPE PROT SYSTEM IP ID ID PRIVATE IP PORT PUBLIC IP PORT ORGANIZATION LOCAL COLOR PROXY STATE UPTIME ID
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
vsmart dtls 10.10.10.242 100 1 192.168.99.242 12446 192.168.99.242 12446 SDWAN-LAB biz-internet No up 0:00:00:51 0
vsmart dtls 10.10.10.243 100 1 192.168.99.243 12446 192.168.99.243 12446 SDWAN-LAB biz-internet No up 0:00:00:49 0
vmanage dtls 10.10.10.240 100 0 192.168.99.240 12646 192.168.99.240 12646 SDWAN-LAB biz-internet No up 0:00:00:52 0Of course, we can also check to see our device status in the vManage dashboard as well.
Over on the Monitor > Network page, we can see that our new Catalyst 8000v is now online:
Extra: How do I check the routing table on an IOS-XE SD-WAN device?
So - if you've only used the vEdge software devices, you may be used to using the show ip route or show ip route vpn 10 commands.
In the IOS-XE world, most SD-WAN commands are prefixed with the sdwan keyword. For example: show sdwan bfd sessions (where on vEdges, it would just be show bfd sessions)
This might lead you to believe that you can use show sdwan ip route or show sdwan ip route vrf 10 - but these won't work! In fact, you'll get the following message:
Cat8k-Site400# show sdwan ip route
% Error: This command is not supportedFor the IOS-XE based devices, they actually just use the standard IOS-XE routing table and VRF constructs.
So on a vEdge, you would have VPN 0 as your transport VPN. On IOS-XE, this is just the default global routing table, shown with show ip route.
But what about our LAN-side service VPNs? In this case, our routes are being dumped into a VRF on the IOS-XE device.
So for example, I have VPN 10 in my lab which is used for LAN-side clients. We can use the show vrf command to see that this exists, and then show ip route vrf 10 to see the routes from our other SD-WAN locations:
Cat8k-Site400# show vrf
Name Default RD Protocols Interfaces
10 ipv4 Gi2
512 ipv4 Gi3
65528 ipv4 Lo65528
Cat8k-Site400# show ip route vrf 10
Routing Table: 10
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
H - NHRP, G - NHRP registered, g - NHRP registration summary
o - ODR, P - periodic downloaded static route, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR
& - replicated local route overrides by connected
Gateway of last resort is not set
10.0.0.0/24 is subnetted, 2 subnets
m 10.2.2.0 [251/0] via 10.10.10.235, 00:09:02, Sdwan-system-intf
m 10.3.3.0 [251/0] via 10.10.10.236, 00:09:02, Sdwan-system-intf Okay, that's it! Pretty quick process overall - and now we can get into applying our device/feature templates.
Hope this was helpful!!