Automating the CLI: Using Scrapli & Cisco Genie to Collect Interface Data

So I just scheduled an upcoming attempt at the Developing Applications using Cisco Core Platforms and APIs (DEVCOR 350-901) exam.

It's coming up quick in two weeks here & I'm starting to do some review / final prep before taking the exam.

Why does this matter? Well - I plan on doing a few write-ups & videos on some of the content I'm studying. There are a lot of people diving into the DevNet exams, and I want to do what I can to help other people succeed.

All that being said, check back here over the next few weeks - or subscribe on YouTube - to see the additional content that will follow. While this specific blog doesn't cover much from the exam blueprint, it's the beginning a simple project I'll be using in the further content.

Getting Started with Scrapli

For this project I opted to start with scrapli, and possibly migrate to using RESTCONF later on in the process. Mostly this was an excuse for me to try out scrapli, as I've heard a few people using it recently & was interested.

Scrapli is a screen scraping module for Python. If you're not familiar with screen scraping, it's the process of connecting to something via telnet/ssh/etc, then literally scraping or dumping the contents of the screen. There are other modules that do this as well, like paramiko, netmiko, or expect scripting.

In networking, too many of our devices still rely on CLI and don't have proper API endpoints. We're working on it, but yet still a ways from having it everywhere. So in the meantime, we still need screen scraping for automation.

On the whole, this isn't necessarily a bad thing. For example, most network engineers are very familiar and competent with the CLI of any network operating system. It's easier to jump into the world of automation if you can start with something you know, the CLI. It's harder to force someone to start their automation journey by giving up everything they know and shoving REST APIs at them.

Okay - enough rambling. Let's get scrapli installed and see what it can do.

Installing the module

Easiest part of the whole project. Install with pip:

pip install scrapli

Next we'll go ahead & import into our script.

Scrapli supports quite a handful of operating systems (including a few non-Cisco platforms as well!). In the case of my project though, I'm only using Cisco IOS-XE switches - so we'll only import the scrapli driver for that.

from scrapli.driver.core import IOSXEDriver

Connecting to a Device

Okay - now that we're all setup with the module, it's time to get working!

First thing, we need to authenticate to our device. Building off of the example code from the scrapli page - we'll create a short dictionary of authentication parameters first:

switch = {
    "host": "10.1.1.1",
    "auth_username": "net_api",
    "auth_password": "net_api_pass",
    "auth_strict_key": False
}

cli = IOSXEDriver(**switch)
cli.open()

Once we build out dictionary, we'll use **switch to unpack our key/value pairs into the IOSXEDriver object & assign it to a variable called cli. Then, all we need to do is call cli.open() and scrapli will open a connection to our target device.

Now we can run any command we want by calling cli.send_command():

sh_int = cli.send_command("show interface")
print(sh_int.output)

So in the above example, I want to issue a show interface - then print the output.

What we get is shown below:

What's that look like? Exactly the same output we would get if we entered show interface on the command line ourselves! (Only Gig1/0/1 shown here to keep this short & clean)

Using Cisco Genie to Parse CLI Output

So what if we wanted to pull a list of every interface on the switch? Maybe tally how many ports are connected vs down vs admin disabled? Or even check operational speeds & media types?

Well that's what I'm looking to do for this project.

Now at first it may seem like we have to use regular expressions to try & match the content we need from the output. However, there is an easier way!

Cisco Genie is an open source project, and part of the larger PyATS automated network testing suite. If you haven't looked at any of that yet, I would highly recommend you check it out.

So what's Genie do? It's a utility that handles all the output parsing for us. There are a ton of pre-existing parsers written, which handle all the hard work so we don't have to.

Best yet - scrapli has native integration into Genie. All we have to do is install one additional component:

pip install scrapli[genie]

And to make use of the power of Genie, we just need to pass our raw output to it.

So our new code will look like this:

sh_int = cli.send_command("show interface")

sh_int_parsed = sh_int.genie_parse_output()
print(sh_int_parsed)

That's it. And now, instead of that raw output - we get a native Python dictionary that's ready to consume by our script:

{
   "GigabitEthernet1/0/1":{
      "port_channel":{
         "port_channel_member":False
      },
      "enabled":True,
      "line_protocol":"up",
      "oper_status":"up",
      "connected":True,
      "type":"Gigabit Ethernet",
      "mac_address":"78bc.1a81.e101",
      "phys_address":"78bc.1a81.e101",
      "description":"Test Port",
      "delay":10,
      "mtu":1500,
      "bandwidth":1000000,
      "reliability":"255/255",
      "txload":"1/255",
      "rxload":"1/255",
      "encapsulations":{
         "encapsulation":"arpa"
      },
      "keepalive":10,
      "duplex_mode":"full",
      "port_speed":"1000mb/s",
      "media_type":"10/100/1000BaseTX",
      "flow_control":{
         "receive":False,
         "send":False
      },
      "arp_type":"arpa",
      "arp_timeout":"04:00:00",
      "last_input":"00:00:05",
      "last_output":"00:00:08",
      "output_hang":"never",
      "queues":{
         "input_queue_size":0,
         "input_queue_max":2000,
         "input_queue_drops":0,
         "input_queue_flushes":0,
         "total_output_drop":198,
         "queue_strategy":"fifo",
         "output_queue_size":0,
         "output_queue_max":40
      },
      "counters":{
         "rate":{
            "load_interval":300,
            "in_rate":8000,
            "in_rate_pkts":11,
            "out_rate":1868000,
            "out_rate_pkts":205
         },
         "last_clear":"never",
         "in_pkts":2658450,
         "in_octets":372811780,
         "in_no_buffer":0,
         "in_multicast_pkts":6783,
         "in_broadcast_pkts":6783,
         "in_runts":0,
         "in_giants":0,
         "in_throttles":0,
         "in_errors":0,
         "in_crc_errors":0,
         "in_frame":0,
         "in_overrun":0,
         "in_ignored":0,
         "in_watchdog":0,
         "in_mac_pause_frames":0,
         "in_with_dribble":0,
         "out_pkts":51057453,
         "out_octets":60309072263,
         "out_underruns":0,
         "out_errors":0,
         "out_interface_resets":2,
         "out_collision":0,
         "out_unknown_protocl_drops":0,
         "out_babble":0,
         "out_late_collision":0,
         "out_deferred":0,
         "out_lost_carrier":0,
         "out_no_carrier":0,
         "out_mac_pause_frames":0,
         "out_buffer_failure":0,
         "out_buffers_swapped":0
      }
   }
}

This format makes our lives a lot easier. For example, let's say we wanted to determine the operational state of a port. Without the Genie integration, we would need to write some regex to comb through the raw output & find what we needed.

However, with Genie involved - its as easy as this:

print(sh_int_parsed['GigabitEthernet1/0/1']['oper_status'])

Collecting the Useful Bits

Now that we have everything in an easy-to-use format, all we need to do is write a few loops to run through our interface list & collect data.

The purpose of my script is to automate the collection of port utilization. For example, think about performing a switch refresh or some form of capacity planning scenario. You might need to inventory how many switches you have, how many ports are utilized vs how many are available, or count the number of copper vs fiber ports.

For this first iteration, we'll just collect the data then dump it out to a CSV file. As I mentioned earlier, this is just the start of a small project I'll be using to study for the DEVCOR exam - so this will be evolving into a web service later on.

The full script can be found here. But I've posted just a snippet of how we count the different interface characteristics:

# Count all Ethernet interfaces
interfaceStats['total'] += 1
# Count admin-down interfaces
if not sh_int_parsed[iface]['enabled']:
    interfaceStats['intdisabled'] += 1
# Count not connected interfaces
elif sh_int_parsed[iface]['enabled'] and sh_int_parsed[iface]['oper_status'] == 'down':
    interfaceStats['intdown'] += 1
# Count up / connected interfaces - Then collect current speeds
elif sh_int_parsed[iface]['enabled'] and sh_int_parsed[iface]['connected']:
    interfaceStats['intup'] += 1
    speed = sh_int_parsed[iface]['bandwidth']
    if speed == 10_000:
        interfaceStats['intop10m'] += 1
    if speed == 100_000:
        interfaceStats['intop100m'] += 1
    if speed == 1_000_000:
        interfaceStats['intop1g'] += 1
    if speed == 10_000_000:
        interfaceStats['intop10g'] += 1
# Count number of interfaces by media type
try:
    media = sh_int_parsed[iface]['media_type']
    if '1000BaseTX' in media:
        interfaceStats['intmedcop'] += 1
    else:
        interfaceStats['intmedsfp'] += 1
except KeyError:
    interfaceStats['intmedsfp'] += 1

Once all that runs, we create a CSV file to dump all of the data to.

For example, running the complete script against my lab Catalyst 9200 switch would output the following:

And that's it! Using the combination of scrapli & genie made this script very quick and easy to write - which means now I can focus more time on the next steps.

As I've mentioned a few times, this is hopefully the start of a short series of blog posts & videos as I wrap up studying for the DEVCOR exam.

If you found this content helpful - or you're interested in the future content - please check back soon! Or consider subscribing to my YouTube channel, where new content will be coming shortly.