Back when I started diving into studying for the Cisco DevNet certifications, I thought a lot of the REST API stuff was fairly easy. But I always struggled a bit with wrapping my head around YANG models & how exactly NETCONF/RESTCONF worked. A lot of it seemed more abstract, and browsing through YANG models isn’t quite as intuitive as most REST API documentation.

These days I end up working quite a bit with NETCONF - and if there’s one tool that’s helped me more than anything, it’s certainly YANG suite. So in this post, I wanted to spend a little bit of time walking through how to get set up with this tool - and walk through a quick example of how it can be used.

What’s YANG Suite?

To put it simply: YANG Suite is a web-based tool that allows you to easily browse through YANG models & find what you need. It also has a bunch of tools to run test queries against devices, which makes developing NETCONF automation easier. It can also auto-generate code snippets - and who doesn’t like that? 😄

YANG suite is an open source tool maintained by Cisco & the GitHub repo here.

Initial Setup

YANG suite is a containerized application, so we’ll need a container host to run it on. You could use something like Docker Desktop to run it on your local PC, but I’ll be using a dedicated Ubuntu machine in my lab with Docker installed.

Note: YANG suite now supports a local Python pip-based install. Check out the GitHub repo for more info on that.

Deploy with Docker

First thing we’ll do is pull down the code from GitHub:

matt@yangsuite:~/yangsuite/docker$ git clone https://github.com/CiscoDevNet/yangsuite.git

Next, we can execute the setup script to help provision the YANG Suite containers, which is located at yangsuite/docker/start_yang_suite.sh. The script will prompt you to set up an admin user and generate self-signed certificates:

matt@yangsuite:~/yangsuite/docker$ cd yangsuite/docker
matt@yangsuite:~/yangsuite/docker$ ./start_yang_suite.sh
Hello, please setup YANG Suite admin user.
username: matt
password:
confirm password:
email: matt@0x2142.local

Setup test certificates? (y/n): y
  -- output omitted -- 
  
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]: US
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:
Email Address []:
Certificates generated...
Building docker containers...

Once all of the above info is provided, the script will build the YANG Suite contianer images & launch them. However, by default it will launch in the foreground - so we can kill the script with Ctrl-C & relaunch the containers with docker-compose up -d.

Allow Access from Remote IPs

We may need to make a quick change depending on how we’ll be accessing YANG Suite.

By default, YANG Suite will only permit access to the web UI from localhost - so if you’re running Docker on your local PC, then no change is necessary.

However, if you’re running Docker on a dedicated VM like I am, then we’ll need to allow external IPs to access the YANG Suite UI. This is controlled by the DJANGO_ALLOWED_HOSTS environment variable that gets supplied to the container.

As long as we’ve already run the setup script once - our environment variables will be saved to yangsuite/docker/yangsuite/setup.env:

DJANGO_SETTINGS_MODULE=yangsuite.settings.production
MEDIA_ROOT=/ys-data/
STATIC_ROOT=/ys-static/
DJANGO_STATIC_ROOT=/ys-static/
DJANGO_ALLOWED_HOSTS=localhost
YS_ADMIN_USER=<removed>
YS_ADMIN_PASS=<removed>
YS_ADMIN_EMAIL=<removed>

We’ll need to change DJANGO_ALLOWED_HOSTS=localhost to the IP address of the Docker host. So if your Docker host is reachable at 192.168.1.1, then we’ll set: DJANGO_ALLOWED_HOSTS=192.168.1.1.

After which, we can bring up the containers again with docker-compose up -d:

matt@yangsuite:~/yangsuite/docker$ docker-compose up -d
Recreating docker-yangsuite-1 ... done
Recreating docker-backup-1    ... done
Recreating docker-nginx-1     ... done

Once everything is running, the web UI can be reached at https://<ip address>:8443.

Loading YANG Models

After we get logged in, we’ll need to load the appropriate YANG models for whatever devices we’re using.

For example, I’ll be using Cisco IOS-XE devices running version 17.6.x code - so I’ll need to download the YANG models from: https://github.com/YangModels/yang/tree/main/vendor/cisco/xe/1761.

Within YANG Suite, we’ll navigate to Setup > YANG files and repositories. Then click the button to add a New repository:

You can name the repo whatever you like. In my case, I named it ios-xe-1761 which describes the models it will hold.

Next we can tell YANG Suite where we want to pull our models from. There are a couple of options, including a direct upload from your local PC or querying models directly from a device - but I’ll choose Git.

After which, we’ll need to fill in the following info:

The Repository URL will be the base Git repo that we’re pulling from. In this case, that’s https://github.com/YangModels/yang.git.

We’ll also need to specify which Git branch to pull from, which for the YangModels repo will be main.

Then we’ll specify which directory to import. The YangModels repository holds models for a large number of vendors & devices - so we only want to pull in what is necessary. In my case, I’ve set this to vendor/cisco/xe/1761 - which will only pull in the YANG models for Cisco IOS-XE 17.6.x devices. I’ve also checked the box for Include subdirectories.

Once that’s all set, we can hit Import YANG files. Depending on how many models are being downloaded & how quick your Docker host is, this may take a few minutes.

After it finishes, mine displayed a message showing that 827 models had been added:

Selecting YANG Module Sets

Once we have all of our models downloaded into YANG Suite, it’s time to set up some module sets. You can think of these as filters for the YANG models.

For example, later on we’ll show how to change a wireless PSK with NETCONF. So since I already know what type of information I’ll be working with, I can create a module set to only contain those models.

YANG Suite will require at least one module set to be configured, even if you don’t want to filter anything out.

To do this, we’ll navigate to Setup > YANG model sets. Then click New YANG set:

Again, since we’ll be working with wireless data in a bit, I’ll name my model set wireless. We’ll also have to select which YANG repository this will pull from, which will be ios-xe-1761.

Next, we’ll get to select which YANG models we want to include in our module set. To make things easy, I’ll filter the models by wireless and then click Add selected.

Once those are added, you’ll likely see an error about missing dependencies. Some of the models we selected may have sub-models or data types that are located in other models.

In order to quickly add these, YANG Suite provides a one-click-button to Locate and add missing dependencies:

That’s it - our model set is ready to use.

Adding Devices

Okay, so now that we have all of our necessary YANG models set up, let’s take a add a device that we can use for testing. I’ll be adding a Catalyst 9800 Wireless controller.

To do this, we’ll head over to Setup > Device Profiles. Then click on Create new device.

Here we’ll fill in a name for our device profile, along with the device’s IP/FQDN and login credentials.

Scrolling down a bit, we’ll also need to make sure we enable the appropriate management protocols. So I’ll be checking the box for both NETCONF and RESTCONF, since I have both enabled. I’ll also add SSH, which isn’t shown in the screenshot below.

Please note, you’ll likely want to check the box for Skip SSH key validation for this device under the NETCONF settings.

After that’s all done, we can click Create Profile at the bottom.

Then, to validate everything works - we’ll select our device & click the button for Check selected device’s reachability.

With any luck, you should see a similar result as shown below:

Example: Changing Wireless PSK

Alright, now let’s take a look at how we might use YANG Suite to test NETCONF / RESTCONF calls to a wireless controller.

In this example, perhaps we’re developing automation to rotate the guest wireless network password on a regular basis. We need to figure out how specifically to accomplish that by finding the right YANG models & understanding what data is sent and received with those calls.

We’ll explore how to do this with both NETCONF and RESTCONF.

Via NETCONF

To start off, we’ll navigate to Protocols > NETCONF. On this page, we’ll have full access to explore models, build XML payloads, and test them against our configured device.

Get-config

First thing we’ll need is to select our YANG set & Modules. In my case, the YANG set will be wireless, and I’ll select the module: Cisco-IOS-XE-wireless-wlan-cfg.

Then click the button to Load Modules:

By default, our NETCONF Operation will be set to get-config, which will work for now since we only want to retrieve configuration and not change anything yet.

Under the Nodes pane, we can expand the tree a bit & dig in to find which call we need.

For our example, we’ll need to examine the current configured WLAN profiles. So we’ll check the box next to wlan-cfg-entries:

This will retrieve all configured WLAN profiles on the controller. For now, let’s do that so we can figure out what data we’ll get back.

We can now click the button to Build RPC, and YANG Suite will automatically build the appropriate XML payload for us:

You’ll also notice that I selected the catalyst 9800 as the target device. Let’s go ahead and run this with the Run RPC(s) button, which should pop open a new window & establish a connection to our device.

We can see in the response above, that we have two WLAN profiles configured: test-lab-network and test-guest-network. For the purposes of this demo, I have opted to have the guest network PSK stored in plain text, so we can validate our changes more easily.

Okay, so what if we wanted to refine our query XML to only retrieve data on the test-guest-network? Maybe we also want to filter our the additional information returned, and only see the current PSK?

Back on the payload editor, we can expand wlan-cfg-entries and input our desired profile-name:

We’ll also find the psk item further down in the list. Now we want to pull back this field, regardless of what the PSK is currently set to. So we’ll click the value field, but leave it blank:

Once we click Build RPC, we can see our payload now shows the additional XML filters asking only for a single profile & it’s associated PSK:

If we run that RPC call, we’ll now see that we only get back a limited set of data:

Now we know exactly what payload to send if we want to get back the current configured PSK for any specific WLAN profile.

Let’s try changing it!

Edit-config

Lucky for us, we’re already nearly set up for that. We’ll need to make two quick changes. First, set the NETCONF Operation to edit-config:

Then, under the YANG tree, we’ll set a new PSK:

Again, we’ll click Build RPC - and take a quick look at the proposed change:

Look good to you? Okay, let’s send it:

Sure enough, the new configuration was sent to the wireless controller and we got a response of: ok.

But of course, we can check quickly by switching our operation back to get-config and re-running our previous XML payload:

Auto-generated snippets

Well if you’re new to NETCONF, you might be asking yourself “Okay great, but how do I get this into Python?!?!”

YANG Suite has some built in tools to auto-generate Python code or Ansible playbooks, which can help us get started on our code more quickly.

To generate these, click the Replays button:

For example, I selected to auto-generate a Python script. Here’s what that looks like:

#! /usr/bin/env python
import traceback
import lxml.etree as et
from argparse import ArgumentParser
from ncclient import manager
from ncclient.operations import RPCError

payload = [
'''
<get-config xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
    <source>
      <running/>
    </source>
    <filter>
      <wlan-cfg-data xmlns="http://cisco.com/ns/yang/Cisco-IOS-XE-wireless-wlan-cfg">
        <wlan-cfg-entries>
          <wlan-cfg-entry>
            <profile-name>test-guest-network</profile-name>
            <psk/>
          </wlan-cfg-entry>
        </wlan-cfg-entries>
      </wlan-cfg-data>
    </filter>
  </get-config>

''',
]

if __name__ == '__main__':

    parser = ArgumentParser(description='Usage:')

    # script arguments
    parser.add_argument('-a', '--host', type=str, required=True,
                        help="Device IP address or Hostname")
    parser.add_argument('-u', '--username', type=str, required=True,
                        help="Device Username (netconf agent username)")
    parser.add_argument('-p', '--password', type=str, required=True,
                        help="Device Password (netconf agent password)")
    parser.add_argument('--port', type=int, default=830,
                        help="Netconf agent port")
    args = parser.parse_args()

    # connect to netconf agent
    with manager.connect(host=args.host,
                         port=args.port,
                         username=args.username,
                         password=args.password,
                         timeout=90,
                         hostkey_verify=False,
                         device_params={'name': 'csr'}) as m:

        # execute netconf operation
        for rpc in payload:
            try:
                response = m.dispatch(et.fromstring(rpc))
                data = response.xml
            except RPCError as e:
                data = e.xml
                pass
            except Exception as e:
                traceback.print_exc()
                exit(1)

            # beautify output
            if et.iselement(data):
                data = et.tostring(data, pretty_print=True).decode()

            try:
                out = et.tostring(
                    et.fromstring(data.encode('utf-8')),
                    pretty_print=True
                ).decode()
            except Exception as e:
                traceback.print_exc()
                exit(1)

            print(out)

Via RESTCONF

What if we prefer working with REST & JSON instead of XML? This time we’ll navigate to Protocols > RESTCONF.

Here, we’ll select our YANG set & modules again. In case you skipped the NETCONF section, we’re working with the wireless YANG set & the Cisco-IOS-XE-wireless-wlan-cfg module.

We’ll also select a device to use, then click Load Module(s):

Again, we’ll see a similar structure to the NETCONF side. But this time, we’ll utilize it quite a bit differently.

In the screenshot above, I’ve already selected the wlan-cfg-entries entry. Next, we’ll click the button to Generate API(s):

We’ll see auto-generated API documentation for all of the available calls. This list will contain every variation possible, so it can be quite long. Since we already went through the NETCONF method, we know what data we need to find / use which should make this easier.

To start with, I’ll expand the GET for /data/Cisco-IOS-XE-wireless-wlan-cfg:wlan-cfg-data/wlan-cfg-entries. We’ll have the ability to test the RESTCONF calls directly from this interface as well by clicking on Try it out:

In the screenshot above, I’ve already submitted the test request & gotten a response from the controller. While the screenshot is cut off a bit, we can see both of our wireless profile names.

If we wanted to see only the PSK, like we did with our previous example - we could scroll down quite a bit to find the GET request for /data/Cisco-IOS-XE-wireless-wlan-cfg:wlan-cfg-data/wlan-cfg-entries/wlan-cfg-entry={wlan-cfg-entry-profile-name}/psk

In the screenshot below, I’ve already input our test-guest-network profile name & executed the API call:

Sure enough, we see the PSK that we had just configured via NETCONF a few minutes ago.

Now let’s change it. We’ll expand the PATCH API call for the same endpoint, and fill in the request body with our new PSK:

Executing that call, the wireless controller returned a status code of 204.

So let’s check again to see if that updated properly:

Now that we have all the appropriate REST calls, it should be much easier to start developing our automation.

Okay, I think that’s about all I wanted to cover in this post. I’ve seen a lot of people struggle with NETCONF much like I used to, and I think the real key to getting familiar is just having the right tools & spending enough time with it. YANG Suite has really helped me get more familiar with the various model structures and how to use them.

Hopefully this information helps others to get started. NETCONF & YANG are certainly big hills to climb, but the right tools can make all the difference.

Earlier this year I started trying out Golang as a new language to learn. Most of my prior experience is with Python, with a handful of other languages sprinkled in here & there.

My experience with Go so far has been good & I’ve been having fun learning something new. It’s been easy enough to pick up & there have been quite a handful of things that I really appreciate about Go vs Python.

One of the best ways to learn something new is to find a good project to work on. Since I’ve done quite a bit historically with Python & building Webex chatbots - I figured one good way to learn Go would be trying to do something similar.

So in this blog post, I’ll walk through how to build a simple Discord bot using a Golang module called DiscordGo. We’ll use the same project idea as my previous Webex bot, where the bot will leverage the OpenWeather APIs to retrieve weather information.

Since I’ve been learning Go recently, I’ll do my best to try to keep this simple - so that hopefully you can follow along if you’re also new to Go 😊.

Code repo for the examples below can be found here

Setting up the Project

So first we’ll run through some quick setup. If you’re familiar enough with Go already, you can likely skip this step & just set up your project in the way you prefer.

I’ve created a new project folder called discord-weather-bot. So first we’ll initialize our Go module using the command go mod init discord-weather-bot.

Then, I’ll create a main.go in this folder, as well as a sub-folder named bot which will contain a bot.go file.

Once we’re done, our initial project layout should look like this:

We’ll add on to this later, but this will work to start.

Next, we’ll need to install the DiscordGo module, which we can do with one simple command: go get github.com/bwmarrin/discordgo

Generating API Keys

Next we’ll need to get our API keys for OpenWeather & Discord.

OpenWeather is going to be the easier of the two - simply sign up for a free account here. Then jump over to your account page & create a new API key here. Free accounts are limited to only using their current weather API, which will be more than enough for our example.

Next, we’ll look at the Discord side of things - which will be a few more hoops to jump though.

First we’ll need to head over to the Discord Applications page & create a new application/integration.

We’ll be prompted to provide a name for our app (Note, this isn’t the name of our bot - we’ll get to that shortly):

Once we finish that, we’ll be taken to a page to add additional information about our app - like tags, a description, an icon, or links to app resources. If you plan to publish your app publicly, you’ll want to invest some time here. However, since we’ll just be building the bot as a personal project, we don’t need to do anything here.

Next, we’ll want to jump down to the Bot section under Settings. This is where we can set up some basics for our bot & get our bot API token.

We’ll get a quick confirmation prompt, where we’ll click on Yes, do it!

Now we have our bot created! We can change the name of our bot here, to be separate from our app name if we want.

Scrolling down the page a bit - by default our bot is public, which means anyone could add it to their server. I’ll switch this off for now, since this is intended to be a private example bot.

In order for our bot to receive & process message content, we’ll also need to enable the toggle for message content. Note that Discord says this is only allowed for servers under a certain size, after which they would need to be verified in order to use this intent. Since this bot will only be used as an example, we don’t need to worry about that yet - but we will still need to enable the intent:

For now, this is all we need to set up on this page. Before we leave, we’ll want to reset our bot access token - and save the token for later:

Last but not least, we’ll need to add our Discord bot to a server. I have a server in Discord that I use for testing, and I’ve created a private channel to experiment with this bot.

In order to do this, we’ll need to create an OAuth authorization link that can be used to give the bot access.

Under our Settings, we’ll click on OAuth2 > URL Generator.

In the screenshot above, I’ve selected the Scope as bot. And checked off permissions for Read Messages, Send Messages, and Send Messages in Threads. For what we’ll be doing in this example, that should be more than enough.

At the bottom of the screenshot, you can see that Discord will auto-generate a URL which we can now use to add the bot to our Discord Server.

If we visit that link while signed into Discord, we’ll be prompted to add it to our server:

We’ll also be prompted to confirm the permissions the bot will have:

Now our bot has been successfully added to our Discord server, but will show as offline since we haven’t built it yet! Let’s get started with that.

Building the Bot - Basic Interaction

So we’ll start off by building out just enough of our bot code for it to send messages to our server.

Collecting API Keys via Environment Variables

For the purpose of this project, I’ll be storing both the Discord & OpenWeather API keys as environment variables. So well keep our main.go simple & just handle importing our API keys and kicking off the bot:

// snippet from: main.go
package main

import (
 "discord-weather-bot/bot"
 "log"
 "os"
)

func main() {
 // Load environment variables
 botToken, ok := os.LookupEnv("BOT_TOKEN")
 if !ok {
  log.Fatal("Must set Discord token as env variable: BOT_TOKEN")
 }
 openWeatherToken, ok := os.LookupEnv("OPENWEATHER_TOKEN")
 if !ok {
  log.Fatal("Must set Open Weather token as env variable: OPENWEATHER_TOKEN")
 }
 
    // Start the bot
 bot.BotToken = botToken
 bot.OpenWeatherToken = openWeatherToken
 bot.Run()
}

In the code above, we’ll import os and log which we’ll use to handle importing our environment variables & generating errors if this fails. We’ll also import our bot folder with discord-weather-bot/bot.

For our main function, we’ll start by trying to find our environment variables. For each of the variables, we’ll use a format of <varname>, ok := os.LookupEnv("<env name>").

Using os.LookupEnv, we’ll try and search for the environment variable by name - so in the first example looking for BOT_TOKEN. If the environment variable exists, it will be assigned to botToken - and ok will be True. Otherwise, we’ll get no value for botToken, and ok will be False.

If either environment variable doesn’t exist, we’ll just print a log message reminding the user to set the variable. log.Fatal will also quit the program after displaying the log message.

Finally, assuming we have both of those API keys - we’ll kick those keys over to the bot module & start the bot with bot.Run(). Let’s hop over & build that piece now.

Connecting to Discord

Next, we’ll start out bot code with enough to receive the variables being passed by main.go:

// snippet from: bot/bot.go
package bot

import (
 "fmt"
 "log"
 "os"
 "os/signal"
 "strings"

 "github.com/bwmarrin/discordgo"
)

var (
 OpenWeatherToken string
 BotToken         string
)

func Run() {
 // Not implemented yet
}

In the snippet above, we’ve defined two variables - OpenWeatherToken and BotToken. Note that both start with a capital letter, meaning that they have been exported & available to code outside this module - which is how we can update these values via main.go by setting bot.BotToken = botToken.

Okay - Let’s get started with connecting our bot to Discord. We’ll update our Run function to the following:

// snippet from: bot/bot.go

func Run() {
 // Create new Discord Session
 discord, err := discordgo.New("Bot " + BotToken)
 if err != nil {
  log.Fatal(err)
 }

 // Add event handler
 discord.AddHandler(newMessage)

 // Open session
 discord.Open()
 defer discord.Close()

 // Run until code is terminated
 fmt.Println("Bot running...")
 c := make(chan os.Signal, 1)
 signal.Notify(c, os.Interrupt)
 <-c

}

To begin with, we create a new discord session with discordgo.New() & passing our BotToken. Discord requires an HTTP Authorization header that contains Bot <token> or Bearer <token> for OAuth. The discordgo.New() function is just setting this header for us.

Next we’ll register an event handler to our Discord client. In a moment, we’ll create a new function called newMessage that will receive & process any new messages that are created in our channel. More to come in just a moment!

Now we can open our websocket tunnel to Discord via discord.Open(). If you’re not familiar with websockets, I wrote a little about them when I built a Webex bot in this post. We’ll also include a defer discord.Close(), which will ensure that we gracefully close the Discord session whenever the Run() function exits.

Lastly, we’ll use Go’s os/signal package to listen for any interrupt/process kill signals. This will allow our Discord bot to run in the background until we stop it. We do this by creating a new channel of type os.Signal where we will listen for termination signals. Then using signal.Notify, we tell the signal package where to send termination signals (our c channel) - and what signals we want to listen for (os.Interrupt). Then, using <-c, our program will hold until something is received on the channel.

An operator like <- is typically used for concurrency, where we have one function passing data to another via a channel. The <- or -> operator indicates which direction that data is being sent. So in this example, we’re receiving data from channel c using the statement <-c. Now the tricky thing here, is that we’re not assigning that input to another variable (for example, incomingSignal <- c), since we don’t care what signal was sent - just that there was a signal received.

After we receive any termination signal, the code will continue to the end of the Run() function. This will then automatically execute our defer discord.Close() from earlier, which will tear down our Discord session.

Processing Incoming Messages

Okay, now that we have our Discord session establishment handled, let’s build out our newMessage() function to handle receiving messages.

So for now, our code for this function will look like this:

// snippet from: bot/bot.go

func newMessage(discord *discordgo.Session, message *discordgo.MessageCreate) {

 // Ignore bot messaage
 if message.Author.ID == discord.State.User.ID {
  return
 }

 // Respond to messages
 switch {
 case strings.Contains(message.Content, "weather"):
  discord.ChannelMessageSend(message.ChannelID, "I can help with that!")
 case strings.Contains(message.Content, "bot"):
  discord.ChannelMessageSend(message.ChannelID, "Hi there!")
 }
}

Let’s take a quick look at the code above.

The newMessage() function is our event handler that we are registering with our Discord client. It will need to receive two values - a pointer to our Discord session (*discordgo.Session), and the type of event we’re listening for. In this case we’re listening for new message creation events (*discordgo.MessageCreate). A full list of events that we could listen for are listed in the Discord Gateway Documentation.

First thing we’ll do when processing the message is ensure that the bot ignores any message from itself. We wouldn’t want our bot creating a loop by responding to it’s own messages, which then generates a new message to respond to!

We accomplish this by checking the incoming message’s Author ID (message.Author.ID) against our current Discord session’s User ID (discord.State.User.ID). If these values match, then the incoming message was created by our bot code - so we’ll just return and stop execution of this function.

If the incoming message is from another user, then we can go ahead and process it. There are a number of ways to do this depending on what you want to achieve. To keep things simple, I am using a quick switch statement to evaluate the message content.

Our bot then is listening for two keywords - “weather” and “bot”. If the incoming message content contains either of those two words, our bot will respond with a message by using discord.ChannelMessageSend - and passing the channel ID from the incoming message (message.ChannelID) and our message.

A quick note: I used a switch/case here since it would be easier to add onto than a long list of if/else… but the example code here still isn’t perfect. For instance, our second case is only looking for someone using the word bot. However this is a simple match, and would still catch on someone talking about a robot or a bottle - since both contain bot in them. If this was a production/public bot, we would want to clean that up a bit.

Testing

Okay, with that we can now give our bot a quick test.

Let’s go ahead and start our bot with go run main.go. Don’t forget to set your BOT_TOKEN & OPENWEATHER_TOKEN environment variables first! (Or comment out the code for the OPENWEATHER_TOKEN, since we’re not using it just yet)

Here’s my test run:

As we can see, the bot responded just as we expected!

Another fun thing to point out, is that since we’re using websockets for the connection (aka Discord Gateway) - our bot can also register presence events. So as long as our bot is connected, it will actually show as online:

Adding Basic Bot Commands

So far we’ve gotten our bot connected to Discord, and responding to regular chat messages. Now we’ll focus on actually adding functionality to our bot by leveraging the OpenWeather API.

For now, we’ll implement this with very simple command matching. In a future blog post, I’ll be showing how to accomplish this with Discord slash commands.

So first we’ll modify our existing switch statement in the newMessage() function. We’ll provide a little help text with our response to someone mentioning weather - and create a new command by catching messages with !zip:

// snippet from: bot/bot.go

// Respond to messages
 switch {
 case strings.Contains(message.Content, "weather"):
  discord.ChannelMessageSend(message.ChannelID, "I can help with that! Use '!zip <zip code>'")
 case strings.Contains(message.Content, "bot"):
  discord.ChannelMessageSend(message.ChannelID, "Hi there!")
 case strings.Contains(message.Content, "!zip"):
  currentWeather := getCurrentWeather(message.Content)
  discord.ChannelMessageSendComplex(message.ChannelID, currentWeather)
 }

Under the !zip command, we’ll have a function called getCurrentWeather() which will return a Discord message. However, in this case we’ll change things up a little - and use an embedded message so we can include some formatting. This also means that we’ll change our response function to discord.ChannelMessageSendComplex().

Where discord.ChannelMessageSend() needs the channel ID & a simple string message, discord.ChannelMessageSendComplex() will require data of the type discordgo.MessageSend instead of a string. In our getCurrentWeather() function, we’ll see how to assemble our response using this structure.

Querying Weather

So to start with, we’ll need to query the OpenWeather API & retrieve the current weather information for a given US ZIP code.

In our project, I’ve created a new Go file called command-weather.go in the same bot subfolder which already contains our bot.go file. This new file will also be part of package bot.

// snippet from: bot/command-weather.go

package bot

import (
 "encoding/json"
 "fmt"
 "io/ioutil"
 "net/http"
 "regexp"
 "strconv"
 "time"

 "github.com/bwmarrin/discordgo"
)

I’ve included the list of imports above, in case you’re following along.

To start with, I’ll define two items that we’ll need for our `getCurentWeather() function:

// snippet from: bot/command-weather.go

const URL string = "https://api.openweathermap.org/data/2.5/weather?"

type WeatherData struct {
 Weather []struct {
  Description string `json:"description"`
 } `json:"weather"`
 Main struct {
  Temp     float64 `json:"temp"`
  Humidity int     `json:"humidity"`
 } `json:"main"`
 Wind struct {
  Speed float64 `json:"speed"`
 } `json:"wind"`
 Name string `json:"name"`
}

The first item is a constant for the base URL for OpenWeather’s API. This should never need to change, which is why we use a constant here.

In addition, I’ve defined a new struct called WeatherData that will be used to unpack our JSON response from OpenWeather’s API. They list a current sample response payload on their documentation page, which we can use to build our data structure. Since we won’t be using all of the data provided in the response, I also don’t need to define every value in our struct.

Note: Need a quicker way to convert a sample JSON payload to a Golang struct? There are a handful of awesome websites that will do this conversion automatically for you. I like to use this one, but that’s only one of a handful of options!

Next, we’ll start building out our getCurrentWeather() function:

// snippet from: bot/command-weather.go

func getCurrentWeather(message string) *discordgo.MessageSend {
 // Match 5-digit US ZIP code
 r, _ := regexp.Compile(`\d{5}`)
 zip := r.FindString(message)

 // If ZIP not found, return an error
 if zip == "" {
  return &discordgo.MessageSend{
   Content: "Sorry that ZIP code doesn't look right",
  }
 }

In our function definition, we’ll be expecting a string input containing the user’s message - for example: “!zip 12345” - and we’ll be returning a discordgo.MessageSend object like I mentioned earlier.

Next, we’ll do a quick (and very lazy) regular expression match against the incoming message & attempt to pull out the ZIP code. This regex is only searching for a pattern of 5 digits. If the r.FindString() call finds a match, it will return the 5-digit ZIP code. If it doesn’t match, it will return an empty string.

In case we don’t match a ZIP code, we’ll check to see if the zip variable is empty. If it is, we’ll have Discord send a message back to the user saying it’s invalid.

Here’s where we’ll see a simple example of using our discordgo.MessageSend object. This object is just a Go struct, so we can use it to store data in the format of Key: Value. In this case, we can still respond with a simple plain-text message - by using the Content key and supplying a string message. The full definition & possible fields for this object are in the discordgo docs.

Next, we’ll handle making our HTTP request to the OpenWeather API:

// snippet from: bot/command-weather.go

        weatherURL := fmt.Sprintf("%szip=%s&units=imperial&appid=%s", URL, zip, OpenWeatherToken)

 // Create new HTTP client & set timeout
 client := http.Client{Timeout: 5 * time.Second}

 // Query OpenWeather API
 response, err := client.Get(weatherURL)
 if err != nil {
  return &discordgo.MessageSend{
   Content: "Sorry, there was an error trying to get the weather",
  }
 }

We’ll start by generating the complete API URL. We’ll use fmt.Sprintf to generate a formatted string & inject variable values. So in this case, we’re combining the base URL with the required parameters: zip, units, and appid. The zip value will contain the ZIP code we just collected from our user. appid will be our OpenWeather API key. Using fmt.Sprintf, we can inject variables using the %s placeholder for string values - then provide those values as inputs.

So for example, a full query URL for ZIP code 12345 may look similar to this: https://api.openweathermap.org/data/2.5/weather?zip=12345&units=imperial&appid=ABCDEF12345

Then we create a new instance of an http.Client - just so we can modify the timeout value to 5 seconds. We then use this client to issue a HTTP GET request to the full weatherURL.

Assuming the call succeeds, we’ll have our JSON response stored in the response variable. If not, we’ll quickly check for an error & send a Discord message back to the user.

Generating a Discord Embed Message

Now we can parse our JSON response:

// snippet from: bot/command-weather.go

 // Open HTTP response body
 body, _ := ioutil.ReadAll(response.Body)
 defer response.Body.Close()

 // Convert JSON
 var data WeatherData
 json.Unmarshal([]byte(body), &data)

 // Pull out desired weather info & Convert to string if necessary
 city := data.Name
 conditions := data.Weather[0].Description
 temperature := strconv.FormatFloat(data.Main.Temp, 'f', 2, 64)
 humidity := strconv.Itoa(data.Main.Humidity)
 wind := strconv.FormatFloat(data.Wind.Speed, 'f', 2, 64)

We’ll read in the HTTP response body using ioutil.ReadAll(). We’ll need to close this object, so we’ll immediately follow that read with a defer response.Body.Close() to make sure that happens.

Next we’ll create a new variable called data, which will be of type WeatherData - our struct that we created earlier to store the JSON data. We can then convert that JSON response to our data object using json.Unmarshal() and passing our HTTP body & the target variable to store the data in.

Lastly, we can start pulling out the information we need to use later. For clarity, I’ve chosen to pull out each individual value here & assign them to their own variable names. Since our Discord response can only be a string, we also handle type conversions here - from integer or float64 to a string.

Finally, we can generate our Discord MessageSend object:

// snippet from: bot/command-weather.go

 // Build Discord embed response
 embed := &discordgo.MessageSend{
  Embeds: []*discordgo.MessageEmbed{{
   Type:        discordgo.EmbedTypeRich,
   Title:       "Current Weather",
   Description: "Weather for " + city,
   Fields: []*discordgo.MessageEmbedField{
    {
     Name:   "Conditions",
     Value:  conditions,
     Inline: true,
    },
    {
     Name:   "Temperature",
     Value:  temperature + "°F",
     Inline: true,
    },
    {
     Name:   "Humidity",
     Value:  humidity + "%",
     Inline: true,
    },
    {
     Name:   "Wind",
     Value:  wind + " mph",
     Inline: true,
    },
   },
  },
  },
 }

 return embed
}

We’ll create a new variable called embed, which will store a discordgo.MessageSend struct.

Earlier we used this to send a Content: value, but this time we’ll be leveraging the Embeds key. Embeds is of the type []*discordgo.MessageEmbed, which is again just a struct to store data. For reference, the format & possible fields of the MessageEmbed object are listed here

Within this MessageEmbed object, we can start filling in data about our message. So we fill in our message Title and Description - and tell Discord that this is going to be a richtext type of response.

We’ll be displaying a box in discord that has a number of key/value pairs - which we’ll use to show the measurement title & value. To do this, we’ll fill in the Fields: key - which takes in a list of structs of the type []*discordgo.MessageEmbedField.

For each of the weather measurements we want to display, we’ll provide a Name:, the Value:, and specify Inline: true. The Inline options tells Discord to try and list each key/value inline with eachother, rather than placing each pair on a new line.

Last, we’ll return our discordgo.MessageSend object via return embed.

Testing

Let’s test this out. We can stop our currently running bot with Ctrl+C, then restart with go run main.go.

We should be able to ask our bot for the weather using our new !zip command:

Look at that! The bot quickly responds with the current weather, and formatted in an embedded message. There’s a lot more we could do with embedded messages, but this is just an example to get started.

Okay - I think that wraps up this blog post. I know it was a long one, so if you’re still with me - Thank you! I hope this was helpful.

I am planning on a follow up post shortly, where I’ll cover how to add Discord slash commands to our example bot. Check back soon or subscribe to the Blog or YouTube if you’re interested!

A few months ago, I had written a blog post on building a simple chatbot with Webex. You can find that here.

At the time I was already thinking about some follow-up content, where I could walk through how to use AdaptiveCards to make the bot a little fancier - but I decided to wait a bit & see how the first post performed first.

Well, now it’s a few months later & I suppose it’s about time I write this 😄.

So in this post - We’ll build on the code I used in the last post. We’ll show how to add cards to both display information, as well as take user input.

What are AdaptiveCards?

AdaptiveCards are actually not a Webex/Cisco-specific thing! They were actually created by Microsoft with the intention of being a platform-agnostic way of displaying information. So in theory, a card schema written for a Webex bot could be taken & re-used on another messaging platform that implements AdaptiveCards - without any/much rework.

Hopefully this should mean a more consistent user experience for a bot that is supported across multiple platforms. The cards themselves can also enhance the experience by making your bot more dynamic & interactive. Maybe a developer or network engineer is okay with issuing specific syntax-based commands to a bit - but what about someone who is less computer savvy? Using cards makes it easy to provide guided input with textboxes, buttons, and toggles.

AdaptiveCards are built on a standard JSON structure with a pre-defined schema of card properties. Let’s take a quick look at a simple example:

{
    "type": "AdaptiveCard",
    "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
    "version": "1.2",
    "body": [
        {
            "type": "TextBlock",
            "text": "Hello there! What's your name?",
            "wrap": true
        },
        {
            "type": "Input.Text",
            "placeholder": "Your name here",
            "id": "user_name"
        },
        {
            "type": "ActionSet",
            "actions": [
                {
                    "type": "Action.Submit",
                    "title": "Submit"
                }
            ]
        }
    ]
}

So in the JSON payload above, we’re creating a card that asks for user input. We have a block of text that asks the user “What’s your name?”, then we provide a text input field, and finally a submit button. Looks pretty straightforward, yeah?

We’ll get into the specifics around how to create that card JSON in a bit, but you can always use Microsoft’s AdaptiveCard Designer to play around with card elements/structure.

So what would that card look like when rendered in Webex? Let’s take a look:

That card seems a lot more user-friendly than trying to explain to your users that they need to remember some sytax like /botcommand username set <firstname> <lastname>. Instead, they can just enter the info on the card & click submit.

Sending a Static, Default Card from a JSON File

So first we’ll start off with the easier scenario. We’ll have a pre-defined JSON card built, and use the bot to load & send this card response.

One note before we get started - I’ll be building on my simple weather chatbot code that I mentioned earlier. If you’re following along, you can pull down that code from here.

Okay, to start off - we’ll be creating a card that allows a user to enter their desired zip code. Instead of having to remember the specific command syntax, a user will instead just issue the weather command & our bot will prompt for information via a card.

I’ve used the Microsoft Adaptive Card Designer to build out the basic card structure, which I’ve kept simple for now. Here’s what that looks like:

{
    "type": "AdaptiveCard",
    "$schema": "http://adaptivecards.io/schemas/adaptive-card.json",
    "version": "1.2",
    "body": [
        {
            "type": "TextBlock",
            "text": "Get Current Weather",
            "wrap": true,
            "size": "Medium",
            "weight": "Bolder"
        },
        {
            "type": "TextBlock",
            "text": "Enter a Zip code:",
            "wrap": true
        },
        {
            "type": "Input.Number",
            "placeholder": "00000",
            "id": "zip_code",
            "min": 1,
            "max": 99950
        },
        {
            "type": "ActionSet",
            "actions": [
                {
                    "type": "Action.Submit",
                    "title": "Get Weather",
                    "data": {
                        "callback_keyword": "weather"
                    }
                }
            ]
        }
    ]
}

This card looks pretty similar to the example I used earlier. We have an initial text block that displays the title “Get Current Weather”, and we’ve added attributes for bold text & a medium size.

Next, we have a normal text block prompting the user to enter their zip code. This is followed by a number input box for that data entry.

The input box has a “min” & “max” value for the zip code from 1 to 99950 - which is the full range of US zip codes. We also need to assign any input a unique “id” value - which we’ll see later is used to pull out the user input from the response.

Last, we have our actions - which similar to earlier. This will present as a simple submit button with the text “Get Weather”. The important note here is adding a data dictionary with a callback_keyword sub-element. We set the callback_keyword to the command we want this submission to be sent to. In our case, we want the input to be sent back to the weather command.

Now to implement. Back in our bot, we’ll edit the weather.py file to load the card & assign as the default command response. Let’s take a look:

### Script snippet

with open("./input-card.json", "r") as card:
    INPUT_CARD = json.load(card)

class WeatherByZIP(Command):
    def __init__(self):
        super().__init__(
            command_keyword="weather",
            help_message="Get current weather conditions by ZIP code.",
            card=INPUT_CARD,
        )
        
### Script snippet

In the above snippet, I’ve extracted just the pieces we’ll be editing. For full examples, please see the final example repo here.

I’ve saved that JSON payload from earlier as input-card.json. Then we’ll open that card file & assign that JSON payload to a variable called INPUT_CARD.

Since we’ll expect this card to be the primary method of collecting a zip code - we’ll make this card the default response from our bot when the command is triggered. In my original code, we initialized the WeatherByZIP class with card=None - which instead passed our input directly to the execute() function. In this case, we’ll set that to card=INPUT_CARD to send our form.

With all that done, let’s test it out:

Collecting & Processing Card Submissions

Now that we have our input card, let’s move on to processing the response.

In the original example, we expected the user to enter a command like weather 12345 - and it was easy enough to just strip the 5-digit zip code from the response.

Since a card could have many input boxes, we instead receive that data back in a structured JSON format.

If we check out the logging in the webex_bot console, we can see the payload that gets returned to us when the card is submitted:

2022-03-03 14:59:26  [INFO]  [webex_websocket_client.root._process_incoming_websocket_message]:62 attachment_actions from message_base_64_id: Webex Teams AttachmentAction:
{
  "id": "--removed--",
  "type": "submit",
  "messageId": "--removed--",
  "inputs": {
    "callback_keyword": "weather",
    "zip_code": "12345"
  },
  "personId": "--removed--",
  "roomId": "--removed--",
  "created": "2022-03-03T19:59:24.820Z"
}

Using this output, we can see exactly what data is passed back to our bot. Under imports, we do see the correct callback_keyword that we defined in our card JSON. But we also see the user input - which is 12345. This is listed under the key zip_code which is the id we assigned to our input field earlier.

Now we can easily dig through this response to pull out the data we need. This JSON payload is delivered to the execute function under the attachment_actions parameter.

So we’ll make some minor modifications to our original code to pull that value out.

def execute(self, message, attachment_actions, activity):
    zip_code = attachment_actions.inputs['zip_code']

As a reminder, zip_code is the variable we’re using inside the execute() function to store the requested zip code - which is then passed to the OpenWeather API.

Previously, we had just assigned the value of message to zip_code, which was the incoming text message from the user.

Instead, this time we’ll fetch the zip code from the incoming card response, using attachment_actions.inputs & pulling out the zip_code key.

With that simple change completed, we should be able to ask the bot for the weather, input our zip code, then receive the same text-based output as before.

Here’s what that looks like now:

As we would expect, upon hitting Get Weather - the bot returns the text-based weather report.

Next, we’ll take a look at dynamically building a card to display weather info.

Dynamically Building Cards

Now we get to the real fun part. We’ll need to dynamically build an AdaptiveCard response, depending on what values we want to present to the user.

I will be using a Python module called adaptivecardbuilder. There are a handful of modules available that can help dynamically build cards, but this was the one I found to be most intuitive for me. While I will walk through an example here, I would also recommend reviewing their docs for additional context/examples.

You could also generate a JSON payload manually, or build one using the card designer & try to edit the JSON dynamically to change card values - but I won’t be doing that here.

So we’ll start off by installing the adaptive card module:

pip install adaptivecardbuilder

Before we build out the full card response, let’s take a quick look at how we can use this module.

In our execute() function, we’ll create a simple card that shows what zip code was entered.

First, we’ll need to add two new imports:

from webex_bot.models.response import Response
from adaptivecardbuilder import *

We’ll import the Response module from webex_bot, which will be required to send any attachments / custom response to the client. We’ll also import our adaptivecardbuilder module.

Next, we’ll modify the execute() function to look like the following:

    def execute(self, message, attachment_actions, activity):
        zip_code = attachment_actions.inputs['zip_code']
        
        card = AdaptiveCard()
        card.add(TextBlock(text=f"Weather for {zip_code}", size="Medium", weight="Bolder"))
        card_data = json.loads(asyncio.run(card.to_json()))

        card_payload = {
            "contentType": "application/vnd.microsoft.card.adaptive",
            "content": card_data,
        }

        response = Response()
        response.text = "Test Card"
        response.attachments = card_payload

        return response

First we create a new AdaptiveCard() instance. With this new object we can add different card elements.

In this case, we add a single card element - a TextBlock. This has multiple attributes, including the actual text content, and we’re specifying that we want a medium font size & bold text.

Next we serialize the card payload to JSON using asyncio.run.

In order to prepare the card object to be attached to our response, we will need to set the appropriate headers. In this case we need to wrap the JSON structure with the contentType header. We could use the response.attachments object to send any number of different file types to the user, but in this case we’ll set that value to application/vnd.microsoft.card.adaptive.

Then we can assemble the response to the client. We’ll create a new instance of the Response() object, where we’ll define some necessary attributes.

We’ll set our response.text first. This will be the text message sent to the client. If the client supports adaptive cards, then this message won’t be shown to the user - however, it may still appear in pop notifications. If the client does not support adaptive cards, only this message will be shown to the user.

Next we attach our complete card payload using response.attachments.

Last but not least, we can return that response object - which is then sent back to the user.

In this case, we can see the bot responds back to our request with a simple card that displays the zip code.

Let’s move onto building out the remainder of the card. For the purposes of this example, I’ll also be pulling out a few additional items from the OpenWeather API - including sunrise, sunset, pressure, and high/low temperatures.

I’ll go ahead and show the full card build here:

card = AdaptiveCard()
card.add(TextBlock(text=f"Weather for {city}", size="Medium", weight="Bolder"))
card.add(ColumnSet())
card.add(Column(width="stretch"))
card.add(FactSet())
card.add(Fact(title="Temp", value=f"{temperature}"))
card.add(Fact(title="Conditions", value=f"{conditions}"))
card.add(Fact(title="Wind", value=f"{wind}"))
card.up_one_level()
card.up_one_level()
card.add(Column(width="stretch"))
card.add(FactSet())
card.add(Fact(title="High", value=f"{high_temp}"))
card.add(Fact(title="Low", value=f"{low_temp}"))
card.add(Fact(title="Humidity", value=f"{humidity}"))
card_data = json.loads(asyncio.run(card.to_json()))

If this looks a little messy, it is. There is a cleaner way to do this, which I’ll show in a moment.

The first thing to know is that the card JSON structure is hierarchical. So individual card items need to be added under the correct hierarchy - for example, you create a ColumnSet first, then add a Column underneath, then the items within that Column last.

See the below visualization of this card:

Top Level
 ↳ Text Block
 ↳ ColumnSet
   ↳ Column
     ↳ Fact Set
       ↳ Fact
       ↳ Fact
       ↳ Fact
    ↳ Column
     ↳ Fact Set
       ↳ Fact
       ↳ Fact
       ↳ Fact

So on our card, we first placed the title TextBlock. Underneath that we created a ColumnSet, which will contain two Columns.

Each one of those Columns contains a FactSet, which is a key/value pair listing of information. And of course, we add each Fact pair underneath the FactSet.

You’ll notice after building one Column and FactSet, we use card.up_one_level() twice - to return back to the ColumnSet level, where we then add a second Column.

This can certainly get a little confusing at first, but I’ll show a more intuitive method in a moment.

Let’s take a peak at what this looks like now:

Sub-cards & Better Card Building

In this section we’ll do two quick things. First we’ll take a look at a simpler, more visual way of building the cards. Then we’ll also expand our example with sub-cards.

The adaptivecardsbuilder module supports a second way of building cards, which I personally find to be much easier to use.

In this method, we just create a single card.add() object - and feed it a list of all the items on the card.

For me, this makes it easier because you can add indentation to each list item - which helps keep track of where you are at in the hierarchical structure.

To navigate within the structure, we use "<" to go up one level, or "^" to return to the top level.

I’ve re-written the earlier example with this new structure below:

card = AdaptiveCard()
card.add(
    [
        TextBlock(text=f"Weather for {city}", size="Medium", weight="Bolder"),
        ColumnSet(),
            Column(width="stretch"),
                FactSet(),
                    Fact(title="Temp", value=f"{temperature}F"),
                    Fact(title="Conditions", value=f"{conditions}"),
                    Fact(title="Wind", value=f"{wind} mph"),
                    "<",
                "<",
            Column(width="stretch"),
                FactSet(),
                Fact(title="High", value=f"{high_temp}F"),
                Fact(title="Low", value=f"{low_temp}F"),
                "^",
        ActionSet(),
            ActionShowCard(title="More Details"),
                ColumnSet(),
                    Column(width="stretch"),
                        FactSet(),
                            Fact(title="Humidity", value=f"{humidity}%"),
                            Fact(title="Pressure", value=f"{pressure}"),
                            "<",
                        "<",
                    Column(width="stretch"),
                        FactSet(),
                            Fact(title="Sunrise", value=f"{sunrise}"),
                            Fact(title="Sunset", value=f"{sunset}"),
    ]
)
card_data = json.loads(asyncio.run(card.to_json()))

Again, you’re welcome to use whichever method suits you - but I find this method to be a bit cleaner & easier to read/troubleshoot.

You can also see that I’ve added a new ActionSet, with a single action: ActionShowCard. This adds a nice little button to our card where we can optionally display additional information.

Here’s what that looks like now:

And if we expand that sub-card:

Sub-cards can be an easy way to hide excess information to keep the response clean & focus on the important information.

Well that’s it for this blog post. There’s a lot more you can do with cards, including adding images, video, tables, dropdown menus, etc. But I just wanted to show a few examples of building cards. They can be a simple way to make chatbots a little less boring!

Enjoy!

I’ve been spending some of my time lately working on a new project that involves an interactive chatbot - where you can send commands to the bot & ask for it to take certain actions or return information.

I didn’t previously have a ton of experience building something like this. Mostly what I’ve done before is just using Webex or Discord APIs to send alerts or messages to a room.

As with anything new - I’ve stumbled my way through a couple of things, but have been enjoying the challenge of trying something different. Building out some of the more interactive parts of the bot hasn’t been quite as difficult as I originally imagined, and some of the newer modules & SDKs have made the process much simpler.

With all that being said - I wanted to throw together a quick guide on how to get started building a basic chatbot, in case anyone else is interested but worried that it may be too complicated.

Repo here for all the sample code used below

Step 1: Getting API Keys

So the first thing we’ll worry about is getting signed up for a Webex developer account & generating API keys.

So head on over to the Webex Developer site & log in (or sign up for a new account).

Once logged in, we’ll go to the upper-right corner of the screen and click on our user icon - and select “My Webex Apps” from the drop down:

Then we’ll be asked what type of app we’re creating. In this case, we’ll select bot:

Next we’ll need to provide some details about our bot. This includes a display name, username, image, and some details. Once the bot is completed, we have the ability to publish it via the Webex App Hub where anyone could interact with it. If you’re looking to do this, make sure you give a good description of your bot!

After we finish all that, we’ll finally get our API key!

Webhook vs Websocket: Which to use?

Next, I wanted to cover the two primary methods of sending & receiving messages from the Webex APIs: Webhooks and websockets.

Webhooks

Likely webhooks are the method most people are familiar with. Using this method, we need to run out bot code on a publicly reachable URL. When we run our bot, one of our first actions will be sending a request to the Webex APIs with our bot URL. This way, every time Webex receives a message from a user that wants to interact with our bot, the Webex cloud knows where to send that chat message.

Now, the potential downside of using webhooks is that we need to 1) manage a web server and 2) expose the bot publicly to the internet. Both of these could be overcome with easy workarounds, if needed. For example, we could use something like FastAPI to quickly build our web listener & handle incoming POST requests. We could also use a tunneling method (like ngrok) to avoid having to directly open ports to our webserver.

Websockets

On the other hand, we could use websockets to accomplish the same function. With a websocket, we’re instead opening a direct, persistent TCP connection to the Webex APIs. Through this persistent connection, we can send & receive messages very quickly and easily.

This method offers a few advantages over webhooks. Primarily not having to expose our app publicly to the internet. Websockets act almost similar to something like ngrok, where we create an outbound tunnel - except in this case, only Webex knows about it and has access to it (where ngrok is still providing you a public URL that anyone could hit, if they knew about it). Additionally, this method removes the need for maintaining a web server & having to handle all of the incoming HTTP requests in your bot’s code. There may also be a handful of API calls which are only supported via websockets today as well - like having your bot show that they’ve read a message, for example.

I’ve also found that the websocket method seems to be much more responsive, likely because there is a persistent TCP connection open - and you no longer have to rely on a full HTTP session establishment & teardown for every message to the bot.

Historically, websockets were only officially supported via the Webex JavaScript SDK. However, it seems recently that someone has built a Python equivalent earlier this year - which we’ll be using throughout this blog post. You can find that package here

Getting started: Registering the Bot with Webex

So we’ll start with some basic functionality. Let’s grab the modules we need, apply our API key, and see if we can get a response from our bot.

Luckily, the module we’re using makes this super easy. So let’s install via pip:

pip install webex_bot

Once that’s installed, we’ll create a new Python file - I’ll call mine bot.py

In that file, we’ll start our script like this:

from webex_bot.webex_bot import WebexBot

api_token = "<TOKEN_HERE>"

bot = WebexBot(api_token, approved_domains=["0x2142.com"])

bot.run()

Looks easy enough, yeah? We’ll import our webex_bot module first, as always. Then we’ll define our API key. Obviously this can be stored as an environmental variable or another more secure location, but we’ll define it here for demonstration.

Then we create a new instance of WebexBot, by providing our API key and an optional approved_domains value. In my case, I’ve provided the 0x2142.com domain - which means the bot will reject messages from anyone who isn’t from that organization.

Last, we start our bot with bot.run().

Let’s go ahead and run our bot with python bot.py and see what happens!

You should see something similar to this:

matt@0x2142:~/demo-webex-bot$ python bot.py
2021-10-25 15:42:52  [INFO]  [webex_bot.webex_bot.webex_bot.__init__]:39 Registering bot with Webex cloud
2021-10-25 15:42:52  [INFO]  [restsession.webexteamssdk.restsession.user_agent]:167 User-Agent: webexteamssdk/0+unknown {"implementation": {"name": "CPython", "version": "3.8.10"}, "system": {"name": "Linux", "release": "5.10.16.3-microsoft-standard-WSL2"}, "cpu": "x86_64"}
2021-10-25 15:42:52  [WARNING]  [command.webex_bot.models.command.__init__]:33 no card actions data so no entry for 'callback_keyword' for echo
2021-10-25 15:42:52  [INFO]  [command.webex_bot.models.command.set_default_card_callback_keyword]:47 Added default action for 'echo' callback_keyword=callback___echo
2021-10-25 15:42:53  [INFO]  [webex_bot.webex_bot.webex_bot.get_me_info]:74 Running as bot '0x2142 Bot' with email ['0x2142@webex.bot']
2021-10-25 15:42:54  [INFO]  [webex_websocket_client.root._connect_and_listen]:151 Opening websocket connection to wss://mercury-connection-partition2-a.wbx2.com/v1/apps/wx2/registrations/06b5c858-174a-4977-a268-04530d777563/messages
2021-10-25 15:42:54  [INFO]  [webex_websocket_client.root._connect_and_listen]:154 WebSocket Opened.

So long as we continue to run this script in the foreground - we’ll be able to see the live log of anything that happens, including messages to our bot. So for now, we’ll leave this up & running.

Now we should be able to try sending our bot a message in Webex.

In the Webex app, if we do a quick search for our bot (by email or name), we should see it pop up pretty quickly:

With our new space, we can interact with our bot. I’ll send a “Hello” message:

And the webex_bot module will automatically respond back with a built-in help card.

Currently the bot only supports one command: echo. We’ll add our own commands shortly, but for now let’s give that a try:

So in the example above, I used the echo command. The bot returns a card with an input field. Once you fill out the textbox & hit submit, the bot will echo back whatever text was provided!

So now we have a running bot. But what about implementing our own commands? Let’s take a look!

Creating custom bot commands

Now we get to the fun part!! To demonstrate, let’s add a command to ask our bot for the current weather conditions. To accomplish this, we’ll use the OpenWeather APIs.

So to begin - We’ll probably want a separate Python file for each command we want to add. First we’ll add weather.py to our current directory.

Next, we’ll throw in a bit of boilerplate code that will be used for each command:

from webex_bot.models.command import Command
import logging

log = logging.getLogger(__name__)

class WeatherByZIP(Command):
    def __init__(self):
        super().__init__(
            command_keyword="weather",
            help_message="Get current weather conditions by ZIP code.",
            card=None,
        )

    def execute(self, message, attachment_actions, activity):
        return f"Got the message: {message}"

So first we’ll import Command from our webex_bot module, which will be used to create our custom command class.

Next we create a new class, in this case WeatherByZIP, and inherit the Command class from webex_bot. Within this new class, we’ll create our __init__ function & define some information about our command.

So using super().__init__(), we’ll provide what keyword should trigger this command, any help message to be provided to the user, and an optional card. We saw how the card could work earlier with the echo command - but to keep this module simple, we’ll go without a card for now.

After that - we just need to create an execute function. This is what gets called by webex_bot when our command is triggered. By default, webex_bot will pass us the user’s message - and, in the case of an action (like a card submission), it will provide those details.

Two things to keep in mind with our execute function:

1. the message variable will contain the user’s message with the command keyword removed. So in our case, if the user’s message was “weather 12345” - then the message variable would just contain “12345”.
2. Any text we return from our function will be sent via the bot as a chat response. If we wanted to provide extras, like cards or attachments, we would need to use the webex_bot Response object. I might cover this in a separate blog post later

Okay! So right now our new command is configured to just echo back any test that is sent to the bot - similar to the echo command earlier, except without the card.

We’ll add our weather functionality shortly - but let’s first add our command to the bot & give it a quick test.

In our bot.py file, we’ll just need to import our command module and call bot.add_command():

from webex_bot.webex_bot import WebexBot
from weather import WeatherByZIP

api_token = "<TOKEN_HERE>"

bot = WebexBot(api_token, approved_domains=["0x2142.com"])

bot.add_command(WeatherByZIP())

bot.run()

That’s it! Now we should be able to try it out. We’ll restart our bot, and send the command weather 12345:

Sure enough, our bot gets the message, passes it to our custom command, and returns the stripped message - all as expected.

So let’s add our weather query & actually make this thing work.

Using the OpenWeather APIs, I’ll create a new API key - then create a variable in the command module called OPENWEATHER_KEY. Then, I’ve updated my execute function to look like this:

def execute(self, message, attachment_actions, activity):
    # Message may include spaces. Strip whitespace:
    zip_code = message.strip()

    # Define our URL, with requested ZIP code & API Key
    url = "https://api.openweathermap.org/data/2.5/weather?"
    url += f"zip={zip_code}&units=imperial&appid={OPENWEATHER_KEY}"

    # Query weather
    response = requests.get(url)
    weather = response.json()

    # Pull out desired info
    city = weather['name']
    conditions = weather['weather'][0]['description']
    temperature = weather['main']['temp']
    humidity = weather['main']['humidity']
    wind = weather['wind']['speed']

    response_message = f"In {city}, it's currently {temperature}F with {conditions}. "
    response_message += f"Wind speed is {wind}mph. Humidity is {humidity}%"

    return response_message

So because of the way that message is passed to us, we’ll first need to strip whitespace. This is because when the user enters a command like “weather 12345”, webex_bot will only remove the “weather” portion - leaving us with " 12345". So by stripping that value, we’re left with just the numeric ZIP Code: “12345”.

Next we assemble our URL, supplying our parameters: zip, units, and appid. Zip is the numeric ZIP code provided by our user, units will be either imperial or metric, and appid is our API key.

After that, it’s as simple as sending the GET request & parsing the data. In the code above, I assemble the final text response into response_message which is then returned to the user.

All that being done - Let’s try it out!!

In this example, I used the ZIP code for Richfield, OH - and our bot quickly returned that information to us.

And that’s it! In just a few steps we created a new Webex bot & added custom commands. Once we have that process down, it’s easy enough to continue extending our bot by adding additional commands.

There are obviously more advanced use cases - and a whole lot of fun that comes with using Adaptive Cards… But I hope this post was helpful in getting you started!

Update 2022/03/12 - If you’re interested in seeing additional content around Cards, please check out the new blog post & video here

“As a network administrator, do I really need to learn to code?”

“How much to I need to learn?”

“This will NEVER be in enterprise networks”

Well, today I’m here to give some thoughts. As with all my content, I reserve the right to be wrong or not 100% in alignment with everyone else’s view of the world.

Do I need to learn code?

Nope.

Wait really?!?

Correct, no one can make you learn if you don’t have an interest in it.

Okay, but will I benefit from it?

100% yes you will. To what degree may vary, but I think it’s worth a look.

Well how much do I need to know?

As the network architect might say, it depends.

Is automation coming for my job?

¯\_(ツ)_/¯

So what’s the right answer here? It depends on what’s right for you - what you’re interested in, what you think will help you in your job, and what you think will help you succeed.

Yes, it’s true that much of the networking industry has been shifting focus to automation & programmability. Open APIs have become much more important in recent years as we seek to expand our capabilities through custom code & integrations. I don’t see this slowing down anytime soon.

But will that spell the end of the network engineer as we know it? I don’t know that anyone can 100% say definitively, but I’m willing to bet that network engineers aren’t going anywhere anytime soon. The role will definitely change & evolve over time (as it already has been, and will continue), but truly go away? I think not.

The benefits of programmability

I’m sure most people are familiar with a lot of this by now, but why should we care about automation? Well there are lots of reasons! For example, here’s a few:

• Reducing human error
• Reducing time spent on repetitive tasks
• Accomplishing things that aren’t natively possible in a product
• Adding custom functionality or integrations with other systems
• And more!

How much you should learn

This one REAALLY depends. To throw out some ideas, here are some factors that could help determine how much you’ll need to know:

• How much you enjoy it
• How much your company cares or has any automation initiatives
• The type (and age) of gear you’re working with
• Whether or not you have an in-house development or devops team

So on one side, I think that most people who get into writing Python & find that they truly enjoy it will likely end up transitioning to a more developer-focused role - whether that be a network-centric devops role or pure development role. Or simply just seeking out more ways to focus on programmability within their current network engineering role. These types of people are finding that they love solving problems & being able to have the creative freedom to write whatever code they want to accomplish a task. They’ve found something that they enjoy - Good for them!

On the other side, even if a network engineer isn’t going to write code very often - just knowing the fundamentals can be hugely beneficial. For example, as a network engineer, have you ever had to learn how VMware works? Likely you didn’t feel the need to become a VMware administrator overnight - but understanding how a vSwitch works has helped you support those teams. How much easier did it become to communicate with the VMware admins?

With programmability, it’s much the same. If you’re not going to dive in, that’s okay. But you’ll likely benefit from a good overall understanding of how things work. For me personally, it was easy to recognize the benefit when I was a network administrator. Application team puts in a ticket saying something isn’t working right? Well guess who understands API calls, HTTP requests, and JSON/XML? It suddenly became much easier to help find root cause, and often easier to shift blame away from the network. Plus, when you speak with an application/development team & you can use their terms, they feel like you understand them. This helps reduce friction between teams. In my experience, what I found was that those application teams would then ask for my help more often - because I was willing to go further than “checked the firewall, it’s not the network”.

In my opinion, whether or not you learn to code & how much you learn comes down to your goals. Can you keep being a network admin & avoid code? Sure - just like you could continue being a network admin & never learn design, business goals, or what other teams in your company do. But will learning those things expand your knowledge & give you a better big-picture view? Yup. Would learning those things give you an advantage? Absolutely.

What happens when you run into a difficult challenge where manual effort would be reckless? Maybe you pass off the job to a development team to write a utility or some custom integration. But guess what happens next? That development team doesn’t understand how your network gear operates. Again, this situation really benefits from having an individual who can speak both languages. Even if you don’t understand functions & classes, you might be able to read API docs for your network gear and help guide the developer through where to start.

What about that in-between state? Maybe you want to learn to code a bit, but not 100% of the time. What then? Well, could you pull & terminate ethernet cables without a cable tester? Yes, but would you be more efficient with one? Also yes. Similarly, being able to code can just be as simple as adding another tool to your skill set. Would you automate a single port change? Nope. But what if you find yourself making that same port change several times a week? That’s when you pull out your Python skills and write a quick script.

And for those who would rather just not do any of it? You’re fine. How many parts of the Internet still run legacy protocols? Someone has to support that. And there are companies out there who want commercially supported products, so they have someone to blame. Vendors know this, which is why we end up with big, centralized controllers & complex GUI apps to help slowly pull networking out of the CLI days. But you can’t tell me that SD-Access, ACI, or EVPN are simple technologies - companies will still benefit from employing people who understand the underlying technologies & protocols. Let’s be honest: automated or not, things still break - and someone will need to have the knowledge to fix it.

So what’s the answer here?

¯\_(ツ)_/¯

It seems like these debates continue because everyone comes from different backgrounds & has different experiences. Some people work in companies that would never entertain the idea of custom code, where others may find their team being pushed to become more agile & learn Python to automate tasks. Neither approach is wrong, but of course our human instinct is that we have to assert that everyone else must be wrong, since their view doesn’t match ours. The world is never a one-size-fits-all for every situation, yet for some reason we’re all inclined to think it is.

All that being said, companies are going to adopt this stuff at different paces. Some find immediate value in it now, and some are scared or just set in their ways. Again, neither is wrong - but they both have their own advantages & disadvantages.

And that’s why I say - if you don’t want to learn any of this, you have nothing to fear. There will always be organizations at different stages of the cycle.

But if you do learn? At worst, you gain knowledge outside of your primary domain. At best, you gain a new tool to leverage & find ways to improve your work.

At the end of the day, you have to focus on what’s right for you, your career, and your current employer. And that answer is going to be different for everyone.

Maybe it’s just me - but I feel like when we want to demonstrate product APIs to someone, we usually jump into Postman first. It’s not without good reason though. Postman is a very easy to use platform for running API calls against REST endpoints & see the nicely formatted output.

That being said, often times I’ve seen that we stop there. We might mention Python as an advanced method of using our APIs, or maybe we talk about how “if APIs are the new CLI, Postman is the new PuTTY”. I’ve seen some engineers who hear these statements and feel like Postman is being pushed on them - even when they’re perfectly happy with an SSH-based CLI.

Network automation isn’t usually accomplished with just one tool - it’s a whole storage closet full of different tools & utilities, each with their own use cases or specializations. In this particular example - learning Python allows you to move beyond one-off API calls in Postman, and into being able to accomplish much more complex automation.

So the purpose of this post is to explore how to get beyond just Postman, and into using Python for REST API calls. This isn’t going to be a complete run-down of all the capabilities of both tools - but rather a few examples of simple GET requests using both methods.

If you don’t have Postman yet, go ahead and snag the download here.

Note: Much of the code below is minimal and does not contain any error handling. Therefore, it should not be used in a production network. Full example code here.

Postman: Simple GET Request

So first, let’s start off with an example of using Postman for a simple GET request.

In this example, we’ll keep things simple & use a non-authenticated API endpoint. We’ll accomplish this using a free website called JSON Placeholder.

Let’s go ahead and start up Postman, and we’ll see a blank workspace:

What we’ll need to pay attention to first, is what type of HTTP request we’re sending - as well as the URL we want to send our request to. If we expand the request type dropdown, we’ll see a handful of options - but we’ll only be using a GET request for now:

In order to create our first request, we’ll just need to enter our API endpoint URL. We’ll use the users endpoint offered by JSON Placeholder, which is located at: https://jsonplaceholder.typicode.com/users

In the screenshot above, you’ll see in the highlighted box that we entered our URL. Next, we just click the big Send button on the right side.

I already sent the GET request, so in the screenshot you’ll also notice that we have our JSON response from the API. The mock data offered by this API provides us with a list of 10 different users, along with the data that’s been entered for each user.

What if we only wanted to get data for one specific user? Well, our test API site supports using query parameters to filter our response. Let’s say we want to find information for a user named Delphine. We would append ?username=Delphine to our URL:

So as we can see above - now our JSON response only contains the data for a single user. If we needed to find a user’s phone number, this could be an easy way to quickly filter our data & get to what we need.

This is really where Postman is great. We’re able to quickly & visually test out an API call. We can see what the response data looks like, and understand the structure of requests & responses.

But what about when we want to iterate through a number of users and pull only a specific few statistics? Or maybe we need a way to take a list of user information, and automatically upload or convert that data into another system.

For use cases like that, we’ll jump over to Python.

Python: Simple GET Request

In this section, we’ll walk through the same example from above - but this time using Python.

First thing we’ll need to do, is install the Python requests library. While there are a handful of libraries that can accomplish this work, requests is fairly popular & easy to use.

So we’ll kick things off by using pip to install our library:

pip install requests

Then, we’ll create a very simple Python script to perform the same initial GET request from earlier. So we’ll pull a list of ALL users from the API.

import requests

URL = "https://jsonplaceholder.typicode.com/users"

response = requests.get(URL)

print(response.text)

The code above should be pretty straightforward. First we’ll import our requests library. Then, just to keep the code clean, we’ll create a variable called URL to hold the URL for the API endpoint.

Next, we send that GET request, using requests.get. Last but not least, we’ll go ahead and print out the text payload that we receive back. That output looks pretty similar:

matt@DESKTOP:~/postman-python$ python3 simpleGET.py 
[
  {
    "id": 1,
    "name": "Leanne Graham",
    "username": "Bret",
    "email": "Sincere@april.biz",
    "address": {
      "street": "Kulas Light",
      "suite": "Apt. 556",
      "city": "Gwenborough",
      "zipcode": "92998-3874",
      "geo": {
        "lat": "-37.3159",
        "lng": "81.1496"
      }
    },
    "phone": "1-770-736-8031 x56442",
    "website": "hildegard.org",
    "company": {
     -- output truncated --

Now, let’s add a little extra logic to our next step. Rather than just specifying a static username that we want to search for, let’s ask our user to specify a username:

import requests

URL = "https://jsonplaceholder.typicode.com/users"

print("Search by Username:")
user = input("> ")
queryURL = URL + f"?username={user}"
response = requests.get(queryURL)

print(response.text)

In the code above, we made just a few changes. One is printing out a prompt to “Search by Username”. Then we use the Python input function to collect input from our user, then store that in the user variable.

Next, we create a queryURL - which is similar to the URL we used in the Postman example. Except in this case, we’re supplying a dynamic username input based on whatever the end user wants to search for. We use a Python f-string to inject the username into our query parameters.

Let’s see what that looks like:

matt@DESKTOP:~/postman-python$ python3 simpleGET.py 

Search by Username:
> Delphine
[
  {
    "id": 9,
    "name": "Glenna Reichert",
    "username": "Delphine",
    "email": "Chaim_McDermott@dana.io",
    "address": {
      "street": "Dayna Park",
      "suite": "Suite 449",
      "city": "Bartholomebury",
      "zipcode": "76495-3109",
      "geo": {
        "lat": "24.6463",
        "lng": "-168.8889"
      }
    },
    "phone": "(775)976-6794 x41206",
    "website": "conrad.com",
    "company": {
      "name": "Yost and Sons",
      "catchPhrase": "Switchable contextually-based project",
      "bs": "aggregate real-time technologies"
    }
  }
]

Perfect! We get the result we expected, which is the single dataset from the user that we specified.

Let’s take this one step further! Maybe we want to build a quick utility to search for a user’s contact information. We can take the raw JSON output, pull out a few pieces of info, then apply some formatting to make it more user-friendly.

In the following example, we’ll update our code to search that JSON response & collect the user’s name, email address, and phone number.

import requests
import json

URL = "https://jsonplaceholder.typicode.com/users"

print("Search by Username:")
user = input("> ")
queryURL = URL + f"?username={user}"
response = requests.get(queryURL)

userdata = json.loads(response.text)[0]

name = userdata["name"]
email = userdata["email"]
phone = userdata["phone"]

print(f"{name} can be reached via the following methods:")
print(f"Email: {email}")
print(f"Phone: {phone}")

So a few things have changed in the above example. First thing you may notice, is that we’ve added an additional import: the json library. We use this in line 11, where we convert the JSON output into a native python object using the json.loads function.

What this allows us to do is easily pull individual data values from the JSON output. In the original JSON data that we received, we saw a bunch of user data organized into key-value pairs. So once we load that JSON into a Python dictionary, we can pull out individual values and assign them to variables.

Finally - we can print all of that data out to our terminal:

matt@DESKTOP:~/postman-python$ python3 simpleGET.py

Search by Username:
> Delphine
Glenna Reichert can be reached via the following methods:
Email: Chaim_McDermott@dana.io
Phone: (775)976-6794 x41206

Nothing super fancy here - but this could be the beginnings of building out a user search web page, or maybe importing certain data fields into another system. Once we get the basics of working with REST APIs out of the way, there are a ton of possibilities for what can be built.

Okay - What About Network Automation?

Many new network technologies are now API enabled. Nearly every cloud-hosted platform or management controller offers some method of interacting programmatically. In fact, most routers & switches even offer a REST-based API that can be used for automated configuration or monitoring.

So let’s take the following example: We have a couple of Meraki networks, and we’ve been tasked with providing a report of how many total clients have connected to our network in the last 30 days. In addition, it would be nice to see a breakdown of the distribution of operating systems.

Let’s take a look at the following screenshot from Postman:

Using Meraki’s API docs, we can determine the exact URL endpoint we need to query. This URL requires that we know the exact network ID for the network we want to get info from, which we’ll pretend we already know. Also not shown here, we have an additional HTTP header that includes our API key - which was generated in Meraki Dashboard.

In addition, we have a few query parameters to help make sure we get the data we need. By default, this API endpoint will return 10 devices. In this particular network, we already know we have more than 10 - so we use the perPage query parameter to request up to 100 devices. We also wanted to query the last 30 days of devices, so we use the timespan parameter. This parameter expects the lookback to be in seconds, so we specify 43,200 seconds.

So we got back a list of devices - and specifically in the example shown, it looks like we have an Android device on our network. Easy enough to spot that info, right?

Well - maybe we have more than a handful of devices. We don’t want to manually sort through all of that data & assemble a list. Or maybe this task needs to be run more than once. This is where we can use Python to automatically assemble that report with just a few lines of code.

import requests
import json

URL = "https://api.meraki.com/api/v1"
APIKEY = {"X-Cisco-Meraki-API-Key": "xxxxxxxxxxxxxxxxx"}

--- Code omitted ---

def getClients(orgID, networkList):
    """
    Query clients for each network, return client list
    """
    clientCount = {}
    total = 0
    # Query Parameters: Return up to 100 devices seen in the past 43,200 seconds (30 days)
    q = {"perPage": "100",
         "timespan": "43200"}
    for network in networkList:
        # Query clients for each network
        queryURL = URL + f"/networks/{network}/clients"
        response = requests.get(queryURL, params=q, headers=APIKEY)
        data = json.loads(response.text)
        # Grab client OS from each device & append to clientCount dictionary
        for client in data:
            try:
                clientCount[client["os"]] += 1
            except KeyError:
                clientCount[client["os"]] = 1
            except TypeError:
                continue
            total += 1
    # Append final count of all devices & return dict
    clientCount["Total Devices"] = total
    return clientCount

def printReport(clientOS):
    """
    Print final output to terminal
    """
    print("Count of clients by operating system:")
    for OS in clientOS:
        print(f"{OS}: {clientOS[OS]}")
        
--- Code omitted ---

To keep things simple - we’ll only look at a snippet of the code. The entire example will be posted to GitHub, and contains two additional functions that will automatically find our Organization ID & Network ID.

In our getClients function, we create an empty Python dictionary to hold our list of client operating systems - and we also create a total variable which we’ll increment for every client in the list.

The HTTP GET request should look fairly similar to what we used previously. The big difference is that we’re creating a dictionary called q to hold our perPage & timespan query parameters. Then we include that in our GET request by adding params=q to the request.

Next - we convert the JSON response into a Python object, and walk through every device in that list. For each device in the list, we look at the “os” value to determine the operating system detected by Meraki. We then use a try/except block to see if we can increment the counter for that operating system. If we get a KeyError, then we know the OS isn’t currently on the list & we can just add it with a new value of 1.

After all that has been completed, we return our clientCount dictionary - which our primary function passes to the printReport function. This just prints out a header, then iterates through our clientCount dictionary and prints each operating system & count.

Let’s take a look at that output below:

matt@DESKTOP:~/postman-python$ python getOScount.py
Count of clients by operating system:
None: 9
Sailfish OS: 2
Nokia: 1
Windows 10: 2
Raspberry Pi: 2
Android: 4
Meraki OS: 1
Total Devices: 21

In the list above, we can see that we have a total of 21 devices in our network - and 7 different operating systems were found as well.

That output may not be the prettiest of reports, but it was accomplished with ~50 lines of Python & takes less than a few seconds to run. If we wanted to format the data in another way, or include data from a non-Meraki source, we absolutely could.

That’s part of what’s exciting about using Python for network automation - most anything we could think of doing is possible.

TIP: Use Postman to Generate Python Code

Okay - so I wanted to save this bit for last. Now that you’ve seen examples in both tools, I wanted to make sure we cover a feature in postman that can save some time.

So over in the right-hand side of your Postman window, you’ll see an icon that looks like this: </>

If you click that, you’ll see a dropdown menu of various languages & tools. Select one of those options and Postman will auto-generate code you can use. This auto-generated code will execute the same request you have built in the Postman GUI.

For example, our last Postman request we used to get Meraki clients:

I wanted to save this for last, because I feel like it’s important to understand how to write the code manually first - then take advantages of shortcuts after having a good understanding of what the code is doing.

That’s about all I had for this example. Again, my intent here wasn’t to create an all-inclusive resource for how to use Postman & Python - but rather to give some examples for each one & show where/why each would be used.

I’ve met a handful of people over the years who have seen many API demos using Postman, but aren’t sold yet on the value of learning Python & getting into network automation.

Hopefully these examples help bridge that gap a little bit 😄

This post has been on my backlog for a while… I’ve been intermittently trying out the IOS-XE guestshell feature for a few different projects, and never feeling like any of them were good enough to write about.

So why not just write about guestshell itself? It’s such a nifty tool.

What is Guestshell?

Now that Cisco’s Catalyst platform has standardized on IOS-XE, we can take advantage of some of the features of the Linux-based platform. One of which being Linux Containers (LXC).

Most of the latest Catalyst routers, switches, and access points support what Cisco calls Application Hosting - which is a fancy way of just saying it can run containers on-box. Another acronym to be aware of is IOx - which is what Cisco calls their IOS-XE/Linux appliction framework.

Guestshell is a specific feature within IOS-XE, where a dedicated pre-built container is enabled for on-box access to a Linux shell.

What Hardware Supports Guestshell?

Okay - so for this example, I’ll be using the new Catalyst 8000V platform (a virtual Catalyst router). I do also have a Catalyst 9200L on hand, but unfortunately it’s one of the few Catalyst platforms that does not support Guestshell (due to lower-end HW specs).

You can also use the Catalyst 8000V to test this in a lab, or you could use most of the Catalyst 8000 / 9000 series platforms if you had them available (or the CSRv & some ISR routers!).

You can find a list of supported platforms & other hardware requirements here.

All that being said - let’s get started!

Enabling IOx

So before we can actually use the guestshell feature, there is some pre-configuration to be done.

First we’ll need to ensure that the IOx application framework is enabled & running. We can check this by using the show iox command:

0x8KV01# show iox

IOx Infrastructure Summary:
---------------------------
IOx service (CAF)              : Not Running
IOx service (HA)               : Not Supported
IOx service (IOxman)           : Not Running
IOx service (Sec storage)      : Not Supported
Libvirtd 5.5.0                 : Running

And as we see above - most components are listed as “Not Running” or “Not Supported”.

So we’ll enter config mode, and use the iox command to enable these services:

0x8KV01# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
0x8KV01(config)# iox 
0x8KV01(config)#
0x8KV01(config)# exit
0x8KV01# show iox

IOx Infrastructure Summary:
---------------------------
IOx service (CAF)              : Running
IOx service (HA)               : Not Supported
IOx service (IOxman)           : Running
IOx service (Sec storage)      : Not Supported
Libvirtd 5.5.0                 : Running

Configuring Guestshell Networking

There’s a good chance we’ll want our container to have access to the outside world, so let’s take a quick look at the network configuration.

Note: The networking config syntax will differ if you are using a Catalyst router or a Catalyst switch. Since I am using a Catalyst 8000V - we will look at the config for a Catalyst router first.

In order to allow external access to our container, we’ll need to configure a gateway interface and NAT translations. Then we can provide a full network configuration to our container:

## Enable NAT on our external interface - in my case, gigabitEthernet1
0x8KV01(config)# interface gigabitEthernet 1
0x8KV01(config-if)# ip nat outside

## Create a virtual interface for the guestshell container
0x8KV01(config-if)# int virtualportgroup0
0x8KV01(config-if)# ip address 192.168.100.1 255.255.255.0
0x8KV01(config-if)# ip nat inside

## Create an ACL to match traffic from guestshell & Enable NAT
0x8KV01(config)# ip access-list standard IOX_NAT
0x8KV01(config-std-nacl)# permit 192.168.100.0 0.0.0.255
0x8KV01(config)# ip nat inside source list IOX_NAT interface gigabitEthernet 1 overload

## Assign network configuration to our guestshell container
0x8KV01(config)#app-hosting appid guestshell
0x8KV01(config-app-hosting)# app-vnic gateway0 virtualportgroup 0 guest-interface 0
0x8KV01(config-app-hosting-gateway0)# guest-ipaddress 192.168.100.5 netmask 255.255.255.0
0x8KV01(config-app-hosting-gateway0)# app-default-gateway 192.168.100.1 guest-interface 0
0x8KV01(config-app-hosting)# name-server0 8.8.8.8

Now, as I mentioned before - this config will look different if you’re using a switch vs a router.

So let’s take a quick look at what this would look like, if we were on a Catalyst Switch instead:

Cat9k(config)# interface AppGigabitEthernet 1/0/1
Cat9k(config-if)# switchport mode trunk

Cat9k(config)# app-hosting appid name
Cat9k(config-app-hosting)# app-vnic AppGigabitEthernet trunk
Cat9k(config-app-hosting-trunk)# vlan 10 guest-interface 0
Cat9k(config-app-hosting-vlan-access-ip))# guest-ipaddress 192.168.100.1 netmask 255.255.255.0

Cat9k(config-app-hosting)# app-default-gateway 192.168.100.1 guest-interface 0
Cat9k(config)# name-server 8.8.8.8 

Once all that config is done - Let’s move on to getting our container started!!

Managing the Guestshell Process

In order to start a container on our Catalyst device, we’ll need to go through a process of deploying the app, activating it, then starting it.

For guestshell, there is a shortcut command that will perform all of these steps for you - and make managing the guestshell process much easier.

So let’s go ahead and spin up our guestshell container with the command guestshell enable:

0x8KV01# guestshell enable
Interface will be selected if configured in app-hosting
Please wait for completion
guestshell installed successfully
Current state is: DEPLOYED
guestshell activated successfully
Current state is: ACTIVATED
guestshell started successfully
Current state is: RUNNING
Guestshell enabled successfully

0x8KV01# show app-hosting list
App id                                   State
---------------------------------------------------------
guestshell                               RUNNING

And as you can see above, everything gets automatically deployed & started. We can verify using the show app-hosting list command to see that our container has been started.

If we need to, we can also stop & deactivate our container using the command guestshell disable.

We could also use the commands below if we wanted to perform these steps manually:

0x8KV01# app-hosting activate appid guestshell
0x8KV01# app-hosting start appid guestshell

0x8KV01# app-hosting stop appid guestshell
0x8KV01# app-hosting deactivate appid guestshell

Before we move onto accessing the guestshell interface, I also wanted to show where you can get more detail about the guestshell container.

Using the show app-hosting detail appid guestshell command, we can see details around the container version, current MAC/IP config, and resource consumption:

0x8KV01# show app-hosting detail appid guestshell
App id                 : guestshell
Owner                  : iox
State                  : RUNNING
Application
  Type                 : lxc
  Name                 : GuestShell
  Version              : 3.2.0
  Description          : Cisco Systems Guest Shell XE for x86_64
  Path                 : /guestshell/:guestshell.tar
  URL Path             :
Activated profile name : custom

Resource reservation
  Memory               : 256 MB
  Disk                 : 1 MB
  CPU                  : 800 units
  CPU-percent          : 3 %
  VCPU                 : 1

Attached devices
  Type              Name               Alias
  ---------------------------------------------
  serial/shell     iox_console_shell   serial0
  serial/aux       iox_console_aux     serial1
  serial/syslog    iox_syslog          serial2
  serial/trace     iox_trace           serial3

Network interfaces
   ---------------------------------------
eth0:
   MAC address         : 52:54:dd:d:bf:da
   IPv4 address        : 192.168.100.5
   IPv6 address        : ::
   Network name        : VPG0

Port forwarding
  Table-entry  Service  Source-port  Destination-port
  ---------------------------------------------------

Accessing the Guestshell CLI

Now for the fun part! Let’s get into our guestshell container.

We’ll just need to use the guestshell command, and we’ll be dropped into the Linux shell:

0x8KV01# guestshell
[guestshell@guestshell ~]$
[guestshell@guestshell ~]$ uname -a
Linux guestshell 5.4.40 #1 SMP Tue Oct 6 17:57:00 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

Let’s also make sure our network & NAT configuration worked:

[guestshell@guestshell ~]$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=64 time=3.57 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=64 time=2.64 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=64 time=2.54 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 4ms
rtt min/avg/max/mdev = 2.543/2.916/3.567/0.465 ms

If needed, we can also run commands on our Catalyst device CLI without leaving the guestshell interface by using the dohost utility:

[guestshell@guestshell ~]$ dohost "show app-hosting list"
App id                                   State
---------------------------------------------------------
guestshell                               RUNNING

[guestshell@guestshell ~]$

Just a note, the reverse is also possible. Using the guestshell run command from our Catalyst CLI, we can run commands in our container:

0x8KV01# guestshell run uname -a
Linux guestshell 5.4.40 #1 SMP Tue Oct 6 17:57:00 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

0x8KV01#

It’s also worth reiterating, that the guestshell container is a full Linux container (It runs CentOS). So if we need to install additional Linux packages, we can use our Linux package manager (YUM) to do so.

Anyways - Let’s wrap this up by looking at the built-in Python modules & capabilities

Using Python in Guestshell

One of the more appealing use cases for guestshell is the ability to run native Python scripts & code directly on your Catalyst device.

A couple of possible use cases could be:

• Troubleshoot intermittent issues - Automatically collect device state (routes, MAC changes, etc) on a regular basis to log locally or send via email
• Automated resolution - If a certain type of state change is observed, we could run a pre-set list of commands to try and automatically fix the issue
• Local monitoring / troubleshooting - We could run a series of ping, web page load tests, etc from a python script to help monitor performance from the switch’s perspective
• Change management / notification - A script could be written to monitor the device configuration for any changes made, then send an automated email when something was detected
• Have a risky change that might sever your connection to a remote device? A script could execute the change for you, and if unsuccessful, revert to a working configuration (without doing a reload after)

These are just a handful of ideas that came to mind while writing this blog post… But there are many more possibilities!

Within the guestshell container, there is a special cli python module that comes pre-loaded. This module allows us to run CLI commands on our catalyst device directly from a python script & receive the command output for parsing.

Let’s look at a quick example using the Python interactive interpreter to save the device configuration:

[guestshell@guestshell ~]$ python3
Python 3.6.8 (default, Aug 24 2020, 17:57:11)
[GCC 8.3.1 20191121 (Red Hat 8.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> import cli
>>> status = cli.execute("write memory")
>>> if "OK" in status: print("Device configuration was saved!")
...
Device configuration was saved!

The example above uses the cli.execute Python function, which will run commands in Cisco’s exec mode.

What if we wanted to make configuration changes? In that case we would use the cli.configure function:

>>> config_data = """interface loopback200
... ip address 172.16.99.10 255.255.255.0
... """
>>>
>>> cli.configure(config_data)
'Line 1 SUCCESS:  interface loopback200\nLine 2 SUCCESS:  ip address 172.16.99.10 255.255.255.0\n'
>>>
>>> cli.execute("show run interface loopback200")
'Building configuration...\nCurrent configuration : 68 bytes\n!\ninterface Loopback200\n ip address 172.16.99.10 255.255.255.0\nend\n'

In the above example, we used a multi-line comment to provide the list of commands to enter. The cli.configure function also supports a Python list - so we could provide the commands in that format as well.

You’ll notice that the Python interpreter returns a status for each command that we attempted to run. This should allow us to error-check & ensure that all of our intended configuration was accepted.

Lastly, we used the cli.execute command again to validate that our running configuration looks correct.

There’s a bit more functionality built into the cli Python module, but I just wanted to cover some of the common use cases. More details can be found here

A Few Common Questions

Does this give me access to the underlying IOS-XE Linux shell?

Nope, anything running in guestshell is running in a container - which runs on-top of the IOS-XE image.

What about resource conflicts?

The linux containers running on the IOS-XE platform are given their own dedicated resources. Any resource constraints that affect the containers will not impact normal network functions.

In the last post, I covered a simple project that I’m working on while studying for the Cisco DevNet certifications. This is part two of a series, which will continue on while I work through this project.

As a brief summary - I started using a combination of Scrapli & Cisco Genie for a short network automation script. This script connects to a list of network switches & outputs a spreadsheet of interface statistics. This provides a quick snapshot view into the current port capacity within your network.

In this post, I’ll be showing off a bit of how to build a simple web frontend - which I’ll put together using Flask and Bootstrap.

Getting Started with Flask

Okay so I have a bad habit of making a lot of Python scripts which are essentially just command-line utilities.

In theory, this isn’t necessarily problem. But what if I want to share some of the tools I build? Maybe not everyone wants to compile config files & figure out what command-line arguments are needed. For some, they may prefer to have an easy-to-use web interface instead.

So for this project I opted to venture a little outside my typical comfort zone & build a web dashboard. I’ll admit I’ve done this once or twice before, but certainly not something I would claim proficiency in!

The good thing is - building a web interface doesn’t have to be complex. My intent with this post is to try & break it down a bit - and hopefully encourage you to try it yourself!

Installing Flask & Hello World

Installing Flask is much like any other python module - we’ll use pip:

pip install flask

Once installed, we can import the module into our python script:

from flask import Flask

Simple enough!

Now let’s look at how easy it is to set up the web server & return a simple web page:

app = Flask(__name__)

@app.route('/')
def main():
    return "Hello there!"

if __name__ == '__main__':
    app.run()

First we create a new instance of Flask, using the syntax app = Flask(name).

Next, we just create a quick python function to return the text Hello there! You’ll notice that there is a decorator above the function - this is used by Flask to route incoming HTTP requests.

In this example, we use @app.route(’/’). This tells Flask to register this python function for any HTTP calls to http://<url>/. If we wanted a function for any HTTP requests to http://<url>/networking, we would modify the decorator to @app.route(’/networking’).

Lastly, we need to start the Flask web server when the script is run. This is accomplished using app.run()

Let’s go ahead and try to run the script - which should start the Flask server:

In the above screenshot, you’ll see that by default Flask will start on http://127.0.0.1:5000. You may also notice that Flask will print out real-time logging of incoming HTTP requests.

If we visit our page, we’ll see pretty much what we might expect:

Before we move onto the next section, I did want to show one more example.

Let’s say that we have a python variable (or dictionary, etc). Could we pass that as part of our web page function?

Let’s look at this modified example:

web_page_text = {"Hello": "There!"}

@app.route('/')
def main():
    return web_page_text

If we reload our local web page, we’ll now see the following:

Keep this in mind - We’ll use this later!

Making life easier with HTML templates

Now then - you could build all of your HTML in python, then return the entire HTML page as a response to the function call. That being said, just because you can doesn’t mean you should 🙃

Let’s take a quick look at how we can use HTML templates to help build our web page.

First we’ll create a new directory called templates, and create an HTML file. In this case, I named my template main.html:

To start with, we’ll just add some simple text into the HTML file:

<body>
    Hello There - From the template!
</body>

On the Flask side, we’ll have to modify our imports to include the render_template module. We’ll also update what we’re returning when an HTTP request is received:

from flask import Flask, render_template

app = Flask(__name__)

@app.route('/')
def main():
    return render_template('main.html')

if __name__ == '__main__':
    app.run()

So now instead of returning plain text, we’ll return render_template(‘main.html’).

Flask uses Jinja2 for templating, which we’ll get into shortly. The above function call tells our app to use main.html as a base template for the content that is returned to the end user.

Let’s check out the page:

Awesome - we receive our HTML file as a response this time.

Extending functionality with Jinja2

Now we can start getting into the fun stuff.

Manually writing out a bunch of HTML is fun and all - but what if we could auto-generate everything programmatically?

Jinja2 allows for exactly that. We’ll be able to insert variables, if/then statements, and looping logic directly into our HTML template.

As an example, let’s say we had the following python dictionary - which we wanted to display on a web page:

switchList = [
        {
        "name": "C9200",
        "serial": "ABCD00010",
        "ip": "192.168.1.1",
        "check": True,
        "total": 28,
        "up": 10,
        "down": 5,
        "disabled": 13,
        "capacity": 35       
    },
    {
        "name": "C9300",
        "serial": "ABCD00011",
        "ip": "172.168.44.2",
        "check": False,
        "total": 48,
        "up": 32,
        "down": 6,
        "disabled": 10,  
        "capacity": 67       
    },
        {
        "name": "C9400",
        "serial": "ABCD00012",
        "ip": "172.33.44.2",
        "check": True,
        "total": 48,
        "up": 40,
        "down": 6,
        "disabled": 2,  
        "capacity": 82       
    }
]

This dictionary is an example from the scrapli script I’ve been working on (more detail in my last post).

One way for us to present this using an HTML template would be the following:

<body>
    {% block content %}
    
    {% for switch in switches %}
    Switch Name: {{ switch.name }} 
      - Serial: {{ switch.serial }} 
      - Reachable at: {{ switch.ip }} 
      
    {% endfor %}
    
    {% endblock %}
</body>

As you can see - that’s a bit of a change from what we were using in our template before.

The Jinja2 syntax above uses a combination of curly braces & percentage signs to indicate which parts of our template are logic that should be processed before returning content to the user. Variables are represented with double curly braces.

Once our template receives the switchList dictionary, we can iterate through it similar to how we might in python. Using the {% for switch in switches %} syntax, we can iterate through each item - and pull out relevant data.

Within our for loop, we can reference the individual values of each item in the dictionary. Using the double braces, we can insert a placeholder for where a piece of content should be inserted. For example, in the above template we have a spot where we want to insert the name of the switch. We use {{ switch.name }} as a placeholder, where the value from the dictionary will be inserted once our template is processed.

Okay - so before we test this, we have one minor modification to our python function:

@app.route('/')
def main():
    return render_template('main.html', switches=switchList)

We’ve changed our function render_template, but also include the python object that we’ll be passing to our template.

Let’s test this out:

Now there’s some magic! Using a combination of python & our Jinja2 templates - we can drastically reduce the amount of HTML code that we need to write.

Okay, but that’s not pretty

Yes I know. But we needed to get the basics done first!

Here’s where Bootstrap comes in.

I went out to Google and searched for good CSS templates. I’m not a web developer, nor am I good at design - so I’ll leave that work to someone else.

In my searching, I found quite a bunch of free templates… but I fell in love with one called Lux by Bootswatch.

They provide a great sample page for the template, which shows off different variations. But the best part is that it also includes code samples!

After we pick & download our CSS template, we’ll place it in a new directory named static:

To get started, we’ll need to install one additional python module:

pip install flask_bootstrap

Then we’ll also need a quick modification to our base Flask app:

from flask_bootstrap import Bootstrap

... code omitted ...

if __name__ == '__main__':
    Bootstrap(app)
    app.run()

I removed some code to focus on just the two parts that change.

First, we need to import our new module. Second, we need to inject our bootstrap plugin into the Flask app. We do this with Bootstrap(app) prior to app.run().

Let’s move back over to our HTML template, since this is where the most of our changes will be.

First, we’ll need to include a few things to make sure our CSS file is loaded:

{%- extends "bootstrap/base.html" %}

<head>
    {% block styles %}
    {{super()}}
    <link rel="stylesheet"
            href="{{url_for('static', filename='bootstrap.css')}}">
    {% endblock %}

</head>

I won’t dive in too deep on this piece, as it’s fairly straightforward. First, we’ll need to include the base/default bootstrap template. Next, we’ll include a reference to where our CSS file is located & it’s name - so that it gets loaded when our page is rendered.

Okay, now for the body of the page:

<body>
    {% block content %}
    <table class="table table-hover">
        <thead>
          <tr>
            <th scope="col">Name</th>
            <th scope="col">Current Capacity</th>
          </tr>
        </thead>
        <tbody>
          {% for switch in switches %}
            {% if switch.capacity > 75 %}
                <tr class="table-danger">
            {% else %}
                <tr class="table-success">
            {% endif %}
            <th scope="row">{{ switch.name }}</th>
            <td>{{ switch.capacity }}%</td>
          </tr>
          {% endfor %}
        </tbody>
      </table> 
    {% endblock %}
</body>

Now instead of just returning a plain-text list of all three switches, we’ll use our CSS template to generate a colorful table. For the purpose of this example, I kept the data to just showing the switch name & it’s current capacity.

Most of the template is fairly simple. First we build our table header using the <thead> tags. Next we’ll build the rows of our table using <tbody> and <tr>.

For the Jinja2 logic - We’ll iterate through our list of switches, and add a new table row (<tr>) for each device. I also added logic to check the current capacity of the switch. If the switch capacity is greater than 75%, we create the table row using <tr class=“table-danger”> - which is a reference in our CSS stylesheet that will color the row red. If capacity is less than 75%, we use table-success for a green color.

Let’s go ahead and check out what this looks like:

And as we expect - we get our nicely formatted table & the correct colors on each row.

Bringing it all together

Now that we’ve gotten some background on how to use Flask & Bootstrap, let me show you a bit of what I’ve been working on.

I won’t cover much code in this section, but you can check out the GitHub repo if you’re interested in seeing the details.

The scrapli script that I wrote previously has been extended a little. The biggest change is that instead of writing its output to a spreadsheet, it will insert the data into a sqlite database.

The web frontend has been assembled using the same basic ideas covered in this blog (plus a lot of banging my head against the desk, trying to figure out why things don’t format or align properly 🙂). Whenever a request is made to the web dashboard, it will query the sqlite database for the requested information.

So, without further delay - here it is!

I currently have one physical device running, plus a handful of virtual nexus 9k & CSRv devices running in Cisco Modeling Labs. So all of the data shown in this dashboard is being pulled from real(ish) equipment - no mock data.

This was one of those projects where I started off thinking my dashboard was going to be really simple… but then I kept having new ideas and getting carried away with the design & functionality.

The table above shows each switch, it’s serial number, management IP, current software version, and the port inventory. Similar to the example earlier in this post, I have a small progress bar that shows switch capacity - and will change color depending on percentage of in-use ports.

If you click on any of the switch names, you’ll be taken to a page with additional detail on that particular device:

On this first tab, you’ll see some quick info - mostly the same that is on the main page of the dashboard. However, I added two additional tabs here - one for a detailed port breakdown & one for a dump of the raw CLI output.

If we click on the port info:

We have a breakdown of how many ports are currently in use vs down, as well as our breakdown of port media & operational speed. Since I account for switches that are anywhere from 10M up to 100G, the data for operational speeds will only show the speeds that are actually in-use on the switch.

Just in case it’s needed, I added a tab to show the raw CLI output from the switch:

This could be a quick way to check port counters & errors, or any other information that isn’t already presented via the dashboard.

Lastly, I built a separate page to provide statistics on ALL devices in the network:

This page will show port count & availability network-wide. However, I also added some spaces to show the top 5 hardware models & software versions that are in-use across the network.

I went into this project a bit worried because frontend development isn’t my strength. That being said, I really enjoyed working on this project. It was a different challenge than I’m normally used to - and it gave me a creative way to try and format & display the data I am collecting.

I hope this post was useful to you. Please leave a comment below - and check out my Github repo for the full code.

So I just scheduled an upcoming attempt at the Developing Applications using Cisco Core Platforms and APIs (DEVCOR 350-901) exam.

It’s coming up quick in two weeks here & I’m starting to do some review / final prep before taking the exam.

Why does this matter? Well - I plan on doing a few write-ups & videos on some of the content I’m studying. There are a lot of people diving into the DevNet exams, and I want to do what I can to help other people succeed.

All that being said, check back here over the next few weeks - or subscribe on YouTube - to see the additional content that will follow. While this specific blog doesn’t cover much from the exam blueprint, it’s the beginning a simple project I’ll be using in the further content.

Getting Started with Scrapli

For this project I opted to start with scrapli, and possibly migrate to using RESTCONF later on in the process. Mostly this was an excuse for me to try out scrapli, as I’ve heard a few people using it recently & was interested.

Scrapli is a screen scraping module for Python. If you’re not familiar with screen scraping, it’s the process of connecting to something via telnet/ssh/etc, then literally scraping or dumping the contents of the screen. There are other modules that do this as well, like paramiko, netmiko, or expect scripting.

In networking, too many of our devices still rely on CLI and don’t have proper API endpoints. We’re working on it, but yet still a ways from having it everywhere. So in the meantime, we still need screen scraping for automation.

On the whole, this isn’t necessarily a bad thing. For example, most network engineers are very familiar and competent with the CLI of any network operating system. It’s easier to jump into the world of automation if you can start with something you know, the CLI. It’s harder to force someone to start their automation journey by giving up everything they know and shoving REST APIs at them.

Okay - enough rambling. Let’s get scrapli installed and see what it can do.

Installing the module

Easiest part of the whole project. Install with pip:

pip install scrapli

Next we’ll go ahead & import into our script.

Scrapli supports quite a handful of operating systems (including a few non-Cisco platforms as well!). In the case of my project though, I’m only using Cisco IOS-XE switches - so we’ll only import the scrapli driver for that.

from scrapli.driver.core import IOSXEDriver

Connecting to a Device

Okay - now that we’re all setup with the module, it’s time to get working!

First thing, we need to authenticate to our device. Building off of the example code from the scrapli page - we’ll create a short dictionary of authentication parameters first:

switch = {
    "host": "10.1.1.1",
    "auth_username": "net_api",
    "auth_password": "net_api_pass",
    "auth_strict_key": False
}

cli = IOSXEDriver(**switch)
cli.open()

Once we build out dictionary, we’ll use **switch to unpack our key/value pairs into the IOSXEDriver object & assign it to a variable called cli. Then, all we need to do is call cli.open() and scrapli will open a connection to our target device.

Now we can run any command we want by calling cli.send_command():

sh_int = cli.send_command("show interface")
print(sh_int.output)

So in the above example, I want to issue a show interface - then print the output.

What we get is shown below:

GigabitEthernet1/0/1 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 78bc.1a81.e101 (bia 78bc.1a81.e101)
  Description: Test Port
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:11, output 00:00:03, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 198
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 9000 bits/sec, 11 packets/sec
  5 minute output rate 2066000 bits/sec, 221 packets/sec
     2647548 packets input, 371883264 bytes, 0 no buffer
     Received 6985 broadcasts (6757 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 6757 multicast, 0 pause input
     0 input packets with dribble condition detected
     50905390 packets output, 60134059101 bytes, 0 underruns
     0 output errors, 0 collisions, 2 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

[**Output Truncated**]

What’s that look like? Exactly the same output we would get if we entered show interface on the command line ourselves! (Only Gig1/0/1 shown here to keep this short & clean)

Using Cisco Genie to Parse CLI Output

So what if we wanted to pull a list of every interface on the switch? Maybe tally how many ports are connected vs down vs admin disabled? Or even check operational speeds & media types?

Well that’s what I’m looking to do for this project.

Now at first it may seem like we have to use regular expressions to try & match the content we need from the output. However, there is an easier way!

Cisco Genie is an open source project, and part of the larger PyATS automated network testing suite. If you haven’t looked at any of that yet, I would highly recommend you check it out.

So what’s Genie do? It’s a utility that handles all the output parsing for us. There are a ton of pre-existing parsers written, which handle all the hard work so we don’t have to.

Best yet - scrapli has native integration into Genie. All we have to do is install one additional component:

pip install scrapli[genie]

And to make use of the power of Genie, we just need to pass our raw output to it.

So our new code will look like this:

sh_int = cli.send_command("show interface")
sh_int_parsed = sh_int.genie_parse_output()
print(sh_int_parsed)

That’s it. And now, instead of that raw output - we get a native Python dictionary that’s ready to consume by our script:

{
   "GigabitEthernet1/0/1":{
      "port_channel":{
         "port_channel_member":False
      },
      "enabled":True,
      "line_protocol":"up",
      "oper_status":"up",
      "connected":True,
      "type":"Gigabit Ethernet",
      "mac_address":"78bc.1a81.e101",
      "phys_address":"78bc.1a81.e101",
      "description":"Test Port",
      "delay":10,
      "mtu":1500,
      "bandwidth":1000000,
      "reliability":"255/255",
      "txload":"1/255",
      "rxload":"1/255",
      "encapsulations":{
         "encapsulation":"arpa"
      },
      "keepalive":10,
      "duplex_mode":"full",
      "port_speed":"1000mb/s",
      "media_type":"10/100/1000BaseTX",
      "flow_control":{
         "receive":False,
         "send":False
      },
      "arp_type":"arpa",
      "arp_timeout":"04:00:00",
      "last_input":"00:00:05",
      "last_output":"00:00:08",
      "output_hang":"never",
      "queues":{
         "input_queue_size":0,
         "input_queue_max":2000,
         "input_queue_drops":0,
         "input_queue_flushes":0,
         "total_output_drop":198,
         "queue_strategy":"fifo",
         "output_queue_size":0,
         "output_queue_max":40
      },
      "counters":{
         "rate":{
            "load_interval":300,
            "in_rate":8000,
            "in_rate_pkts":11,
            "out_rate":1868000,
            "out_rate_pkts":205
         },
         "last_clear":"never",
         "in_pkts":2658450,
         "in_octets":372811780,
         "in_no_buffer":0,
         "in_multicast_pkts":6783,
         "in_broadcast_pkts":6783,
         "in_runts":0,
         "in_giants":0,
         "in_throttles":0,
         "in_errors":0,
         "in_crc_errors":0,
         "in_frame":0,
         "in_overrun":0,
         "in_ignored":0,
         "in_watchdog":0,
         "in_mac_pause_frames":0,
         "in_with_dribble":0,
         "out_pkts":51057453,
         "out_octets":60309072263,
         "out_underruns":0,
         "out_errors":0,
         "out_interface_resets":2,
         "out_collision":0,
         "out_unknown_protocl_drops":0,
         "out_babble":0,
         "out_late_collision":0,
         "out_deferred":0,
         "out_lost_carrier":0,
         "out_no_carrier":0,
         "out_mac_pause_frames":0,
         "out_buffer_failure":0,
         "out_buffers_swapped":0
      }
   }
}

This format makes our lives a lot easier. For example, let’s say we wanted to determine the operational state of a port. Without the Genie integration, we would need to write some regex to comb through the raw output & find what we needed.

However, with Genie involved - its as easy as this:

print(sh_int_parsed['GigabitEthernet1/0/1']['oper_status'])

Collecting the Useful Bits

Now that we have everything in an easy-to-use format, all we need to do is write a few loops to run through our interface list & collect data.

The purpose of my script is to automate the collection of port utilization. For example, think about performing a switch refresh or some form of capacity planning scenario. You might need to inventory how many switches you have, how many ports are utilized vs how many are available, or count the number of copper vs fiber ports.

For this first iteration, we’ll just collect the data then dump it out to a CSV file. As I mentioned earlier, this is just the start of a small project I’ll be using to study for the DEVCOR exam - so this will be evolving into a web service later on.

The full script can be found here. But I’ve posted just a snippet of how we count the different interface characteristics:

# Count all Ethernet interfaces
interfaceStats['total'] += 1
# Count admin-down interfaces
if not sh_int_parsed[iface]['enabled']:
    interfaceStats['intdisabled'] += 1
# Count not connected interfaces
elif sh_int_parsed[iface]['enabled'] and sh_int_parsed[iface]['oper_status'] == 'down':
    interfaceStats['intdown'] += 1
# Count up / connected interfaces - Then collect current speeds
elif sh_int_parsed[iface]['enabled'] and sh_int_parsed[iface]['connected']:
    interfaceStats['intup'] += 1
    speed = sh_int_parsed[iface]['bandwidth']
    if speed == 10_000:
        interfaceStats['intop10m'] += 1
    if speed == 100_000:
        interfaceStats['intop100m'] += 1
    if speed == 1_000_000:
        interfaceStats['intop1g'] += 1
    if speed == 10_000_000:
        interfaceStats['intop10g'] += 1
# Count number of interfaces by media type
try:
    media = sh_int_parsed[iface]['media_type']
    if '1000BaseTX' in media:
        interfaceStats['intmedcop'] += 1
    else:
        interfaceStats['intmedsfp'] += 1
except KeyError:
    interfaceStats['intmedsfp'] += 1

Once all that runs, we create a CSV file to dump all of the data to.

For example, running the complete script against my lab Catalyst 9200 switch would output the following:

And that’s it! Using the combination of scrapli & genie made this script very quick and easy to write - which means now I can focus more time on the next steps.

As I’ve mentioned a few times, this is hopefully the start of a short series of blog posts & videos as I wrap up studying for the DEVCOR exam.

If you found this content helpful - or you’re interested in the future content - please check back soon! Or consider subscribing to my YouTube channel, where new content will be coming shortly.

In a recent post, I walked through setting up a Netgear LTE cellular modem with Google Fi. Might seem random - but I had a plan for this!

In trying to ensure I keep a stable internet connection for work, I eventually led myself down the path of buying a cellular modem. That was part one. The next step was being able to somewhat-intelligently monitor my home internet connection, and then fail over to the cell modem when connectivity was poor (think lazy SDWAN).

At first I figured this would be easy…. but of course nothing is :)

I currently have a Cisco FirePower 1010 appliance that I’m using for a home firewall. Unfortunately, while this thing does support IP SLA - It wouldn’t quite accomplish what I wanted to do. And to be fair, this box is intended to be a security appliance - not SDWAN.

Anyways - I talked myself into just writing some Python automation to monitor my home internet, and dynamically inject/remove static routing entries. I needed a new project anyways

For those of you wanting to jump straight to the code, the GitHub repo is here

Part 1 - Monitoring Packet Loss & Latency

So first thing - I regularly experience both complete internet outages (always at the worst times), as well as very high latency. Packet loss seems to be less common with my ISP - but hey, it happens sometimes too.

I opted to use the pythonping module as an easy way to collect some of the info I needed.

After importing the module, it’s as easy as specifying a destination, packet size, and number of ICMP messages to send:

from pythonping import ping
result = ping('8.8.8.8', size=2, count=10, verbose=True)

To simplify at least one part of the operation, pythonping has a built in function to return the average round trip:

>>> print(result.rtt_avg_ms)
74.23

The actual contents of our ping results are going to be in the following format: Reply from 8.8.8.8, 10 bytes in 43.82ms

So a quick way for me to figure out packet loss was to simply search for the “Reply from” text in each response:

lost = 0
for packet in result:
    if "Reply from" in str(packet):
        pass
    else:
        lost += 1

Easy enough - and we can use that to calculate the amount of packet loss against the 10 total packets sent:

lossperc = 100 * lost / 10

Once all that is figured out, we can use a simple expression to determine whether or not the loss or latency thresholds have been exceeded:

if response >= MAX_LATENCY or loss >= MAX_LOSS:
    print("Loss/Latency violate thresholds.")
    return True
else:
    print("Loss/Latency within thresholds.")
    return False

The code I wrote then takes one of two paths.

• If the loss and latency is ABOVE our thresholds, call to the FirePower module to inject a static default route over the LTE modem
• If the loss and latency is BELOW our thresholds, call to the FirePower module to delete the static default route - which returns traffic to the primary internet

These functions within the FirePower module were written so that each change will only be made once. For example, if the script checks and finds that the loss/latency is bad, but the static route towards the LTE modem already exists - then no additional changes are made.

So next - Let’s take a look at the FirePower module.

Part 2 - Authenticating to FDM & Initial Checks

This was my first time getting into the FirePower FDM APIs - but they ended up being fairly straightforward to use.

Turns out that FDM has built-in API documentation, which is extremely helpful. This can be found at https://<FDM IP>/#/api-explorer

Okay - So first thing’s first. Authenticating to the firewall:

oauth_data = {
    "grant_type": "password",
    "username": "admin",
    "password": "Cisco1234!"
}
headers = {
          "Content-Type": "application/json",
          "Accept": "application/json"
}
baseurl = "https://" + FDM + "/api/fdm/latest"

authurl = baseurl + "/fdm/token"
print("Posting AUTH request to FDM")
# POST request to FDM with headers / oauth data
resp = self.s.post(authurl, headers=headers,
                   data=json.dumps(oauth_data),
                   verify=False)
# If success - only pull out & return the auth token
if resp.status_code == 200:
    print("Auth success - got token.")
    return json.loads(resp.text)['access_token']
else:
    print("Authentication Failed.")
    print(resp.text)

In the code above - we take the headers and some basic authentication info (username, password, grant type) and send it in an HTTP POST request to https://<FDM IP>/api/fdm/latest/fdm/token

This returns an authentication token, which we’ll need to include in any further HTTP request to the firewall. This can be done by sending a new set of headers in the future:

headers = {
          "Content-Type": "application/json",
          "Accept": "application/json",
          "Authorization": "Bearer " + self.token
}

Next we need to figure out which routing table to insert the route into. Since I am only using the default routing table, the script will just grab the UID for the default global routing table:

vr_url = baseurl + "/devices/default/routing/virtualrouters"
routing_table = requests.get(vr_url, headers=headers)

I mentioned earlier that the script will not make any changes if the firewall is already in the desired state. So we will take time now to check what state the firewall is in, before attempting to make any changes.

This is accomplished by making our first primary action scanning the routing table to see if our route already exists:

route_url = baseurl + "/devices/default/routing/virtualrouters/" + \
            self.globalVR + "/staticrouteentries"
route_data = requests.get(route_url, headers=headers)

# Convert returned JSON object
current_routes = json.loads(route_data)['items']
# Run through each route returned and see if it matches the one we're looking for
for route in current_routes:
    # For each route, we need to manually go look up the uid of our gateway & network objects
    gateway = self.getNetworkObject(route['gateway']['id'])
    dest_network = self.getNetworkObject(route['networks'][0]['id']).split('/')[0]
    # Match based on route prefix & upstream next hop gateway
    if gateway == GATEWAY and dest_network == ROUTE.split('/')[0]:
        print("Found route to %s via %s" % (dest_network, gateway))
        return route

Depending on whether we were looking to add or remove a route, the script may proceed with doing so - or just quit if the action has already been taken previously.

Part 3 - Failover (Add Static Route)

Assuming that we need to Add a route, we need to sent a POST request to https://<FDM IP>/api/fdm/latest/devices/default/routing/virtualrouters/<Virtual Router ID>/staticrouteentries with some required information (like subnet info, next-hop, etc)

Unfortunately, we can’t just send a POST with the target subnet & gateway. Those need to be created as network objects on the FirePower box beforehand. For each network object, we need to build a quick JSON object with the required info:

host_data = {}
host_data['name'] = name
host_data['description'] = "Created by ISP Failover automation"
host_data['subType'] = subtype
host_data['value'] = address
host_data['type'] = "networkobject"

Then we can send a POST to the FDM API to create the object with our supplied parameters:

posturl = baseurl + "/object/networks"
requests.post(posturl, headers=headers, json=host_data)