Hey everyone. I’ve got another present for you - Free study resources to use while studying for Juniper Network’s JNCIA-Cloud cert (As a reminder, I also gave a list of Free study resources for the JNCIA-Junos exam).

1) Juniper Open Learning

If you were familiar with Junos Genius, it was discontinued, but all the Associate level material can be now found Juniper Open Learning for free. Lots of videos & practice exams to get your prepared for the Associate level exams!!

2) Juniper Networks Day One Books

Description: “Day One Books cover networking technologies using step-by-step instructions and practical examples written by working engineers.” Here’s the link to the Day One Books; there are numerous Day One Books, and I guarantee you’ll learn a lot. Me, personally, I like to read with physical books, so I bought the “Day One: Data Center Fundamentals” book for myself. But here’s the pdf version !!!!

3) Documentation (aka ‘TechLibrary’)

First: For any cert, print & follow the Exam Objectives.

Second: As you’ll see from the Exam Objectives, there are a lot of Juniper solutions on this exam. Luckily their ‘TechLibrary’ is FILLED with all the info you need about the specific products. I really hope you like to read; you’ll be doing a lot of that. Tehehehee!

4) Network Fun Times blog

This guy Chris (who is a Juniper Ambassador!!) made a LENGTHY blog post about what he used to study & pass the JNCIA-Cloud. I visited this blog post countless times b/c it is just THAT GOOD. Use it. Bookmark it.

And that’s it! Those are all the resources I used to pass the JNCIA-CLOUD. All FREE! I hope you found this thread useful.

• Nicole

Hi, hello. Nicole here.

Do you want to diversify your skillset & learn JunOS? Do you want to be able to add JNCIA-Junos to your resume? Here’s a thread of the free resources I used!

Link to the original twitter thread

1) Junos Genius

Junos Genius is an amazing, stupendous, fantastic resource. USE THIS!! Create an account, scroll down to Juniper Open Learning, & select JNCIA-Junos (or whichever cert track in which you’re interested). At the time of this post, once you finish all the videos & practice test(s), there will be a voucher test. If you pass the voucher test, you’ll receive a voucher for 75% off any Associate Level exam. Again, USE JUNOS GENIUS!! Watch all the vids, take all the practice tests.

UPDATE All the material from Junos Genius has been moved to Juniper Open Learning within the Learning Portal. The content is the same, just in a new location.

2) “Day One: Beginner’s Guide to Learning Junos”

It’s easy to read and is a very nice complement to Junos Genius. I HIGHLY recommend it. Here’s the pdf.

## 3) “Junos for IOS Engineers”

If you’re familiar with Cisco IOS, “Junos for IOS Engineers” is another option for you. (click the link & scroll down until you see the book). There are a bunch of Day One books, I’ve already read 2, so much information!!!

Side note: Juniper Networks has a collection of books written by industry professionals on a variety of topics. These books are called Day One books. They can be found here. And if you’re like me and you like physical books, you have the option to buy books from the virtual bookstore !!

4) Juniper vLabs

Don’t have any physical equipment?? No problem!! Use vLabs for practice!!! Definitely watch the video on the home page before you get started; it’s a good introduction to vlabs & how to use it. Super highly recommend vLabs.

5) Youtube

Here’s some random youtube videos that I really liked:

• “Using Juniper for the First Time | JunOS CLI”
• “Interface Naming Conventions”
• “Juniper Device Interfaces”
• “Juniper Networks JNCIA-Junos Certification Practice Test”

And that’s all! Those are the resources I used - all free. I knew nothing about Junos, and it took about 2 months to study for the exam using those resources and I passed on the first attempt. I hope you found this thread useful.

(Shoutout to Matt for letting me use his site to make this post)

The future is APIs! SD-EVERYTHING! Automation! Orchestration! Artificial Intelligence and Machine Learning! Sound familiar? It’s all part of the messaging going around in just about everything IT-related. With as much as you keep hearing about it, you might think that it’s all anyone is doing anymore. Yet it still just seems like not a whole lot of people are really getting into it in my area. Every vendor event I’ve gone to this year has asked attendees the same questions: “How many of you are leveraging the APIs in your network hardware/software?”. And every time the same answer - maybe two or three people in a room of 40 raise their hands.

So where is the problem? Is all of this just marketing fluff or am I just talking to the wrong people?

Let’s think about this from a typical network admin’s perspective. Shifting from traditional CLI to automation and APIs can seem difficult or overwhelming. Let’s say I want to automate a new VLAN deployment. Oh, you’re telling me I need to stop and learn vendor APIs… but before that I need to understand how to write scripts. But I’ve never even programmed something before. There are dozens of languages - how do I pick one? How much fundamental programming knowledge do I really need to have before starting? I don’t want to be a developer!

Okay, okay - just stop there for a second. No one is asking you to drop networking and write code for a living. The end goal of all this programmability stuff isn’t to turn networkers into developers - It’s to enable network/systems admins to be more efficient at their jobs. Why  copy/paste the same config change to 100+ devices, if you can mass-deploy the change via an API? That’s a lot of time savings that could be used toward educating yourself on new products, planning other projects, or thinking about your ideal network design.

I’ve heard a lot of the same things over the past few years:

“Programming is difficult” or “I don’t know where to start”

Try learning Python. It’s simple to get started and you can build from there.

“I don’t know what an API is or how to use it”

Don’t worry about that yet - start with learning the basics and APIs will make sense later.

“I’m not a developer”

No one is asking you to be one! But learning the basics of scripting and automation gives you a whole new toolset to solve problems.

For me personally - I would never want to be a developer. I can’t stand the thought of coming into work every day and just writing code. Some people might enjoy that, but for me it doesn’t sound like fun. However - I enjoy writing scripts to solve problems, especially when it ends up making my job easier. I think that’s the part where some people tend to get stuck though. A lot of automation sounds like I need to be able to develop a huge 10,000+ line application to pull data from 15 sources and aggregate it to make intelligent network changes. Ehhh… Nope, not really. But what about just a quick script that runs every 5 minutes to check an interface statistic, and email you when a particular threshold is exceeded? Realistically that could be done in less than 50-100 lines of a script and maybe 30 minutes worth of work.

Still not interested? That’s okay too. Traditional networking isn’t going away any time soon, and over time the vendors will write all of that automation for you. They will package it up in a pretty GUI and sell it off to companies that want it. In fact, this has already happening and has been for quite some time. This isn’t a bad thing - vendors need to make money, and not all companies will have the time or skilled resources to automate all the things. However, a network admin who can write their own scripts/automation won’t be exclusively tied to a vendor to help them - and instead they will be empowered to solve more problems themselves.

Where do you get started? I already wrote a bit earlier this year on a few resources for learning Python - which you can find here. I also wanted to point out some other great resources that are a bit more specific to using those skills for network automation:

• Python For Network Engineers - Don’t know anything about Python yet? Start here! This is a free course provided by Kirk Byers for anyone who is interested in using Python for network automation. Once a week you’ll get an email with all the great free content, but it will be up to you to spend time going through it. Go sign up, and set aside an hour or two each week to practice.

• Cisco DevNet - There is a ton of great content here. While DevNet does offer some tutorials on basic Python fundamentals,  the real value here is examples on how to use some network APIs (NX-OS, Meraki, UCS, etc). Also - one of the best parts about DevNet is the sandboxes they offer. Want to write scripts against the FirePower Management Center, but you don’t have one to test with? Well with DevNet you can get access to one!  Get familiar with your Python basics, then come here to see where you can start using those skills with your existing infrastructure.

• Network Programmability and Automation - This is a fantastic book. Not free, but it is well worth the ~$30. Once you have a good handle on how to write some basic network automation with Python, I highly recommend picking this up. While Python is covered here, the book does a great job of introducing you to all of the other toolsets available. Curious about how Linux or Ansible fit into network automation? You can find out here - and learn about APIs and source control systems too!

So - What are you waiting for? Go get started, and see what you can accomplish. Learn the basics - and keep an open mind for opportunities to use those skills.

Have suggestions on where else to learn? Comment below!

On the other side of things, I’ve worked with people over the years who haven’t had quite the same experiences as I have. Some of them don’t typically call vendors for support - or maybe they’ve just never been lucky enough to be the first responder on a Severity-1/Production-down case. This has occasionally resulted in a number of vendor support cases being closed with highly unsatisfactory results. In fact, there have been times where this has been bad enough that it gives management the impression that we received bad support from the vendor - but what if it was just our own inability to work effectively with that vendor? Maybe we didn’t push hard enough on an issue or stress the importance.

I feel that in a majority of cases, it shouldn’t be difficult to get the results you want out of a vendor support case. So I’ve put together a list of my own personal tips and guidelines for working with technical support.

The Vendor is your partner - Not the enemy

I’ve seen a lot of people call vendor support for help, then treat the support engineer as the bad guy. If you want to get a quick resolution, then you need to look at them as your partner. They’re here to help you figure out your issue and get everything fixed. When you open a new case, give them a concise summary of your issue with any relevant details you think are important. Don’t hide information, as this will just prolong finding a resolution - Give them everything they need. If you know your vendor always asks for the same diagnostics information every time you open a case, then have that information ready before you contact them.

Always remember how difficult the job of a technical support rep can be. They’re likely sitting in a call center, just waiting for the next case. They may have in-depth knowledge of their product, but they’re walking into your network completely blind. They won’t know your traffic flows, or that some systems are redirected through a proxy, or that one band-aid fix that Joe put in a year ago and never documented. Your support rep is going to do the best job they can to get a handle on what your network looks like, but be prepared to guide them. How would you like to troubleshoot a completely different and unknown network every time your phone rings?

Have realistic expectations

Especially when you’re new to a career in IT, it can be hard to gauge what you can and can’t ask of your tech support rep. One of my former jobs had a policy that for every ticket with AT&T we opened, we would be required to call back every hour for a status update. If there wasn’t one, then we were supposed to demand escalation to a manager. That policy might make sense if it’s a critical issue, but what about something that’s not? Have realistic expectations about what your vendor can do.

Troubleshooting takes time. If your support engineer grabs a bunch of logs and says they’ll need to get back to you - then you’ll need to give them the time they need. Feel free to ask for a time estimate - but if they say they’ll have something to you by the following day, don’t start bugging them every hour.

Remember that your support engineer’s job is to help you. If you don’t feel like that’s happening, you have a few options. You can ask for a case escalation or ask for the case to be reassigned. You never know the skillset of the person receiving your case, and you might get someone who isn’t super familiar with the problem you’ve raised. As long as there is no urgency, I will usually give the person time to work the issue - but be prepared to request a case transfer if it becomes apparent that they’re not getting anywhere. For example, I once had a case for a web-based firewall management system. The engineer I got was very good with the GUI side of things, but wasn’t very knowledgeable when troubleshooting took a turn toward the underlying linux system. A quick request to transfer the case to someone more experienced in this side of the system and we had the case solved within an hour. If an escalation or case transfer doesn’t help, you can also usually reach out to your local account representative and ask them to help push the issue for you.

It’s also very helpful to have an idea of your vendor’s support policies. Have a question about how to set up a new feature? Some vendors don’t permit you to open a case for a new configuration, and will refer you to their professional services team. On the other hand, some vendors are perfectly okay with helping you figure out how to set up something. Even better, some support teams are willing to stand by during migrations and upgrades, just in case you need their help. In my experience, if you’re not 100% confident in your changes, then it’s better to open a proactive case beforehand.

Be clear about the impact

If your entire datacenter is offline because of an issue, make sure that you immediately stress the importance. Again, your support engineer is jumping into your environment blind. Does this firewall performance issue impact twenty people, where it is just a minor inconvenience? Or is this issue prohibiting 50,000 customers from using the services you provide? The last thing you want is a misunderstanding of impact when it’s a high priority issue for your business.

Usually when I open a high severity case, I’ll let the engineer know: “Just so we’re on the same page, this issue impacts a large datacenter impacting 600+ customers. We need to get this back into a stable state as soon as possible”. High severity cases can be stressful for both sides, and I try to be clear about the impact without making that worse.

On the other side of things, if there is a low severity issue - don’t blow it out of proportions. I’ve worked with too many engineers who open up a Sev 1/Prod-Down case for every issue, even if the issue is just a minor inconvenience. Categorize your issues appropriately when you open them - and do your best to be realistic. A slow download for three users probably doesn’t warrant getting half a dozen TAC engineers on a conference bridge.

In case of emergency

Emergency situations are a completely different subject - so I want to spend a bit of time covering them separately. It’s really important to know what constitutes a true emergency in your environment. Is it when an office (or datacenter) goes offline? Or maybe even a single extremely critical business application? Things break - so have a plan and be prepared.

Step one - Always call into your vendors support line. You don’t want to open a web/email case and wait around for a technician. This might seem obvious, but I’ve known a lot of people who complain about the vendor’s response on a critical issue when they opened a case via a support portal. Find the vendor’s support number (or have it saved somewhere) and call them.

Step two - Ask for a warm handoff. In most cases, the person answering the support line is just creating a case and routing it to the appropriate team/ticket queue. They may just give you a ticket number and tell you to expect a call back shorty. If the issue is truly critical, ask them for a warm handoff to a technician. Most vendors I have worked with have had no problem doing this, and it helps you get to troubleshooting faster.

Step three - Clarify your issue and set expectations. You may be in a rush to get the issue fixed, but take a minute to explain your issue thoroughly and clearly. The more information you give to your support technician, the more easily they can dive into troubleshooting. And as I had mentioned earlier, be sure to set expectations and be extremely clear about the impact of your issue.

Step four - Keep troubleshooting on track. As I’ve stated before, you know your environment/network better than your vendor does. If they start looking at something you don’t believe is related, you need to guide them back to the main problem.

In addition, if you feel after a bit that the troubleshooting isn’t making progress - then request an escalation or a second set of eyes. There is no harm in asking for more eyes on the problem. I’ve even had situations before where the technician said “Well, I think we need to do X to fix it”, and I’ve just asked them to see what their peers think. You would rather be sure about a change, than make the issue worse.

Step five - See the issue through to resolution. Make sure you get your environment back to a stable state before ending the call. If the technician wants to drop off and call you back after reviewing logs, let them know you’re willing to just wait on hold. Once they’re off the phone, it’s easy for your technician to get dragged into another issue.

If the call ends with everything in a temporary state - then take follow ups on your next steps and make sure you accomplish them! Maybe you were able to restore connectivity, but need to wait for a maintenance window to make a change that is more service impacting than the original issue. Or maybe your support technician needs you to gather additional logs that they can forward to development. Whatever it ends up being, make sure you take note of it and follow through.

These are just some of my own personal tips that have worked for me. Support calls with vendors don’t always need to be a massive pain to deal with. Sure, sometimes you might have bad luck and get an inexperienced technician - but I find most issues with vendors can be solved easily enough once you know how much you can push them, and what you can and cannot ask for.

I hope these are useful - Let me know in the comments if you have any additional tips!

I have always said that I’m not sure I could write code for a living, but I do really enjoy writing scripts that make my life easier. Today’s post is a great example of that. The ease of use offered by the Juniper SRX firewalls and JunOS is something that I wish I had in all of my networking infrastructure. Even better, a brand new SRX 300 can be purchased on Amazon for less than $300 - which made a great addition to my home lab. Now I have a place to develop and test automation without breaking production 🙂

Anyways - I had a requirement to monitor my SRX clusters for route changes. Specifically I’m using this with devices where I have implemented customer peering via BGP. The SRXs don’t natively offer any form of monitoring and alerting for this, and the current monitoring applications at my disposal don’t either. So I decided to write something myself, which took significantly less time than I had assumed.

Code has been posted up to my GitHub - but I’m going to walk through some of it here. This script is intended to run as a cron job on a 5 or 10 minute interval.

# Imports
from jnpr.junos import Device
from jnpr.junos.op.routes import RouteTable

On my initial research to see how easy this would be to pull off, I found a nifty thing in the JunOS PyEZ package. Turns out they already offer a module literally called RouteTable that can pull the information I need.

Originally, I spent a bit of time trying to figure out how to use the standard device API to pull this info, but this module made everything 10x easier.

#################
# Required values
#################
deviceName = 'SRXFirewall'
deviceIP = '0.0.0.0'
apiuser = 'username'
apipassword = 'password'
SMTP = 'smtp-alias'
fromName = 'BGP Monitor'
fromAddr = 'bgpmonitor@domain.com'
toName = 'contactName'
toAddr = 'contact@domain.com'
prefixDict = {
        "FriendlyName": "Prefix",
        "Route-to-Inet": "0.0.0.0/0"
    }
################

The section above contains a list of the required variables for this script to function. A lot of them are going to be self-explanatory - but I wanted to take a moment to look at the prefixDict. This is a Python dictionary that maps a friendly name to a route prefix, which is used when we check our routing table. For example, if you wanted to monitor the SRX device for a route for 10.10.10.0/24 which belongs to Customer1, then we would just add and entry in this dictionary for “Customer1”: “10.10.10.0/24”

Alright - now let’s skip to the good stuff:

# Function to check received BGP routes
def checkBGP():
    try:
        # Open SRX session
        dev.open()
    except:
        sys.exit(0)

    # Pull device routing table, then keep only BGP originated routes
    allroutes = RouteTable(dev)
    bgp = allroutes.get(protocol="bgp").keys()

    # Close SRX session
    dev.close()

The section above is the beginning of the checkBGP function. Simple enough - just open an API session to the SRX and grab the entire routing table. That module I was talking about earlier makes it super easy! Next, we pull only the routes originating from BGP and assign them to a variable called bgp.

So in order to run this script repeatedly as a cron, I needed a place to persistently store the last retrieved routing info. For the time being, this is done by simply writing a temp file with the information:

    if not os.path.isfile(tempfile):
        with open(tempfile, 'w+b') as a:
            a.write(str(bgp))
        sys.exit(0)

    # Local file used to keep track of BGP learned routes
    with open(tempfile, 'ab') as a:
        lastroutes = a.readlines()
        # Compare if routes are different
        if str(bgp) == str(lastroutes[0]):
            sys.exit(0)
        if str(bgp) != str(lastroutes[0]):
            pass
    # Delete file, then re-create with new route list
    #os.remove(tempfile)
    with open(tempfile, 'w+b') as a:
        a.write(str(bgp))

The logic in the comparison is simple enough. If the current string of routes pulled from the SRX equals what is in the temp file (from the last run), then we assume no changes have occurred - and the script ends. Otherwise, if they don’t match then something has changed.

Once we know something has changed, we’ll go ahead and find out exactly which route entry is missing:

    # Create Status list, by checking received routes in bgp object
    status = []
    for name,prefix in prefixDict.items():
        if prefix in bgp:
            status.append("%s - RECEIVED" % name)
        if not prefix in bgp:
            status.append("%s - MISSING" % name)

This is where our prefixDict from earlier comes into play. We’ll look at every BGP prefix that we defined in that dictionary, and see if it exists in the current SRX routing table. All of this information (both routes received and missing) get added to an alert email.

Lastly, we’ll go ahead and compose our alert email to send out:

    # Send alert message
    sendMail(str(lastroutes[0]), str(bgp), status)

I’m not going to cover the email function here, since it’s pretty straightforward.

That’s it! The existence of the JunOS RouteTables module made creating this script a piece of cake. I’m considering adding onto this later with some additional functionality, so be sure to check my GitHub repo if you’re interested.

Hope this is useful to someone else out there - Let me know in the comments!

In a lot of jobs, the initial training you receive is fairly straightforward. You are usually taught how to respond to a task by following a series of steps to get an intended result. Training like this is great - It helps to achieve consistency and efficiency. You bring in any new person and give them the exact same troubleshooting steps, implementation steps, and/or validation steps - and you’re likely to get a similar result every time.

This is the point where I have seen far too many people stop though. They are happy with doing their job, and don’t necessarily want to progress their career or maybe they don’t know how. These engineers will continue to produce decent work at the quality that they were taught at. Even for those who try and progress further (maybe through certifications or otherwise), there is a difference between learning new technologies/concepts and truly understanding them. For some people out there, having this basic level of skill is all they really want - and if that’s their goal in life, then this type task-based knowledge is perfect. But if you really want to master the domain of technology that you are interested in, then you need to put forth the time and effort into gaining that understanding.

For me, a true understanding of a technology means that you’re able to speak confidently about how something works, abstract concepts to apply to similar products, and mentally walk though how the technology might handle a given situation.

Let me provide an example or two that might help to frame this a bit better. Given a particular network, an engineer might know that for traffic to get from point A to point B, it travels through two firewalls. Every time there is a new request to permit a new traffic flow, that engineer knows that they must make a configuration change to one or both of those firewalls to allow that traffic through. However, to this engineer, the inner workings of that firewall are a complete mystery. The firewalls are complete black boxes which take in traffic through one interface and spit it out another. So when there is a technical issue within the firewall appliance, they may be extremely limited in their troubleshooting abilities - and they may have no choice but to call the vendor for support.

Another engineer who has a deeper understanding of how firewalls work might see the problem differently. This engineer knows that for every packet received, the firewall follows a specific flow of processing. That flow could include any number of things, including NAT, routing, firewalling, IPS, VPN, etc. This engineer knows which order those things get processed and what effects those processes can have on the traffic. So when we have a technical issue with this firewall appliance, this engineer may be able to mentally walk though the packet flow/processing and determine where the problem may be - sometimes without even looking further than general log files.

Another example is something that I see quite often. An engineer is asked to implement something - let’s say a new port configuration for a server. They follow their known process for implementing this change, but something doesn’t quite work right. So they change settings or maybe delete the entire port configuration and start over - but eventually they get it working. They don’t know why it didn’t work the first time, or what caused it to work the second time - but it works now, so they aren’t concerned with it. However, it’s possible this engineer ends up running into this same problem more than once. The ideal step here would be to step back and look at what was different between the original configuration and the working configuration. Maybe there is an additional command in the original configuration which seems suspect - a quick search of the internet could turn up an explanation behind why that command was preventing the port from working as expected. After that research, that engineer would not only know why their configuration didn’t work - but now they know what that command actually does, which could be beneficial in a future scenario.

As I briefly mentioned earlier, not all IT admins or engineers are concerned with gaining a significant level of understanding. There are those who want to come to work, get their job done, then go home to their families - and there is absolutely nothing wrong with that. For me personally, I can’t handle running into a problem and not knowing exactly what the cause was. An issue that “fixes itself” is never an acceptable answer to me, because if something caused the problem once then it can certainly happen again. I don’t enjoy having to blindly configure an option on a system without knowing what’s going on in the background. Some people might call me crazy, but this seems to be a skill/trait shared by many higher-level engineers I have worked with. So how do you get to a point where you really understand a system? For me, it’s been a lot of playing in labs, reading vendor documentation, and not settling until I feel like I can speak confidently to how something works. I never feel truly comfortable in a new company until I can mentally walk through every device a packet touches from source to destination - and know which devices configs/routes may have an impact on that flow. Any time there is a problem with something, I spend time digging into it until I know what caused it - even if the problem is only momentary and goes away. Not only do I then understand why the problem happened, but I also learn how to quickly identify similar issues again.

Especially if you’re still in the beginning stages of your career, I can’t stress enough how important it is to understand the technologies you’re responsible for. Take the extra time and study it, play with it, break it and fix it again. Know how things work and what their behaviors are under different conditions. Don’t settle for “It just works because it does”. One of the key skills I’ve seen in engineers who truly understand their domain of technology, is the ability to abstract concepts to apply to other systems. Someone who has a deep understanding of routing and switching technologies might prefer to work with a certain vendor, but given any router/switch they can make it work.

Have you worked with anyone who you think has a great understanding of what they do? What other skills or traits do they display that makes them successful? Comment below!

Anyways - after I cut over to my new firewall, I’ve been digging through logs to make sure that I didn’t miss anything. I have all of my device/lab logs going into an instance of Splunk Light (their free product). It makes it easy to collect and search through logs, and it’s extremely easy to set up and use. A few quick queries and I came across one or two minor things that needed to be tweaked on my firewall - but I also saw some traffic that I wasn’t sure about.

So that brings me to my question of the day: Do you know what’s going out of your network?

A lot of people I know only use firewalls to block inbound access, both in homes and businesses. For homes it’s more understandable since most average people aren’t network admins. However, it still surprises me how many businesses are willing to add a ‘permit any any’ out to the internet. Yes, I block all traffic by default through my home firewall, both inbound and outbound. Yes, it’s a bit of a pain sometimes when something isn’t quite working right - but it’s usually a quick ACL change, and overall I would rather take the minor inconvenience for the security gains.

When I originally built the firewall policy for my network, I started off simple. I know we need DNS, HTTP, and HTTPS outbound - easy enough, right? Then I started watching logs for blocked traffic and trying to decipher what else was trying to communicate outbound using another port. Some things were very easy to determine - TCP 5228 out to a Google owned IP? Yep that’s actually a known thing - a lot of Google services, like Chrome, will use this. Some other things were harder to figure out - like game consoles which use a very wide range of non-standard ports. Many of these weren’t really documented well by the console manufacturer, and meant that I spent a while between browsing forums and some trial and error.

This really gets interesting when you start digging past the stuff you know about. What about a PC in my home network that is trying (and getting blocked) to reach a few random IPs in Korea and Russia over a bunch of non-standard TCP ports? Yeah that doesn’t make me feel comfortable. Could it be a legitimate application, or is it malware? A few quick searches on the internet don’t turn up anything immediately helpful. For the time being, I’ll keep stuff like this blocked until I have time to spin up some packet captures to see what this traffic is actually doing.

For a business I feel like this type of thing is even more important than just what I’m doing at home. You certainly don’t want end users (or servers) possibly running strange applications, which might be transferring data to some unknown external party. It seems like larger companies seem to have a better handle on restricting outbound access than most smaller companies, who likely don’t have the time or see the value. However, I’ve also worked with a few larger organizations who still permit all user and server traffic out to the internet with no filtering in place.

If you’re not already blocking outbound traffic - Get some good logging in place. Use something like Splunk Light and start collecting firewall logs for everything going out of your environment. Start with the basics - create a list of the software/ports you know you’ll need to open. After a few weeks, start digging through the logs to figure out what else might need to be added to your list. Once you feel comfortable that you’ve compiled a sufficient base ruleset, schedule a time to make the change and put it in place. Start blocking the unknown traffic - and only permit when necessary.

How do you have your firewalls configured today? Do you permit everything or are you very restrictive? Comment below - I’m curious to see what other people are doing.

In a few occasional cases, the CPU issue doesn’t resolve itself and someone needs to manually investigate the cause. Luckily, the httpd issue is pretty easy to spot and fix - so I wanted to cover that briefly today. This issue can crop up randomly after someone uses the JWeb GUI to administer an SRX firewall. You could avoid this issue entirely by disabling the web interface entirely - but that’s not always possible.

So the first thing we want to do is log into our SRX firewall and check the current CPU utilization for our RE processor:

{primary:node0}
root@test-srx> show chassis routing-engine node 0 
node0:
--------------------------------------------------------------------------
Routing Engine status:
    Temperature                  41 degrees C / 105 degrees F
    CPU temperature              70 degrees C / 158 degrees F
    Total memory               4096 MB Max 1556 MB used ( 38 percent)
      Control plane memory     2976 MB Max 804 MB used ( 27 percent)
      Data plane memory        1120 MB Max 773 MB used ( 69 percent)
    5 sec CPU utilization:
      User                       41 percent
      Background                  0 percent
      Kernel                     59 percent
      Interrupt                   0 percent
      Idle                        0 percent
    Model                           RE-SRX345
    Serial ID                       XX1000XX0002
    Start time                      2016-09-01 02:49:50 UTC
    Uptime                          351 days, 13 hours, 28 minutes, 47 seconds
    Last reboot reason              0x1:power cycle/failure
    Load averages:                  1 minute   5 minute   15 minute
                                        1.29       1.27        1.10

So we can see that over the past 5 seconds, there is 0% idle CPU - It’s all split between User and Kernel. Some higher-end SRX models will also show utilization for 1 minute, 5 minutes, and 15 minutes.

Next, we want to confirm which process is consuming that CPU:

{primary:node0}
root@test-srx> show system processes extensive node 0
node0:
--------------------------------------------------------------------------
last pid: 25330;  load averages:  1.16,  1.24,  1.10  up 351+13:29:51    16:19:11
165 processes: 21 running, 132 sleeping, 12 waiting

Mem: 354M Active, 191M Inact, 1253M Wired, 585M Cache, 112M Buf, 1595M Free
Swap:


  PID USERNAME     THR PRI NICE   SIZE    RES STATE  C   TIME   WCPU COMMAND
1635 root           7  76    0  1192M   113M RUN    0    ??? 281.93% flowd_octeon_hm
14607 nobody         3  76    0 14848K  6308K ucondt 0  25:03 83.45% httpd
   21 root           1 171   52     0K    16K RUN    0 6952.9  0.00% idle: cpu0
 1679 root           1  76    0 48580K 24476K select 0  90.2H  0.00% mib2d
 1715 root           1  76    0 35264K 19520K select 0  49.0H  0.00% snmpd
   23 root           1 -20 -139     0K    16K RUN    0  29.9H  0.00% swi7: clock
 1681 root           1   4    0   101M 68284K kqread 0  28.0H  0.00% rpd
   22 root           1 -40 -159     0K    16K WAIT   0  26.0H  0.00% swi2: netisr 0
  <-- Output Truncated -->

In this case it’s pretty clear that httpd is the top offender for CPU usage. You might also notice the process named ‘flowd_octeon_hm’. This is part of the firewall processes, so don’t be surprised if this process is also one of the top. It’s pretty normal for this process to show >100% CPU, so this is safe to ignore. If you see eventd as a top consumer, then you might have your logging configured to use event mode rather than stream mode - which I’ll cover in another post.

So how do we fix the httpd problem? Reboot the SRX? Well, yeah that would probably fix it - but there is an easier way:

{primary:node0}
root@test-srx> restart web-management
Web management gatekeeper process started, pid 25343

One quick command and we’ve restarted all of the web management processes, including httpd. So now you’ll want to give the SRX a few seconds to recover itself - then run the show system processes extensivecommand again:

{primary:node0}
root@test-srx> show chassis routing-engine node 0
node0:
--------------------------------------------------------------------------
Routing Engine status:
    Temperature                 41 degrees C / 105 degrees F
    CPU temperature             69 degrees C / 156 degrees F
    Total memory              4096 MB Max  1556 MB used ( 38 percent)
      Control plane memory    2976 MB Max   804 MB used ( 27 percent)
      Data plane memory       1120 MB Max   773 MB used ( 69 percent)
    5 sec CPU utilization:
      User                       6 percent
      Background                 0 percent
      Kernel                     3 percent
      Interrupt                  0 percent
      Idle                      91 percent
    Model                          RE-SRX345
    Serial ID                      XX1000XX0002
    Start time                     2016-09-01 02:49:50 UTC
    Uptime                         351 days, 13 hours, 32 minutes, 52 seconds
    Last reboot reason             0x1:power cycle/failure
    Load averages:                 1 minute   5 minute  15 minute
                                       0.35       0.99       1.04

Looks much better, with 91% idle CPU!

Even though this issue can be annoying, its a quick fix - I recommend that you perform some sort of CPU monitoring/alerting on your SRX clusters (I use Observium for this). This will help to identify the issue quickly and then get it resolved quickly. If this issue is left unchecked, it can sometimes cause some latency and performance issues.

Hope this helps!

Redundant Ethernet Interfaces

So first thing is first - Once you have a cluster configured, you’ll probably want to configure a few sets of redundant ethernet interfaces. These interfaces are also often referred to as reth interfaces. This will create a shared interface between your SRX pair, where you can configure IP address and VLAN information to be shared between the two. Let’s say that we have a Juniper SRX 1500 cluster, and we want to create a redundant interface for one of our 10Gb ports. Here is how we would do that:

root@testsrx# set interfaces xe-0/0/16 gigether-options redundant-parent reth1
root@testsrx# set interfaces xe-7/0/16 gigether-options redundant-parent reth1
root@testsrx# set interfaces reth1 redundant-ether-options redundancy-group 1

In the config above, we first take both of our interfaces (xe-0/0/16 on node0, and xe-7/0/16 on node1) and tell them that they now belong to a redundant interface group (reth1). Next, we enter into the reth1 config, and associate it to a redundancy group.

You’re also going to need to keep in mind that the SRX requires you to specify how many redundant ethernet interfaces will be configured. This is likely a memory thing, since each SRX also has a different maximum number of reth interfaces that can be configured. For example, if you tell the SRX that you need 5 reth interfaces, then the SRX will allocate system resources to manage those interfaces. In order to set the number of available reth interfaces, we’ll use the following command:

root@testsrx# set chassis cluster reth-count 5

Redundancy Groups

A redundancy group, or RG, is used as a container for logically grouping redundant interfaces/virtual routers which must fail over together. A single RG can be configured as primary on one of the two active SRX firewalls is a cluster - with the ability to fail over to the other node. For example, we might want be planning on only using one virtual routing instance on our SRX - so we would create RG1 and assign out interfaces to belong to it.

A quick note - all interfaces in a single virtual router must belong to the same RG. This way the virtual routing instance and all of it’s associated interfaces will always run on the same SRX node. In order to achieve an active/active firewall configuration, you would need to create two separate virtual routers, each with their own reth interfaces and different RGs. Then you would make RG1 primary on node0, and RG2 primary on node1.

In most configurations, dumping all of your reth interfaces into RG1 will be sufficient. You’re likely going to want to set up a priority for each RG - and maybe even preemptive fail-over. In order to do that - you’ll have to configure each cluster member with a priority:

root@testsrx# set chassis cluster redundancy-group 0 node 0 priority 200
root@testsrx# set chassis cluster redundancy-group 0 node 1 priority 50
root@testsrx# set chassis cluster redundancy-group 1 node 0 priority 200
root@testsrx# set chassis cluster redundancy-group 1 node 1 priority 50
root@testsrx# set chassis cluster redundancy-group 1 preempt

The higher priority wins here - so if you set node0 to a higher priority and preempt is enabled, then node0 will actively try to take ownership of RG1. I would rather not set preempt on RG0 for a few reasons - which we’ll cover in the next section. Priorities can also be modified using interface monitoring, so if a particular interface goes offline we can decrement the priority of that node (also covered below).

A Note About RG0

You might notice from the last post, that you’re output of show chassis cluster status already showed two redundancy groups: RG0 and RG1. RG0 is only used for management traffic and manages the routing engine for your SRX. Unfortunately, this can lead to some weird behaviors that you might not be expecting.

For example, whichever node is primary for RG0 is the only node that collects interface and monitoring statistics. If you’re using a monitoring tool that polls data from both of your SRXs, then the secondary for RG0 will report nothing about it’s interfaces, CPU, etc. This is also true if you log into the actual SRX itself - a show interfaces will actually return a bunch of default values, including showing that your ports are half-duplex. Don’t panic though, this is just an oddity of RG0. If you log back into the primary node for RG0, then it will show all of the proper statistics for both SRX firewalls.

Due to these weird things about RG0 - I prefer to always leave it on node0. Therefore I know which one to log into whenever I need to look at something, or which SRX to check in our monitoring tools. It’s also worth noting that whichever SRX is primary for RG0 is also the node you’re going to need to log into for configuration changes - even if all of your other redundancy groups are the other SRX.

Weird, right?

Oh, and be warned that since RG0 controls the routing engine, a failover of this RG can cause brief outages. This is primarily because the routing table and firewall state information will be lost. The secondary node has to spin up new processes for the routing engine, and at least currently there isn’t a graceful sync of all of that data.

Interface Monitoring

I mentioned setting device priorities a bit earlier. Setting interface weights is going to be the primary method for dynamically affecting those priorities, and therefore possibly causing a preemptive failover. One example might be that you’re using an SRX cluster for your edge firewall, and you want it to automatically fail over if the primary loses it’s internet uplink.

Note that you must configure the physical interfaces here, not the redundant ethernet interfaces:

root@testsrx# set chassis cluster redundancy-group 1 interface-monitor xe-0/0/16 weight 160

Remember when we set the priorities of our firewalls earlier? Node0 was set to 200, and node1 at 50. So here we are saying that xe-0/0/16 on node0 is worth 160 points. So if xe-0/0/16 goes down, then node0 will decrement it’s priority by 160 - which will be 40. This will trigger a preemtive failover by node1. The reverse is also true - when xe-0/0/16 comes back up, then node0’s priority will go back up to 200. Then node0 will take back ownership of RG1.

Manual Failover

There is a pretty good chance at some point you might need to perform a manual failover of your SRX redundancy groups. Maybe you need to do some maintenance or upgrades, or you just want to make sure failover works as you expect. In either case, the commands to do this are pretty straightforward:

root@testsrx> show chassis cluster status
Cluster ID: 5
Node       Priority       Status       Preempt       Manual failover
Redundancy group: 0 , Failover count: 0
node0      200            primary      no            no 
node1      50             secondary    no            no

Redundancy group: 1 , Failover count: 0
node0      200            primary      yes           no 
node1      50             secondary    yes           no

root@testsrx> request chassis cluster failover redundancy-group 1 node 1 

root@testsrx> show chassis cluster status 
Cluster ID: 5
Node       Priority       Status       Preempt       Manual failover
Redundancy group: 0 , Failover count: 0
node0     200             primary      no            no 
node1     50              secondary    no            no

Redundancy group: 1 , Failover count: 1
node0     200             secondary    yes           yes
node1     255             primary      yes           yes

Okay - so let’s talk about a few things that have happened here. I always recommend that you run a show chassis cluster status first, so you know where things already stand. Then we can proceed by requesting a failover. To do this, you have to specify which redundancy group you want to fail over, and which node you want to become the new primary. So in this case, we made node1 the new primary of RG1.

You might also notice that the priorities have changed, and the devices are marked as being in a manual failover state. This is important, because you cannot manually fail back until you reset this state. That’s right - if you tried to run the failover command again to move RG1 back to node0, it will not work. An automatic failover due to hardware failure or interface monitoring will still be permitted. In order to perform a manual fail-back to node0, we have to run the following reset command:

root@testsrx> request chassis cluster failover reset redundancy-group 1

Hopefully between last weeks post and this one, you should have a good handle on the basics of configuring a chassis cluster on your new pair of Juniper SRX firewalls. Let me know in the comments below if this helped you!

So let’s take a look at what we need to do!

Physical configuration

First thing we need to do is get both devices unboxed and cabled appropriately. In order to get a successful cluster configured, we will need to get two critical ports connected: the HA control port and the cluster fabric port. Technically only the HA control port is required to get a cluster working, but you’ll want to get the fabric port working as well - here is what both ports are used for:

HA Control Port - This is used for communication between the cluster members. This connection is used just for control-plane stuff - like keepalives/heartbeats and config sync between the two nodes.

Fabric Port - This port is used for data sync between the cluster members. All routing/firewall state information is synced using this port, and any cross-cluster traffic is also transferred using this port (for example, if one SRX was primary for a redundancy group, but the secondary was the active BGP speaker for your upstream connection - then the traffic would come in through the secondary and cross this link to reach the primary RG)

The fabric port is the easiest to connect - because you can use any port you like, then specify which port to use in the CLI. The control port, however, must be the assigned port that Juniper allocates for this use. Unfortunately, this port varies between device models. The two most common SRXs that I’ve deployed are the 345 and 1500. The SRX 1500 has a dedicated 10G HA control port, but the SRX 345 actually uses ge-0/0/1 on both nodes for this. Juniper lists what all those port assignments are over on this page.

Once those ports are connected, go ahead and power on both devices!

JunOS Config

Okay - Once the physical configuration has been completed, there are a few things that need to be configured on both devices before you can establish a cluster.

When you first boot each device, you’ll log in with root and no password. Then you’ll be dropped into the JunOS shell, and you’ll need to type cli to start the JunOS command-line interface. Then type configure to get into the configuration mode.

root@testsrx% cli
root@testsrx> configure
root@testsrx#

In the config mode, we’ll need to set a root password before we can enable the clustering. This password must match on both devices!

root@testsrx# set system root-authentication plain-text-password
New Password: <type your root password here>
Retype new password: <and again...>

For the SRX 1500 series, where there is a dedicated HA Control port, this is enough to get the cluster working. But for some of the branch SRXs, like the 300 series, you’ll need to make a few additional changes. These devices come with a default config, which includes IP addresses on certain interfaces. Unfortunately, this will conflict with your cluster config and will not allow your cluster to reach a healthy state.

In my config, I already plan on re-configuring all of the interfaces and security-zones to fit my needs - so I will just delete those entire config sections:

root@testsrx# delete interfaces
root@testsrx# delete security

After all that is done, we need to commit our changes:

```text
root@testsrx# commit

Finally we can go ahead and set up the cluster! This config is actually done outside of configure mode, so you will need to exit that.

So one thing to note here - each cluster will be configured with a cluster-id. This MUSTbe unique across any layer 2 subnet. So if we had multiple SRX clusters within a single broadcast domain, we would need to assign each one a different cluster ID.  I’ll use cluster-id 5 in this example.

On whichever SRX you want to be the primary node:

root@testsrx# exit
root@testsrx> set chassis cluster cluster-id 5 node 0 reboot

I personally like to give the primary a minute or to into the boot process before I configure the secondary, but we’ll do so with a similar command (just specifying node 1 instead of 0):

root@testsrx> set chassis cluster cluster-id 5 node 1 reboot

After both nodes come back online, log into node 0 and run the following command:

root@testsrx> show chassis cluster status 
Cluster ID: 5
Node       Priority      Status      Preempt      Manual failover

Redundancy group: 0 , Failover count: 0
node0      100           primary     no           no 
node1      100           secondary   no           no

Redundancy group: 1 , Failover count: 0
node0      100           primary     yes          no 
node1      100           secondary   yes          no

Perfect! Now let’s go configure our fabric ports! Interface fab0 will be configured as the fabric port on node0, and interface fab1 will be configured as the fabric port on node1.

root@testsrx> configure
root@testsrx# set interface fab0 fabric-options member-interfaces ge-0/0/10
root@testsrx# set interface fab1 fabric-options member-interfaces ge-5/0/10
root@testsrx# commit

Now that we’re in a cluster, all of this configuration can be done on node0 - but note that in this case the secondary device’s ports all start with ge-5/x/x. This is another oddity of JunOS - that numbering scheme isn’t always the case. In the SRX1500s, the node1 ports all start with ge-7/x/x - so this will vary depending on what devices you’re working with. If you ever need to check this - you can run show interface terse to list all interfaces in the cluster.

As a final verification that all our ports are up, drop out of config mode and run show chassis cluster interfaces:

root@testsrx> show chassis cluster interfaces
Control link 0 name: ge-0/0/1
Control link status: Up

Fabric interfaces:
Name Child-interface Status
fab0 ge-0/0/10 up
fab0
fab1 ge-5/0/10 up
fab1
Fabric link status: up

Hooray! We now have a functioning SRX cluster!

Sometimes if this doesn’t work, the output of show chassis cluster status will show the secondary node as disabled or lost. I’ve found that lost usually indicates a conflicting configuration on the cluster interfaces (like leaving the default IPs configured). If you see disabled, try rebooting the secondary node again - and if that doesn’t work, then you may need to disable clustering on both nodes and re-configure. This can be done using the set chassis cluster disable reboot command.

Next week, we’ll look at redundancy-groups, performing manual failovers, and setting up interface monitoring for automatic failovers. Hope this was helpful!

• Part 1 - Initial Thoughts
• Part 2 - Leaning Django
• Part 3 - Creating the Dashboard

Alright - We’re finally at the last step to this project. We already have Django running with a somewhat decent-looking web dashboard. Now the only thing that remains is polling our SRX firewalls for VPN connections, then populating this information into the database.

For this to happen, I built a custom management command for Django - the cool thing here is that we can easily use this within a cron job to automatically schedule updates at regular intervals. So within our application (the vpn folder), we’re going to create a management folder then a commands folder within that. In here you can name your script whatever you want, so I went with cronpoller.py.

The script that I wrote is extremely simplistic and could probably use quite a number of improvements - which I will slowly make over time. But for now, it is what it is - and it does exactly what I need it to.

First thing we need to do is import a few things we will need:

from django.core.management.base import BaseCommand, CommandError
from vpn.models import Firewall, Datacenter
from jnpr.junos import Device
from lxml import etree

We have some Django modules for treating this as a management command - and we’re also importing our existing models into this script. This part is really cool to me, because it means that this cron script can just reference the existing objects that we created to get their attributes. Next, we import the JunOS stuff from their pyEZ packages (don’t forget to install pyEZ!). The last one is going to be used to parse the responses from our SRX into something we can use.

Alright - so in order to build a management command, we have to create a class called Command, then define our functions within that. Within the Command class, Django will look for a function called handle to execute. In my final script, I have that function plus one called getVPNStatus. Let’s start with putting together getVPNStatus:

# This function will poll the device for status 
def getVPNStatus(self, fw, dc):
    connectedlist = []
    # So we generate a JunOS pyEZ device connection using the information about the 
    # firewall that we gather from the database object
    device = Device(fw.firewall_manageip,
                    user=fw.firewall_user,
                    password=fw.firewall_pass)
    # Try to open a connection out to the target device
    try:
        dev.open()
    except:
        # If for some reason this doesn't work, just return UNREACHABLE - which
        # we'll assume means the device is down
        return "UNREACHABLE"
 
    # Here is where we poll the SRX for a list of all IPSec Security Associations.
    # The equivalent of the 'show security ipsec sa' command
    response = etree.tostring(dev.rpc.get_security_associations_information())
    # The SRX returns a response in XML, which we'll need to dig through
    # Credit to the guys over at <a href="http://packetpushers.net/parsing-junos-xml-python/" target="_blank" rel="noopener">Packet Pushers</a> for a great post explaining how to
    # parse these responses
    with open(response) as a:
        xmldoc = etree.parse(a)
        docroot = xmldoc.getroot()
        rootchildren = docroot.iter()
        for child in rootchildren:
            # For each IPSec SA returned, we need to find the remote gateway IP, which 
            # we use to tie the connection back to the connected datacenter
            if child.tag == "sa-remote-gateway":
                connectedlist.append(child.text)
   
    # Once we've built our list, send it back!
    return connectedlist

That already is major part of our script. It will connect out to each device, grab a list of every connected IPSec tunnel, then return back a list of connected gateways. Now we just need to write our handle function, which will do all of the remaining work.

def handle(self, *args, **options):
    # Let's quickly create two lists - based on our firewall and datacenter models
    # This actually polls the database for anything that's been created
    dclist = Datacenter.objects.order_by('datacenter_code')
    fwlist = Firewall.objects.order_by('firewall_name')
 
    # This will loop through each firewall, then call the getVPNStatus function
    for fw in Firewall.objects.order_by('firewall_name'):
        statuslist = self.getVPNStatus(fw, dclist)


    # Once we have our response, we're going to check the returned list of connected
    # gateways against our list of datacenters from the database
    for dc in dclist:
        for remoteFW in fwlist:
            # If we find another datacenter in the connected gateway list, add 
            # that datacenter to the vpnstatus list as "VPNUP", otherwise assume it's down
            if remoteFW.firewall_vpnip in statuslist:
                vpnstatus[dc.datacenter_code] = "VPNUP"
                break
            else:
                vpnstatus[dc.datacenter_code] = "VPNDOWN"
 
    # After all that is done - we just save the new vpnstatus list to the firewall_vpnstatus
    # field in the database
    fw.firewall_vpnstatus = vpnstatus
    fw.save(update_fields=["firewall_vpnstatus"])

I’ve said this before, but I think it’s worth noting again - This probably isn’t the best or most efficient/reliable way of doing this. I think that over time I would like to revisit this and refine it a bit - but this is what I’ve put together so far. In fact, I think it’s worth stating that this will probably break in a fantastically horrific manner. This isn’t a finished product, but pretty much just version 0.1 - the base functionality works, but it’s in desperate need of refinement.

Alright - so now we just have to add a cron job in our system to call this script and run it. Depending on how many firewalls you have and how the latency is, you might need a different interval than I am using - but I went ahead and set mine to run once every 10 minutes. I would like to get this down to 5 minutes or less, but I might need to figure out a way to potentially multi-thread the cronpoller.py script. So here is what I did in /etc/cron.d/vpnstatuspoller:

*/10  * * * * * python /root/junos-dashboard/manage.py cronpoller

After letting the script poll my firewalls - I ended up with a dashboard that looks like this:

Future improvements

Of course, I still have a few things I would like to add or improve on. I’ve been keeping a list of some ideas that I’ve had, which I’ll share here:

• Add a timestamp to each firewall that contains the last poll time (in case something gets missed or hasn’t updated yet)
• Possibly add ability to click a ‘down’ VPN cell to force clear SA
• Set up email alerts from the cronpoller to automatically notify me of a failure
• Ability to click on any VPN cell and get additional info - like VPN uptime, subnets routed across it, or maybe the IKE/IPSec negotiation parameters
• Ability to click on any firewall name and get additional info - like uptime, JunOS version, CPU/Memory

I finally got around to getting myself a GitHub account, so I might put the final code up there once I’m done. I also have a number of other JunOS scripts that are probably worth posting up there as well.

Well, I hope you enjoyed reading about this project - because I certainly enjoyed working on it. This was one of my first real projects using the JunOS pyEZ libraries, and I got to pick up and learn how to use Django as well. The experience of building this has given me ideas for other JunOS automation projects - so look out for some of those in the future!

• Part 1 - Initial Thoughts
• Part 2 - Leaning Django

In the last post, we got Django working well enough to start serving up web requests. That’s a great start - So now it’s on to actually building the project. We’re going to cover this in two parts: the front-end dashboard, and the back-end scripts. In this post, I’m going to show you how I began putting everything together to assemble the front-end HTML dashboard.

As a quick refresher note, we created a project in the last post called junos-dashboard. Within that project, we created our app named vpn. This folder is primarily where we’ll be spending our time today. There are two files in our app folder that will need a few edits:

views.py - This is where we originally created our basic index page, which currently just spits out a message. So we’ll need to add code in here to render our dashboard instead.

models.py - In here we will define our objects and the characteristics about those objects that we want to store/retrieve in the sqlite database.

Let’s go ahead and open up our models.py file. For my dashboard, I have two objects I want to track - Firewalls (the SRX hosting the VPN) and Datacenter (as a location tracking). So in order to create those objects, we just have to add two classes into the models.py file:

from __future__ import unicode_literals
from django.db import models

class Datacenter(models.Model):
    def __str__(self):
        return self.datacenter_code

class Firewall(models.Model):
    def __str__(self):
        return self.firewall_name

Easy enough, right? So here is the really cool thing about Django (or probably any web framework really) - These model objects are now something that we can reference in our code to change or query attributes about them. Even cooler is that Django automatically builds an admin page to our site, where we can create new instances of each model right there (which I’ll cover later). So this means that all I have to do is define these two models and their attributes, then anyone on my team can log into the dashboard admin page and add new firewalls or datacenters.

Okay, so in order to actually make these models useful, we need to add our attributes. For the datacenter object, I only care about two things - What is the datacenter code (identifier), and is the datacenter an active location. So here is where we add those options under our datacenter class:

class Datacenter(models.Model):
    datacenter_code = models.CharField('Datacenter Code', max_length=10)
    datacenter_active = models.BooleanField('Datacenter Active?', default=True)

The firewall object gets a little more complicated because I want to be able to define quite a few things. The obvious attributes are: the firewall name, which datacenter it belongs to, and whether or not it is an active firewall. I’ll also need to collect this firewall’s VPN peer IP, so that I can compare it against other devices to check the connection. But wait - how am I actually checking that VPN status? Oh, I guess I’ll also need to collect a username/password for the SRX API, as well as a valid management IP to reach the API.

Let’s take a moment though - because I stopped here and realized that I’ll need to collect user/password data in a form, then it’s going to be stored in a sqlite database. Oh, and of course it will be unencrypted. I didn’t really like that idea - so I did some research on how to encrypt one of the object attributes. Turns out, it’s actually pretty easy since someone has already created a module to do exactly that.

So let’s grab that module real quick:

[root@0x2142-Centos vpn]# pip install django-encrypted-fields

Once we have the module, it requires us to generate some keys which will be used for encrypting our data. The following is based on the steps listed on the GitHub page for django-encrypted-fields:

[root@0x2142-Centos junos-dashboard]# mkdir fieldkeys
[root@0x2142-Centos junos-dashboard]# keyczart create --location=fieldkeys --purpose=crypt
[root@0x2142-Centos junos-dashboard]# keyczart addkey --location=fieldkeys --status=primary --size=256

Next, we just need to add this keyfile into our settings.py:

ENCRYPTED_FIELDS_KEYDIR = '/root/junos-dashboard/fieldkeys'

Alright - back to actually creating the attributes we need for our firewalls. Since we’re using this new module, we’ll need to add an additional import statement - otherwise, we are just adding the fields I covered earlier:

from encrypted_fields import EncryptedCharField

class Firewall(models.Model):
    firewall_name = models.CharField('Firewall Name', max_length=50)
    firewall_location = models.ForeignKey('Datacenter', on_delete=models.CASCADE)
    firewall_active = models.BooleanField('Firewall Active?', default=True)
    firewall_manageip = models.GenericIPAddressField('Management IP')
    firewall_vpnip = models.GenericIPAddressField('VPN Interface IP')
    firewall_user = models.CharField('API User', max_length=50, blank=True)
    firewall_pass = EncryptedCharField('API Pass', max_length=50, blank=True)
    firewall_vpnstatus = models.TextField(editable=False, blank=True)

A few things to note here - The firewall_location attribute will actually query the table of datacenter objects - so we’re not creating any data twice. You might also notice that I added a firewall_vpnstatus text field, which is going to be hidden in the administrative console. Because I’m a terrible programmer, I’ll be storing all of the VPN status info in this text field in the database. Yes, there is probably a much more elegant way of handling this - but again, I’m a network admin not a professional coder. For what I need, this gets the job done. If you have a better way of accomplishing this - let me know! I would be very interested in another method.

I also found out later that just because I encrypted the field doesn’t mean that Django knows the field is a password. I wanted the field to be treated like a password input, so the data wasn’t visible to anyone who just logged in. I found out that you can create a forms.py file within the vpn directory, and pretty much override the field type to account for this.

So here is what I threw into the forms.py file to handle password input:

from django.forms import ModelForm, PasswordInput
from .models import Firewall

class FirewallForm(ModelForm):
    firewall_pass = CharField(widget=forms.PasswordInput)
    class Meta:
        model = Firewall
        fields = ['firewall_pass']

That was actually way easier than I had anticipated.

Awesome, now our work on the models is completely done - why don’t we log into the administrative  web interface and take a look at how it all turned out?

Here is a view of the datacenter page - where we can add a new location:

And now for the firewall objects - with all the fields we need to get the job done:

Looks pretty good, huh? And we didn’t even need to do any work to get the free admin functionality! This is honestly my favorite part of Django. Without posting a ton of additional screenshots, the admin page also provides audit logs - so we can see who changed what objects. Very helpful.

Alright - now let’s start work on our views.py file, so we can actually start putting all this together in our dashboard!

Within our views.py file, we already had our index function, which just returned a string of text. We’ll be adding to that in a moment - but first I created a separate function called buildrows, which does exactly what you think. This function will pull each firewall and it’s status, and build our little HTML table. I’m not going to go into great detail on this - but check out the comments for more information on what’s going on here:

# This is our function to build the HTML table - We're going to be expecting this function
# to be fed a list of firewalls and datacenters
def buildrows(firewall_list, datacenter_list):
    firewallstatus = []
    # We will iterate through each firewall and compile each row for our table
    for fw in firewall_list:
        # The first cell in our row is going to be the firewall name and it's own VPN peer IP
        onerow = "<td><b>%s</b><i>%s</i></td>" % (fw.firewall_name, fw.firewall_vpnip)
        # Now we iterate through every datacenter, in order to see which ones we have a
        # connection to from this firewall
        for dc in datacenter_list:
            # First thing is first - If this firewall contains the name of the datacenter
            # then we'll print N/A, since we wouldn't expect a VPN to itself 
            if str(dc.datacenter_code) in str(fw.firewall_name):
                onerow += ("<td>N/A</td>")
                continue
            # Also, if there is no VPN status in the database, then just print 'No Status'
            if not fw.firewall_vpnstatus:
                onerow += ("<td>No Status</td>")
                continue
     
            # This portion is pretty self-explanatory - We parse the vpnstatus field in the
            # database. If the state of the connection is 'VPNUP', then we print a cell for
            # that datacenter with the text 'UP'. If the state is 'VPNDOWN', then we print
            # 'DOWN'. And if nothing matches, print 'No Status'
            statuslist = ast.literal_eval(fw.firewall_vpnstatus)
            try:
                status = statuslist[dc.datacenter_code]
                if status == "VPNUP":
                    onerow += ("<td class=\"vpnup\">")
                    onerow += ("UP")
                elif status == "VPNDOWN":
                    onerow += ("<td class=\"vpndown\">")
                    onerow += ("DOWN")
                else:
                    onerow += ("<td>")
                    onerow += ("No Status")
                    onerow += ("</td>")
            except KeyError:
                onerow += ("<td>")
                onerow += ("No Status")
                onerow += ("</td>")
        # Once complete, we append this row to the status list
        firewallstatus.append(onerow)
    
    # All done! Return our list of statuses!
    return firewallstatus

That block might not make a ton of sense at the moment - but it will shortly. Essentially I have a script that is connecting to each firewall via the SRX API and polling the VPN status for each peer. The script is then writing all of this information to a field in the database called vpnstatus in one long string, which kinda looks like this:

{u'US-1': 'VPNUP', u'US-2': 'VPNUP', u'EU-1': 'VPNUP', u'EU-2': 'VPNUP', u'CN-1': 'VPNDOWN'}

The buildrows function is then just grabbing each firewall and then parsing this data to figure out what it has connections to - then generates a row in our table.

With that done, our index function within views.py needs a little work to start displaying our table. You’ll also notice we’re importing the models we created, as well as a template loader - which I’ll get to in a moment. So here is what that function will look like when we’re done:

from .models import Firewall, Datacenter
from django.template import loader
import sys, ast

def index(request):
    firewall_list = []
    datacenter_list = []
    firewall_status = []
    # This will grab each datacenter object (if they're marked active) from the 
    # database and add them to a list
    for each in Datacenter.objects.order_by('datacenter_code'):
        if each.datacenter_active == True: datacenter_list.append(each)
    # Same thing here - grab our firewalls and append to a list
    for each in Firewall.objects.order_by('firewall_name'):
        if each.firewall_active == True: firewall_list.append(each)
    # Now we pass those lists to the buildrows function to assemble our HTML table
    firewall_status = buildrows(firewall_list, datacenter_list)
    # We're also going to apply a template to make the page look fancy
    template = loader.get_template('web/index.html')
    # This is taking all of our lists, and passing them into our HTML template
    # so that it can build the page semi-automatically
    context = {
        'datacenter_list':datacenter_list,
        'firewall_list': firewall_list,
        'firewall_status':firewall_status,
    }
    return HttpResponse(template.render(context,request))

Alright - our views.py file is now complete. I wanted to keep the index function kind of simple, so we just have it pull data from the database, pass it to another function for processing, then return it to the browser for display.

So in the code above, you might have noticed that I was using an HTML template to make the page look fancier. I did this by going out to Google and finding a free CSS template for HTML tables. I won’t post the CSS here, but there are plenty of free templates out there - just find whatever suits your taste. Once you have that CSS file, create a directory (within our app folder) called static and place the CSS file in there.

Then, it’s one simple template HTML file to load our CSS and render our table. Within the app directory, I created a templates folder, then a folder called web within that. I added a new file called index.html - which looks like this:

<title>SRX VPN Dashboard</title>
<div align="center">
    {% load static %}
    <!-- Load the CSS template here -->
    <link rel="stylesheet" type="text/css" href="{% static 'style.css' %}" />
    <!-- Add a header to the table -->
    <div class="table-title">
        <h3>SRX VPN Dashboard</h3>
    </div>

    <!-- Create our first row, which will be a list of each data center location -->
    {% if firewall_list %}
        <table class="container">
        <tr>
        <th></th>
        {% for datacenter in datacenter_list %}
            <th>{{ datacenter.datacenter_code }}</th>
        {% endfor %}
        </tr>

        <!-- Then we add a row for each firewall, followed by its status 
             for each datacenter -->
        {% for firewall in firewall_status %}
            <tr>
            {{firewall|safe}}
            </tr>
        {%endfor%}

        </table>
        <!-- If something goes wrong, we just print an error -->
    {% else %}
        No firewalls were found.
    {% endif %}
</div>

Guess what? The dashboard is complete! Now we can run our Django server using ‘./manage.py runserver’ and take a look at what we have created.

Pretty simplistic - but we’re getting very close to what I wanted to accomplish. The dashboard loads and generates the table, but it doesn’t have any data yet.

In the next post - We’ll take a look at building the backend script to poll all of our SRX firewalls for VPN connections. I hope you enjoyed reading this - let me know what you think in the comments below!

This is a multi-part series - Check out the other posts:

• Part 1 - Initial Thoughts
• Part 2 - Leaning Django
• Part 4 - Polling the SRX

• Part 1 - Initial Thoughts

Once I figured out the overall idea behind what I wanted to do - the next big part was figuring out where to start. As I mentioned in part 1, I had never used Django before starting this project. So I figured that learning how to to use a new web framework might be the best place to begin.

The great thing, is that Django already has some great tutorials on how to get the packages installed and begin working on your first project. Their tutorials can be found here:

• Installing Django
• Writing your first app - Credit to this page for some parts of the examples below

I used both of these tutorials when I wrote my dashboard, and pretty much just modified what they were doing to fit my needs. After I got a baseline down, it was pretty easy to keep going and complete what I wanted to do.

Note: Just as a pure warning - I’m a network admin, not a professional programmer. Python, Django, and the Junos PyEZ libraries make it easy enough for me to make my ideas into a reality. However, that doesn’t mean I write perfect or even great code. (I learned Python from Learn Python The Hard Way - I highly recommend this site, as it was a great resource!)

Alright, with that out of the way - Let’s get started!

Step One - Installing Django

First thing is first, I learned Python on the 2.7.x chain, and I’ve been terrible about forcing myself to switch to 3.x. So for the purposes of this tutorial, everything I’ve written was intended for Python 2.7.5. Installing the actual Django package is a super simple task (if you use pip):

[root@0x2142-Centos ~]# pip install django
Collecting django
 Downloading Django-1.11.2-py2.py3-none-any.whl (6.9MB)
 100% |████████████████████████████████| 7.0MB 140kB/s
Collecting pytz (from django)
 Downloading pytz-2017.2-py2.py3-none-any.whl (484kB)
 100% |████████████████████████████████| 491kB 1.9MB/s
Installing collected packages: pytz, django
Successfully installed django-1.11.2 pytz-2017.2

Step Two - Get JunOS pyEZ libraries

I already wrote about this earlier in Getting Started with JunOS PyEZ. If you don’t already have the JunOS packages installed, check out that page for the instructions.

Step Three - Create the Project

Django is pretty wonderful from my brief experiences with it. They package a number of tools that make starting a new project very simple.

So to start off with our project, we’re going to use the django-admin tool:

[root@0x2142-Centos ~]# django-admin startproject junos-dashboard

This will create a base directory structure for us to begin our project. We’ll dive into what these files and folders are as we touch them - but for now, you’ll need to make one minor edit to ~/junos-dashboard/junos-dashboard/settings.py. Open that file up in your favorite text editor, and find the ALLOWED_HOSTS setting. Add in the IP address of your linux VM, like this:

ALLOWED_HOSTS = ['10.2.32.90']

Once that’s done, go to the root of your project (~/junos-dashboard/) and test out the web server:

[root@0x2142-Centos junos-dashboard]# ./manage.py runserver 10.2.32.90:8000

Django includes it’s own web server for development and testing, which is extremely helpful. Once you run that command, try to hit your page in a browser and just make sure it loads.

Once that’s done, make sure you’re in the root of your project - and we will create our dashboard application:

[root@0x2142-Centos junos-dashboard]# django-admin startapp vpn

This command just creates another directory structure within your project. I think this is helpful though, because then I don’t have to worry about which files I need to create or how they should be laid out - Django just handles that for us.

Step Four - Getting our web server to return our app page

Okay - So next we need to make a couple of minor edits in a few files. Within our app directory, there are two files (views.py and urls.py) that handle the main parts of our web interface. Note - urls.py doesn’t exist by default and you will have to create it!

Within views.py, we’re going to create our dashboard homepage with a quick test response:

# Import stuff
from django.shortcuts import render
from django.http import HttpResponse
from django.template import loader
import sys, ast

# This is our index page
def index(request):
    # Return some plain text - just so we know it worked!
    return HttpResponse("This page will eventually be a magical dashboard!")

This file is where we will eventually be putting in all the code for our dashboard page. When we hit our webserver, this file is executed to generate the view that we’ll see.

Next, we need to create a urls.py file within the same folder as views.py. This file is just a way to direct traffic. If we receive traffic to a regex of ^$, which is no path, then we are going to redirect it to the index function in views.py:

# Import stuff
from django.conf.urls import url
from . import views

#Redirect to index function in views.py
urlpatterns = [
 url(r'^$', views.index, name='index'),
]

Finally, we’re going to grab the urls.py file in our project folder - not the one we just edited within the app. So this should be ~/junos-dashboard/junos-dashboard/urls.py. In this file, we just need to tell the web server to include the urls.py file we created when evaluating traffic. This file should already exist and have content in it for routing traffic to the admin page, so just add an entry under urlpatterns - should look like this:

urlpatterns = [
 url(r'^$', include('vpn.urls')),
 url(r'^admin/', admin.site.urls),
]

I just want to point out that in this case, we’re catching traffic again with the ^$ regex. This is so that any traffic to the main page (in this case http://10.2.32.90:8000) is automatically sent to our vpn app. However, if you had multiple apps running, you might want to give each one it’s own directory.

Go ahead and execute the runserver tool again, and validate that you’re able to successfully hit the page and retrieve the test text we put in.

Okay, I think we’re going to stop here for this post. At this point we have Django installed and our project/app created. We also have the beginnings of our dashboard page. In the next post, we’ll actually start building out the dashboard interface!

This is a multi-part series - Check out the other posts:

• Part 1 - Initial Thoughts
• Part 3 - Creating the Dashboard
• Part 4 - Polling the SRX

One thing my current job has never had is a good way to view VPN tunnel status between our firewalls. Our Check Point firewalls don’t really provide a good high-level view, which has unfortunately caused some confusion around whether or not a site-to-site VPN tunnel is currently established. Luckily, over the past year I’ve had the opportunity to install over 20 new Juniper SRX firewalls - mostly made up of SRX 1500 and SRX 345 models. I’ve written a bit before about the JunOS APIs and pyEZ (their Python library), but I’m always really excited at a new use case for network automation.

So recently I decided that it would be great to build a web-based dashboard, which would query all of our SRX firewalls for currently connected VPN tunnels. Approximately 90% of the current tunnels are site-to-site between our own data center locations, with the other 10% being external to customers. My idea was that I would have a simple HTML table, which would show each data center along the top and bottom and whether or not each was connected to each other, kinda like this:

I wasn’t really concerned about making it look fancy - just dynamically updatable by whatever backend mechanism I used. Speaking of which, I had also assumed I would just be writing some simple Python script to pull VPN info from each device, then just write it to an HTML file. I haven’t really done much web stuff with Python in the past, so I set out to learn a bit and figure out what the best approach might be.

I ended up settling on trying out Django to write the frontend web stuff. I had never used it before, but I’ve been interested in trying - and what’s a better time to learn, than when you have something that needs to be accomplished? One thing that really pushed me towards Django for this project was the built-in administration page. This was huge for me, because it meant that I could easily have a way for other people to update the web dashboard. Whenever a new location came online, I wouldn’t have to go update the script directly - anyone on my team could log into the admin page and make the changes.

In this post I just wanted to get through what my ideas were behind this project. In the next few weeks, I’ll begin explaining how I built the dashboard using Django and use pyEZ to scrape VPN status from each firewall.

Thoughts? Drop a comment below!

This is a multi-part series - Check out the other posts:

• Part 2 - Leaning Django
• Part 3 - Creating the Dashboard
• Part 4 - Polling the SRX

The IP addressing design that I came up with limited our typical data center deployment from 4-6 /16 blocks to a single /16 block for each location. For all new locations since then, this new design has been used and it has proved to be extremely beneficial. The ability to use proper address summarization has made firewall rules, routing, and VPN tunnel configuration much simpler. But what about all the old locations which still had several /16 blocks? None of these needed more than a single /16, but we have thousands of systems that would need to be re-addressed. Not something that was going to happen overnight. So let’s take a look at some of the methods we employed for migrating from one IP addressing scheme to another.

• Have a plan - The first step is to have a good handle on the overall situation and how to get from point A to point B. You’re likely going to need buy-in from other teams to help get there, and this could easily be a multi-year project depending on the number of systems. When you meet to discuss the re-addressing project, you need to be pretty strong when describing the benefits of the new system - otherwise no one will want to help.

• Enforce the standard for anything new - The easy target for any transition is to hit new stuff first. For example, we started using the new range in a brand new location first. Anything being deployed that requires a new IP address allocation needs to be using the new scheme. We don’t want to perpetuate the scheme we are trying to get rid of.

• Transition (Network Config) - This can be a difficult step that requires a bit of planning. For any existing sites, we need to configure the ability to use both IP address schemes side-by-side until the transition is completed. There are two primary ways to accomplish this that I’ve used - either build out a new segmented (VLANed) network, or overlay the existing using secondary IP addresses. Don’t forget to propagate routes to the new subnets and ensure that firewall rules match the existing functionality.

• Transition (Infrastructure/Servers) - Once the underlying networking pieces are done, the next step is to begin transitioning services. Again, make sure any new systems getting deployed are now using the new ranges. Then we can take either an active or passive approach. In the passive approach, we are going to essentially just build new systems in the new scheme and wait until the older systems are eventually removed from service. This probably isn’t the ideal way to do this - but it’s certainly an option. In a more active approach, we would start identifying the older systems to move and making plans to do so (likely in a phased manner). Either method is going to require a serious investment of time, depending on the size of your network.

• Long-Term - This process is never going to be quick or easy, but the end result should be a much better state than we began. In the meantime, maintaining both IP addressing schemes can be quite painful. Make sure that everyone on the team understands the goals of the new scheme, the plan for getting there, and how everything is configured to make it happen. The last thing you want is for someone to try and back out of the move, just because they’re not confident in what’s going on.

I also wanted to stress the importance of research throughout this whole process. It’s important to try and understand why the original IP addressing was designed the way it was, and what goals they had in mind at the time. It’s also important to check the technologies you’re using to understand how everything will work. For example, Juniper’s SSG (ScreenOS) platform doesn’t support utilizing a secondary IP address on an ‘untrust’ interface (KB5527) - but it works if you use a custom zone name. And Check Point doesn’t support secondary IP addresses at all when you are using their ClusterXL protocol (SK89980), instead they actually recommend that you deploy a new VLAN and tagged sub-interface. However, they do support it if you are using VRRP instead.

This is in no way a definitive guide on the various ways you might accomplish this - but I wanted to give a bit of background on how we tackled the problem. Unfortunately in my case, most of the older locations have several thousand systems - so I’ll be working on this migration for quite a while.

Ever had to migrate to a new IP addressing scheme? What methods did you use? How large was the network? Run into any big problems? Comment below!

Securing your BGP peering (Know who you’re connecting to)

BGP is a little different from most other routing protocols, since it uses a single unicast TCP connection between peers to exchange routing updates. Lucky for us, that means that we can easily filter traffic from only known peers. Once you have direct connectivity up between your edge router/firewall and your direct peer, lock down that connection with an ACL. Permit TCP port 179 traffic ONLY from your directly connected peer IP - no one else.

While you’re at it, let’s take it another step further: Request that your ISP set up BGP authentication. Sure, a majority of BGP implementations today still require use of MD5 for auth (which is terrible) - but some authentication is still better than none. This can usually be arranged at the time of turning up peering. Both sides configure the same authentication password and with any luck the peering still establishes.

BGP by nature is unfortunately not the most secure protocol - but a few simple steps like this will help ensure you’re only connecting out to authorized peers.

Route filtering (Don’t trust anyone)

Usually when you’re filling out the BGP peering paperwork for your service provider, they will ask you what kinds of routes you want. In most cases, you should be able to request one of the following:

Default only - Exactly what it sounds like. Your provider will only advertise a route for 0.0.0.0/0. In many cases, this is probably what you’re going to want. With this type of advertisement, each upstream provider will just give us the same default route to the internet. From there we can weight which one we want to use, and traffic will automatically fail-over to the secondary connection should the primary fail.

Partial - If for any reason you want to weight routes to certain destinations differently, then we might request this. In this case, you’re probably going to still receive 0.0.0.0/0 plus any specific routes you ask for. A good example of this is if we wanted to specifically manipulate routes for a remote office we have. Maybe we want to weight Internet traffic for one uplink, and VPN traffic to a remote office on the other uplink.

Full - In 99% of typical business cases, this won’t be required. This option means the upstream providers will be dumping the entire Internet routing table on you. While this offers you a ton of control over path manipulation, it also requires significant memory resources on your routers in order to maintain that routing table.

After we figure this out, the next step is to make sure we are filtering the routes we accept from the upstream provider. Wait - didn’t we just tell them exactly what routes to send us? Why do we need to filter them? Well you can never be too safe here - and we would rather perform an unnecessary filtering than have an ISP accidentally misconfigure route advertisements. So if you’re only expecting a route for 0.0.0.0/0, then filter your inbound route advertisements so you only accept that route.

Same thing goes for outbound route advertisement  - if we own a /24 of public IP space, then we only want that range to be advertised out. Some providers may already filter this on their end, but again it doesn’t hurt here to be extra cautious. If we are accepting anything other than a default route from our provider, then we run the risk of leaking those additional routes between the two providers - which would lead to inadvertently becoming a transit AS. Chances are pretty good that you don’t want that, so make sure you configure filtering for all outbound route advertisements.

Minimum Advertisement (Oh no, we have to re-address everything)

I mentioned this in the original post - but typically when you are peering with two separate upstream providers, you need to advertise no less than a /24. We ran into this at my last job, where we had been provided a /25 by AT&T but we needed to bring in a second carrier via BGP. The reasoning behind this is to keep global routing tables as small as possible, by not allowing them to end up flooded with a ton of routes for smaller subnets. It makes sense, but on the other hand I feel like requiring a /24 in all cases can be a bit wasteful. My last job only required maybe 30 publicly addressable hosts - which meant that the remaining addresses went unused.

At any rate - should you find yourself in this scenario then you’re going to have to face the inevitable: Renumbering into a new IP space. Any time you have to do this, it’s going to be a bit of a pain - but for external addressing like this it might be easier. So in our case, the entire /25 space was hosted on our external firewall then NAT’ed into DMZ servers.

Here is the quick steps that I used to do a side-by-side migration without taking any significant downtime:

• Get the new subnet up and running - assign the interface addresses on your firewall and BGP up and running
• Assign new IP addresses to all of your existing services
• Configure NAT rules for the new external IP addresses to the DMZ hosts - while leaving the existing NAT rules for the old subnet (Also make sure your firewall rules permit the same traffic to either IP)
• Migrate DNS entries externally to point to the new IP space
• Once traffic stops flowing to the old IP, remove the old NAT

As a side note - if you procure redundant internet connections through the same upstream provider, then you might be able to work out something else. They may be able to provide you a private ASN to use, and they will likely accept any minimum advertisement - since they will be summarizing upstream within their network anyways.

I had a few more things I originally intended to cover here - but it seems that these topics are filling way more space than I thought they would. Specifically, I’m thinking about a dedicated post to BGP path manipulation - which is probably something you’re going to want to implement after peering is established. Hopefully these tips help! If you have any questions, throw them in the comments below.

We run multiple locations around the world, and unfortunately have to keep full mesh VPN connectivity due to the way our systems have been deployed. Today each SRX cluster has around 15 different VPN peers, which are made up of other SRXs, older SSGs, CheckPoint firewalls, Cisco ASAs, and Watchguard firewalls. This is still an on-going process - but I wanted to throw out some of the issues I’ve run into so far, and what I’ve been able to do to fix them or work around them..

Issue #1 - VPN is up, but no traffic is flowing across it

This one initially took me a minute to figure out. All of our tunnels are route-based, using secure tunnel interfaces. So each VPN is configured with a set security ipsec vpn vpn_name bind-interface st0.x command. I had a set of VPN tunnels between two locations that were not passing traffic, even though a show security ipsec sa showed the tunnels as established. For reference, here is what the config looked like:

root@SRX-SITE-A> show configuration security ike
respond-bad-spi 1;
proposal ike-aes256 {
 authentication-method pre-shared-keys;
 dh-group group2;
 authentication-algorithm sha-256;
 encryption-algorithm aes-256-cbc;
 lifetime-seconds 28800;
}
policy ikepolAES256 {
 mode main;
 proposals ike-aes256;
 pre-shared-key ascii-text xxxxxxxxx; ## SECRET-DATA
}
gateway gateway-siteB {
 ike-policy ikepolAES256;
 address XXX.XXX.XXX.XXX;
 no-nat-traversal;
 external-interface reth0.0;
}

root@SRX-SITE-A> show configuration security ipsec
proposal ipsec-aes256 {
 protocol esp;
 authentication-algorithm hmac-sha1-96;
 encryption-algorithm aes-256-cbc;
 lifetime-seconds 28800;
}
policy ipsecpolAES256 {
 perfect-forward-secrecy {
 keys group2;
 }
 proposals ipsec-aes256;
}
vpn vpn-to-SITE-B {
 bind-interface st0.1;
 df-bit clear;
 ike {
 gateway gateway-siteB;
 ipsec-policy ipsecpolAES256;
 }
 establish-tunnels immediately;
}

root@SRX-SITE-A> show configuration interfaces st0
unit 1 {
 description vpn-to-SITE-B;
}

The config on both sides practically matched, but there was one thing missing that was preventing the tunnel from passing traffic. Under the st0 configuration, unit 1 (or whichever tunnel interface you might be using) needs to have family inet configured. Even though I’m using an unnumbered tunnel interface, this command still needs to exist to tell the SRX that the interface is used for IPv4 traffic. Quick fix, but it’s easy to miss.

Issue #2 - VPN drops every 2-4 hours and doesn’t re-establish for another 2-4 hours (or manual SA clearing)

The original SRXs that I installed were running JunOS 15.1X49-D40.6. I had at least half a dozen of these devices interconnected with full mesh VPNs, and experienced no issues. However, when I picked up a new set of SRX 1500s a few months back, Juniper had just released 15.1X49-D70.3 - so I upgraded before these were put into production. Strangely enough, when I began migrating tunnels to the new cluster we started to see the VPNs to remote SRXs drop sporadically. The first remote sites to migrate were less of a priority to keep connectivity established, so I took this opportunity to spend a little time figuring out what was going on.

The initial issue seemed to be that the VPNs would establish, but only for about 2-4 hours. Then they would drop and not re-establish for 2-4 hours. This seemed a bit weird to me, because the re-key interval was set for 8 hours - which means that re-key wasn’t playing into this. Even more weird, whenever the issue occurred - one of the two SRX clusters would always still show the IPSec tunnel as up, while the peer SRX would just keep logging errors about bad SPIs. Clear the stale IPSec security association, and the tunnels re-establish immediately.

In order to resolve this, I had to configure both Dead-Peer-Detection and Juniper’s VPN monitoring on both sides of the connection - so that each SRX would more actively monitor the tunnel status. Juniper’s documentation states that they enable DPD by default, but in an ‘optimized’ method which only sends a DPD R-U-THERE message under certain conditions. I had to change this to force the SRX to send the DPD messages at regular intervals. Here are the changes I made to fix the issues:

root@SRX-SITE-A# set security ike gateway gateway-SITE-B dead-peer-detection always-send
root@SRX-SITE-A# set security ipsec vpn vpn-SITE-B vpn-monitor optimized

After these changes were in place, I stopped experiencing the issue. Again, these had to be implemented on BOTH sides of the connection. These weren’t necessary on the tunnels in-between the SRX clusters on the older firmware version - so there may be some sort of bug between those and the newer firmware.

Issue #3 - VPN between SRX and CheckPoint duplicates IPSec SA on re-key (sometimes causes tunnel to stop passing traffic)

This issue was a complete mess - mostly because of the effort involved in trying to coordinate two separate vendors to work on an issue. New SRX clusters (on 15.1X49D40.6 at the time) had been deployed and all of them had to connect back into our existing CheckPoint locations via IPsec tunnels. All was great, until about two weeks after installation we started seeing some weird tunnel drops. After some troubleshooting on my end, I discovered that watching what happened during the regularly scheduled re-key interval was helpful to see what was going on. Right at the eight hour re-key, the tunnels would try to re-establish but couldn’t - and sometimes this led to uni-directional traffic flows across the VPN.

The SRX tries to start a soft reset process prior to the re-key interval, so that it can gracefully migrate traffic to the new SPIs. However, something was happening that was causing the SRX to never terminate the old SPIs - so after a while the SRX would try to begin the soft reset process and fail because it had already reached its maximum SPIs for a given peer. Once the re-key interval was reached, the SRX would initiate the hard reset process on the tunnel. The CheckPoint side typically wouldn’t notice that anything was going on, and would keep sending traffic down the bad (expired) SPIs. A quick clear security ipsec sa and clear security ike sa would bring the tunnels back up.

I worked with some great guys on the Advanced JTAC team - but ultimately the SRX configuration and behavior seemed to be exactly what was expected. The only thing we couldn’t figure out is why the SRX was holding onto the old IPSec SPIs. So we opened a support case with CheckPoint to see what they had to say. After a few troubleshooting sessions and running a bunch of debugs, the CheckPoint engineer seemed to believe that the issue was on their side. All of our CheckPoint clusters were running R77.10 at the time, but we also tried upgrading to R77.30 which still experienced the issue.

Ultimately, the CheckPoint guy pointed to SK97746, which states that CheckPoint has interoperability issues due to the way it handles the tunnel renegotiation between other vendors. Essentially, as soon as the Phase 1 IKE tunnel re-negotiates, the CheckPoint deletes the Phase 2 tunnel immediately (even when we are working in a tunnel soft reset). This means that the SRX would have believed the tunnel has re-established and keep using the old one until the hard re-key time. However, the CheckPoint had already deleted the old tunnel - which caused the traffic drops.  This is fixed using CheckPoint’s GUI DB editor tool and making the modifications listed in the support article linked above.

While the CheckPoint side seemed to be responsible, it’s still odd that the SRX was never clearing the old SPIs. It might be that it kept them open because the old tunnels were never gracefully closed with the CheckPoint.

So there you have it - I hope that these might help someone out who is currently banging their head against a SRX VPN issue. If you’ve run into similar issues, drop a comment below!

I talked about this briefly in my initial background story posts, but I went straight from Cisco Networking Academy in high school out to working a full time job at a local IT consulting company. By the time I finished high school, I had already passed the Cisco Certified Network Associate (CCNA) exams and become certified. Having that certification is what got me in the door for a number of interviews, and eventually got me the job at the consulting company. At that point, I really didn’t have much else going for me - I didn’t have a college education nor any real-world experience. In my time working at this company, I spent a significant amount of time doing self-study and labs for my certification goals. When I got my CCNP certification, I used it along with the experience I had gathered to get my next job. This new employer was heavily focused on their IT staff needing to have a college education - so they pressured me for a while to go back until I eventually gave in.

I spent a while reviewing many colleges in the area and online, trying to figure out what would meet my needs. I ended up picking out a four-year degree in network security, and opted to go the online-only route because it benefited my schedule better. I packed my classes up to a full-time schedule, because I didn’t want a four-year degree to take any longer than four years. At this point, I also had the benefit that my employer was willing to reimburse 100% of the costs - which certainly helped convince me to go back.

Over the course of the past four years, I have taken many classes that include general IT, development, networking, and security (not including the normal required materials). I found that a significant portion of these classes didn’t directly benefit me. A lot of the material was much more focused toward beginners who haven’t already been working in the field for six years - which is completely understandable. The most I really got out of this was improving my abilities to push myself through work that I didn’t want to do. I did have a few interesting classes, like an Android development course, which I found to be extremely fun even if I probably won’t use the knowledge much.

Four years later and I’m done - did I benefit from it? On some level yes, I think I did. At the time of my degree completion, I have now been Cisco certified for ten years and I’ve been working in networking nearly the same amount of time. I’m already further in my career than I thought I would be at this point, and I’m happy with my position and pay (the degree isn’t going to change either of these things). At this point in time, finishing the degree is not much more than an accomplishment that I can add to my resume. Sure, having the degree on my resume may get me past HR screening for new jobs and opportunities - but it likely won’t actually play much into a company’s decision to hire me.

In the end I think that both certifications and college education are useful - they can both be great indications to an employer that you’ve been trained on certain technologies or fields. However, I think that the actual on-the-job experience is what really matters - and I experienced a direct benefit from getting in the field early and working while all of my friends were still in college. I would not be as far in my career as I am today if I had waited four more years to start working.  Unfortunately, I think that we place a little too much importance on completing a formalized degree program, when equivalent experience and certifications may benefit a company more.

I understand that I had a bit of a unique situation, but I figured it would be worth sharing my experiences and how they have affected my view of college education. I’m still happy that I went through with it and completed the degree, but you won’t see me throwing a big celebration - except that I’m just super glad it’s all finished. At this point, I will take a few months to relax and spend time on hobbies - but I do plan on going back to certification studies (Juniper stuff and likely begin working on a CCIE).

Any thoughts? Comment below with your experiences - I’m interested to see if there are many people who have had similar experiences to me, or possibly even the complete opposite.

If you really want to become great at something, you practice it a ton, right? Well networking and IT work exactly the same. You’re not going to become an expect by just reading a ton of tech books and blogs. While those certainly help, there is nothing better than simply getting your hands dirty. Having a good home lab setup is key to truly understanding how things work.

So how do you get started? Well the way that I built a home lab over the past 10 years is probably much different from you could today, given the amount of virtualization technologies available. Still, I believe that some physical pieces of equipment are necessary. I took classes in high school toward CCNA certification, and we had a lab of several routers and switches there. Once I got into the real world, I wanted to start working on additional certifications and just improve my skills overall. So I picked up an old Cisco 2611 router and a 2950 switch. I played with these for a bit and used them to get my CCNA Security, which at the time covered the basics of securing Cisco IOS routers and switches.

Another year or so down the road and I expanded by picking up a power-over-ethernet switch, and two Cisco 7900 series IP phones. Since I had discovered that the 2611 router could run Cisco’s Call Manager Express, I decided to go for the CCNA Voice certification. Having this equipment to work on gave me experience that was much closer to real world, than if I had just studied the textbooks. I could configure things, break things, then sit there for hours until I figured out how to fix my problem. I could configure the entire system, test it all, then tear it down and completely rebuild. Being able to configure the entire CME system from memory gave me a lot of confidence toward taking the certification exam.

So do I still have a home lab today? Oh yeah, you bet I do! It’s changed quite a bit from what it used to be, but the same concept still applies. I have an entire environment to play with, which allows me to test and learn new technologies outside of work. In fact, my ‘home lab’ has evolved into just part of my home networks. So here is what I’ve got running today:

• Cisco ASA 5505 (Probably soon to be replaced with a Juniper SRX 300)
• Two Cisco 2960G-8TC-L switches
• Ubiquiti UniFi 802.11n wireless access point
• Synology DS411 Network Attached Storage with 4x 3TB drives (Soon to be replaced, as it is over 5 years old(Update: Got myself a DS918+!))
• A few spare PCs running VMware ESX 6.0

The ASA, switches, and AP run just about all of my home network. I even have the ASA running AnyConnect SSL VPN so I can access my storage at home from anywhere. The Synology has been one of the best additions to my network and lab. For one, it acts as a centralized storage device for my home network. I back up all of my PCs to it, and any digital media I own is also stored on it so I can stream it to devices within my home. For two, the Synology acts as an iSCSI backend to my VMware hosts. This setup allows me much more flexibility with my lab.

On the ESX hosts, I have a few VMs for lab use and a few that are for my home network. A GitLab server hosts all of my Git repositories for my own personal coding projects. I have a CentOS box for running the Ubiquiti management web interface. Another few CentOS VMs for running bind DNS, Observium, and Splunk. I also run a personal Minecraft server on there, so it’s not all work here 🙂

I love the idea that at any time I can just go home, spin up a few VMs, and start playing with something new. When I was learning Juniper’s SRX platform, I downloaded their free trial of the vSRX and had it running for a while. When I changed jobs, I needed to learn a new web proxy software - so I downloaded their free trial and stood up a VM. You really learn a lot by building a platform from scratch, because you gain a better understanding of what impact certain configuration options have. You also have the freedom to change whatever settings you want and see what they do. I once had an idea for a coding project, so I turned up a VM running RabbitMQ - and spent a weekend learning how it works to see if it would accomplish what I needed for the project.

So to sum it up - I just want to say that having a home lab has really contributed a lot to my success. It offers way more flexibility than trying to test something at work, unless they also offer you a complete lab environment. Your lab doesn’t have to start off perfect, nor does it need to have expensive equipment - it just needs to help facilitate your ability to learn and gain experience. Have a lab at home? Tell me about it in the comments below! I would love to hear what other people have done.

This guide is written for CentOS 7. If you’re running another distro, find your dependencies here.

Last year we had to begin migrating off of some of our older Juniper SSG firewalls since we were beginning to push them to their throughput limits. We evaluated a couple of vendors but ultimately decided to stay with Juniper and purchase SRX 1500 firewalls, which are capable of up to 10G throughput. After a while of working with these firewalls, I have to say they’re pretty solid devices and I’m extremely happy with them. One of the main reasons I like them so much is the ease of automation, which is what we’re going to dive into today. If you need a device for lab/test - Amazon has the SRX 300 for less than $300. I’ll likely be picking one up in the near future for easier automation testing.

Juniper provides an awesome library for SRX management called PyEZ. Here is what we need to get the toolkit ready to use in our scripts:

Install dependencies

I run CentOS here, so I just needed to grab the following packages:

yum -y install python-devel libxml2-devel libxslt-devel gcc openssl openssl-devel libffi-devel

Just a quick note - Juniper’s web page doesn’t actually mention openssl-devel, but their tools will fail to install without it

Get pip

If you don’t have it already, then download the pip installer here. Then just run the following:

python get-pip.py

Install PyEZ

Once everything else is set, this is the easy part:

pip install junos-eznc

After all that is done, we can get to the exciting part: automating something so you don’t have to do it anymore! So here is what we’re going to throw in our script just to get started:

# Import the JunOS modules we need
from jnpr.junos import Device

# Define the login credentials and address for the target firewall

# Need to provide device address, user account, and password

# Note: Juniper also provides authentication via key-pair, which would be more secure than username/password

srx = Device('10.10.10.10', user='deviceuser', pass='devicepass')

# Open the connection

srx.open()

Once you have that going, it’s pretty easy to start making calls to collect data or change the configuration. In one of the first projects I used this for, I was making use of the API to reset VPN tunnels. This was due to an issue with a cross-vendor tunnel, which would occasionally break during VPN re-keys and only negotiate a uni-directional tunnel. So the script was written to detect a uni-directional flow of traffic, then log into the SRX and reset the VPN - which would force a renegotiation and fix the issue.

So in order to accomplish the SRX-side of that script, I used the following to reset the IKE and IPSec security associations:

# Clear IKE security association for a given peer
response = srx.rpc.clear_ike_security_association(peer_address='20.20.20.20')

# Check to see if command was accepted
if response == True: print "IKE SA Cleared"

# Clear IPSec security association for a given peer
# Note: In this case, you need to provide the index ID
# Get the index ID from running '*show security ipsec sa' *on the SRX
response = srx.rpc.clear_ipsec_security_association(index='123456')

# Same thing - Check to make sure the command succeeded
if response == True: print "IPSec SA Cleared"

This provides a pretty basic example of how easy it is to control the SRX via a Python script. You might be wondering: This is great, but how do I find the names of those calls? Well, Juniper made that extremely simple as well! Just take any command on the SRX and pipe it through display xml rpc and the device will tell you exactly what you need. For example:

root@SRX123> show security ipsec sa | display xml rpc
<rpc-reply xmlns:junos="http://xml.juniper.net/junos/15.1X49/junos">
    <rpc>
        <get-security-associations-information>
        </get-security-associations-information>
    </rpc>
    <cli>
        <banner>{primary:node0}</banner>
    </cli>
</rpc-reply>

In this case, we are running the command which would normally print all of the connected IPSec tunnels. So the section under rpc is where we want to look. Now just take the get-security-associations-information, and change the dashes to underscores. So to use this in a script we would call srx.rpc.get_security_associations_information(). I’ve actually used this command to build a VPN status dashboard using Python and Django, which I accomplished by just pulling all the IPSec tunnels and parsing them into a web table.

As a final note, I would highly recommend creating a separate service account on the SRX for scripting. A separate user account with limited permissions would be the recommended way to go here. It might seem like a pain to set up, but it’s worth it in terms of security. Here is what I have configured on my SRX firewalls for the API service account:

system {
    login {
        class api-class {
            permissions security-control;
            allow-commands "(clear security ike)|(clear security ipsec)";
            deny-commands "(clear)|(file)|(file show)|(help)|(load)|(monitor)|(op)|(request)|(save)|(set)|(start)|(test)";
        }
        user apiuser {
            full-name API_Service_Acct;
            uid 125;
            class api-class;
            authentication {
                encrypted-password *<encrypted string here>*
            }
        }
    }
}

So using the above example, you should only define the necessary commands in the allow-commands section. If the account is ever compromised, we are severely limiting the amount of damage that could be done. A little extra effort, but potentially a big payoff.

So what did you think of this tutorial? Helpful? Have questions? Let me know in the comments below!