About ARIN

If you haven’t already worked with them - ARIN is the American Registry for Internet Numbers. They are a non-profit organization and their purpose is to assign/manage Internet number resources for all of North America. This includes IPv4/IPv6 addresses and BGP Autonomous System Numbers (ASNs). ARIN is one of five Regional Internet Registries (RIRs) - each managing Internet resources for it’s own individual region. All of these report back to a top-level organization, the Internet Assigned Numbers Authority (IANA).

What I didn’t know: ARIN actually used to manage resources for all of South America and Africa as well. LACNIC formed and took ownership of South America in 2001, and AFRINIC took Africa in late 2004. ARIN itself has only been around since 1997, and will be celebrating it’s 20th anniversary this December.

Outside of assigning/managing number resources - ARIN manages a huge manual of numbering policies and standards (The Number Resource Policy Manual). A good note here is that these policies are heavily influenced by the community - so if any individual or group of network operators want to change/modify or add new policies, then they can submit proposals to do so.

IPv4 Depletion

I was very interested to hear about what’s going on with IPv4/IPv6 - mostly because I’ve been trying to push for IPv6 in many of the places I have worked. The ARIN group spent a little bit of time talking about how the depletion of IPv4 addresses has affected their workload. Overall, it seems like their work has remained about the same - but it has transitioned from mostly IPv4 allocations to more IPv4 transfer requests.

An interesting note from this discussion was that ARIN only performs the backend registration changes for IPv4 block transfers. They play no part in the actual negotiations between two organizations. However, they do perform their own investigations during transfers to ensure that the source organization legitimately owns the IP block, and the destination organization can justify the use of the space.

I had heard previously that ARIN kept a block of IPv4 addresses for transition to IPv6 - but I never researched it further. This was a topic ARIN touched on during the event. Essentially, they have kept ownership of a /10 block of addresses, which is split up into individual /24 blocks for assignment. Any organization can request one of the /24s when they request a block of IPv6 addresses. The organization must fill out a justification form, in which they demonstrate how the IPv4 blocks will be used to help transition to IPv6. Organizations can request one of these blocks every 6 months, provided they can still justify the need for them. This is all documented in NRPM section 4.10.

The somewhat surprising thing here is that ARIN was actively encouraging people to take advantage of this. Probably because they need to push IPv6 adoption in any way they can. As of the date of the event, ARIN stated that only ~60 /24 blocks had been assigned so far.

IPv6 Adoption

This part of the event wasn’t quite everything I wanted it to be. Overall ARIN touched on statistics from Google and other organizations that show the trending uptake in IPv6 network access. They also spoke briefly about how the structure of IPv6 addresses makes life easier - because the last 64 bits can always be used for host-based MAC autoconfig, then network operators only worry about subnetting above that.

Interestingly enough, ARIN was advocating for the method of ‘assign way more addresses than you’ll ever need’ mentality for IPv6. Another attendee asked the question ‘Won’t we run into the same thing as IPv4, if we just throw out v6 blocks like candy’? This actually led to hearing something I wasn’t aware of - IANA has currently only made 1/8th of IPv6 blocks public available for use. The current numbering scheme/standard will be used for this first block of addresses. If we run through them too quickly, then we can step back and re-evaluate best practices before handing out the next 1/8th block of addresses.

DNSSec

Initially I was a bit confused that DNSSec was on the topic list - but I figured maybe ARIN was just trying to push this for the betterment of the Internet. While they spoke a bit about DNSSec for forward DNS, their primary topic was how DNSSec for reverse DNS isn’t something people are normally thinking about. As it turns out, ARIN offers reverse-lookup DNSSec for any IP blocks that they assign out. This is good to know, since reverse DNS can be important for things like email security - and its certainly something I’ve never really considered in the past.

If you have purchased IPv4/v6 blocks directly from ARIN - I would recommend that you check this out.

RPKI

Resource Public Key Infrastructure (RPKI) is a way of cryptographically validating ownership of IP address space or routing objects. Since BGP is primarily a trust-based protocol between organizations, RPKI allows network operators to implement additional security by providing a certificate-based system of trust. The majority of this discussion was around how bad BGP security is, and that overall North America is far behind on implementing RPKI.

ARIN has a service available where they will act as your Certificate Authority (CA) for RPKI - so it only requires network operators to sign records then implement a few device changes.

My Thoughts

Overall the event was fairly informative! It wasn’t quite everything I wanted it to be, but I did walk away with additional knowledge that I didn’t have before. I was really hoping to learn more about how other organizations are implementing IPv6, or even how other people are convincing their employers to take IPv6 adoption seriously. When I spoke with some other attendees, it seemed like not many people had IPv6 running in a production environment yet - only a few of them had even started testing. Surprisingly, even the ARIN reps were repeatedly asking people to contact them if they had an IPv6 success story to share.

One thing I found really interesting was surrounding DNSSec/RPKI. A few attendees asked about how many people are actually validating signed resources. It’s one thing to implement signing, but it won’t matter if no one validates the resources, right? Surprisingly, ARIN had no statistics about this - and stated the point that they cannot enforce adoption of these standards. It certainly makes sense, but it’s not something I gave much thought to previously. Since they’re just a registry, they can only make these services available - not enforce their usage. This is why they put on events such as this to raise awareness and provide education.

ARIN pushed the fact that all of their policies are community driven. There were quite a few examples throughout the event of how individual members of the community could impact changes to their policies. My primary concern is that it seemed like a majority of the individuals in attendance represented government or educational organizations - and not a lot who worked in similar network environments to what I manage. They raised their own concerns and questions, which were certainly valid for the types of infrastructure and designs that they maintain. However, a number of these things don’t really apply to my infrastructure in quite the same ways.

If I have to make one point here: If you’re a network operator, go subscribe to ARINs mailing lists and get involved. Maybe you don’t have any ideas for policy changes, but you never know what might come up that you could provide meaningful input on. The ARIN reps provided an example or two of when a smaller group of people suggested policy changes which drastically affected bigger companies - and almost no one opposed it until it took effect. Only you have the ability to voice your opinion and concerns about how a proposed policy could affect your network. If not, the next time you try to request a block of IP addresses or a BGP ASN, you could potentially run into roadblocks because of a policy change proposed by someone with very different needs.

The staff at ARIN don’t live and work in the networks that we do. They try to work with network operators to understand use cases and the possible ramifications of policy changes - but ultimately they are a small non-profit. They can’t think of everything, nor can they force network operators to contribute their opinions. Get involved and make a difference.

As a final note, ARIN has a Fellowship Program where you can apply to attend one of their Public Policy meetings for free. Fill out an application and if you’re chosen they’ll provide a ticket, hotel room, and travel expenses. It’s a great opportunity to experience one of these meetings, especially if you might not have the financial means to otherwise.

The slide deck from the event is publicly available on ARIN’s website: here.

One of the big things that helped me was just the experience I had prior to starting on the CCIE. My experience going into the studying likely gave me a huge step up compared to if I tried the exam earlier in my career. If I tried the CCIE eight years ago like I originally wanted to, it would have been a lot more difficult and much more time consuming. I would have had much more to learn from scratch, and much less practical experience to help.

Additionally - the other huge benefit was going into the lab with a solid strategy around time and task management. There were several places through the exam that I felt like I could have easily lost 30-45 minutes on one item. It was very important for me to be able to step back and admit I couldn’t solve something. Instead, it let me focus my time on completing the tasks that I could do - and working on the unknown stuff if I had time later.

On the task management side - I spent time early in the study process on finding a good strategy that worked for me. Once I had this figured out - I used it on every single practice lab. I ended up using a combination of a few things other people have written about previously. My base task management was using a great blog post by Chris Miles (Read it here). In Chris’ blog, he suggests breaking up the tasks per location - then completing all the tasks for a location, one location at a time. That part didn’t work for me. Instead, I only used his method of organizing all of the tasks under individual locations - that way I could easily see what tasks were left and where I still needed to work. For example, if I needed to configure EIGRP - I could easily look at the sheet and see every location that needed some form of EIGRP config.

For the actual order in which I implemented tasks, I followed the guidance of a LinkedIn post by Kim Bartlett (Link here). In that article, Kim suggests a logical order of operations - like L2, IGP for MPLS, then MPLS, etc. Doing things in this way made sense to me. So I worked out what order worked for me, and decided to follow it. The big difference in my strategy, was that I found it easier to complete all tasks for a certain protocol/technology at once. For example, if I was configuring OSPF - then I would configure it at every location at the same time before moving onto the next piece. My overall order of operations was something like this: L2 -> all IGP -> VPN/MPLS -> MP-BGP -> iBGP -> eBGP -> BGP -> IPv6 -> Anything else. I found this to be a good flow for me. It allowed me to configure things like BGP only after I had already configured all of the underlying dependencies - which meant I could test immediately to see if everything was working as intended.

All of the above combined with constant labbing for months prior to the exam was absolutely critical to helping me pass on the first try. I had found a good strategy that worked for me and applied it to every practice lab, which meant that I walked into the actual exam feeling like I had a good way to guide myself through the onslaught of work. Had I walked in with just labbing experience and no good strategy, I don’t think I could have gotten close at all.

Okay, Now What?

I’m now getting around to posting this over three months after I passed the CCIE. I’ve spent a lot of time catching up on things around the house, reading books, running through a few video games, and overall just trying to enjoy the free time.

That being said - it wasn’t long for me to start feeling guilty and itching to start working on something else. My first thought was to begin working on the DevNet certifications. I’ve been doing a bit of Python & network scripting over the past few years, and I’m excited that Cisco is launching a certification program around it. I’ve been working on this a bit recently, which has also helped me get back into a few Python projects I hadn’t touched in a while. My current plan is to try taking some of these exams shortly after they launch.

I’ve also kept thinking back to one of the other certifications I considered going after: the CCDE. In my current job as a Systems Engineer at Cisco, the content behind this certification applies a lot more to my job than the CCIE. That’s not saying the CCIE doesn’t help me - it absolutely does. However, my job today is more understanding the technologies and how they fit into a customer’s network, rather than performing in-depth configuration work.

I don’t know yet whether I will fully pursue the CCDE and take the exams. But I have started reading a few of the recommended books, and I’m already finding bits of information that are valuable to me. I’m also really enjoying the content and getting much more interested in some of the topics. For now - I am planning on continuing to read through the information just to learn it and see where I can apply it. Once I get a good feel for everything, I’ll decide whether to chase the actual certification or not. For now, I think I’ll just enjoy not looking at a PuTTY window for a while 🙂

Thanks for reading - and thanks to all the people who have supported me over the past few years. It’s was a long journey, and not always an easy one - but I think it was well worth it.

Started here? Read the rest of my story:

Part 1: Getting Started

Part 2: Written Exam & Lab Prep

Part 3: Lab Day

Part 4: Lab Strategy & What’s Next

Lab day finally came - and I arrived at Cisco building 5 in Richardson, TX around 7:45am. There were already a handful of other CCIE candidates waiting outside for the building to open. Once it hit 8am, we all went in to get signed in and fill out our lunch order forms. Then it was time to wait.

The exam proctor showed up around 8:17 and guided us to the exam room. I figured there would be more time allotted to the proctor talking through rules, guidelines, etc… but instead he just said a few quick things and we were told to begin.

Troubleshooting

The troubleshooting section had me a bit concerned. It’s always difficult to jump into a completely unknown network and try to fix a problem - and this was no different. My first question immediately made me start panicking a little. I read the ticket, looked at the expected output - and began wondering where to start while being very aware of my short time limit. Every question felt like “I’m never going to figure this out in time” - yet after a few minutes of troubleshooting I was able to find the answers to the first few questions.

Halfway through the section I received a few tickets that required a lot more work. Some of these I didn’t make much progress on, and some I was able to get half-way resolved. For each of these I tried very hard to keep to a reasonable time limit per question, then mark it down as something to come back to later if I had time.

A lot of people talk about counting your points during the exam to know where you stand. I had originally assumed that this would just be a waste of time. Yet when I finished going through the remaining tickets, I knew I had to make sure I had enough points. Turned out I was barely on the edge of a passing score - assuming I had resolved all of the tickets correctly. My first two hours ran out, and I got the 30 minute warning. I was hoping to avoid using the extra 30 minutes, but I knew I needed to go back to the 3-4 questions I hadn’t completed.

About 15 minutes later - I had managed to figure out one or two more of the tickets and decided to give up on the remaining items. Based on my estimated point count - I should have been in a good spot on the troubleshooting section….. But I still wasn’t confident in all of my answers. I knew I had a ticket or two that might not be resolved in the correct way. I decided to save the remaining 15 minutes and just move onto the next part of the exam.

Diagnostics

Next was the diagnostics section. My biggest complaint here (and it’s somewhat minor) is that the on-screen timer is located in a completely different place than troubleshooting & config. At first (probably because I was in a rush), I couldn’t find the timer - and I also had not kept track of when I began the section. That was a big mistake on my part. So I forced myself to rush through the section, knowing it could end unexpectedly at any second.

Once I wrapped up my diag questions - I finally found the timer… and to my surprise had just under five minutes left. Not a ton of time, but enough for me to go back and double check a few answers that I had rushed myself through. I also used the last minute or two to run for a restroom break before starting the config section.

I honestly had no idea how well I was doing on this section. One of the questions seemed straightforward, but the answer I picked felt too simple. But maybe I was just overthinking it? The other questions made me waffle back and forth between a few answers. In the end, I just went with what my instincts told me was the most likely answer and just stuck with that.

Config

The config section is extremely overwhelming at first. Well, I suppose it doesn’t get any less overwhelming during the exam - but you quickly get busy enough to stop caring about that 🙂

I had about 30-45 minutes in the config section before we took lunch. That was enough time for me to get through all of the Layer 2 tasks quickly and then build out my task list on the scratch paper. During this time, I thought I was doing okay until I got to the end of one of my first tasks. I had just completed all of the items within that task when I read the last item - which made me realize I had done the entire task incorrectly. That was not a pleasant feeling. Luckily, I caught my mistake before moving on - but the time had already been wasted and now I had to go back and re-configure that entire section.

Lunch was quick. We went out, ate our food, then got back to the exam in less than 15-20 minutes. There was a bit of minor discussion - but not a whole lot.

The remainder of the day went by very quickly. As I had practiced during the prior weeks of practice labs, I placed my trust in strategy & order of operations - then just went heads down and got to work. I tried not to look at the clock and instead just focused on getting the tasks done as quickly and efficiently as possible. I’ll share a little more on my strategy in the next post.

I ran into a few problems here and there throughout the exam, but nothing too crazy. The strategy I used allows for quick connectivity/functionality testing after completing a task, which allowed me to find and fix my errors quickly. Similar to the troubleshooting section, I hit a few tasks that I could only figure out parts of - so I marked them down to follow up later and just moved on. Since you don’t get partial credit for tasks, I knew I would need to circle back to these if I wanted a shot at passing - but there is no sense in wasting too much time on one task if I couldn’t figure it out quickly.

By the time I had finished every task, I finally let myself check the clock. I was shocked to see I still had almost a full hour remaining. I quickly took advantage of the time to go back to the several sections I needed more work on. A few of these I stumbled through until I was able to find my problems - and some of it I had to crack open the documentation site to figure out what I needed to do.

Running through a lot of the verification steps - there was still a few things not working as they should. I spent time troubleshooting, changing configs, and finally figuring out a few things. I made quite a few configuration changes here to force a few things to work, but I wasn’t sure if they were valid solutions - or if I would end up losing points for doing things I shouldn’t have.

In the last 10 or so minutes, I tried to very quickly add up my points while performing a quick skim through the tasks again. Being that close to the end of the exam - it made me feel a bit sick to start finding additional items I had missed. I rushed to throw in a few last-minute changes, then retest to make sure nothing broke in the process. I didn’t make it through re-reading all of the tasks, so I was left wondering what else I might have missed.

Assuming I had not missed anything else - my count of points placed me in a fairly decent spot on config. However, since there is an overall cut score for the entire exam - I had no idea if I would have enough total points between all three sections to pass. I was already like I might have just barely scraped enough points together for troubleshooting, and diag felt like a complete wildcard.

When I left the exam center, I found myself feeling much better than when I had entered. If I passed, then that would be awesome. And if I had failed, then at least I was confident in what I needed to go back and study. Rather than having to keep worrying about what tricks the exam might hold, I now had the experience of knowing what to expect. I was happy to have attempted the exam once - and knew I would be far better prepared the next time.

That evening I went to dinner with a few CCIE candidates who would be attempting the exam the following day. Just tried to have a good time, and not check my email too much :). When I got back to the hotel that night, I still had no results yet - so I just went to bed and tried to get some sleep.

The Next Day

I woke up probably a dozen or more times throughout the night. Every time my first instinct was to grab my phone and see if I had gotten my results yet. Every time I forced myself to not check, and just go back to sleep. Around 5am, I finally let myself check once - but still had nothing.

I finally got up around 6:30 - and the CCIE exam site was down. I had a bunch of text messages from people back home asking if I had anything to report - but now I couldn’t even check the site. Later I would find out that the site was broken due to an internal issue at Cisco, but for the time I couldn’t do anything. I tried a few more times throughout the morning, but mostly just gave up and decided to wait it out.

My flight left around 10:30 am. While waiting in the airport, I still kept checking every so often but could not get to the site.

Once I got onto the plane, the site finally loaded! But my results were the same: No score yet. A this point I figured I would just give up, enjoy the flight - and check when I got back home.

Boarding took a little longer than usual for the remaining passengers. Right as it was announced that they were shutting the doors and we would be taking off shortly, I decided to try checking one more time.

As the site loaded - this time I was greeted with a new status: Pass.

My initial reaction was just absolute relief to finally be done - knowing that I didn’t have to keep worrying about trying to pass before the upcoming certification changes. I sat back for a minute before refreshing the site again to make sure the result didn’t change. Nope - the result still said pass.

With that - on October 9th, 2019 - I was done. I had my number. CCIE #63461.

Keep going for the rest of my story:

Part 1: Getting Started

Part 2: Written Exam & Lab Prep

Part 3: Lab Day

Part 4: Lab Strategy & What’s Next

Finally in early 2019 I gave up on trying to gauge where I was at - and figured it was time to just give the exam a shot. I had already been studying for almost a year and a half, and I was craving some definitive way of figuring out where I was at. I went ahead and scheduled an exam for Tuesday, March 12th.

When I walked into the written exam, my first question immediately made me feel unprepared. It was something specific to provider WAN switching - not a topic I had spent enough time on yet. I did my best to take an educated guess, but that first question gave me a lot of doubt about how well prepared I was.

The written exam overall felt very… all over the place. It didn’t feel like a single cohesive exam - instead it felt like 20 different banks of questions shuffled into one. Some people call the exam just random networking trivia - and in some ways that might be accurate. For example, I might have a question on very basic L2, followed immediately by a very in-depth question on MPLS. Then probably over to something completely different. I didn’t want to admit it at the time, but I probably felt far less confident in answering many of the questions I got - and gave my best effort on guessing at quite a few.

Already not feeling great about how well I was doing, the test finally made its way into the evolving technologies section. This section did nothing to ease my nerves :). I completely understand why this section exists, but it felt like there was almost no effort put into some of the questions. Many of the questions I got made no sense, had grammatical errors, or gave a set of possible answers that didn’t line up with what the question was asking. Even for technologies that I did have a lot of experience with, it felt like the question was just written by someone who had no understanding of it.

As I finished my last question, there was no doubt in my mind that I had failed. To me, it was just a matter of how badly did I miss and how can I better prepare for next time. I was already making several mental notes on what topics I desperately needed to go back and review for the next attempt.

However - when I clicked through the remaining screens on the exam, I was extremely surprised to see that I had passed. It was only by a few points - but a pass is a pass!

Walking out of the exam, I sent a message to a few people at work to let them know I had passed. Even with the score sheet in my hand, I didn’t feel comfortable saying that I had passed. At no point during the exam did I feel like I was doing well. Maybe that’s just part of the difficulty? I don’t know… I’m honestly glad to see the written exam requirement is being dropped from the new exam blueprints.

Studying for the Lab Exam

Once I had gotten past the written exam, my full attention went into working toward the lab. I spent too much time initially trying to get my lab environment all sorted out. Went back and forth trying to choose between EVE-NG and GNS3, before finally settling on GNS3. Then I wasted a bunch of time trying to find the right images to use and testing them to make sure everything worked.

Finally - I picked up a copy of “CCIE Routing and Switching v5.1 Foundations: Bridging the Gap Between CCNP and CCIE” and got started. Going through this first book was far less enjoyable than I had hoped. Each lab was a completely different topology with a lot of pre-work to get going - and in many cases completing the actual practice lab would take a fraction of the time it took to get set up. I got frustrated with this a lot - but tried to keep pushing through to at least finish the book as a starting point. This ultimately amounted to a rocky start to labbing for me. Not working on it as much as I should, and not necessarily looking forward to it.

My next set of materials would be the INE workbooks - which honestly are structured far better. These labs were all on a shared topology that I could easily clone in GNS3 every time I started a new section. All of the pre-config is done for you - so that you can just focus on the pieces relevant to the topic. For example, if you’re working on a BGP lab - you don’t have to start from scratch with IP addressing or L2 configs. This made the content much easier to consume, and did a lot to help me spend more time working on practice labs. I got through these labs pretty quickly and repeated quite a few for additional practice.

At Cisco Live US 2019 - there was a huge announcement regarding certification changes. The CCIE exam & content was changing (along with pretty much everything else). I wasn’t entirely surprised to hear the announcement since the existing track was several years old, and I had come across a few rumors on the internet of possible changes. Even still, I was finding myself now up against a very finite amount of time to pass the lab exam. The old test would be phased out in just eight months (in February 2020).

After the announcement, I talked to my manager about what to do. We decided it would probably be in my best interests to schedule a lab date, and do whatever I can to try and pass ahead of the exam changes. So - only a few days after the new content was announced, I had scheduled a lab date for October 9th, 2019. This was less than four months away, and I still had a ton of content / practice labs to get through.

Having the looming deadline did great things for my motivation :). On the good side of things - It helped me to spend more and more time studying for the lab exam. I was able to focus more than before, and I was finding it much easier to push myself to practice even when I wasn’t necessarily excited to. Over the summer I nearly doubled the amount of time I had spent labbing compared to before the announcement. On the not-so-good side - I had also put together a week-by-week plan of what I still needed to accomplish between now and October. It was a tighter timeline than I was originally looking at, and now it felt like I didn’t have enough time to accomplish everything. I pushed through it anyways, knowing that October was just my first attempt. If I couldn’t finish everything in time, then I would still have time before the second try.

Remember back when I mentioned that six year gap between getting the CCNP and starting on the CCIE? This is the big part where that helped me a ton. Going through a lot of the workbooks - I didn’t necessarily feel like anything was too crazy. Over the past 10+ years I’ve worked at a number of different companies and had the opportunity to play with a lot of networking gear. I had a great base of experience with most L2/L3 technologies, including quite a bit of practice with all the fun that BGP has to offer.

One of the other big things that I think helped was that not all of my prior experience was on Cisco equipment. Having to learn how to configure BGP, VRFs, or switching on multiple vendors forces you to think beyond the syntax. Every vendor implements things in their own unique way - and this helps you to get beyond just memorizing what commands to enter. Instead, you begin having to learn much more about the underlying technologies and how they operate - and understanding what you’re actually trying to accomplish. Then it’s just a matter of researching whatever syntax that specific vendor uses to implement that function.

Having that good base of knowledge and experience helped me burn through the practice labs fairly quickly. A lot of content felt very familiar, with maybe a few new variations of commands - or maybe a new option that I hadn’t previously used. Even some of the pieces that I hadn’t used much of before, like DMVPN or multicast, still seemed easy enough to grasp how it worked and learn the necessary syntax.

That being said - In a lot of ways it also gave me a false sense of security. Feeling like maybe I knew more than I realized and therefore maybe I was better prepared. Yet at the same time, knowing how difficult the lab is supposed to be - and constantly wondering what I could be missing.

Keep going for the rest of my story:

Part 1: Getting Started

Part 2: Written Exam & Lab Prep

Part 3: Lab Day

Part 4: Lab Strategy & What’s Next

In about a month, most of the exams will be changing over to the new blueprints so I’m not sure how relevant any of this will be - but it’s still worth throwing out there, right?

Why CCIE? Why now?

The two years I spent working on the CCIE dragged on for what seems like forever. Back in late 2017, I had hit a point where I felt like I wasn’t being challenged enough technically - and I missed the old days of excitement when I was studying/labbing for certifications exams. I had always wanted to go after the CCIE for a number of reasons, but it never made sense before. I had decided that maybe it was finally time to give it a shot.

To step back for just a moment - I originally began my career in networking by taking advantage of the Cisco Networking Academy program, which had been offered at my high school. It’s hard to believe I started that over 14 years ago - but it was likely the single most influential thing in getting me where I’m at in my career today. After two years of classes, I walked out in late 2007 with my CCNA and eager to begin working in networking.

Over the next few years - I worked on a number of additional certifications. I always had fun going after certifications because they gave me a path to follow and a goal to achieve. They helped to make the process of learning a bit more fun. On the Cisco side of things, I worked on the CCDA, CCNA Voice (now retired), and my CCNA Security. Finally in 2011 I finished up my CCNP and had to figure out what was next. I was super interested in the CCIE - but there was no way my company would pay for it. For the time I shelved the idea - but I didn’t give up on it as a goal. Instead, I just continued to maintain & recertify my existing certs, and picked up the CCDP along the way.

Fast forward to late 2017. I had officially passed my 10 year anniversary on my CCNA. I was also feeling like I was hitting a wall in my technical abilities. I wanted to do something different and fun - and my first thought went back to pursuing a new certification because of how much I used to enjoy the process. I debated between a handful of certs, including CISSP, CCNP Security, CCDE, and CCIE R&S. After giving it some thought and talking to a few people, I decided it was finally time to tackle the CCIE and work toward one of my long-standing goals. That six year gap between CCNP and starting on the CCIE would come back to cause me a lot of problems, but also help me in a few ways I hadn’t expected - both of which I’ll talk about later.

Time to Study

On October 4th, 2017 - I ordered by first set of books and began studying for the CCIE Routing & Switching written exam.

To be absolutely honest, I had no plan going into this. Historically when I took certification exams my process was usually watching a set of training videos (usually CBT Nuggets), reading through the official cert guides a few times, picking up maybe another book or two, taking a bunch of notes, then a lot of labbing. It was never enough for me to just watch/read about the stuff - I needed to get hands on and break it to really learn. Usually by the time I had finished all of that, I would be feeling confident enough to go give the test a shot. I went into the CCIE written assuming this strategy would still probably work - and I was absolutely wrong.

When I began working through the books and videos I had - I found that I wasn’t getting as excited about it as I had hoped. In fact, it just felt like so much of the content was just review of things I had learned years ago during CCNP studies. That long gap since my CCNP also left me reluctant to want to memorize all of the little details again. How many things had I studied for the CCNP that I never used in my actual job? I certainly didn’t want to waste the time trying to re-learn/re-memorize those things now… But I knew I would need to if I wanted to pass the exam. This kinda killed my motivation in some ways - because I would end up having to force myself to try and retain information that I didn’t want to.

Studying for the written was hard for me - and probably more than it should have been. Between the mixed motivation, I was also working through a lot of stress and nonsense in both my personal and work life. I would eventually work through these issues - but sometimes it would mean having to take a few weeks off from studying.Every time I took a break, I knew I needed to - yet it was still very demoralizing.

I got some help toward my goal in June 2018: I had the opportunity to take a job working at Cisco as a Systems Engineer. In terms of working toward the CCIE, this was an absolute key step in getting there. I was finally working for a company that was willing to encourage and help me toward my goal. I was also surrounded by a ton of engineers and enthusiastic networking professionals who were there to support me. I got to spend time with other people who were working on certifications, and even network engineers at my customers who always wanted to ask how my studies were going. This helped a lot to get me back into being excited about the content - and brought a bit of motivation back.

Even though I was spending a lot of time studying for the written exam - I never really felt like I was making true progress. I believe this was likely caused by the fact that the exam blueprint is so large and diverse. I never settled on a good method to reliably track how far I had progressed on all of the content. While I felt like I had learned a lot, I also perpetually felt like I was nowhere close to where I needed to be. I also have an old habit of waiting to schedule the exam until after I already already feel confident I have a good shot at passing. With the CCIE written, I felt like that level of confidence was never going to happen.

Keep going for the rest of my story:

Part 1: Getting Started

Part 2: Written Exam & Lab Prep

Part 3: Lab Day

Part 4: Lab Strategy & What’s Next

I have always said that I’m not sure I could write code for a living, but I do really enjoy writing scripts that make my life easier. Today’s post is a great example of that. The ease of use offered by the Juniper SRX firewalls and JunOS is something that I wish I had in all of my networking infrastructure. Even better, a brand new SRX 300 can be purchased on Amazon for less than $300 - which made a great addition to my home lab. Now I have a place to develop and test automation without breaking production 🙂

Anyways - I had a requirement to monitor my SRX clusters for route changes. Specifically I’m using this with devices where I have implemented customer peering via BGP. The SRXs don’t natively offer any form of monitoring and alerting for this, and the current monitoring applications at my disposal don’t either. So I decided to write something myself, which took significantly less time than I had assumed.

Code has been posted up to my GitHub - but I’m going to walk through some of it here. This script is intended to run as a cron job on a 5 or 10 minute interval.

# Imports
from jnpr.junos import Device
from jnpr.junos.op.routes import RouteTable

On my initial research to see how easy this would be to pull off, I found a nifty thing in the JunOS PyEZ package. Turns out they already offer a module literally called RouteTable that can pull the information I need.

Originally, I spent a bit of time trying to figure out how to use the standard device API to pull this info, but this module made everything 10x easier.

#################
# Required values
#################
deviceName = 'SRXFirewall'
deviceIP = '0.0.0.0'
apiuser = 'username'
apipassword = 'password'
SMTP = 'smtp-alias'
fromName = 'BGP Monitor'
fromAddr = 'bgpmonitor@domain.com'
toName = 'contactName'
toAddr = 'contact@domain.com'
prefixDict = {
        "FriendlyName": "Prefix",
        "Route-to-Inet": "0.0.0.0/0"
    }
################

The section above contains a list of the required variables for this script to function. A lot of them are going to be self-explanatory - but I wanted to take a moment to look at the prefixDict. This is a Python dictionary that maps a friendly name to a route prefix, which is used when we check our routing table. For example, if you wanted to monitor the SRX device for a route for 10.10.10.0/24 which belongs to Customer1, then we would just add and entry in this dictionary for “Customer1”: “10.10.10.0/24”

Alright - now let’s skip to the good stuff:

# Function to check received BGP routes
def checkBGP():
    try:
        # Open SRX session
        dev.open()
    except:
        sys.exit(0)

    # Pull device routing table, then keep only BGP originated routes
    allroutes = RouteTable(dev)
    bgp = allroutes.get(protocol="bgp").keys()

    # Close SRX session
    dev.close()

The section above is the beginning of the checkBGP function. Simple enough - just open an API session to the SRX and grab the entire routing table. That module I was talking about earlier makes it super easy! Next, we pull only the routes originating from BGP and assign them to a variable called bgp.

So in order to run this script repeatedly as a cron, I needed a place to persistently store the last retrieved routing info. For the time being, this is done by simply writing a temp file with the information:

    if not os.path.isfile(tempfile):
        with open(tempfile, 'w+b') as a:
            a.write(str(bgp))
        sys.exit(0)

    # Local file used to keep track of BGP learned routes
    with open(tempfile, 'ab') as a:
        lastroutes = a.readlines()
        # Compare if routes are different
        if str(bgp) == str(lastroutes[0]):
            sys.exit(0)
        if str(bgp) != str(lastroutes[0]):
            pass
    # Delete file, then re-create with new route list
    #os.remove(tempfile)
    with open(tempfile, 'w+b') as a:
        a.write(str(bgp))

The logic in the comparison is simple enough. If the current string of routes pulled from the SRX equals what is in the temp file (from the last run), then we assume no changes have occurred - and the script ends. Otherwise, if they don’t match then something has changed.

Once we know something has changed, we’ll go ahead and find out exactly which route entry is missing:

    # Create Status list, by checking received routes in bgp object
    status = []
    for name,prefix in prefixDict.items():
        if prefix in bgp:
            status.append("%s - RECEIVED" % name)
        if not prefix in bgp:
            status.append("%s - MISSING" % name)

This is where our prefixDict from earlier comes into play. We’ll look at every BGP prefix that we defined in that dictionary, and see if it exists in the current SRX routing table. All of this information (both routes received and missing) get added to an alert email.

Lastly, we’ll go ahead and compose our alert email to send out:

    # Send alert message
    sendMail(str(lastroutes[0]), str(bgp), status)

I’m not going to cover the email function here, since it’s pretty straightforward.

That’s it! The existence of the JunOS RouteTables module made creating this script a piece of cake. I’m considering adding onto this later with some additional functionality, so be sure to check my GitHub repo if you’re interested.

Hope this is useful to someone else out there - Let me know in the comments!

Securing your BGP peering (Know who you’re connecting to)

BGP is a little different from most other routing protocols, since it uses a single unicast TCP connection between peers to exchange routing updates. Lucky for us, that means that we can easily filter traffic from only known peers. Once you have direct connectivity up between your edge router/firewall and your direct peer, lock down that connection with an ACL. Permit TCP port 179 traffic ONLY from your directly connected peer IP - no one else.

While you’re at it, let’s take it another step further: Request that your ISP set up BGP authentication. Sure, a majority of BGP implementations today still require use of MD5 for auth (which is terrible) - but some authentication is still better than none. This can usually be arranged at the time of turning up peering. Both sides configure the same authentication password and with any luck the peering still establishes.

BGP by nature is unfortunately not the most secure protocol - but a few simple steps like this will help ensure you’re only connecting out to authorized peers.

Route filtering (Don’t trust anyone)

Usually when you’re filling out the BGP peering paperwork for your service provider, they will ask you what kinds of routes you want. In most cases, you should be able to request one of the following:

Default only - Exactly what it sounds like. Your provider will only advertise a route for 0.0.0.0/0. In many cases, this is probably what you’re going to want. With this type of advertisement, each upstream provider will just give us the same default route to the internet. From there we can weight which one we want to use, and traffic will automatically fail-over to the secondary connection should the primary fail.

Partial - If for any reason you want to weight routes to certain destinations differently, then we might request this. In this case, you’re probably going to still receive 0.0.0.0/0 plus any specific routes you ask for. A good example of this is if we wanted to specifically manipulate routes for a remote office we have. Maybe we want to weight Internet traffic for one uplink, and VPN traffic to a remote office on the other uplink.

Full - In 99% of typical business cases, this won’t be required. This option means the upstream providers will be dumping the entire Internet routing table on you. While this offers you a ton of control over path manipulation, it also requires significant memory resources on your routers in order to maintain that routing table.

After we figure this out, the next step is to make sure we are filtering the routes we accept from the upstream provider. Wait - didn’t we just tell them exactly what routes to send us? Why do we need to filter them? Well you can never be too safe here - and we would rather perform an unnecessary filtering than have an ISP accidentally misconfigure route advertisements. So if you’re only expecting a route for 0.0.0.0/0, then filter your inbound route advertisements so you only accept that route.

Same thing goes for outbound route advertisement  - if we own a /24 of public IP space, then we only want that range to be advertised out. Some providers may already filter this on their end, but again it doesn’t hurt here to be extra cautious. If we are accepting anything other than a default route from our provider, then we run the risk of leaking those additional routes between the two providers - which would lead to inadvertently becoming a transit AS. Chances are pretty good that you don’t want that, so make sure you configure filtering for all outbound route advertisements.

Minimum Advertisement (Oh no, we have to re-address everything)

I mentioned this in the original post - but typically when you are peering with two separate upstream providers, you need to advertise no less than a /24. We ran into this at my last job, where we had been provided a /25 by AT&T but we needed to bring in a second carrier via BGP. The reasoning behind this is to keep global routing tables as small as possible, by not allowing them to end up flooded with a ton of routes for smaller subnets. It makes sense, but on the other hand I feel like requiring a /24 in all cases can be a bit wasteful. My last job only required maybe 30 publicly addressable hosts - which meant that the remaining addresses went unused.

At any rate - should you find yourself in this scenario then you’re going to have to face the inevitable: Renumbering into a new IP space. Any time you have to do this, it’s going to be a bit of a pain - but for external addressing like this it might be easier. So in our case, the entire /25 space was hosted on our external firewall then NAT’ed into DMZ servers.

Here is the quick steps that I used to do a side-by-side migration without taking any significant downtime:

• Get the new subnet up and running - assign the interface addresses on your firewall and BGP up and running
• Assign new IP addresses to all of your existing services
• Configure NAT rules for the new external IP addresses to the DMZ hosts - while leaving the existing NAT rules for the old subnet (Also make sure your firewall rules permit the same traffic to either IP)
• Migrate DNS entries externally to point to the new IP space
• Once traffic stops flowing to the old IP, remove the old NAT

As a side note - if you procure redundant internet connections through the same upstream provider, then you might be able to work out something else. They may be able to provide you a private ASN to use, and they will likely accept any minimum advertisement - since they will be summarizing upstream within their network anyways.

I had a few more things I originally intended to cover here - but it seems that these topics are filling way more space than I thought they would. Specifically, I’m thinking about a dedicated post to BGP path manipulation - which is probably something you’re going to want to implement after peering is established. Hopefully these tips help! If you have any questions, throw them in the comments below.

I’ve been looking for a good chance to use VRFs in the past, but in most cases it didn’t really make much sense. About a year ago, I had a great opportunity when we needed to build a new data center. The data center was aimed at being lower capacity than most of our other locations, so we had to cut some costs here and there. In all of our other locations we use two physically separate sets of firewalls, one for external traffic and one for internal traffic. In this new location, we opted to save some money by picking up only a single pair of Juniper SRX 345 firewalls.

I made the decision here to make use of Juniper’s virtual routing instances to keep logical separation of internal vs external firewalling, even though it was only a single physical cluster. For one, this would allow existing staff to maintain their current understanding of network architecture. Every data center has the same overall logical traffic flow, even if the physical devices are different. Second, this allowed us to split load across the two devices. Normally we have two physical clusters to handle the traffic load, but in this case we were essentially going to pump the same traffic through one pair of firewalls. Assigning each virtual routing instance into its own redundancy group allowed us to run each firewall instance on a separate device - yet still allow for both instances to run on one in the event of a failure.

Once we got that firewall cluster into production, there seemed to be a lot less fear regarding virtualized network contexts. I was able to prove that it worked, and worked well for what we needed at the time. Soon enough I was able to find a few additional places where we could make use of the same concepts. We recently procured quite a few Cisco Nexus 9372PX switches for both new deployments and hardware refreshes. By default these switches already come pre-configured with a out-of-band management VRF, which is already super useful to me. We run all of our device management traffic on a segregated network, so a management VRF allowed me to configure the IP/route information to make all that work - while not interfering with the normal layer 3 operations of the device.

Being a cloud provider, most of our customers are completely abstracted from the hardware/software that runs their hosted applications. However, in a few cases there are instances where a customer negotiates for a contract change to say otherwise. For example, a customer might have a special software integration they want to run and have the ability to control - or some customers want a dedicated point-to-point Ethernet connection into one of our data centers for increased reliability. A lot of the background networking work for this in the past was a bit of a pain - but it opened up another opportunity to make use of VRFs. I now have a dedicated customer VRF, which has separate routing configurations than our normal production environment. Customer wants to stand up BGP peers across their direct connection to our data center? Sure, I can isolate that BGP instance in the customer VRF, so there is no conflict with our production routing tables.

I’m sure that my current use cases are probably not the ideal implementations of virtual networking contexts - but they work for what we need and they make life a lot easier. I can see these becoming more and more common in our environment to logically segregate traffic. I am interested to hear how other companies have integrated this type of technology into their networks - so leave a comment below!

The easy part of the whole process is the first step - ordering a second Internet connection. Our CIO at the time placed a few calls and had a quote back pretty quickly. A local carrier was willing to run new fiber cables to our building in less than a month. Depending on how important uptime is to your organization, this is the point where you might want to ask about a diverse path into the building. If both connections run though the same physical paths, then a single incident could still cause an outage. For example - I once worked somewhere where the redundant Internet connections shared the same telephone poll across the street. So even though the connections were redundant, a single accident involving that telephone poll and both connections were severed.

Next - Ask about IP space. In terms of IPv4, the general rule for external BGP peering is that ISP’s don’t like to accept any prefixes smaller than a /24. In our case, we had a single /25 block already allocated by our current provider - which wasn’t going to work. Luckily, the new service provider offered to give up a free /24 block along with the installation costs. Unfortunately, this meant that we had to re-address all of our public-facing services, which is almost always a pain to do. I have a few tips for this, which helped us to minimize downtime - but that’s a story for another time.

Next, we need to obtain a globally unique Autonomous System (AS) number, which will be used to advertise our network to the world. Since we were located in North America, we went though ARIN for this process - which was fairly painless. Sign up for an account, prove that you’re associated with the business, fill out a few forms to justify your need, and then just wait for the approval. One thing to watch out for is 2-byte vs 4-byte AS numbers. 2-byte is the standard and has been around forever, but only allows for up to 65,535 unique IDs. A 4-byte ASN allows for significantly more unique IDs, but I have actually run into instances where an ISP doesn’t support these. I would hope that in most cases a 4-byte ASN will be just fine, but it might be worth asking your ISP just in case.

At this point, you should be ready to hit the ground running as soon as that second Internet uplink is installed. This is also assuming you already run a router or multilayer switch on the edge of your network, which also has BGP capabilities. So let’s get down to the fun stuff - an extremely basic configuration to peer between two ISPs. I’ll dedicate another post to additional recommended settings and configurations - but for now let’s focus on getting this running. The configuration sample below is aimed at Cisco devices, but the same concepts apply to most vendors:

EdgeRouter(config)# router bgp *<YOUR AS NUMBER>  *! The AS number provided by ARIN
EdgeRouter(config-router)# network *<YOUR LOCAL SUBNET>*   ! The subnet we need to advertise out both ISPs
EdgeRouter(config-router)# neighbor *<ISP1 PEER IP>* remote-as *<ISP1 ASN>* ! Provided by the first ISP - Their remote peer IP and ASN
EdgeRouter(config-router)# neighbor *<ISP2 PEER IP> *remote-as *<ISP2 ASN> *! Provided by the second ISP

As I mentioned, this config is very basic and will just accomplish what we need to get going. Follow up with a quick show ip bgp neighbors and hopefully you’ll see two peers in the established state. Any other state indicates a problem bringing up the peer connection. I won’t get into too much detail here - but check the physical connection, ping the peer, and make sure there are no firewalls blocking TCP port 179 between the peer addresses.

Hope this was helpful! Comment below and let me know how your experiences have gone with this type of setup - and look forward to a few more posts regarding BGP peering setup with multiple ISPs.