Matt Schmitz/ May 9, 2018

This post has been on my mind for a while now. I’ve worked as a network admin for long enough, and opened more technical support cases with vendors than I want to think about. Over the years I’ve developed my own process for how I handle those support cases in an effort to get a quick and efficient resolution. Some of this stems from starting off in a NOC, where calling vendor support was practically step 1 of any troubleshooting procedure. A lot of this is based on my own experiences or things I’ve been taught by former co-workers.

On the other side of things, I’ve worked with people over the years who haven’t had quite the same experiences as I have. Some of them don’t typically call vendors for support – or maybe they’ve just never been lucky enough to be the first responder on a Severity-1/Production-down case. This has occasionally resulted in a number of vendor support cases being closed with highly unsatisfactory results. In fact, there have been times where this has been bad enough that it gives management the impression that we received bad support from the vendor – but what if it was just our own inability to work effectively with that vendor? Maybe we didn’t push hard enough on an issue or stress the importance.

I feel that in a majority of cases, it shouldn’t be difficult to get the results you want out of a vendor support case. So I’ve put together a list of my own personal tips and guidelines for working with technical support.

The Vendor is your partner – Not the enemy

I’ve seen a lot of people call vendor support for help, then treat the support engineer as the bad guy. If you want to get a quick resolution, then you need to look at them as your partner. They’re here to help you figure out your issue and get everything fixed. When you open a new case, give them a concise summary of your issue with any relevant details you think are important. Don’t hide information, as this will just prolong finding a resolution – Give them everything they need. If you know your vendor always asks for the same diagnostics information every time you open a case, then have that information ready before you contact them.

Always remember how difficult the job of a technical support rep can be. They’re likely sitting in a call center, just waiting for the next case. They may have in-depth knowledge of their product, but they’re walking into your network completely blind. They won’t know your traffic flows, or that some systems are redirected through a proxy, or that one band-aid fix that Joe put in a year ago and never documented. Your support rep is going to do the best job they can to get a handle on what your network looks like, but be prepared to guide them. How would you like to troubleshoot a completely different and unknown network every time your phone rings?

Have realistic expectations

Especially when you’re new to a career in IT, it can be hard to gauge what you can and can’t ask of your tech support rep. One of my former jobs had a policy that for every ticket with AT&T we opened, we would be required to call back every hour for a status update. If there wasn’t one, then we were supposed to demand escalation to a manager. That policy might make sense if it’s a critical issue, but what about something that’s not? Have realistic expectations about what your vendor can do.

Troubleshooting takes time. If your support engineer grabs a bunch of logs and says they’ll need to get back to you – then you’ll need to give them the time they need. Feel free to ask for a time estimate – but if they say they’ll have something to you by the following day, don’t start bugging them every hour.

Remember that your support engineer’s job is to help you. If you don’t feel like that’s happening, you have a few options. You can ask for a case escalation or ask for the case to be reassigned. You never know the skillset of the person receiving your case, and you might get someone who isn’t super familiar with the problem you’ve raised. As long as there is no urgency, I will usually give the person time to work the issue – but be prepared to request a case transfer if it becomes apparent that they’re not getting anywhere. For example, I once had a case for a web-based firewall management system. The engineer I got was very good with the GUI side of things, but wasn’t very knowledgeable when troubleshooting took a turn toward the underlying linux system. A quick request to transfer the case to someone more experienced in this side of the system and we had the case solved within an hour. If an escalation or case transfer doesn’t help, you can also usually reach out to your local account representative and ask them to help push the issue for you.

It’s also very helpful to have an idea of your vendor’s support policies. Have a question about how to set up a new feature? Some vendors don’t permit you to open a case for a new configuration, and will refer you to their professional services team. On the other hand, some vendors are perfectly okay with helping you figure out how to set up something. Even better, some support teams are willing to stand by during migrations and upgrades, just in case you need their help. In my experience, if you’re not 100% confident in your changes, then it’s better to open a proactive case beforehand.

Be clear about the impact

If your entire datacenter is offline because of an issue, make sure that you immediately stress the importance. Again, your support engineer is jumping into your environment blind. Does this firewall performance issue impact twenty people, where it is just a minor inconvenience? Or is this issue prohibiting 50,000 customers from using the services you provide? The last thing you want is a misunderstanding of impact when it’s a high priority issue for your business.

Usually when I open a high severity case, I’ll let the engineer know: “Just so we’re on the same page, this issue impacts a large datacenter impacting 600+ customers. We need to get this back into a stable state as soon as possible”. High severity cases can be stressful for both sides, and I try to be clear about the impact without making that worse.

On the other side of things, if there is a low severity issue – don’t blow it out of proportions. I’ve worked with too many engineers who open up a Sev 1/Prod-Down case for every issue, even if the issue is just a minor inconvenience. Categorize your issues appropriately when you open them – and do your best to be realistic. A slow download for three users probably doesn’t warrant getting half a dozen TAC engineers on a conference bridge.

In case of emergency…

Emergency situations are a completely different subject – so I want to spend a bit of time covering them separately. It’s really important to know what constitutes a true emergency in your environment. Is it when an office (or datacenter) goes offline? Or maybe even a single extremely critical business application? Things break – so have a plan and be prepared.

Step one – Always call into your vendors support line. You don’t want to open a web/email case and wait around for a technician. This might seem obvious, but I’ve known a lot of people who complain about the vendor’s response on a critical issue when they opened a case via a support portal. Find the vendor’s support number (or have it saved somewhere) and call them.

Step two – Ask for a warm handoff. In most cases, the person answering the support line is just creating a case and routing it to the appropriate team/ticket queue. They may just give you a ticket number and tell you to expect a call back shorty. If the issue is truly critical, ask them for a warm handoff to a technician. Most vendors I have worked with have had no problem doing this, and it helps you get to troubleshooting faster.

Step three – Clarify your issue and set expectations. You may be in a rush to get the issue fixed, but take a minute to explain your issue thoroughly and clearly. The more information you give to your support technician, the more easily they can dive into troubleshooting. And as I had mentioned earlier, be sure to set expectations and be extremely clear about the impact of your issue.

Step four – Keep troubleshooting on track. As I’ve stated before, you know your environment/network better than your vendor does. If they start looking at something you don’t believe is related, you need to guide them back to the main problem.

In addition, if you feel after a bit that the troubleshooting isn’t making progress – then request an escalation or a second set of eyes. There is no harm in asking for more eyes on the problem. I’ve even had situations before where the technician said “Well, I think we need to do X to fix it”, and I’ve just asked them to see what their peers think. You would rather be sure about a change, than make the issue worse.

Step five – See the issue through to resolution. Make sure you get your environment back to a stable state before ending the call. If the technician wants to drop off and call you back after reviewing logs, let them know you’re willing to just wait on hold. Once they’re off the phone, it’s easy for your technician to get dragged into another issue.

If the call ends with everything in a temporary state – then take follow ups on your next steps and make sure you accomplish them! Maybe you were able to restore connectivity, but need to wait for a maintenance window to make a change that is more service impacting than the original issue. Or maybe your support technician needs you to gather additional logs that they can forward to development. Whatever it ends up being, make sure you take note of it and follow through.


These are just some of my own personal tips that have worked for me. Support calls with vendors don’t always need to be a massive pain to deal with. Sure, sometimes you might have bad luck and get an inexperienced technician – but I find most issues with vendors can be solved easily enough once you know how much you can push them, and what you can and cannot ask for.

I hope these are useful – Let me know in the comments if you have any additional tips!

About Matt Schmitz

Cisco certified since 2007 with a wide variety of IT and networking experiences. Just looking to share a bit of my own knowledge and experiences. All opinions are my own, and do not represent any vendor or current/former employer.

6 Comments

  1. That’s actually some really clear and excellent advice. Having worked with TAC on cases at Cisco, setting clear expectations and severity is critical to getting the best and most appropriate assistance.

    1. Thanks Jason! Yes, it certainly does help when everyone is on the same page.

  2. Great advice so far, I’d like to suggest defining a proper Problem Description to steer the investigation.

    In my experience I found this helpful to document problems:

    What:
    – what is expected behavior?
    – what is the observed behavior? the deviation is usually the problem statement we’re investigating

    When:
    – when does the problem occur? timestamp of the first reported incident and additional events
    logs are best evidence here, but sometimes all we have is an end user giving approximate time of day

    Extent:
    – how many objects (eg. network device) show the problem
    – how many occurrences of the problem were observed on each object (single event or multiple events)
    – is there a pattern in the event history, or are the events sporadic (usually we don’t see the pattern yet)
    – how many other objects could have experienced the problem, but did not? compare similar objects (eg. same functional role)

    Changes:
    – was this working previously?
    if no, should this work as-is? it might not be a tested feature by the vendor, with the same environment
    if yes, what changed? hardware, software version, configured features, surrounding environment (start with the specific object that you are investigating, but be willing to broaden the scope of your search to all objects in your environment, even environmental factors could be relevant)

    Usually if you can answer these questions (with supporting evidence) then you have enough information to postulate a problem statement and begin formulating possible causes. Test each possible cause, first the most probable cause, or if that would be very time-intensive (coordinated effort among multiple groups), try ruling out the low-hanging fruit first. Document what troubleshooting steps have occurred, what was the outcome, if the problem state changed (did performance improve even a little bit, or was it degraded further?).

    Focus on data points that rely on the presence of something, rather than the absence of something, because there are usually multiple reasons why something might be missing. If something is missing, then usually you can take a look at the same problem from a different viewpoint to find something present that is out-of-place / not right — triage here.

    1. Mark – Thanks for the additional detail! I’ve definitely found that the more information you can provide up front about an issue, the more likely you can get it resolved quickly. It’s all the same questions we would ask our users (or other teams), if they came to us for support.

  3. Great overview! This should really help folks better interact with vendor TAC. I wrote a similar piece about the tension between IT end user and IT vendors: https://ryburn.org/2018/05/07/it-vendors-vs-it-end-users/

    It shouldn’t have to be an adversarial relationship. When both sides work together, things get done much faster as you highlight.

    1. Justin – Thanks! You’ve got a good article there as well. It’s amazing how much easier troubleshooting calls can be, when we’re all willing to work together.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.