As part of NIL’s customer support division, my colleagues and I face customers’ networking problems daily. Trying to solve these problems quickly and efficiently – and get the maximum benefit out of available resources (time, engineers, etc.) – requires us to stay open to new techniques and tools that might improve our troubleshooting ability and decrease time and other resources spent, consequently increasing customer loyalty.
Recently a customer reported high packet delays between his central site and one of his remote sites. These two sites were connected over the Internet by secure IPSec tunnel, while some local traffic was destined directly to the Internet. Examination revealed high packet delays during certain times of day for traffic transmitted over the IPSec tunnel, whereas traffic transmitted directly to the Internet (i.e., ICMP, TCP, UDP) – and not secured by the IPSec tunnel – was not subject to high packet delays at any period of the day.
At first I thought that something might be wrong with the customer’s networking equipment, or that the customer was generating a high amount of traffic at certain times of the day. But a few weeks previously we had placed the customer’s network into the NIL Monitor system, which could provide long-term network statistics (e.g., interface and other resource utilization, packet delay, and so on). Looking at the statistics, I was astonished by the following graph, which revealed daily changing of packet delays over the previous month. After checking RAM usage and interface utilization, I determined that the problem was not in the customer’s network, but rather in the service provider’s network.
Because only encapsulated traffic encountered high packet delays, it was obvious that the encapsulated traffic (protocol ESP) was subject to quality of service issues in the provider’s network. After I informed the customer, he asked the service provider to fix the problem. At first, the provider insisted that nothing on his end would contribute to high packet delay. But after reviewing the previous graph, which revealed the periodic nature of the traffic delay, the service provider immediately solved the problem. The following graph speaks for itself.
Sometimes we need to take a few steps back when we can’t see the forest for the trees. I found NIL Monitor to be a powerful tool that helps me to see the whole problem from a distance, contributing to faster and more efficient problem-solving and finally to a higher level of customer satisfaction.