How to Troubleshoot Intermittent Network Issues After They Happen

Introduction  

One of the most frustrating challenges I faced as a network engineer was troubleshooting problems that had already disappeared.  

A user would call and report that an application was performing terribly. A site would become unusable for several minutes. Voice calls would break up. File transfers would suddenly slow to a crawl.  

Then, by the time I started investigating, everything would be working normally again.  

Over the years, I’ve learned that some of the most difficult problems to solve are not necessarily the most severe ones. They are the intermittent issues that appear unexpectedly, impact users, and then disappear before anyone can gather the evidence needed to understand what happened.  

Every network engineer has heard some version of:  

“It was broken this morning, but it seems fine now.”  

Users know something happened.  

Monitoring tools may show that performance degraded.  

Alerts may have been generated.  

But by the time troubleshooting begins, the application is working, the network appears healthy, and the evidence seems to have vanished.  

If the traffic was captured when the problem occurred, engineers can select the incident window, find the affected conversations, and analyze what happened after the issue has disappeared.  

Why Intermittent Issues Are So Difficult to Troubleshoot  

When an application is consistently slow, troubleshooting is usually more straightforward. Engineers can collect data while the issue is occurring, test hypotheses, and validate results in real time.  

Intermittent network issues are different.  

The issue may appear for a few minutes, then disappear for hours or even days. Users may experience serious disruption, but by the time IT starts collecting data, everything looks normal again.  

This often leads to a frustrating cycle:  

• Users report a problem  

• Monitoring confirms something happened  

• Engineers begin investigating  

• The issue disappears  

• No evidence exists to validate the root cause  

Many organizations attempt to solve this problem with ad hoc packet captures using tools like Wireshark.  

Don’t get me wrong, I still use Wireshark and consider it one of the most valuable troubleshooting tools ever created. The challenge is not Wireshark itself. The challenge is knowing where to capture, when to start capturing, and how to find the right packets when the issue may have lasted only a few minutes somewhere inside a network carrying millions, billions, or even trillions of packets.  

That is where traditional packet capture workflows can break down.  

Modern enterprise networks often carry tens to hundreds of gigabits of traffic per second. Capturing traffic continuously, storing it, and then finding the few packets relevant to a sporadic issue can quickly become overwhelming without the right tools and architecture.  

If the problem already happened, starting a packet capture now may not help.  

You need access to the packets from when the issue was actually occurring.  

A Six-Month Problem Solved in a Week  

One troubleshooting engagement has always stood out to me.  

An energy company was experiencing significant performance issues with its IBM Maximo application. The challenge was not simply that the application was slow. The challenge was that the issue was intermittent.  

Users would experience severe delays, report the problem, and then find that everything was functioning normally by the time troubleshooting began.  

The organization had been attempting to resolve the problem for nearly six months. Multiple teams had investigated the issue. Monitoring systems confirmed that users were experiencing delays. Engineers knew there was a problem. Nobody knew why.  

Teneo were brought in as part of a professional services engagement to help identify the root cause.  

Rather than relying on a single packet capture location, we deployed visibility at multiple points throughout the network. Packet aggregation technology was used to collect traffic close to the users, in the middle of the network, and near the application servers. Those packet streams were then synchronized and analyzed together.  

The goal was simple:  

Capture the evidence before the next occurrence disappeared.  

We continuously collected and retained packet data over an extended period while waiting for the issue to occur again.  

When it finally happened, we were able to go back in time and examine the exact conversations that occurred during the period users were reporting poor performance.  

That last part is important.  

We were not guessing from a dashboard after the fact.  

We were looking at the packets that actually made up the application conversations during the incident.  

The network itself was extremely complex and regularly exceeded 100 Gbps during peak operating periods. There was no realistic way someone was going to plug in a laptop running Wireshark, capture from one point in the network, and somehow find the root cause of an intermittent application problem affecting a large enterprise environment. It would be like looking for a needle in a thousand haystacks, except the haystacks are made of different types and sizes of needles, and you do not yet know which characteristic makes the one you need different from all the others. 

Historical packet analysis ultimately revealed what months of troubleshooting had failed to uncover: the performance issues were not being caused by a single problem, but by several issues occurring across different parts of the environment.  

The issue ultimately turned out to be far more complex than anyone initially suspected.  

Through historical packet analysis and packet replay, we identified several contributing factors, including virtual machine configuration issues impacting server performance, HTTP header size limitations on intermediary WAN optimization devices, latency introduced by intermittent bandwidth bursts, and perhaps most significantly, a client-side issue that resulted in intermittent resets on nearly 50% of connections passing through the load balancer.  

By examining synchronized packet captures from multiple locations and replaying the relevant traffic, we could see exactly where and when packets were being delayed, retransmitted, reset, or dropped.  

This was not a case of simply proving the application was at fault.  

It was not a case of simply proving the servers were at fault.  

It was not a case of simply proving the network was at fault.  

It was a combination of issues spanning multiple components of the environment, each contributing to the overall user experience.  

Within a week-long engagement, the root causes had been identified and validated.  

Without historical packet analysis, there is a good chance the organization would have continued chasing theories instead of working from evidence.  

Why Packet Replay Changes Everything  

One of the biggest challenges with intermittent issues is that engineers are often forced to investigate yesterday’s problem using today’s data.  

That rarely works.  

Historical packet analysis changes the conversation because it preserves the evidence.  

Packet replay allows teams to revisit traffic from a specific period and analyze conversations as if they were investigating the event in real time.  

This approach provides visibility into:  

• Application conversations  

• Packet loss  

• TCP retransmissions  

• Latency spikes  

• Network path behavior  

• Traffic bursts  

• Infrastructure events  

• Client-to-server communication patterns  

Most importantly, packet replay allows engineers to investigate the conditions that existed during the incident rather than attempting to recreate them later.  

In many real-world environments, intermittent issues are almost impossible to reproduce on demand. A load balancer behaves differently under peak load. A route changes for a few minutes. A storage backup creates unexpected traffic. A faulty device or cable introduces retransmissions only under specific conditions.  

If the packets were not captured when the issue occurred, the investigation can quickly turn into guesswork.  

If the packets were captured, retained, indexed, and made searchable, the investigation can continue even after the issue disappears.  

A Practical Historical Packet Analysis Workflow  

When an intermittent issue occurs, historical packet analysis provides a structured approach to investigation.  

Step 1: Identify When the Issue Occurred  

The first step is determining when users experienced degradation.  

This may come from:  

• User reports  

• Monitoring alerts  

• Application logs  

• Service desk tickets  

• Performance dashboards  

• DEX or endpoint experience data  

• Business process timestamps  

The goal is to establish the investigation window.  

You do not need to start by knowing the root cause. You need to know when the issue was reported and which users or applications were affected.  

Figure 1. Historical traffic analysis showing application activity during the selected investigation period.  

Step 2: Replay Traffic During the Event  

Once the timeframe is known, engineers can replay traffic captured during that period.  

Rather than looking at current conditions, they can analyze the exact communications that occurred while users were experiencing the issue.  

Engineers can review traffic patterns, connection behavior, response times, retransmissions, and packet loss from the actual incident window.  

Figure 2. Historical traffic replay showing application communications and response characteristics during a previously recorded event.  

Step 3: Isolate the Affected Conversations  

In high-speed environments, most packets are irrelevant to the specific issue being investigated. The objective is finding the specific users, applications, servers, and conversations associated with the incident.  

An engineer should not have to manually sift through massive PCAP files hoping to find the right conversation. The tool should make it possible to narrow the investigation quickly by time, IP address, protocol, application, server, user location, or other relevant attributes.  

Figure 3. Historical conversation analysis narrowing the investigation to the communications associated with the performance issue.  

Step 4: Correlate Traffic Across the Network  

Many intermittent problems cannot be solved from a single capture point. That was one of the most important lessons from the Maximo engagement.  

• If you only capture near the user, you may see retransmissions but not know where they started.  

• If you only capture near the application, you may see delayed traffic but not know whether the delay occurred upstream.  

But if you can correlate packets from multiple points in the network, the picture becomes much clearer. You can then determine whether packets left one location successfully, whether they arrived at another location, whether they were retransmitted along the way, and where loss or delay was introduced.  

This is especially important in complex networks with:  

• Load balancers  

• Multiple WAN paths  

• Data center interconnects  

• Cloud connectivity  

• Firewalls  

• Packet brokers  

• Segmented network zones  

• High-speed core infrastructure  

The goal is not simply to see that packet loss occurred. The goal is to determine where it occurred.  

Figure 4. Multi-point packet analysis showing traffic observed from different locations during the same application transaction.  

Step 5: Validate the Root Cause  

Once the relevant traffic has been isolated and correlated, engineers can use the packet data to determine what actually happened. That may include identifying packet loss, retransmissions, delayed responses, network path issues, application bottlenecks, infrastructure failures, load-balancing problems, or asymmetric traffic behavior. 

The value here is that the investigation can move beyond symptoms and high-level metrics to the actual communications that occurred during the problem. That makes it much easier for the right teams to focus on the specific component or part of the path that needs attention.  

How is AI Is Changing the Investigation Process? 

Historically, engineers had to manually sift through enormous packet repositories to locate the relatively small number of conversations associated with an incident.  

AI-assisted investigation capabilities are beginning to change that.  

Rather than requiring engineers to manually correlate thousands of conversations and performance indicators, AI can help identify anomalies, surface likely causes, and guide investigations toward the most relevant evidence.  

The quality of those insights depends heavily on the quality of the underlying data. Packet-derived metrics such as latency, retransmissions, packet loss, MTU behavior, TCP window advertisements, response times, and connection resets provide rich context that helps AI recognize patterns and highlight abnormal behavior.  

As AI continues to evolve, the combination of packet-level visibility, historical packet analysis, and packet replay provides the operational evidence needed to investigate complex intermittent issues with greater speed and confidence.  

Speed Matters More Than Ever 

Historical packet analysis is not simply about storing packets. It is about being able to find the right packets quickly enough to make them useful. 

Capturing traffic at modern network speeds is already challenging. As bandwidth increases, so does the volume of data that has to be written, retained, indexed, and analyzed. The real challenge often comes later, when engineers need to find a relatively small number of relevant conversations buried inside a massive amount of historical traffic. 

Fast search, packet indexing, historical visibility, and replay capabilities make that investigation practical. They let engineers narrow the scope quickly, focus on the traffic associated with the incident, and spend their time analyzing the evidence instead of hunting for it. 

That is the real value of historical packet analysis. It makes retained packet data something teams can actually use during an investigation, rather than simply a large archive of traffic.  

When Historical Packet Analysis Matters Most  

Not every issue requires historical packet replay.  

If a problem is happening right now, and it is easy to reproduce, a targeted packet capture may be enough.  

Historical packet analysis becomes especially valuable when problems are:  

• Intermittent  

• Difficult to reproduce  

• Reported after the fact  

• Spread across multiple network paths  

• Impacting only certain users or applications  

• Occurring in high-speed environments  

• Involving multiple technical teams  

• Buried inside large volumes of traffic  

These are the situations where many troubleshooting efforts stall.  

Not because the teams lack skill. Not because the tools they already have are useless. But because the evidence needed to validate the root cause is no longer available unless it was captured and retained.  

Conclusion  

Intermittent issues are difficult to investigate because the network and application may appear healthy by the time engineers begin troubleshooting. Historical packet analysis gives teams a way to return to the period when users experienced the problem, find the relevant conversations, and examine what happened across the observed network. 

In the energy company example, months of investigation had established that Maximo users were experiencing delays, but had not revealed why. Capturing traffic at multiple points and analyzing it after the next occurrence helped the team isolate several contributing issues. The evidence gave each team a clearer place to focus its investigation. 

That is the value of historical packet analysis: captured traffic remains available after an issue disappears, and search and correlation help engineers find the evidence they need within it. Instead of waiting for the problem to happen again, teams can investigate the event that already occurred.  

See Historical Packet Analysis in Action 

The problem may have disappeared, but your investigation doesn’t have to stop. 

See how Teneo’s Network Performance Monitoring & Diagnostics (NPMD), powered by Allegro Packets, helps engineers search historical traffic, isolate affected conversations, and analyze retained packet evidence to investigate intermittent network and application issues. 

Book a Teneo NPMD demo to see how the workflow helps your team move from “it was slow earlier” to a clearer understanding of what happened.