The Day the Scanners Stopped

In the world of high-volume logistics, silence is the scariest sound there is.

Years ago, I was air-dropped into the data center of a global logistics provider. Their primary sorting facility—the beating heart of a continent-wide supply chain—had gone quiet. This facility processed a peak of 8,000 scanner requests per minute. Every time a driver scanned a package, a request hit our backend. If the backend didn't answer, the driver couldn't move the box. The trucks didn't leave.

The dashboard said the system was "Green." The sorting floor said otherwise. The scanners were timing out, retrying, and creating a storm of traffic that eventually locked the entire system.

We didn't fix it by rewriting the application. We fixed it by understanding the physics of the runtime.

Here is the technical breakdown of how we diagnosed a "death by a thousand cuts" outage—and why today’s AI Architects are about to make the exact same mistakes.

The architecture was a classic high-availability setup for its time. Handheld scanners sent XML requests via REST to a load-balanced array of Apache Web Servers, which passed them to a cluster of Front End Communication Integration Servers (FEC IS). From there, data was pushed to a Front End Broker (FEC Broker) and then to a Back End Broker (BEP Broker) for asynchronous processing by the backend (BEP IS).

Under normal conditions, this facility effortlessly handled 8,000 requests per minute.

On the morning of the outage, an ISP failure severed connectivity for approximately one hour. When the connection was restored, the "thundering herd" arrived. Thousands of handheld scanners, which had been queuing transactions locally, simultaneously attempted to flush their backlogs.

The volume of "Process Shipment" requests—normally a manageable 3,000 per minute—hit the firewall in a single, instantaneous wave.

The latency climbed immediately. The scanners, programmed to be impatient, timed out and retried, adding duplicate loads to an already drowning system. Message sizes bloomed as metadata piled up. And then: deadlock.

We started with the thread dumps and the pidstat output. We weren't looking for a logic error; we were looking for resource exhaustion. We found three critical "smoking guns" that had turned a robust architecture into a fragile house of cards.

The production servers were running on a standard JVM configuration that was hostile to low-latency, high-throughput workloads.

Parallel GC is a "stop-the-world" collector. It prioritizes throughput over latency. With a MaxTenuringThreshold of only 2, short-lived objects (like the XML payload of a scan) were being promoted to the "Old Generation" of memory almost immediately. Perfect for long-running transactions; disastrous for massive amounts of short-lived ones.

Because the "Young Generation" was capped at 1GB, it filled up instantly during the burst. The system was spending more time pausing to clean up memory than it was processing packages. The "outage" wasn't a crash; it was a series of freezes while the JVM took out the trash.

While the JVM was gasping for air, the integration layer was suffocating it. We looked at the JMS Trigger configuration and found the default setting:

watt.server.jms.trigger.reuseSession=false

This innocuous setting meant that for every single one of those 8,000 requests per minute, the server was forcing a new session creation. We were burning CPU cycles just shaking hands with the broker.

Worse, Producer Caching was disabled. Every time the system tried to send an acknowledgement back to the scanner, it opened a new connection.

Finally, we looked at the trigger concurrency. The triggers were configured with 200-400 threads and a pre-fetch size of 10.

Do the math: 400 threads * 10 messages = 4,000 messages.

The server was eagerly pulling 4,000 messages into local memory per trigger. This flooded the heap, triggering the Garbage Collector, which paused the CPU, which caused the scanners to timeout, which caused them to retry.

It was the perfect storm.

We didn't deploy a single line of new application code to fix this. We tuned the engine.

1. We Changed the Physics of Memory We swapped the garbage collector to CMS (Concurrent Mark-Sweep), which cleans up memory concurrently without stopping the application threads.

2. We Stopped the Thrashing We enabled session reuse and connection caching.

3. We Throttled the Flood We reduced the thread count on the triggers. It sounds counter-intuitive to lower concurrency to increase speed, but by stopping the context switching, we allowed the CPU to focus on actual work.

The result? The latency line on the graph dropped like a stone. The silence ended. The facility roared back to life.

Why am I telling a story about JVM flags from 2016?

Because Physics is immutable.

Right now, I see Enterprise Architects designing "Agentic AI" systems that are making the exact same mistakes we made with the JVM. As I discussed in The Agentic Strangler, the rush to modernize often ignores the underlying plumbing.

We moved from Monoliths to Microservices, and now to Agents. The abstraction layers get higher, but the fundamentals of engineering—resource management, caching, and concurrency—remain the only things standing between a working system and a silent facility.

Stop building Chatbots. Start engineering Systems.

If you enjoyed this deep dive into the physics of runtime architecture, check out these related analyses on how to build safe, scalable Agentic systems: