CyberSecurity

The 11-Byte Attack That Can Freeze an OpenSSL Server’s Memory

Published

on

Eleven Bytes, 131 KB of Frozen Memory

Eleven bytes. That’s all it takes to make an unpatched OpenSSL server set aside up to 131 KB of memory for a message that never arrives. On glibc systems, that memory stays locked until the process restarts. Not great for a production server.

This is the HollowByte flaw, a denial-of-service bug that Okta’s Red Team found, named, and reported. The team published its findings after OpenSSL shipped a fix — quietly, with no CVE, no advisory, and no changelog entry pointing at it.

So how does a tiny TLS request cause so much damage? The trick lies in how OpenSSL handles certain message fragments. A crafted 11-byte request triggers an allocation that never gets freed. Repeat it enough times, and you’ve got a memory leak that grinds the server to a halt.

What Exactly Is HollowByte?

HollowByte is a memory exhaustion vulnerability in OpenSSL’s TLS handling. It doesn’t require authentication. It doesn’t need special privileges. Just a network connection and a carefully constructed request.

Okta’s Red Team discovered that sending a specific 11-byte TLS message causes the server to allocate memory for a response that never comes. On systems using glibc — the standard C library on most Linux distributions — that allocated memory isn’t reclaimed. It sits there, frozen, until the process dies.

The impact? An attacker can send repeated requests to exhaust available memory, effectively freezing the server. It’s a classic DoS vector, but with a twist: the trigger is absurdly small.

Why glibc Makes It Worse

The memory behavior isn’t universal. On some systems, the allocation gets cleaned up. But glibc’s allocator handles certain patterns differently, and that’s where the freeze happens. Okta’s testing showed the memory staying put until restart — no garbage collection, no cleanup, just a slow leak that compounds.

For organizations running OpenSSL on glibc-based systems, this is a real problem. A single connection isn’t dangerous. Thousands of them? That’s a different story.

OpenSSL’s Quiet June Fix

Here’s the part that’s raising eyebrows. OpenSSL patched HollowByte in June — but did it without a CVE, without a security advisory, and without a changelog entry that mentions the vulnerability.

That’s unusual. Security fixes typically get publicized so administrators know to update. A silent patch means many systems remain vulnerable, simply because nobody knows to apply the update.

Okta’s Red Team, which reported the bug and gave it the HollowByte name, published its research to fill that gap. The disclosure includes technical details on how the attack works and which versions are affected.

Who’s Affected and What to Do

If you’re running OpenSSL on a glibc-based system, you need to check your version. The fix shipped in June, so any version before that is vulnerable. The exact version numbers are in Okta’s disclosure.

Here’s what to do right now:

  • Update OpenSSL to the latest patched version. Don’t wait for a CVE announcement.
  • Check your changelog — if you’re on a version from June or later, verify it includes the fix.
  • Monitor memory usage on TLS-facing servers. Unexpected spikes could indicate an attack.
  • Restrict network access to TLS endpoints where possible, limiting who can send requests.

The update itself is straightforward. The challenge is knowing you need it.

The Bigger Problem: Silent Security Fixes

HollowByte highlights a broader issue in open-source security: fixes without fanfare. When a vulnerability is patched silently, the window of exposure stretches. Attackers who reverse-engineer the patch can exploit systems that haven’t updated — and they’ll do it before the news spreads.

Okta’s decision to publish the research after the patch is a pragmatic move. It alerts the community while giving administrators a heads-up. But it also raises questions: how many other HollowByte-style flaws are out there, patched but unannounced?

For security teams, the lesson is clear. Don’t rely solely on CVE alerts. Regularly audit your dependencies, track upstream changes, and test for unusual behavior. A silent patch is still a patch — but only if you apply it.

Interested in related security topics? Check out our guides on TLS certificate management and denial-of-service attack prevention for more context on keeping your infrastructure safe.

Leave a Reply

Your email address will not be published. Required fields are marked *

Trending

Exit mobile version