If you have ever run a fuzzer like AFL++ on a file parser (like libpng or ffmpeg), you know the drill: you throw mutated bits at a binary until it crashes. But try to point that same logic at a stateful network server, and you will hit a wall.

Network fuzzing is a systematic approach to discovering security and robustness flaws in software that communicates over structured protocols. It differs from file fuzzing or command-line fuzzing because network protocols involve timing, ordering, and interaction among multiple peers. Recent surveys cover the landscape of protocol fuzzing and stateful system testing in depth (Zhang et al., 2024; Daniele et al., 2024).

In this post we’re going to look at why naïve random mutation fails against network protocols, and how modern stateful fuzzing architectures address this using structure modeling and deterministic harnessing. We will also look at practical examples targeting the MQTT protocol (OASIS, 2021).

Here’s the flow:

  • Why network fuzzing fails with naïve mutation.
  • How to model syntax and enforce determinism.
  • Three practical MQTT setups: white-box, black-box, and gray-box.

The Problem: It’s Not Just About the Bytes

In file fuzzing, the input is static. In network fuzzing, the input is a conversation.

As defined in modern System Under Test (SUT) models, a network protocol is a state machine. If your fuzzer sends a valid GET_DATA packet, but sends it before the HANDSHAKE is complete, the server won’t crash, it will just close the connection. The server logic is correct; the fuzzer is just “speaking out of turn.”

There are four specific challenges that make this difficult:

  1. Statefulness: Each message is interpreted relative to prior state. Without valid handshakes, most messages are rejected, so random mutation remains stuck in the connection setup phase.
  2. Structure: Messages contain complex fields whose values depend on each other. Checksums, length prefixes, and magic bytes create dependencies. If you flip a bit in the payload but don’t update the CRC32, the packet is dropped before it ever hits the vulnerable logic.
  3. Asynchrony: Network I/O, timers, and multithreading introduce nondeterminism. A timeout may arise from scheduling rather than input semantics, producing noise in feedback.
  4. Diversity: Implementations interpret the same specification differently. Vendors add extensions or tolerate errors differently, requiring fuzzers that remain robust under dialect variation.

Syntax Modeling

To get past the “Structure” challenge, we cannot simply cat /dev/urandom into a socket. We need to model the protocol syntax.

When a formal grammar isn’t available, we often use tools like Scapy (Python) to define the packet boundaries. This allows the fuzzer to mutate fields rather than bytes.

Here is an example of modeling a custom binary protocol using Python. Instead of fuzzing the raw stream, we fuzz the fields, and let the generator handle the “hard” constraints like Length and Checksum.

class MyCustomProto(Packet):
    name = "MyProto"
    fields_desc = [
        # The fuzzer shouldn't mutate this randomly,
        # or the parser rejects it immediately.
        IntField("magic", 0xDEADBEEF),

        # Field dependencies: The length must match the payload
        ShortField("len", None),

        # This is where we want the fuzzer to go wild
        StrLenField("payload", "", length_from=lambda x: x.len),

        # Checksum must be recalculated on every mutation
        IntField("checksum", None)
    ]

    def post_build(self, p, pay):
        # Automatically fix length and checksum before sending
        if self.len is None:
            l = len(p) - 8 # simplified calculation
            p = p[:4] + struct.pack("!H", l) + p[6:]
        if self.checksum is None:
            ck = crc32(p[:-4])
            p = p[:-4] + struct.pack("!I", ck)
        return p + pay

By using a model like this, we ensure that 100% of our generated test cases pass the initial packet validation checks. This sharp reduction in “wasted executions” allows the fuzzer to reach semantic logic much faster.

The Harness: Enforcing Determinism

A major bottleneck in network fuzzing is the kernel. Opening a TCP socket, performing a handshake, and waiting for recv() is incredibly slow (in CPU time) and introduces timing noise. If a test case crashes the server, can you reproduce it? If the network was laggy, was the crash caused by the packet content or the delay?

Modern approaches (like NSFuzz, AFLNet, or StateAFL) often use “desocketing” or LD_PRELOAD hooks to replace network calls with memory buffers (Qin et al., 2022; Pham et al., 2020; Natella, 2022). Also lock down reproducibility: fix seeds, avoid background threads, and use deterministic timers where possible.

Here is a conceptual C++ harness compatible with LibFuzzer. Instead of running the server and connecting via localhost, we link directly against the server library and feed data directly to the parsing function.

// harness.cc
#include "server_lib.h"
#include <stdint.h>
#include <stddef.h>

extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
    // 1. We must ensure every run starts from a clean slate to maintain
    // auditability.
    ServerState* SUT = server_init();

    // 2. We inject the data directly into the library
    server_process_buffer(SUT, data, size);

    // 3. Collect Feedback / Clean up
    server_teardown(SUT);

    return 0;
}

This harness controls Timing (no network waits), Isolation (fresh memory), and Synchronization (single-threaded execution).

The Feedback Loop

Once we have structured inputs and a deterministic harness, we need feedback. Classical greybox fuzzers use edge coverage. Protocol fuzzers extend this with state coverage (Pham et al., 2020; Natella, 2022):

  • Response-based state modeling: Clustering responses (e.g., 200 OK vs 500 Error) to infer the protocol state machine.
  • Variable-based state modeling: Instrumenting the SUT to watch key variables, like session_state.

The scheduler uses this info to prioritize inputs. If a specific sequence of messages transitions the server from INIT to AUTHENTICATED, the fuzzer saves that sequence and uses it as a prefix for future mutations.

Here is a quick comparison of the three approaches:

ApproachStructureStateThroughputSetup effort
White-box (AFL++)OptionalOptionalHighHigh
Black-box (Boofuzz)RequiredRequiredLowMedium
Gray-box (AFLNet)OptionalRequiredMediumHigh

Fuzzing MQTT

To demonstrate these concepts, let’s look at two practical approaches to fuzzing the MQTT protocol. The MQTT specification and its security guidance provide the baseline protocol behavior and constraints we are testing against (OASIS MQTT 5.0 and NIST guidance).

Whitish-Box: AFL++ (Source Available)

In a whitish-box scenario, we have access to the source code (e.g., the Mosquitto broker). Our goal is to bypass the operating system’s networking stack entirely to increase throughput and determinism.

By using socketpair, we create a bidirectional communication channel that resides entirely in memory. The fuzzer writes mutated data to one end of the pair (sv[1]), and we trick the Mosquitto instance into reading from the other end (sv[0]). To Mosquitto, it looks like a standard network client, but to the kernel, it is just a memory copy. This allows us to use standard sanitizers (like ASAN) to detect memory corruption immediately, without the flakiness of timeouts or dropped packets.

#include <stdint.h>
#include <stddef.h>
#include <unistd.h>
#include <sys/socket.h>
#include <fcntl.h>

extern "C" {
#include "mosquitto.h"
#include "mosquitto_internal.h"
#include "packet_mosq.h"
}

static int g_initialized = 0;

extern "C" int LLVMFuzzerInitialize(int *argc, char ***argv) {
    if (mosquitto_lib_init() != MOSQ_ERR_SUCCESS) return 0;
    g_initialized = 1;
    return 0;
}

extern "C" int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) {
    if (!g_initialized || size < 2 || size > 65535) return 0;

    // Create a fresh Mosquitto instance for every iteration
    struct mosquitto *mosq = mosquitto_new(NULL, true, NULL);
    if (!mosq) return 0;

    // Create a socket pair to mock the network connection
    // sv[0] is for the server, sv[1] is for the fuzzer
    int sv[2];
    if (socketpair(AF_UNIX, SOCK_STREAM, 0, sv) == -1) {
        mosquitto_destroy(mosq);
        return 0;
    }

    // Write the fuzzer's mutation directly into the socket buffer
    write(sv[1], data, size);

    // Assign the server-side socket to the mosquitto instance
    mosq->sock = sv[0];
    mosq->protocol = mosq_p_mqtt5;
    mosq->state = mosq_cs_connected;

    // Execute the protocol parsing logic synchronously
    packet__read(mosq);

    close(sv[0]); close(sv[1]);
    mosquitto_destroy(mosq);
    return 0;
}

Running this harness allows AFL++ to fuzz the Mosquitto broker at speeds exceeding 8,000 executions per second on a single core, as shown in the status screen below:

       american fuzzy lop ++4.35a {default} (./fuzz_mosquitto) [explore]
┌─ process timing ────────────────────────────────────┬─ overall results ────┐
│        run time : 0 days, 0 hrs, 3 min, 37 sec      │  cycles done : 30    │
│   last new find : 0 days, 0 hrs, 3 min, 28 sec      │ corpus count : 548   │
│last saved crash : 0 days, 0 hrs, 3 min, 11 sec      │saved crashes : 3     │
│ last saved hang : none seen yet                     │  saved hangs : 0     │
├─ cycle progress ─────────────────────┬─ map coverage┴──────────────────────┤
│  now processing : 78.97 (14.2%)      │    map density : 3.98% / 17.74%     │
│  runs timed out : 0 (0.00%)          │ count coverage : 2.81 bits/tuple    │
├─ stage progress ─────────────────────┼─ findings in depth ─────────────────┤
│  now trying : havoc                  │ favored items : 101 (18.43%)        │
│ stage execs : 73/100 (73.00%)        │  new edges on : 163 (29.74%)        │
│ total execs : 1.76M                  │ total crashes : 71 (3 saved)        │
│  exec speed : 8048/sec               │  total tmouts : 14 (0 saved)        │
├─ fuzzing strategy yields ────────────┴─────────────┬─ item geometry ───────┤
│   bit flips : 1/96, 1/95, 1/93                     │    levels : 2         │
│  byte flips : 0/12, 1/11, 2/9                      │   pending : 0         │
│ arithmetics : 1/812, 0/1344, 0/1032                │  pend fav : 0         │
│  known ints : 0/99, 2/386, 0/484                   │ own finds : 44        │
│  dictionary : 0/228, 0/247, 0/0, 0/0               │  imported : 425       │
│havoc/splice : 36/1.71M, 0/0                        │ stability : 100.00%   │
│py/custom/rq : unused, unused, unused, unused       ├───────────────────────┘
│    trim/eff : 0.66%/38.8k, 83.33%                  │          [cpu000: 31%]
└─ strategy: explore ────────── state: started :-) ──┘

However, this approach has limitations. It requires access to source code and often necessitates significant changes to the build process to link statically against the target library. It also mocks out the network layer, which means bugs specific to the actual socket handling (like epoll race conditions) might be missed.

Black-Box: Boofuzz

When source code is unavailable, we cannot bypass the network stack. Instead, we use tools like Boofuzz to model the protocol grammar and state machine explicitly. Related black-box work like Snipuzz shows how far message-structure inference can go without source access (Feng et al., 2021).

In this script, we solve two problems:

  1. Structure: We define a mqtt_varlen_encoder to handle MQTT’s variable-length integers automatically. If the fuzzer expands the payload, this encoder ensures the length field remains valid so the packet is accepted by the broker.
  2. State: We use session.connect to define a graph. The fuzzer learns that it must successfully send a MQTT_CONNECT packet before it attempts to send a MQTT_PUBLISH packet.
def mqtt_varlen_encoder(value):
    n = int.from_bytes(value, byteorder="big", signed=False) if value else 0
    if n < 0 or n > 268_435_455: raise ValueError(f"Remaining Length out of range for MQTT varint: {n}")
    out = bytearray()
    while True:
        encoded = n % 128
        n //= 128
        if n > 0: encoded |= 0x80
        out.append(encoded)
        if n == 0: break
    if len(out) > 4: raise ValueError("MQTT varint produced >4 bytes, which is invalid.")
    return bytes(out)

def build_mqtt_packet(name: str, control_header: Union[int, dict], variable_header_fields=None, payload_fields=None):
    variable_header_fields = variable_header_fields or []
    payload_fields = payload_fields or []

    def build_fields(field_defs):
        elements = []
        for f in field_defs:
            ftype, fname, fval, fuzzable, endian, max_len = f.get("type"), f.get("name"), f.get("value", 0), f.get("fuzzable", True), f.get("endian", "big"), f.get("max_len", None)

            if ftype == "group":
                values, default_value = f.get("values", []), f.get("default_value", None)
                elements.append(Group(name=fname, values=values, default_value=default_value, fuzzable=fuzzable))
            elif ftype == "byte": elements.append(Byte(name=fname, default_value=fval, fuzzable=fuzzable))
            elif ftype == "word": elements.append(Word(name=fname, default_value=fval, endian=endian, fuzzable=fuzzable))
            elif ftype == "string":
                elements.append(Size(name=f"{fname}_len", block_name=fname, length=2, endian=">", fuzzable=False))
                elements.append(String(name=fname, default_value=fval, fuzzable=fuzzable, max_len=max_len))
            elif ftype == "raw": elements.append(Bytes(name=fname, default_value=fval, fuzzable=fuzzable))

        return elements

    if type(control_header) == dict:
        fvalues, fdef, ffuzz = control_header.get("values", None), control_header.get("default_value", None), control_header.get("fuzzable", False)
        if fvalues == None and fdef == None:
            print("[FATAL] At least values or default value has to be specified for the control header")
            exit()
        ch = Group(name="ControlHeader", values=fvalues, default_value=fdef, fuzzable=ffuzz)
    else:
        ch = Byte(name="ControlHeader", default_value=control_header, fuzzable=False)

    return Request(name, children=(
        Block(name="FixedHeader", children=(
            ch, # Control Header
            Block(name="RemainingLength", children=Size(name="RemainingLengthRaw", block_name="Body", fuzzable=False, length=4, endian=">"), encoder=mqtt_varlen_encoder, fuzzable=False)
        )),
        Block(name="Body", children=(
            Block(name="VariableHeader", children=build_fields(variable_header_fields)),
            Block(name="Payload", children=build_fields(payload_fields))
        ))
    ))

def build_connect_request():
    variable_header = [
        {"type": "string", "name": "ProtocolName", "value": "MQTT", "fuzzable": False},
        {"type": "byte", "name": "ProtocolLevel", "value": 5, "fuzzable": False},
        {"type": "byte", "name": "ConnectFlags", "value": 0x02, "fuzzable": False},
        {"type": "word", "name": "KeepAlive", "value": 60},
        {"type": "byte", "name": "PropertiesLength", "value": 0, "fuzzable": False},
    ]

    payload = [
        {"type": "string", "name": "ClientID", "value": "boofuzz", "max_len": 30}
    ]
    return build_mqtt_packet("MQTT_CONNECT", 0x10, variable_header, payload)

# ... ... ...

The session.connect calls internally create a Finite State Machine representing the protocol evolution. This can be shown with session.render_graph_graphviz().create_png().

The output from my script produced the following

MQTT State Graph

Boofuzz generates a web/tui interface to track progress that provides immediate feedback on the state protocol graph traversal:

# NOTE: this is an extract of the tui as the output was longer and not really
# usefull to paste it here

│[2025-12-29] Test Case: 1460: MQTT_CONNECT->MQTT_SUBSCRIBE:[MQTT_SUBSCRIBE.Body.Payload.TopicFilter:751]
│[2025-12-29]     Info: Type: String
│[2025-12-29]     Info: Opening target connection (127.0.0.1:1883)...
│[2025-12-29]     Info: Connection opened.
│[2025-12-29]    Test Step: Monitor CallbackMonitor#140523418095968[pre=[],post=[],restart=[],post_start_target=[]].pre_send()
│[2025-12-29]    Test Step: Transmit Prep Node 'MQTT_CONNECT'
│[2025-12-29]     Info: Sending 22 bytes...
│[2025-12-29]     Transmitted 22 bytes: 10 14 00 04 4d 51 54 54 05 02 00 3c 00 00 07 62 6f 6f 66 75 7a 7a b'\x10\x14\x00\x04MQTT\x05\x02\x00<\x00\x00\x07boofuzz'
│[2025-12-29]     Info: Receiving...
│[2025-12-29]     Received: 20 09 00 00 06 22 00 0a 21 00 14 b' \t\x00\x00\x06"\x00\n!\x00\x14'
│[2025-12-29]    Test Step: Callback function 'conn_callback'
│[2025-12-29]     Info: Received CONNACK: 200900000622000a210014
│[2025-12-29]    Test Step: Fuzzing Node 'MQTT_SUBSCRIBE'
│[2025-12-29]     Info: Sending 1000010 bytes...
│[2025-12-29]     Error!!!! SIGINT received ... exiting

While Boofuzz is excellent for logic and state testing, it is significantly slower than AFL++ because it operates over the real network stack. It typically manages 10-50 executions per second compared to AFL++’s thousands. Furthermore, defining the protocol grammar manually is time-consuming and prone to errors if the specification is complex.

Gray-Box: AFLNet

//TBD

Conclusion

We covered why naïve mutation fails on protocols, how syntax modeling keeps inputs valid, and how deterministic harnesses reduce noise so feedback can guide exploration. The MQTT examples show the trade-offs: white-box harnesses buy speed, black-box frameworks buy reach, and gray-box fuzzers bridge the two. The practical takeaway is simple: start by making inputs valid enough to reach deep logic, then let coverage or state feedback do the exploration. LLM-guided approaches are emerging as another way to learn protocol structure and guide mutations (Meng et al., 2024).

References

  • Zhang et al. (2024), A Survey of Protocol Fuzzing, ACM Computing Surveys.
  • Daniele et al. (2024), Fuzzers for Stateful Systems: Survey and Research Directions, ACM Computing Surveys.
  • Qin et al. (2022), NSFuzz: Towards Efficient and State-Aware Network Service Fuzzing, Proceedings of the 1st Fuzzing Workshop.
  • Pham et al. (2020), AFLNet: A Greybox Fuzzer for Network Protocols, IEEE ICST.
  • Natella (2022), StateAFL: Greybox Fuzzing for Stateful Network Servers, Empirical Software Engineering.
  • Meng et al. (2024), Large Language Model Guided Protocol Fuzzing, NDSS.
  • Feng et al. (2021), Snipuzz: Black-box Fuzzing of IoT Firmware via Message Snippet Inference, ACM CCS.
  • OASIS MQTT Version 5.0 Specification .
  • OASIS MQTT NIST Cybersecurity .