Streaming an LLM response that arrives all at once

Eleven seconds of nothing, then the whole answer

The endpoint is a streaming endpoint. The route returns response()->stream(), the closure yields, the provider is streaming — and the browser shows an empty box for eleven seconds and then paints the complete answer in one frame.

Nothing failed. There is no error to grep for. The response is a 200 with the right body, delivered the way a normal slow request is delivered. Streaming is not a thing you switch on once; it is a property that has to survive four separate hops, and any one of them will happily collect the whole stream and hand it over as a single lump. Finding out which one is doing it is the whole job.

Work outward from the provider. The hops, in order: your HTTP client, PHP's output buffer, the web server or proxy, and the browser.

Hop one: your HTTP client already has the whole thing

This is the most common cause and the least visible, because the code reads as though it streams:

$response = Http::withToken(config('services.openai.key'))
    ->post('https://api.openai.com/v1/chat/completions', [
        'model' => 'gpt-4o',
        'stream' => true,
        'messages' => [['role' => 'user', 'content' => $prompt]],
    ]);

foreach (explode("\n\n", $response->body()) as $chunk) {
    // ...
}

'stream' => true in the payload tells the provider to stream. It says nothing about what Laravel's HTTP client does with the bytes as they arrive, and by default Guzzle reads the body to completion before Http::post() returns. By the time that foreach starts, generation finished seconds ago. You are iterating over a recording.

The switch is a Guzzle transfer option, not a payload key:

$response = Http::withOptions(['stream' => true])
    ->timeout(120)
    ->withToken(config('services.openai.key'))
    ->post('https://api.openai.com/v1/chat/completions', [...]);

$body = $response->toPsrResponse()->getBody();

Against a local server that emits three chunks 400ms apart, the difference is not subtle. With withOptions(['stream' => true]) the call returns once the headers land — 0.02 seconds — and the chunks surface at 0.02s, 0.42s and 0.83s as they arrive. Without it the same call blocks for 1.22 seconds and returns all 167 bytes at once. Same provider, same payload, same endpoint.

Reading it is manual work, because SSE frames do not align with TCP reads. Buffer, split on the blank line, strip the data: prefix:

$buffer = '';

while (! $body->eof()) {
    $buffer .= $body->read(1024);

    while (($break = strpos($buffer, "\n\n")) !== false) {
        $frame = trim(substr($buffer, 0, $break));
        $buffer = substr($buffer, $break + 2);

        $data = str_starts_with($frame, 'data: ') ? substr($frame, 6) : $frame;

        if ($data === '[DONE]') {
            return;
        }

        yield json_decode($data, true)['choices'][0]['delta']['content'] ?? '';
    }
}

Note the inner loop. A single read() can return two frames or half of one, and code that assumes one read equals one token drops content quietly on a fast connection.

Hop two: stream() only flushes when it gets a generator

This one is worth knowing exactly, because the two versions look interchangeable and behave completely differently.

response()->stream() inspects your callback with ReflectionFunction::isGenerator(). If the callback is a generator — if its body contains yield — Laravel wraps it in a loop that echoes each chunk, calls ob_flush() when a buffer is active, calls flush(), and merges X-Accel-Buffering: no into the headers. If the callback is an ordinary closure that echoes, you get none of that. Not the flushing, and not the header.

// No flushing, no X-Accel-Buffering header. This is a normal buffered response.
return response()->stream(function () use ($tokens) {
    foreach ($tokens as $token) {
        echo $token;
    }
});

// Flushed per chunk, X-Accel-Buffering: no added for you.
return response()->stream(function () use ($tokens) {
    foreach ($tokens as $token) {
        yield $token;
    }
});

Check it on a response object rather than trusting the shape of the code — the closure version comes back with no X-Accel-Buffering header at all, the generator version with no. If you have inherited a streaming endpoint that does not stream, look for echo where a yield was intended. The older hand-rolled form still works, and the docs still show it, but then the ob_flush(), the flush() and the header are yours to remember.

Hop three: eventStream writes the SSE framing, including the end sentinel

If the client is an EventSource, do not assemble the protocol by hand. response()->eventStream() takes a generator and emits properly framed events, and it sets the three headers that matter: Content-Type: text/event-stream, Cache-Control: no-cache, and X-Accel-Buffering: no.

Route::post('/chat', function (Request $request) {
    return response()->eventStream(function () use ($request) {
        foreach (streamCompletion($request->input('prompt')) as $token) {
            yield $token;
        }

        yield new StreamedEvent('usage', ['total_tokens' => 1200]);
    });
});

Every yielded value goes out as event: update unless you yield a StreamedEvent, which lets you name the event and send structured data — anything that is not a string or a number is JSON encoded for you. When the generator finishes, Laravel sends one last frame containing the literal string </stream>:

event: update
data: token one

event: usage
data: {"total_tokens":1200}

event: update
data: </stream>

That sentinel is the contract with the client. EventSource will reconnect on its own unless you close it, so the browser side has to watch for it:

const source = new EventSource('/chat');

source.addEventListener('update', (event) => {
    if (event.data === '</stream>') {
        source.close();

        return;
    }

    appendToken(event.data);
});

The failure you will not see: an exception mid-stream is still a 200

This is the part that costs an afternoon. The generator inside eventStream() runs wrapped in try { ... } catch (Throwable $e) { report($e); }. The exception is reported and then swallowed.

Think about when that happens. The status line went out with the first byte, long before anything failed, so the response is a 200 and there is no way to change it. The tokens generated before the throw are already on the wire. The </stream> sentinel is inside the try, after the loop — so it is skipped. The client gets a truncated answer, no error, and a connection that just stops.

Run a generator that yields one token and then throws, and the response is status 200 with a body containing exactly that one token and no sentinel. Your exception handler logs it; your user reads a sentence that ends mid-word and assumes the model is bad.

So a streaming endpoint needs its own error channel, inside the stream:

return response()->eventStream(function () use ($prompt) {
    try {
        foreach (streamCompletion($prompt) as $token) {
            yield $token;
        }
    } catch (RequestException $e) {
        report($e);

        yield new StreamedEvent('error', ['message' => 'The model stopped responding.']);
    }
});

Catch inside the generator, yield an error event, let the stream terminate normally so the sentinel is sent. On the client, treat "the connection closed without </stream>" as a failure in its own right, because that is the only signal a dropped upstream gives you.

Hop four: the buffer that is not yours

You can get every line above right and still ship a response that lands in one piece, because something between PHP and the browser is accumulating it.

X-Accel-Buffering: no is an nginx instruction, and only nginx reads it. It disables proxy_buffering for that response, which is the single most common cause. It does nothing for anything else in the path:

• gzip. Compression is inherently buffered — the encoder holds bytes until it has enough to compress. Exclude text/event-stream from the compressed types.
• A CDN or load balancer. Many proxy layers buffer responses by default and have their own setting for it, unrelated to the nginx header.
• PHP-FPM and output_buffering. Laravel calls ob_flush() only when ob_get_level() > 0, then always calls flush(). Nested buffers still added by a middleware that captures output will hold everything until the response ends.
• Any middleware that reads the response body. Wrapping a StreamedResponse in something that inspects its content forces it to materialise, which is the end of the stream.

Test the endpoint with curl -N against the app server directly, then through the proxy. If it streams on one and not the other, you have located the buffer without touching PHP:

curl -N -X POST http://127.0.0.1:8000/chat -d 'prompt=hello'

What to change

Go and look at the streaming endpoint you already have. Does the outbound call pass withOptions(['stream' => true]), or only 'stream' => true in the JSON payload? Does the stream() callback actually yield, or does it echo? Does the client close its EventSource on </stream>, and does it treat a close without that sentinel as an error rather than a completion? Is text/event-stream excluded from gzip?

And if the answer to all of those is yes and it still arrives in one lump, run curl -N against the app server before you change any more PHP. The token is being held somewhere, and four times out of five it is not being held by your code.