⬡ Dev Tools/2026-04-13Advanced

LLM Streaming in Rork Apps: Building ChatGPT-Style Real-Time AI Responses with Expo and SSE

Field notes on shipping LLM streaming (SSE) in React Native and Expo. Anthropic, OpenAI and Gemini behind one interface, AbortController cancellation and retries, a Cloudflare Workers proxy, context compaction, and a mock SSE server for testing at zero API cost.

streamingSSELLM³ AI³¹ Anthropic⁴ OpenAI⁵ React Native²³⁹ Expo²¹⁹ Cloudflare Workers²⁵ Rork⁵⁸⁵

✦ Premium Article

Twelve seconds of a spinner. That was the first version of the AI chat in my app: tap send, watch a circle rotate, and by the time the answer finally landed the tester's thumb was already reaching for the home button. The model wasn't slow. The design was — waiting for the full generation before showing anything turns the entire wait into dead air.

LLM streaming, powered by Server-Sent Events (SSE), solves this at the source. Instead of waiting for the model to finish generating, tokens stream to the client as they're produced. Users see text flowing within milliseconds, and that psychological shift — from "waiting" to "watching it think" — has a measurable impact on retention.

The catch is that streaming in React Native requires a fundamentally different approach than the browser. EventSource doesn't exist in the RN runtime. Reading a ReadableStream from fetch requires specific handling. Wiring up AbortController correctly is trickier than it looks. This guide covers all of it, with production-tested code you can drop into your Rork project.

Why Streaming Is a UX Inflection Point

Before diving into implementation, it's worth understanding what's actually happening from a user perspective.

With blocking (non-streaming) responses, the pattern is always the same: send a message, sit through silence, receive a wall of text. Short answers survive it. The longer, more considered responses that make an LLM worth paying for do not — asking Claude Sonnet for a few hundred words routinely put me in the 15–20 second range.

You don't need analytics to see what that does. Watch someone use it. Around five seconds they glance away from the screen. Around ten they switch apps. When they come back they've forgotten what they asked. The conversation stops before any retention metric has a chance to record it.

Streaming collapses this experience. The first token arrives in under a second. The user's cognitive state shifts from passive waiting to active reading. Even if the total generation time is identical, the perceived speed is dramatically different.

There's also a practical cost angle. When you implement streaming with proper cancellation, users who get enough of an answer midway can stop generation. That cuts API costs on incomplete requests. Depending on your use case, this can reduce monthly LLM spend by 30–50%.

The Core Challenge: React Native Has No EventSource

If you've implemented SSE in a web app, your first instinct might be to reach for EventSource:

// Works in browsers — fails silently in React Native
const es = new EventSource('https://api.anthropic.com/v1/messages');
es.onmessage = (e) => console.log(e.data);

In React Native, this throws or does nothing depending on your setup. The EventSource Web API simply isn't part of the RN runtime.

Streaming means reading a ReadableStream off a fetch() response. And this is where the second trap sits — the one that cost me half a day.

React Native's global fetch does not give you response.body. RN's fetch is a polyfill layered on XMLHttpRequest, and stream support was never implemented (facebook/react-native#27741 has been open for years). Upgrading your Expo SDK does not change this.

What you want is expo/fetch, added in Expo SDK 52. It is a separate WinterCG-compliant implementation that you pull in as a named import. It does not replace the global, so if you forget the import line your code quietly falls back to the old implementation — and that is genuinely hard to spot.

// ❌ The global fetch — the request succeeds, but body is undefined
const bad = await fetch(url, { method: 'POST', body });
bad.body; // → undefined (RN polyfill)
 
// ✅ Expo SDK 52+. The named import is not optional
import { fetch } from 'expo/fetch';
 
const response = await fetch('https://api.anthropic.com/v1/messages', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'anthropic-version': '2023-06-01',
    'x-api-key': 'YOUR_API_KEY', // Use a proxy in production — more on this below
  },
  body: JSON.stringify({
    model: 'claude-haiku-4-5-20251001',
    max_tokens: 1024,
    messages: [{ role: 'user', content: 'Hello' }],
    stream: true,
  }),
});
 
const reader = response.body?.getReader();
const decoder = new TextDecoder();
 
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  const chunk = decoder.decode(value, { stream: true });
  // Parse SSE events from chunk
}

When response.body comes back as undefined, it is almost always one of three causes.

Symptom	Cause	Fix
`body` is undefined, no error thrown	You are on the global fetch	Add `import { fetch } from 'expo/fetch';`
Still undefined after importing	Expo SDK 51 or earlier	Upgrade to SDK 52+, or fall back to a library such as `react-native-sse`
Works on web, fails on device only	`EXPO_PUBLIC_USE_RN_FETCH=1` is set	That flag restores the RN implementation as the global. Named imports are unaffected, so check your import path first

Rork scaffolds projects on a recent SDK, but whether the generated code actually calls expo/fetch depends on what it wrote for you. When adding streaming, read the import line with your own eyes before anything else.

Parsing SSE Format Correctly

The SSE format that Anthropic sends looks like this:

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":", world"}}

event: message_stop
data: {"type":"message_stop"}

OpenAI uses a slightly different format:

data: {"id":"chatcmpl-xxx","object":"chat.completion.chunk","choices":[{"delta":{"content":"Hello"},"index":0}]}

data: [DONE]

A critical detail: one reader.read() call can return multiple SSE events, and a single event can span across multiple reads. If you parse each chunk directly without buffering, you'll see intermittent JSON parse errors in production that are maddeningly hard to reproduce.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦Developers stuck on SSE fetch implementation in React Native can now run LLM streaming in production starting today

✦You'll get copy-paste-ready code that covers AbortController cancellation, error retries, and API cost optimization as a complete set

✦You'll learn a multi-provider abstraction pattern covering Anthropic, OpenAI, and Gemini — enabling provider switching at runtime without a single line of app code change

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

Production-Ready Streaming Hook

Here's the full useStreamingChat hook used in production Rork projects. It handles buffer management, AbortController cancellation, exponential backoff retries, and status tracking.

// hooks/useStreamingChat.ts
import { useState, useRef, useCallback } from 'react';
// ⚠️ The global fetch cannot stream. This named import is required.
import { fetch } from 'expo/fetch';
 
type Message = {
  role: 'user' | 'assistant';
  content: string;
};
 
type StreamingState = {
  messages: Message[];
  isStreaming: boolean;
  error: string | null;
  sendMessage: (text: string) => Promise<void>;
  stopStreaming: () => void;
  resetConversation: () => void;
};
 
// Always proxy through your backend — never put API keys in the app binary
const PROXY_URL = 'https://your-worker.your-subdomain.workers.dev/api/chat';
 
// For error events that arrive after an HTTP 200. A dedicated class keeps them
// distinguishable from ordinary per-chunk JSON parse failures.
class StreamAbortedError extends Error {
  constructor(message: string) {
    super(message);
    this.name = 'StreamAbortedError';
  }
}
 
export function useStreamingChat(): StreamingState {
  const [messages, setMessages] = useState<Message[]>([]);
  const [isStreaming, setIsStreaming] = useState(false);
  const [error, setError] = useState<string | null>(null);
  const abortControllerRef = useRef<AbortController | null>(null);
  // Whether we've painted at least one character. Gates retry eligibility.
  const hasEmittedRef = useRef(false);
 
  const stopStreaming = useCallback(() => {
    if (abortControllerRef.current) {
      abortControllerRef.current.abort();
      abortControllerRef.current = null;
    }
    setIsStreaming(false);
  }, []);
 
  const sendMessage = useCallback(async (text: string) => {
    if (isStreaming) return; // Prevent duplicate sends
 
    const userMessage: Message = { role: 'user', content: text };
    const updatedMessages = [...messages, userMessage];
    setMessages(updatedMessages);
    setIsStreaming(true);
    setError(null);
 
    const abortController = new AbortController();
    abortControllerRef.current = abortController;
 
    // Add a placeholder for the assistant response
    setMessages([...updatedMessages, { role: 'assistant', content: '' }]);
 
    let retryCount = 0;
    hasEmittedRef.current = false; // reset on every send
    const maxRetries = 2;
 
    const attemptFetch = async (): Promise<void> => {
      try {
        const response = await fetch(PROXY_URL, {
          method: 'POST',
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify({ messages: updatedMessages, stream: true }),
          signal: abortController.signal, // This is what makes stopStreaming actually work
        });
 
        if (!response.ok) {
          throw new Error(`HTTP ${response.status}: ${await response.text()}`);
        }
 
        const reader = response.body?.getReader();
        if (!reader) throw new Error('ReadableStream not supported in this environment');
 
        const decoder = new TextDecoder();
        let buffer = ''; // Accumulates data across chunk boundaries
        let accumulatedText = '';
 
        while (true) {
          const { done, value } = await reader.read();
          if (done) break;
 
          buffer += decoder.decode(value, { stream: true });
 
          // SSE events are separated by double newlines
          const events = buffer.split('\n\n');
          buffer = events.pop() ?? ''; // Keep the incomplete trailing event in the buffer
 
          for (const event of events) {
            for (const line of event.split('\n')) {
              if (!line.startsWith('data: ')) continue;
              const data = line.slice(6);
 
              if (data === '[DONE]') continue; // OpenAI termination marker
 
              try {
                const parsed = JSON.parse(data);
 
                // ⚠️ Easy to miss: an error event can arrive *after* a 200 OK.
                // Drop it and a truncated answer looks like a clean finish.
                if (parsed.type === 'error' || parsed.error) {
                  const detail =
                    parsed.error?.message ?? parsed.error?.type ?? 'stream error';
                  throw new StreamAbortedError(detail);
                }
 
                // Handle Anthropic format
                const deltaText = parsed.delta?.text
                  // Handle OpenAI format
                  ?? parsed.choices?.[0]?.delta?.content
                  ?? '';
 
                if (deltaText) {
                  hasEmittedRef.current = true;
                  accumulatedText += deltaText;
                  // Use functional setState to avoid stale closure issues
                  setMessages(prev => {
                    const next = [...prev];
                    next[next.length - 1] = { role: 'assistant', content: accumulatedText };
                    return next;
                  });
                }
              } catch (e) {
                // Do not swallow StreamAbortedError — let it reach the handler
                if (e instanceof StreamAbortedError) throw e;
                // Individual chunk parse failures are common and safe to ignore
              }
            }
          }
        }
      } catch (err: unknown) {
        if (err instanceof Error && err.name === 'AbortError') {
          return; // User-initiated cancellation — not an error
        }
 
        // ⚠️ Retrying after even one character has painted rewinds the UI.
        // accumulatedText is re-initialised inside attemptFetch, so the first
        // paint of the retry replaces visible text with a shorter string.
        if (hasEmittedRef.current) {
          const message = err instanceof Error ? err.message : 'Unknown error';
          setError(`Response ended early: ${message}`);
          return;
        }
 
        if (retryCount < maxRetries) {
          retryCount++;
          console.warn(`Streaming error, retrying (${retryCount}/${maxRetries}):`, err);
          await new Promise(resolve => setTimeout(resolve, 1000 * retryCount));
          return attemptFetch();
        }
 
        const message = err instanceof Error ? err.message : 'Unknown error';
        setError(`Connection error: ${message}`);
        setMessages(prev => prev.slice(0, -1)); // Remove the empty placeholder
      } finally {
        setIsStreaming(false);
        abortControllerRef.current = null;
      }
    };
 
    await attemptFetch();
  }, [messages, isStreaming]);
 
  const resetConversation = useCallback(() => {
    stopStreaming();
    setMessages([]);
    setError(null);
  }, [stopStreaming]);
 
  return { messages, isStreaming, error, sendMessage, stopStreaming, resetConversation };
}

A few design decisions worth explaining.

The buffer variable is not optional. TCP packets don't respect SSE event boundaries. A single read() can return half an event or three events. Splitting the buffer on \n\n and keeping the trailing incomplete chunk solves this cleanly. Without it, you'll chase intermittent parse errors that only appear under real network conditions.

Functional setMessages(prev => ...) prevents the common stale closure bug where the callback captures an outdated snapshot of messages. This matters during fast streaming where state updates overlap.

Retry logic targets network errors specifically. AbortError is excluded — that's intentional user cancellation. Two retries with linear backoff handles the typical mobile network interruption without burning API credits on a genuinely failed session.

UI Integration with Stop Button and Cursor

// components/ChatUI.tsx
import { useState, useEffect } from 'react';
import { View, TextInput, Pressable, Text, FlatList, AppState } from 'react-native';
import { useStreamingChat } from '../hooks/useStreamingChat';
 
export function ChatUI() {
  const { messages, isStreaming, error, sendMessage, stopStreaming } = useStreamingChat();
  const [input, setInput] = useState('');
 
  // Stop streaming when the app backgrounds — iOS cuts connections after a few seconds anyway
  useEffect(() => {
    const sub = AppState.addEventListener('change', (state) => {
      if (state === 'background' && isStreaming) stopStreaming();
    });
    return () => sub.remove();
  }, [isStreaming, stopStreaming]);
 
  // Cleanup on unmount — prevents setState on unmounted component warnings
  useEffect(() => {
    return () => { stopStreaming(); };
  }, [stopStreaming]);
 
  const handleSend = async () => {
    if (!input.trim() || isStreaming) return;
    const text = input;
    setInput('');
    await sendMessage(text);
  };
 
  return (
    <View style={{ flex: 1 }}>
      <FlatList
        data={messages}
        keyExtractor={(_, i) => String(i)}
        renderItem={({ item, index }) => {
          const isLastAssistant = item.role === 'assistant' && index === messages.length - 1;
          return (
            <View style={{
              alignSelf: item.role === 'user' ? 'flex-end' : 'flex-start',
              backgroundColor: item.role === 'user' ? '#007AFF' : '#F2F2F7',
              padding: 12,
              borderRadius: 16,
              margin: 4,
              maxWidth: '80%',
            }}>
              <Text style={{ color: item.role === 'user' ? '#fff' : '#000' }}>
                {item.content}
                {isStreaming && isLastAssistant ? '▋' : ''}
              </Text>
            </View>
          );
        }}
      />
 
      {error && (
        <Text style={{ color: '#FF3B30', padding: 8, textAlign: 'center' }}>{error}</Text>
      )}
 
      <View style={{ flexDirection: 'row', padding: 8, gap: 8 }}>
        <TextInput
          value={input}
          onChangeText={setInput}
          placeholder="Type a message..."
          style={{
            flex: 1,
            borderWidth: 1,
            borderColor: '#C7C7CC',
            borderRadius: 20,
            paddingHorizontal: 16,
            paddingVertical: 8,
          }}
          editable={!isStreaming}
          onSubmitEditing={handleSend}
          returnKeyType="send"
        />
        {isStreaming ? (
          <Pressable onPress={stopStreaming} style={{ justifyContent: 'center', padding: 8 }}>
            <Text style={{ fontSize: 24 }}>⏹</Text>
          </Pressable>
        ) : (
          <Pressable
            onPress={handleSend}
            disabled={!input.trim()}
            style={{
              backgroundColor: '#007AFF',
              borderRadius: 20,
              paddingHorizontal: 16,
              justifyContent: 'center',
              opacity: !input.trim() ? 0.5 : 1,
            }}
          >
            <Text style={{ color: '#fff', fontWeight: '600' }}>Send</Text>
          </Pressable>
        )}
      </View>
    </View>
  );
}

The ▋ block cursor is a small detail with outsized impact. It visually communicates "the AI is currently generating" without requiring a separate loading indicator. It disappears automatically when streaming completes.

Cloudflare Workers Proxy — Never Embed API Keys in App Binaries

This is the security element most solo developers skip, and it's the one that leads to unexpected billing spikes.

Embedding an API key in your app's source code means it ends up in the compiled binary. Tools like strings on iOS/Android binaries can extract them. App Store / Google Play apps are public downloads. Anyone can pull the binary, extract your key, and rack up charges.

The solution is a thin proxy layer — in this case a Cloudflare Worker that holds the key server-side and forwards requests. See also the Hono × Cloudflare Workers REST API guide for related patterns.

// worker/src/index.ts
import { Hono } from 'hono';
import { cors } from 'hono/cors';
 
type Env = { ANTHROPIC_API_KEY: string };
 
const app = new Hono<{ Bindings: Env }>();
 
app.use('/api/*', cors({ origin: ['*'] })); // Restrict to your app's domain in production
 
app.post('/api/chat', async (c) => {
  const { messages, stream = true } = await c.req.json();
 
  // Input validation is not optional — without it, anyone can generate unlimited tokens at your expense
  if (!Array.isArray(messages) || messages.length === 0) {
    return c.json({ error: 'Invalid messages' }, 400);
  }
  if (messages.length > 50) {
    return c.json({ error: 'Conversation too long' }, 400);
  }
  const totalChars = messages.reduce((sum: number, m: {content: string}) => sum + m.content.length, 0);
  if (totalChars > 100_000) {
    return c.json({ error: 'Message content too large' }, 400);
  }
 
  const upstream = await fetch('https://api.anthropic.com/v1/messages', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'anthropic-version': '2023-06-01',
      'x-api-key': c.env.ANTHROPIC_API_KEY, // Pulled from Worker secret — never in code
    },
    body: JSON.stringify({
      model: 'claude-haiku-4-5-20251001',
      max_tokens: 2048,
      messages,
      stream,
    }),
  });
 
  if (!upstream.ok) {
    return c.json({ error: await upstream.text() }, upstream.status as 400 | 401 | 429 | 500);
  }
 
  // Pass the streaming response directly through to the client
  return new Response(upstream.body, {
    headers: {
      'Content-Type': 'text/event-stream',
      'Cache-Control': 'no-cache',
      'Connection': 'keep-alive',
    },
  });
});
 
export default app;

Deploying the secret without it appearing in your shell history:

wrangler secret put ANTHROPIC_API_KEY
# Prompts for input interactively — doesn't appear in bash history
 
wrangler deploy

Beyond security, the proxy gives you a clean place to add rate limiting, per-user token quotas, logging, and provider switching — none of which belong in the mobile client.

Multi-Provider Abstraction Layer

Any production AI feature will eventually need to switch providers — whether for cost, availability, or capability reasons. Building provider-specific code directly into your proxy makes this painful. A clean abstraction makes it trivial.

// worker/src/providers/types.ts
export type ChatMessage = { role: 'user' | 'assistant'; content: string };
export type StreamProvider = (
  messages: ChatMessage[],
  env: Record<string, string>
) => Promise<ReadableStream>;
 
// worker/src/providers/anthropic.ts
export const anthropicProvider: StreamProvider = async (messages, env) => {
  const res = await fetch('https://api.anthropic.com/v1/messages', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'anthropic-version': '2023-06-01',
      'x-api-key': env.ANTHROPIC_API_KEY,
    },
    body: JSON.stringify({
      model: 'claude-sonnet-4-6',
      max_tokens: 2048,
      messages,
      stream: true,
    }),
  });
  if (!res.ok || !res.body) throw new Error(`Anthropic ${res.status}`);
  return res.body;
};
 
// worker/src/providers/openai.ts
export const openaiProvider: StreamProvider = async (messages, env) => {
  const res = await fetch('https://api.openai.com/v1/chat/completions', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${env.OPENAI_API_KEY}`,
    },
    body: JSON.stringify({ model: 'gpt-4o-mini', messages, stream: true }),
  });
  if (!res.ok || !res.body) throw new Error(`OpenAI ${res.status}`);
  return res.body;
};
 
// worker/src/providers/gemini.ts
export const geminiProvider: StreamProvider = async (messages, env) => {
  // Gemini uses a different message format
  const contents = messages.map(m => ({
    role: m.role === 'assistant' ? 'model' : 'user',
    parts: [{ text: m.content }],
  }));
 
  const res = await fetch(
    `https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:streamGenerateContent?alt=sse&key=${env.GEMINI_API_KEY}`,
    {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ contents }),
    }
  );
  if (!res.ok || !res.body) throw new Error(`Gemini ${res.status}`);
  return res.body;
};
 
// worker/src/index.ts — provider selection with automatic fallback
app.post('/api/chat', async (c) => {
  const { messages, provider = 'anthropic' } = await c.req.json();
  const env = {
    ANTHROPIC_API_KEY: c.env.ANTHROPIC_API_KEY,
    OPENAI_API_KEY: c.env.OPENAI_API_KEY ?? '',
    GEMINI_API_KEY: c.env.GEMINI_API_KEY ?? '',
  };
 
  const providers: Record<string, StreamProvider> = {
    anthropic: anthropicProvider,
    openai: openaiProvider,
    gemini: geminiProvider,
  };
 
  const primary = providers[provider] ?? anthropicProvider;
 
  try {
    const stream = await primary(messages, env);
    return new Response(stream, {
      headers: { 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-cache' },
    });
  } catch (primaryErr) {
    console.error('Primary provider failed, attempting fallback:', primaryErr);
    try {
      const fallback = await openaiProvider(messages, env);
      return new Response(fallback, {
        headers: { 'Content-Type': 'text/event-stream', 'Cache-Control': 'no-cache' },
      });
    } catch {
      return c.json({ error: 'All providers unavailable' }, 503);
    }
  }
});

The app client never needs to know which provider is actually serving the request. You can switch from Haiku to Sonnet, add Gemini as a primary option, or route specific use cases to different models — all without touching the mobile app.

Performance: Maintaining 60fps During Fast Streaming

During active streaming, setMessages can fire dozens of times per second. Each call triggers a re-render. On long message lists, this creates visible jank, especially on older devices.

The fix is throttled state updates — only call setState when at least 50ms has passed since the last update:

const lastUpdateRef = useRef<number>(0);
const pendingTextRef = useRef<string>('');
const flushTimerRef = useRef<ReturnType<typeof setTimeout> | null>(null);
 
const flushPendingText = useCallback(() => {
  if (flushTimerRef.current) {
    clearTimeout(flushTimerRef.current);
    flushTimerRef.current = null;
  }
  setMessages(prev => {
    const next = [...prev];
    next[next.length - 1] = { role: 'assistant', content: pendingTextRef.current };
    return next;
  });
  lastUpdateRef.current = Date.now();
}, []);
 
// Replace the direct setMessages call inside the streaming loop with:
const updateText = (newText: string) => {
  pendingTextRef.current = newText;
  const elapsed = Date.now() - lastUpdateRef.current;
 
  if (elapsed >= 50) {
    flushPendingText();
  } else {
    if (flushTimerRef.current) clearTimeout(flushTimerRef.current);
    flushTimerRef.current = setTimeout(flushPendingText, 50 - elapsed);
  }
};

Call flushPendingText() when streaming completes to ensure the final text is committed. This reduces setState call volume by roughly 80%, and the improvement in scroll smoothness during streaming is immediately noticeable.

If you're seeing garbled multibyte characters (Japanese, Chinese, emoji) in Hermes, try decoding without { stream: true } or accumulate Uint8Array chunks and decode them as a batch at the end.

Common Pitfalls Reference

Having shipped several AI chat features in React Native apps, these are the mistakes that cost the most debugging time.

Not passing signal to fetch: The most common reason stopStreaming appears to do nothing. AbortController only works when its signal is connected to the fetch call.

Omitting max_tokens: Without this, the model generates to its default ceiling (often 8,192 tokens for Sonnet). Chat responses rarely need that. Set 1,024–2,048 for normal chat, higher only for specific generation tasks.

Skipping the buffer: Parsing each read() result directly without accumulating across \n\n boundaries causes sporadic JSON errors that only manifest under real network conditions. These bugs are almost impossible to reproduce in development.

Missing useEffect cleanup: If a user navigates away during generation, the component unmounts but streaming continues. setMessages on an unmounted component generates warnings and can cause subtle state corruption. Always return () => stopStreaming() from a cleanup useEffect.

Ignoring iOS background transitions: iOS terminates network connections within seconds of backgrounding. Use AppState.addEventListener to detect this and stop streaming proactively rather than letting it fail and confuse the user.

Embedding API keys in app code: The binary is downloadable and inspectable. Even obfuscated keys get extracted. Always use a server-side proxy. See also the hiding API keys behind a Worker with rate limiting and key rotation for more on this architecture.

Discarding error events that arrive after a 200 OK: This is the one that scared me most. Anthropic's stream can connect successfully, deliver a few tokens, and then send event: error — overloaded_error being the classic case.

A parser that only looks for delta text treats that event as "an event with no text" and walks straight past it. The stream reaches done, the UI records a clean finish, and the reader is left staring at a truncated answer. No error surfaces. No retry fires.

I pulled the parser into a standalone Node script and fed it nothing but a ping and an overloaded_error. Result: zero characters extracted, no exception, no error detected. It fails in complete silence. That's why the hook above checks parsed.type === 'error' explicitly and converts it into a typed exception.

Retrying after partial output, which rewinds the screen: Exponential backoff is the right call when the connection itself never opened. But calling the same function again after five characters have already painted re-initialises accumulatedText, and the first paint of the retry replaces the visible text with something shorter.

Reproducing it with the same parser: five characters on screen at the moment of disconnect, two characters on the retry's first paint. To the reader, a half-written sentence vanishes and a different one grows in its place. On top of that, LLM calls are not idempotent, so you pay for the generated tokens twice.

The fix is small. Once a single character has shipped, stop retrying. Keep the partial text and append the error beneath it. That's all hasEmittedRef is for.

Cost Monitoring in Production

Add lightweight logging to your Worker to track token consumption per request. Anthropic's streaming response includes a message_delta event at the end with actual usage numbers — capture that in your logs.

// Basic request logging at the Worker level
console.log(JSON.stringify({
  ts: new Date().toISOString(),
  model: 'claude-haiku-4-5-20251001',
  messages: messages.length,
  estimatedInputTokens: messages.reduce(
    (sum: number, m: {content: string}) => sum + Math.ceil(m.content.length / 4),
    0
  ),
}));

For per-user quota enforcement, Cloudflare KV works well: store daily token counts keyed by user ID, check before forwarding to the LLM, and return 429 when the limit is hit. This prevents a single power user (or a bot) from exhausting your monthly budget overnight.

Advanced Pattern: Streaming with Tool Use and Structured Outputs

Modern LLM applications often need more than raw text streaming. Tool use (function calling) and structured JSON outputs require additional parsing logic on top of the basic SSE implementation.

Anthropic's streaming API includes specific events for tool use that interleave with text generation. Here's how to handle them correctly in the streaming loop:

// Extended parser that handles both text deltas and tool use events
const parseStreamingEvent = (data: string): {
  type: 'text' | 'tool_use' | 'stop';
  text?: string;
  toolName?: string;
  toolInput?: Record<string, unknown>;
} => {
  try {
    const parsed = JSON.parse(data);
 
    // Standard text token
    if (parsed.type === 'content_block_delta' && parsed.delta?.type === 'text_delta') {
      return { type: 'text', text: parsed.delta.text ?? '' };
    }
 
    // Tool use started — capture the tool name
    if (parsed.type === 'content_block_start' && parsed.content_block?.type === 'tool_use') {
      return { type: 'tool_use', toolName: parsed.content_block.name };
    }
 
    // Tool input delta — accumulate JSON input for the tool
    if (parsed.type === 'content_block_delta' && parsed.delta?.type === 'input_json_delta') {
      return { type: 'tool_use', toolInput: parsed.delta.partial_json };
    }
 
    // Stream ended
    if (parsed.type === 'message_stop') {
      return { type: 'stop' };
    }
  } catch {
    // Ignore parse errors for individual chunks
  }
 
  return { type: 'text', text: '' };
};

For structured output use cases — where you want the LLM to return JSON that the app can act on immediately — you can stream into a JSON accumulation buffer and parse incrementally as valid JSON fragments arrive. Libraries like partial-json handle this gracefully, allowing your app to start acting on partial results before generation completes.

Conversation Context Management

A streaming hook that doesn't manage context length will eventually hit token limit errors in production. Claude Sonnet has a 200K context window, but that doesn't mean you want to send the entire conversation history on every request — both for cost reasons and because very long contexts degrade response quality on focused tasks.

A practical context management approach:

// utils/contextManager.ts
 
const MAX_CONTEXT_TOKENS = 4000; // Conservative limit for cost control
 
// Rough estimate: 1 token ≈ 4 characters for English, ≈ 2-3 for Japanese
const estimateTokens = (text: string): number => Math.ceil(text.length / 3.5);
 
export const trimConversationContext = (messages: Message[]): Message[] => {
  // Always keep the system prompt (if present) and the last user message
  if (messages.length <= 2) return messages;
 
  let totalTokens = messages.reduce(
    (sum, m) => sum + estimateTokens(m.content),
    0
  );
 
  // Trim from the beginning (oldest messages) until we're under the limit
  // Keep at least the last 4 messages for conversational coherence
  let trimmedMessages = [...messages];
  while (totalTokens > MAX_CONTEXT_TOKENS && trimmedMessages.length > 4) {
    const removed = trimmedMessages.shift();
    if (removed) totalTokens -= estimateTokens(removed.content);
  }
 
  return trimmedMessages;
};

Integrate this into sendMessage before constructing the request body:

const contextMessages = trimConversationContext(updatedMessages);
 
const response = await fetch(PROXY_URL, {
  body: JSON.stringify({
    messages: contextMessages, // Send trimmed context
    stream: true,
  }),
  // ...
});

This keeps costs predictable and prevents the "conversation got too long and the API returned a 400" error that catches many developers off guard.

Streaming Responses to Disk for Offline Access

A feature users often request in AI apps is the ability to revisit past conversations. Since you're already accumulating the full response text in accumulatedText inside the hook, persisting it is straightforward.

Using AsyncStorage (or MMKV for better performance) to save completed conversations:

import AsyncStorage from '@react-native-async-storage/async-storage';
 
// Called when streaming completes in the finally block
const saveConversation = async (messages: Message[]) => {
  try {
    const key = `conversation_${Date.now()}`;
    const conversationData = {
      id: key,
      timestamp: new Date().toISOString(),
      messages,
      preview: messages[messages.length - 1]?.content.slice(0, 100) ?? '',
    };
    await AsyncStorage.setItem(key, JSON.stringify(conversationData));
 
    // Maintain an index of all conversation keys
    const indexRaw = await AsyncStorage.getItem('conversation_index');
    const index: string[] = indexRaw ? JSON.parse(indexRaw) : [];
    index.unshift(key);
    // Keep only the most recent 50 conversations
    if (index.length > 50) index.pop();
    await AsyncStorage.setItem('conversation_index', JSON.stringify(index));
  } catch (err) {
    console.warn('Failed to save conversation:', err);
    // Non-fatal — streaming still worked, just not persisted
  }
};

Call saveConversation(messages) in the finally block of attemptFetch (after the streaming loop completes successfully and accumulatedText is non-empty). This gives users a conversation history screen with no backend required.

For apps that need server-side persistence (shared history across devices, analytics, fine-tuning data collection), pass the completed conversation to your Cloudflare Worker endpoint after the stream closes and store it in D1 or Supabase.

Testing Streaming Without Real API Costs

During development, making real API calls on every test run adds up quickly. A simple mock streaming server lets you iterate on UI behavior without touching the actual LLM:

// dev-proxy/mock-server.ts (run locally with tsx or ts-node)
import { createServer } from 'http';
 
const MOCK_RESPONSE = 'This is a mock streaming response from the development server. It arrives token by token to simulate real LLM behavior.';
 
createServer((req, res) => {
  if (req.method !== 'POST' || req.url !== '/api/chat') {
    res.writeHead(404);
    res.end();
    return;
  }
 
  res.writeHead(200, {
    'Content-Type': 'text/event-stream',
    'Cache-Control': 'no-cache',
    'Access-Control-Allow-Origin': '*',
  });
 
  const words = MOCK_RESPONSE.split(' ');
  let index = 0;
 
  const interval = setInterval(() => {
    if (index >= words.length) {
      res.write('data: [DONE]\n\n');
      res.end();
      clearInterval(interval);
      return;
    }
 
    const token = index === 0 ? words[index] : ` ${words[index]}`;
    const event = JSON.stringify({
      choices: [{ delta: { content: token } }],
    });
    res.write(`data: ${event}\n\n`);
    index++;
  }, 80); // ~12 tokens per second, realistic for fast LLMs
 
  req.on('close', () => clearInterval(interval)); // Handle client disconnect
}).listen(3001, () => {
  console.log('Mock streaming server running on http://localhost:3001');
});

Point PROXY_URL to http://localhost:3001/api/chat during development (using Expo's local development tunnel or __DEV__ conditional), and you can iterate on the UI at full speed without API costs.

What Actually Changed After Shipping It

The clearest signal wasn't a metric. It was the kind of feedback that started arriving.

Before streaming, almost every report was about the mechanism: the AI is slow, it's frozen, nothing happened. After streaming, those messages stopped and were replaced by requests about the answers themselves — make it shorter, remember what I said earlier, tell me where this came from. The interface had become transparent enough that people could finally see the feature behind it.

The build cost is real. SSE buffering, AbortController wiring, a proxy to stand up — none of it is glamorous. But it amortizes hard. Once the hook and the Worker exist, the second AI feature is a prompt change and a new screen. Mine took an afternoon.

Assembled, the pieces are small: a ChatUI component backed by a useStreamingChat hook that owns message state, cancellation, and retries; a Cloudflare Worker that holds the secrets and forwards the stream; a provider layer that keeps Anthropic, OpenAI, and Gemini behind one interface. Each part is replaceable on its own, which is the property that has mattered most as the models kept changing underneath me.

If you're starting today, build the mock SSE server from the testing section first. Lock down the UI, the stop button, and the retry path against a fixture where every failure is reproducible and nothing costs money. Connect a real provider only once that behaves. Then add persistence along the lines of these Supabase Edge Functions and RLS design notes, and single-shot Q&A turns into a conversation people can come back to.

Thank you for reading this far. If the code here removes one spinner from your app, it was worth writing down.

Thank You for Reading

Rork Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.

✦Copy-paste ready implementation code
✦New advanced guides published daily
✦$5/mo or $10 for lifetime access

View Membership →