The Three-Minute Video That Kept Failing — Moving Rork's Gemini Uploads Off the Worker

Hand out the destination, not the pipe

Resumable upload in the Files API happens in two stages.

Stage one sends metadata only, using the start command, and the response comes back with an X-Goog-Upload-URL header. Stage two sends the actual bytes to that URL.

The useful property is in that URL. The destination you get back carries its own session token embedded in the URL itself. Stage two needs no API key at all.

Which means only stage one has to live on the server. The key stays put, the device receives a single-use destination, and the video goes from the phone straight to Google without touching Cloudflare.

// ✅ The Worker only issues a destination. No video bytes pass through it.
interface Env {
  GEMINI_API_KEY: string;
  SESSIONS: KVNamespace;
}
 
const MAX_BYTES = 2 * 1024 * 1024 * 1024; // Files API caps a single file at 2GB
 
async function issueUploadUrl(request: Request, env: Env): Promise<Response> {
  const { sizeBytes, mimeType, userId } = await request.json<{
    sizeBytes: number;
    mimeType: string;
    userId: string;
  }>();
 
  // Reject before issuing. Letting someone upload 2GB and then saying no wastes their data plan.
  if (!Number.isInteger(sizeBytes) || sizeBytes <= 0 || sizeBytes > MAX_BYTES) {
    return Response.json({ error: 'invalid_size' }, { status: 400 });
  }
  if (!['video/mp4', 'video/quicktime'].includes(mimeType)) {
    return Response.json({ error: 'unsupported_type' }, { status: 415 });
  }
 
  // Issuing a destination is effectively reserving an analysis, so meter it per user
  const quotaKey = `upload_quota:${userId}:${new Date().toISOString().slice(0, 10)}`;
  const used = Number((await env.SESSIONS.get(quotaKey)) ?? '0');
  if (used >= 20) {
    return Response.json({ error: 'daily_quota_exceeded' }, { status: 429 });
  }
 
  const initResponse = await fetch(
    `https://generativelanguage.googleapis.com/upload/v1beta/files?key=${env.GEMINI_API_KEY}`,
    {
      method: 'POST',
      headers: {
        'X-Goog-Upload-Protocol': 'resumable',
        'X-Goog-Upload-Command': 'start',
        'X-Goog-Upload-Header-Content-Length': String(sizeBytes),
        'X-Goog-Upload-Header-Content-Type': mimeType,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({ file: { display_name: `v_${Date.now()}` } }),
    }
  );
 
  const uploadUrl = initResponse.headers.get('X-Goog-Upload-URL');
  if (!uploadUrl) {
    return Response.json({ error: 'init_failed' }, { status: 502 });
  }
 
  await env.SESSIONS.put(quotaKey, String(used + 1), { expirationTtl: 86400 });
 
  // The session token lives in this URL. The API key never leaves the Worker.
  return Response.json({ uploadUrl, sizeBytes });
}

Taking sizeBytes from the client may look sloppy. It is self-reported, and nothing stops someone from lying.

But a lie here only hurts the liar. Send more bytes than declared and Google rejects them; send fewer and the finalize never completes. What the server is protecting is the key and the issuance count, not the honesty of a number. I made peace with that distinction.

The per-user daily quota exists because issuing a destination is, in practice, a reservation for one analysis. For enforcing the actual spend ceiling at runtime, I wrote up a three-layer approach in Enforcing AI Cost Ceilings at Runtime.

Sending straight from the device

The app side gets simpler: request a destination, then send the file to it.

The classic React Native trap here is FormData. Reach for a Blob the way you would in a browser and nothing works — React Native wants an object with uri, type, and name.

Except this time it is not multipart at all. Stage two of a resumable upload just wants raw bytes in the body. Expo's upload task will stream a file straight from disk, which means the video never enters the JS heap on the device either. That turns out to matter as much on the phone as it did on the Worker.

// app/lib/uploadVideo.ts
import * as FileSystem from 'expo-file-system';
import * as ImagePicker from 'expo-image-picker';
 
const API_BASE = 'https://your-worker.example.workers.dev';
 
export type UploadResult = { fileUri: string; sizeBytes: number };
 
export async function pickAndUploadVideo(
  userId: string,
  onProgress?: (ratio: number) => void
): Promise<UploadResult | null> {
  const permission = await ImagePicker.requestMediaLibraryPermissionsAsync();
  if (!permission.granted) return null;
 
  const picked = await ImagePicker.launchImageLibraryAsync({
    mediaTypes: ['videos'],
    quality: 1,
  });
  if (picked.canceled || !picked.assets[0]) return null;
 
  const asset = picked.assets[0];
  const info = await FileSystem.getInfoAsync(asset.uri);
  if (!info.exists) return null;
 
  const sizeBytes = info.size ?? 0;
  const mimeType = asset.mimeType ?? 'video/mp4';
 
  // Step 1: ask the server for a destination
  const issued = await fetch(`${API_BASE}/api/upload-url`, {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ sizeBytes, mimeType, userId }),
  });
  if (!issued.ok) {
    const { error } = await issued.json();
    throw new Error(`Could not get an upload destination (${error})`);
  }
  const { uploadUrl } = await issued.json();
 
  // Step 2: device to Google, directly. The Worker is not in this path.
  const task = FileSystem.createUploadTask(
    uploadUrl,
    asset.uri,
    {
      httpMethod: 'POST',
      uploadType: FileSystem.FileSystemUploadType.BINARY_CONTENT,
      headers: {
        'Content-Length': String(sizeBytes),
        'X-Goog-Upload-Offset': '0',
        'X-Goog-Upload-Command': 'upload, finalize',
      },
    },
    (p) => onProgress?.(p.totalBytesSent / p.totalBytesExpectedToSend)
  );
 
  const response = await task.uploadAsync();
  if (!response || response.status >= 300) {
    throw new Error(`Upload failed (${response?.status})`);
  }
 
  const body = JSON.parse(response.body) as {
    file: { uri: string; state: string };
  };
  return { fileUri: body.file.uri, sizeBytes };
}

BINARY_CONTENT is the load-bearing choice. Switch it to MULTIPART and boundary strings wrap the payload, and Google can no longer read it as a video. It fails quietly, which cost me an evening.

Honest progress reporting came free with the change. Relaying through the Worker, the progress bar hit 100% the moment the upload to Cloudflare finished — and then the user stared at a full bar for another 40 seconds on a three-minute video while Cloudflare pushed the bytes to Google. Now the bar tracks the last byte.

How long ACTIVE takes scales with the video

A finished upload is not a usable file. The Files API marks freshly uploaded content as PROCESSING, and generateContent cannot reference it until the state turns ACTIVE.

My original code waited in a fixed loop: 2 seconds, ten times. Twenty seconds total.

Fine for 28 seconds of footage. Not fine for three minutes. This was the second reason short clips felt reliable — a fixed loop manufactures confidence on small inputs and quietly collapses on large ones.

If the wait scales with the video, the budget should too.

// workers/src/waitForActive.ts
type FileState = 'PROCESSING' | 'ACTIVE' | 'FAILED';
 
export async function waitForActive(
  fileName: string,
  durationSec: number,
  env: Env
): Promise<void> {
  // Measured around one sixth of the video's length; allow 3x headroom, floor at 15s
  const budgetMs = Math.max(15_000, Math.ceil(durationSec / 6) * 3 * 1000);
  const deadline = Date.now() + budgetMs;
 
  let waitMs = 1_000;
  while (Date.now() < deadline) {
    const res = await fetch(
      `https://generativelanguage.googleapis.com/v1beta/files/${fileName}?key=${env.GEMINI_API_KEY}`
    );
    const { state } = await res.json<{ state: FileState }>();
 
    if (state === 'ACTIVE') return;
    if (state === 'FAILED') throw new Error('file_processing_failed');
 
    await new Promise((r) => setTimeout(r, waitMs));
    waitMs = Math.min(waitMs * 1.6, 8_000); // back off, cap at 8s
  }
  throw new Error(`file_not_active_within_${budgetMs}ms`);
}

Backing off is partly a cost decision. Polling is not billed, but it spends Worker CPU time and subrequests. One-second polling across three minutes means 180 calls, which walks you toward the subrequest ceiling for no benefit. Starting at one second and capping at eight brought a three-minute wait down to roughly 30 calls.

Checking for FAILED explicitly matters more than it looks. Corrupt files and unsupported codecs land there. If you only watch the clock, a heavy video and a broken video produce the same error message — and one of them gets better if you wait while the other never will. Blur that line and you have built a retry button that can be pressed forever.

A fileUri expires in 48 hours

Files uploaded through the Files API are deleted automatically after 48 hours. While they exist you can read metadata, but you cannot download them back.

I filed that away as irrelevant. The analysis runs seconds after the upload; who needs 48 hours?

Then I added a re-ask feature — open a past result, ask a different question about the same footage. I stored the fileUri on the device and reused it. Everything worked beautifully until the next morning, when history entries started returning 403.

A fileUri is not an identifier. It is a 48-hour lease. Conflate the two and you ship something that works perfectly through launch day and starts generating support mail on day two.

So I split what gets stored.

The KV TTL is 47 hours rather than 48 on purpose. A matching TTL creates a window where KV still has the reference but Google has already dropped the file. Giving up an hour moves the expiry decision entirely to my side of the line — better to treat it as gone than to fetch it and catch a 403.

Inside the window, follow-up questions skip the upload entirely. Outside it, the app quietly re-uploads from the copy still sitting on the device. Either way the user just asked again.

Resolution and frame rate decide the bill

With the path sorted out, cost was next.

Video token consumption turned out to be far more legible than I assumed. By default the API samples one frame per second and spends roughly 300 tokens per second of video. Drop media_resolution to low and each frame costs 66 tokens, which — with audio at about 32 tokens per second and some metadata — lands near 100 tokens per second of video.

That is a 3x spread. A model with a 1M context window can take about an hour of video at default resolution, or three hours at low.

I had started writing my own frame-decimation code. I did not need it. videoMetadata accepts an fps argument, and startOffset / endOffset will clip a range server-side.

// workers/src/analyze.ts
type Intent = 'quick_summary' | 'form_coaching' | 'lecture_notes';
 
interface VideoPlan {
  model: string;
  mediaResolution: 'MEDIA_RESOLUTION_LOW' | 'MEDIA_RESOLUTION_MEDIUM';
  fps: number;
}
 
// Branch on what you need to see — not on how long the video is
function planFor(intent: Intent): VideoPlan {
  switch (intent) {
    case 'quick_summary':
      // Knowing what is on screen is enough; static scenes survive 0.5fps
      return { model: 'gemini-flash-latest', mediaResolution: 'MEDIA_RESOLUTION_LOW', fps: 0.5 };
    case 'lecture_notes':
      // Only needs to catch the moment a slide changes
      return { model: 'gemini-flash-latest', mediaResolution: 'MEDIA_RESOLUTION_LOW', fps: 1 };
    case 'form_coaching':
      // Joint angles live in single frames. Decimate here and the conclusion changes.
      return { model: 'gemini-pro-latest', mediaResolution: 'MEDIA_RESOLUTION_MEDIUM', fps: 4 };
  }
}
 
export async function analyze(
  fileUri: string,
  intent: Intent,
  clip: { startSec?: number; endSec?: number },
  env: Env
) {
  const plan = planFor(intent);
 
  const body = {
    contents: [
      {
        parts: [
          {
            fileData: { fileUri, mimeType: 'video/mp4' },
            videoMetadata: {
              fps: plan.fps,
              ...(clip.startSec !== undefined && { startOffset: `${clip.startSec}s` }),
              ...(clip.endSec !== undefined && { endOffset: `${clip.endSec}s` }),
            },
          },
          { text: PROMPTS[intent] },
        ],
      },
    ],
    generationConfig: {
      temperature: 0.2,
      maxOutputTokens: 2048,
      mediaResolution: plan.mediaResolution,
    },
  };
 
  const res = await fetch(
    `https://generativelanguage.googleapis.com/v1beta/models/${plan.model}:generateContent?key=${env.GEMINI_API_KEY}`,
    {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify(body),
    }
  );
  return res.json();
}

Notice that planFor branches on intent rather than duration. That was a deliberate reversal.

I used to decide by length: under 30 seconds, analyze precisely; over three minutes, go coarse. But looking at a five-second swing coarsely tells you nothing, and studying a ten-minute lecture in high resolution adds nothing either. Length drives the bill; it has no bearing on the precision the task needs. Confuse the two and you overpay on short clips while missing the point on long ones.

Clipping earned its keep immediately. A swing analysis really only needs about two seconds. Let the user record thirty and cut the range server-side: pulling the relevant window at 4fps came out cheaper than scanning the whole thing at 1fps, and more accurate.

For the broader cost picture, How I Cut My Rork App's AI Costs from $350 to $35/Month covers the caching layer, and When Your Rork Hybrid AI Quietly Drifts to the Cloud covers routing drift.

What was left after the rewiring

Same three-minute 720p clip, measured before and after.

The CPU time going from ~3,900ms to ~12ms is the number that settled it for me. Issuing a destination is not much work. Which means that for every video, the old design spent nearly four seconds carrying bytes without making a single decision about them.

There were losses too, and they belong in the record.

With video no longer crossing the Worker, I cannot inspect it server-side anymore. Things I could have rejected by reading the bytes now go through. For now I accept that. As a solo indie developer, detecting bad input from the analysis output is more realistic than running content inspection on a server I do not want to operate. For other subject matter, that trade would not hold.

The other loss: low media resolution stops reading fine print. On lecture slides it caught the body text and dropped the footnotes. Fine for summaries, disqualifying if you promise a transcript of the whiteboard. The 3x discount is charging you for something.

Ask whether the video needs to pass through at all

If I were adding video analysis to a Rork app today, the first thing I wrote would not be the relay handler. It would be a one-sentence answer to "what is this server protecting?"

For me the answer was the API key. That was the entire list. Routing video through a Worker to protect a key was like rerouting a delivery truck through my living room because that is where I keep the spare key.

If you have this running already, pull up your Worker logs and look at CPU time for a single upload. If it reads in the thousands of milliseconds, most of those milliseconds are probably spent making no decisions at all.

For handling footage straight from the camera, Rork × Vision Camera v4 covers that side, and LLM Streaming in Rork Apps covers showing results before they finish.

I resent that three-minute video more times than I want to admit before the path made sense. If it saves someone else the same detour, it was worth writing down.