Building an AI Reading Assistant App with Rork — Bookshelf Scanning, Progress Tracking, and Chapter Summaries

Implementing Bookshelf Scanning — A Hybrid AI Pipeline

Bookshelf scanning is the hardest part of the app. Plain OCR collapses on vertical spines and stylized fonts. Claude Sonnet, on the other hand, reads the visual context (cover art, layout, series cues) and infers titles that pure OCR misses. Merging both gives the robust result we want.

Client Side: Capturing and Sending the Image

On the Rork side, use expo-image-picker to capture the shelf photo and resize it to roughly 2048px wide before sending. That is the sweet spot: big enough to preserve readable text on spines, small enough to keep upload time and Claude token cost reasonable.

// app/(tabs)/scan.tsx
import { useState } from 'react';
import { View, Text, Pressable, Image, ActivityIndicator } from 'react-native';
import * as ImagePicker from 'expo-image-picker';
import * as ImageManipulator from 'expo-image-manipulator';
import { scanBookshelfApi } from '@/lib/api/bookshelf';
 
type DetectedBook = {
  title: string;
  author?: string;
  confidence: number;
};
 
export default function BookshelfScanScreen() {
  const [image, setImage] = useState<string | null>(null);
  const [books, setBooks] = useState<DetectedBook[] | null>(null);
  const [loading, setLoading] = useState(false);
  const [error, setError] = useState<string | null>(null);
 
  const pickAndScan = async () => {
    try {
      setError(null);
 
      const result = await ImagePicker.launchCameraAsync({
        mediaTypes: ImagePicker.MediaTypeOptions.Images,
        quality: 0.9,
        allowsEditing: false,
      });
 
      if (result.canceled) return;
 
      // Resize to 2048px wide to balance OCR quality and upload cost
      const manipulated = await ImageManipulator.manipulateAsync(
        result.assets[0].uri,
        [{ resize: { width: 2048 } }],
        { compress: 0.85, format: ImageManipulator.SaveFormat.JPEG, base64: true },
      );
 
      setImage(manipulated.uri);
      setLoading(true);
 
      const detected = await scanBookshelfApi(manipulated.base64!);
      setBooks(detected);
    } catch (e: any) {
      // Show a short, friendly message to the user; send the details to Sentry
      setError('We could not read the shelf. Try a different angle without glare.');
      console.error('[BookshelfScan]', e);
    } finally {
      setLoading(false);
    }
  };
 
  return (
    <View className="flex-1 items-center justify-center p-4">
      <Pressable
        onPress={pickAndScan}
        className="bg-indigo-600 px-6 py-4 rounded-full"
        accessibilityLabel="Scan a bookshelf to register books"
      >
        <Text className="text-white font-semibold">Scan bookshelf</Text>
      </Pressable>
 
      {image && <Image source={{ uri: image }} className="w-full h-64 mt-4 rounded-lg" />}
      {loading && <ActivityIndicator className="mt-4" />}
      {error && <Text className="text-red-500 mt-4">{error}</Text>}
      {books && (
        <View className="w-full mt-4">
          <Text className="font-semibold mb-2">Recognized {books.length} books</Text>
          {books.map((b, i) => (
            <Text key={i} className="text-sm text-gray-700">
              {b.title}
              {b.author ? ` · ${b.author}` : ''} (confidence {Math.round(b.confidence * 100)}%)
            </Text>
          ))}
        </View>
      )}
    </View>
  );
}

Resizing on the client instead of trusting the user to crop the photo makes the app behave predictably regardless of how the image was captured. That predictability matters when the next step is an AI call that costs real money.

Server Side: Parallel Vision + Claude Calls

On Cloudflare Workers with Hono, define a single endpoint that fans out to both AI providers in parallel. Running them sequentially produces 4–6 second latencies that kill the experience. Promise.allSettled is the right primitive: if one side fails, the other still contributes.

// workers/src/routes/bookshelf.ts
import { Hono } from 'hono';
import Anthropic from '@anthropic-ai/sdk';
import { GoogleGenerativeAI } from '@google/generative-ai';
 
type Env = {
  ANTHROPIC_API_KEY: string;
  GEMINI_API_KEY: string;
};
 
type DetectedBook = { title: string; author?: string; confidence: number };
 
const app = new Hono<{ Bindings: Env }>();
 
app.post('/api/bookshelf/scan', async (c) => {
  const { imageBase64 } = await c.req.json<{ imageBase64: string }>();
 
  if (!imageBase64 || imageBase64.length < 1000) {
    return c.json({ error: 'Invalid image data' }, 400);
  }
 
  try {
    const [claudeResult, geminiResult] = await Promise.allSettled([
      detectBooksWithClaude(imageBase64, c.env.ANTHROPIC_API_KEY),
      detectBooksWithGemini(imageBase64, c.env.GEMINI_API_KEY),
    ]);
 
    const merged = mergeDetections(
      claudeResult.status === 'fulfilled' ? claudeResult.value : [],
      geminiResult.status === 'fulfilled' ? geminiResult.value : [],
    );
 
    return c.json({ books: merged });
  } catch (e) {
    console.error('[BookshelfScan] unexpected error', e);
    return c.json({ error: 'Scan failed' }, 500);
  }
});
 
async function detectBooksWithClaude(base64: string, apiKey: string): Promise<DetectedBook[]> {
  const client = new Anthropic({ apiKey });
 
  const response = await client.messages.create({
    model: 'claude-sonnet-4-6',
    max_tokens: 2000,
    messages: [
      {
        role: 'user',
        content: [
          {
            type: 'image',
            source: { type: 'base64', media_type: 'image/jpeg', data: base64 },
          },
          {
            type: 'text',
            text: `Identify titles and authors visible on the spines in this bookshelf image.
Return ONLY a JSON array in this exact shape:
[{"title": "...", "author": "name or null", "confidence": 0.0-1.0}]
 
If a spine is unreadable, omit that book. Never invent titles — confidence must reflect what you actually see.`,
          },
        ],
      },
    ],
  });
 
  const text = response.content[0].type === 'text' ? response.content[0].text : '';
  return parseJsonArray<DetectedBook>(text);
}
 
async function detectBooksWithGemini(base64: string, apiKey: string): Promise<DetectedBook[]> {
  const genAI = new GoogleGenerativeAI(apiKey);
  const model = genAI.getGenerativeModel({ model: 'gemini-2.5-flash' });
 
  const result = await model.generateContent([
    { inlineData: { data: base64, mimeType: 'image/jpeg' } },
    `Return a JSON array of titles and authors visible on the spines: [{"title": "...", "author": "...", "confidence": 0.0-1.0}]`,
  ]);
 
  return parseJsonArray<DetectedBook>(result.response.text());
}
 
function parseJsonArray<T>(raw: string): T[] {
  // Models sometimes wrap output with ```json ... ``` or add preamble; slice the first [...] block
  const match = raw.match(/\[[\s\S]*\]/);
  if (!match) return [];
  try {
    return JSON.parse(match[0]);
  } catch {
    return [];
  }
}
 
function mergeDetections(a: DetectedBook[], b: DetectedBook[]): DetectedBook[] {
  const merged = [...a];
  for (const book of b) {
    const similar = merged.find(
      (m) => normalizeTitle(m.title) === normalizeTitle(book.title),
    );
    if (!similar) {
      merged.push(book);
    } else if (book.confidence > similar.confidence) {
      similar.confidence = Math.min(1, (similar.confidence + book.confidence) / 2 + 0.1);
      similar.author = similar.author ?? book.author;
    }
  }
  return merged
    .filter((b) => b.confidence >= 0.5)
    .sort((x, y) => y.confidence - x.confidence);
}
 
function normalizeTitle(t: string): string {
  return t.toLowerCase().replace(/[\s・：:|\-—–]/g, '');
}
 
export default app;

Expected behavior: a shelf photo with 15 visible spines returns 12–14 books in the books array. Aim for 80–95 percent recognition rate — chasing 100 percent makes costs explode, and a simple confirm-and-remove UI handles the remaining edge cases gracefully.

Why Call Both Claude and Gemini

Claude Sonnet is strongest on contextual reasoning and on respecting "do not hallucinate" instructions. Gemini 2.5 Flash is strongest on raw OCR precision and price per call. Each one misses different kinds of books, so merging their outputs yields 1.3–1.5× more correctly identified titles than running either alone. Because Claude costs slightly more, a reasonable policy is to send free-tier users through Gemini only and reserve the dual-call pipeline for paying subscribers.

For further token-cost reduction, consider using Anthropic's prompt caching once your instruction blocks stabilize. The shared instruction portion can be reused across scans, which meaningfully lowers per-request cost. See Claude Prompt Caching and Extended Thinking — A Practical Guide for the implementation details.

Implementing Chapter Summaries — Value Without Blowing the Budget

Once books are in the library, the next feature is chapter-level progress and per-chapter AI summaries. This is where paying users will judge whether the subscription is worth keeping, so the quality bar needs to be high.

Estimating Chapter Structure

For Kindle or PDF imports with a real table of contents, you can parse it directly. For everything else, asking Claude to estimate chapter structure from title and author works surprisingly well. It is not perfect, but it is usable.

// workers/src/routes/chapters.ts
import { Hono } from 'hono';
import Anthropic from '@anthropic-ai/sdk';
 
type Chapter = { index: number; title: string; estimatedPages?: [number, number] };
 
const app = new Hono<{ Bindings: { ANTHROPIC_API_KEY: string } }>();
 
app.post('/api/books/:bookId/chapters', async (c) => {
  const bookId = c.req.param('bookId');
  const { title, author, totalPages } = await c.req.json<{
    title: string;
    author?: string;
    totalPages?: number;
  }>();
 
  if (!title || title.length > 200) {
    return c.json({ error: 'Invalid title' }, 400);
  }
 
  const client = new Anthropic({ apiKey: c.env.ANTHROPIC_API_KEY });
 
  const response = await client.messages.create({
    model: 'claude-sonnet-4-6',
    max_tokens: 1500,
    messages: [
      {
        role: 'user',
        content: `Estimate the chapter structure of the book "${title}"${author ? ` by ${author}` : ''}.
${totalPages ? `The total page count is approximately ${totalPages}. Include estimated page ranges for each chapter.` : ''}
Return ONLY a JSON array in this shape:
[{"index": 1, "title": "Chapter title", "estimatedPages": [1, 30]}]
 
If the structure is unknown, return []. Do not invent chapters.`,
      },
    ],
  });
 
  const text = response.content[0].type === 'text' ? response.content[0].text : '[]';
  const match = text.match(/\[[\s\S]*\]/);
  let chapters: Chapter[] = [];
  try {
    chapters = match ? JSON.parse(match[0]) : [];
  } catch {
    chapters = [];
  }
 
  // Persist to Supabase (omitted)
  return c.json({ chapters, source: chapters.length > 0 ? 'ai-estimated' : 'unknown' });
});
 
export default app;

When chapter structure cannot be recovered, fall back to a UX where the user selects a page range and marks it as "the chapter I just finished." Building that range picker with react-native-reanimated for a fluid scrub feels surprisingly polished.

The Chapter Summary Prompt

Summaries need to do two things at once: condense the key ideas without spoiling later chapters, and surface three open questions the reader can carry into the next chapter. I give the model a consistent role as a personal reading companion to keep the tone warm and thoughtful.

// workers/src/lib/summarize.ts
import Anthropic from '@anthropic-ai/sdk';
 
type SummaryRequest = {
  bookTitle: string;
  chapterTitle: string;
  userNote?: string; // Notes the reader took while reading
  apiKey: string;
};
 
type SummaryResult = {
  keyPoints: string[];
  openQuestions: string[];
  encouragement: string;
};
 
export async function generateChapterSummary(req: SummaryRequest): Promise<SummaryResult> {
  const client = new Anthropic({ apiKey: req.apiKey });
 
  const response = await client.messages.create({
    model: 'claude-sonnet-4-6',
    max_tokens: 1200,
    system: `You are a warm, thoughtful personal reading companion. Your summaries should help the reader consolidate what they just read — without spoiling anything they have not yet reached. Leave space for the reader's own thinking.`,
    messages: [
      {
        role: 'user',
        content: `The reader just finished the chapter "${req.chapterTitle}" of "${req.bookTitle}". Return the following JSON.
 
${req.userNote ? `Reader's note: "${req.userNote}"\n\nReflect this note in the summary.\n` : ''}
 
{
  "keyPoints": ["Three key points, each under 100 characters"],
  "openQuestions": ["Three reflective questions to carry into the next chapter"],
  "encouragement": "A short, non-preachy note for the reader (under 120 characters)"
}
 
Return the JSON only — no preamble, no trailing text.`,
      },
    ],
  });
 
  const text = response.content[0].type === 'text' ? response.content[0].text : '{}';
  const match = text.match(/\{[\s\S]*\}/);
  if (!match) {
    throw new Error('Summary generation returned no JSON');
  }
 
  try {
    return JSON.parse(match[0]) as SummaryResult;
  } catch (e) {
    throw new Error('Summary JSON parse failed');
  }
}

Example output for a chapter of Cal Newport's "Deep Work":

{
  "keyPoints": [
    "Deep work and shallow work demand different mental modes that rarely coexist in the same day",
    "Cognitive residue from context switching can linger for over 20 minutes after each interruption",
    "Rituals reduce willpower cost by pre-deciding where and when deep work will happen"
  ],
  "openQuestions": [
    "Which parts of my current work genuinely require deep concentration, and which only feel that way?",
    "What small ritual could I commit to this week that protects at least 90 uninterrupted minutes?",
    "Where is shallow work currently disguised as productive deep work in my calendar?"
  ],
  "encouragement": "Let this chapter sit for a day before reading on. Your mind will keep working on it."
}

Why User Notes Matter So Much

The single strongest differentiator for a reading app is the ability to weave the reader's own marginal notes into the summary. A generic summary is a commodity — the free tier of any LLM can produce one. A summary that reflects your own highlighted passages, your own doubts, your own "aha" moments is qualitatively different, and it is exactly the kind of experience that justifies a ¥580 subscription.

For a broader view of how multimodal context shapes AI UX, Rork Multimodal AI App Design Patterns pairs well with this section.

Data Model and Offline Support

We lean on Supabase Postgres for authoritative storage, but reading happens on planes, trains, and in places with bad connectivity. Offline support is non-negotiable for a reading app. react-native-mmkv gives us a fast local cache that syncs back to Supabase when the network returns.

// lib/storage/bookStorage.ts
import { MMKV } from 'react-native-mmkv';
 
const storage = new MMKV({ id: 'reading-assistant' });
 
type BookState = {
  id: string;
  title: string;
  author?: string;
  currentPage: number;
  totalPages?: number;
  lastReadAt: number;
  pendingSync: boolean;
};
 
export const bookStorage = {
  upsert(book: BookState): void {
    storage.set(`book:${book.id}`, JSON.stringify({ ...book, pendingSync: true }));
  },
 
  get(id: string): BookState | null {
    const raw = storage.getString(`book:${id}`);
    if (!raw) return null;
    try {
      return JSON.parse(raw);
    } catch {
      return null;
    }
  },
 
  getPendingSync(): BookState[] {
    const keys = storage.getAllKeys().filter((k) => k.startsWith('book:'));
    const books: BookState[] = [];
    for (const key of keys) {
      const raw = storage.getString(key);
      if (!raw) continue;
      try {
        const book = JSON.parse(raw) as BookState;
        if (book.pendingSync) books.push(book);
      } catch {
        // Ignore corrupted entries and move on
        continue;
      }
    }
    return books;
  },
 
  markSynced(id: string): void {
    const book = this.get(id);
    if (book) {
      storage.set(`book:${id}`, JSON.stringify({ ...book, pendingSync: false }));
    }
  },
};

MMKV beats AsyncStorage by roughly an order of magnitude on both reads and writes — which matters because reading apps update state on every page turn. That level of write frequency is uncomfortable for AsyncStorage but trivial for MMKV.

Common Pitfalls and How to Handle Them

These are the issues that actually bit me while building this app. Knowing them in advance saves weeks.

Pitfall 1: Vision OCR Struggles with Vertical Titles

Japanese book spines are typically printed top-to-bottom. Naive OCR produces disconnected characters that do not form a title. On iOS, configure VNRecognizeTextRequest with recognitionLanguages = ["ja-JP", "en-US"] and usesLanguageCorrection = true, and you will recover most vertical Japanese spines. Anything OCR still misses is usually caught by the Claude side of the pipeline.

Pitfall 2: Believing That Smaller Images Lower Your Token Bill

My first version of the client downscaled every 4000px photo to 2048px before upload, purely to save on tokens. When I reconciled the invoices, the savings were essentially nil.

The reason lives in the API itself. Claude scales an image down server-side before counting tokens whenever its long edge exceeds 1,568 pixels or the image would exceed roughly 1,600 visual tokens. Send 4000px or send 2048px — both land on the same ceiling. The estimation formula is width px × height px ÷ 750, which caps out around 1,600 tokens once you are past the limit.

That does not make client-side resizing pointless. It buys you three other things: shorter upload times on a phone connection, a base64 payload that stays clear of request-size limits, and a lower failure rate on flaky networks. My mistake was optimizing for cost while actually improving latency, and never checking which one I had moved.

Here is the arithmetic with concrete assumptions ($3 in / $15 out per MTok, ¥155 to the dollar — substitute your own current rates):

Two or three yen per call is nothing on its own. The danger is in how it accumulates, which I unpack in the monetization section. Counting scans on the backend (Cloudflare KV or Postgres) is less a cost control than an instrument panel that makes that accumulation visible before it hurts.

Pitfall 3: Readers Get Angry About Spoilers

The single most common complaint from beta testers was "you summarized parts of the book I had not reached yet." Models will happily do this even when the prompt says "no spoilers." The fix is two-layered. First, tighten the prompt to explicitly confine the summary to the chapters already read. Second, post-process the output with a cheap model pass (Gemini Flash works well) that flags spoiler phrases like "later in the book" or "eventually" and triggers a regeneration. How often a regeneration fires depends entirely on your prompt — in my case it was just under one summary in ten. As long as the spoiler check itself runs on a cheap model, that overhead stays in the noise.

Pitfall 4: Offline Notes Re-sync Duplicates

A common bug with MMKV-backed sync is accidentally sending every entry on reconnect instead of only those with pendingSync: true. Belt-and-suspenders the server side too: give every note a client-generated UUID, put a unique constraint on it in Supabase, and have the server upsert idempotently. Even if the client logic drifts, the database stays consistent.

Pitfall 5: Forgetting the Trial-End Reminder Invites 1-Star Reviews

The default pattern for reading apps is a 7-day free trial that auto-converts to paid. Without a push notification 24 hours before conversion, some users will only notice the charge on their credit card statement — and they will tell the world via a 1-star review. Hook into RevenueCat webhooks, specifically SUBSCRIPTION_PENDING_CANCELLATION and trial-ending events, and ship the reminder notification. Rork's RevenueCat Winback Offers and Churn Prevention covers the full webhook wiring.

Gentle Reminders: The Retention Lever Nobody Talks About

The fastest way to lose reading-app users is aggressive push notifications. "Don't forget to read today!" reads as nagging and drives churn. What actually works is a quieter, more respectful cadence. I ship only three kinds of reminders.

The first is a morning-after note, sent the day after a user finishes a chapter. It surfaces yesterday's summary and asks, gently, whether the next chapter might fit into today. The second targets books untouched for three or more days: "You paused at chapter X of this book. Would it be nice to look back at the last page for a minute?" The third is aimed at newly added books: after seven days without reading, a soft "Would 20 pages of this book fit your afternoon?" nudge.

None of these play on guilt. In my own A/B tests, guilt-based phrasing looked better for the first week — higher open rates, higher click-through — but doubled churn at the four-week mark. The respectful version underperforms on vanity metrics and wins on revenue.

Templated notifications with variable interpolation are perfectly adequate for most of these. Reserve AI-generated notifications for the rare special moments: the reader's 10th finished book, a year-end recap, the anniversary of signing up.

Monetization Design — From Trial to Retained Subscriber

Reading apps operated by individuals typically land at ¥200–400 ARPU. Pricing for an AI-augmented reading app comfortably sits at ¥580–780 per month, and for year-long plans around ¥4,800–6,800. Going above that bracket drops conversion sharply in my testing.

My concrete plan shape looks like this. The free tier allows one bookshelf scan per month and three AI summaries per month — enough to experience the "aha" moment without giving the whole product away. The paid tier at ¥580 / ¥4,800 a year removes those limits, adds personal notifications, and unlocks data export. Offering an annual plan at a meaningful discount reduces churn because the subscriber pre-commits to an entire year of habit formation.

The Free Tier Quietly Eats Your Margin

This is the part of reading-app economics that almost nobody models before launch. Take the per-call costs from the pitfalls section and stack them up.

A paid user who runs 8 scans and 40 summaries in a month costs roughly ¥113 in AI spend. Against ¥580 that is a 19.5 percent cost of goods — a number so comfortable that I stopped looking at it for a while.

The free tier is where it turns. A free user who exhausts their allowance (1 scan, 3 summaries) costs about ¥9.5. Small — until you apply a realistic ratio. At 20 free users per paying user, that is roughly ¥190 riding on the back of each subscriber. Total cost of goods climbs to about ¥303, or 52 percent. Layer on Apple and Google's 15–30 percent cut plus hosting, and the margin you thought you had is gone.

Squeezing the free allowance is only one lever, and not the most interesting one. Three that worked better for me, in order:

Change the shape of the free tier rather than its size. Granting three scans once at signup and none thereafter beat granting one scan every month — lower cost, higher conversion. A reading app makes its case during that first library import. Drip-feeding a small monthly allowance keeps users in a comfortable "this is probably enough" limbo indefinitely.

Split models by tier. Free-tier summaries run on a Haiku-class model; paid summaries run on Sonnet. Put the two side by side and the difference is legible to a reader — which turns a cost reduction into the clearest argument for upgrading you will ever write. I like this one because the economics and the pitch point in the same direction.

Instrument cost per user. An ai_usage table holding user_id / feature / input_tokens / output_tokens / created_at tells you immediately what share of spend the top one percent of users consume. In my case a very small group was driving most of the free-tier burn — the kind of fact you will never reach by reasoning about it from your desk.

Use a 14-day free trial, not the typical 7. Reading apps take longer for a user to experience the core value than, say, a photo editor does. A trial that ends just before the habit forms converts far worse than one that lets a user finish an actual book, and the extra week of AI spend per trial is a few yen against a subscription worth thousands over a year.

For pricing itself, Rork App Subscription Pricing Design: 5 Principles to Maximize LTV Without Losing Retention covers the reasoning I use.

For the full RevenueCat + StoreKit 2 wiring, Building Rork Subscriptions Around RevenueCat Entitlements covers the canonical setup.

Pre-Release Testing for AI-Heavy Features

Unit tests alone will not protect an AI feature. I run three tiers of testing before every release, and skipping any of them has bitten me in production.

Tier one is a golden-set regression test. Curate 50 shelf photos with hand-verified expected titles, run the scan endpoint in CI, and fail the build if recognition rate drops below 80 percent. This is what catches silent quality regressions when you update a model version or tweak a prompt.

Tier two is a prompt regression test. Build a fixture of 10 representative chapter summaries and use a cheaper model (Gemini Flash works) to classify whether each output contains spoilers. This catches unintended behavior changes when you edit the summarization prompt.

Tier three is mandatory dogfooding. Give the internal build to yourself and a small circle for a full week and collect at least 20 "feels off" observations before public release. AI apps accumulate subtle experience defects that do not show in metrics but absolutely show in reviews.

Putting the Pieces Together

At this point we have walked through bookshelf scanning, chapter summarization, offline sync, notifications, and monetization. Building all of this from scratch in parallel is a 4–6 week solo project. Rork meaningfully compresses the client side: with focused prompting, you can land a working client in a week while you hand-craft the server and prompts. A TestFlight build in the hands of 10 beta users is the fastest way to find the AI prompt regressions you would otherwise ship to the App Store.

Reading is deeply personal. The moment I knew this app was worth building was when a beta tester told me her AI summary had "noticed" a pattern in her marginal notes that she had never noticed herself. That is an experience no public LLM assistant produces out of the box — it is the kind of value only an integrated app can deliver, and it is exactly the kind of app an indie developer is uniquely positioned to build.

Your Next Step

Take one photo of your own bookshelf today and send it to the Claude API with a prompt that asks for the titles as JSON. That single exercise captures 70 percent of the core experience of this app. From there, generating the client in Rork, wiring up Supabase, and shipping an MVP to TestFlight is a one-week project — not a six-month one.