I Let AI Read My Rork App Reviews — It Found My Next Feature in 3 Minutes

When I first launched a wallpaper app on the App Store, I read every single review by hand. One hundred reviews, then five hundred, then a thousand. I didn't mind it. There was something grounding about reading what real people had to say about something I'd built.

Then downloads crossed a few million, and Google Play alone started bringing in over 2,000 new reviews a month. Reading them all stopped being feasible.

"Just focus on the important ones," I told myself. But that raised a harder question: which ones are important? Prioritizing one-star reviews introduced negative bias. Skimming only five-star reviews hid real problems. A few particularly emotional complaints once pulled me into a six-month feature development cycle that barely moved the needle on ratings.

The answer I eventually landed on: let AI read the reviews, then show me the numbers.

Using Claude API to automatically analyze reviews and score feature priorities, I can now see "what I should build next" in about three minutes. This article walks through building that system inside a Rork app, with working code at every step.

Why "Loud Users" Distort Your Roadmap

Every indie developer I know has been burned by the same pattern. Someone who's frustrated writes a review ten times more likely than someone who's satisfied. The result is a feedback feed that skews toward problems — and within problems, toward whoever feels most strongly about them.

I spent six months building dark mode for my wallpaper app because the requests kept coming up. After shipping it, ratings barely moved. When I ran the analysis system I'll describe here, I found that dark mode requests represented 8% of all reviews. "App is slow to load" covered 23% — but those complaints were written calmly, without emotional intensity, so they didn't stand out when I was reading manually.

What AI analysis does well is separate emotional intensity from actual frequency. A quietly mentioned issue mentioned by 23% of users matters more than an angry request from 8%, even if the angry request feels louder. That's the core insight this system is built on.

Architecture Overview

The system has four layers:

Data collection layer: App Store Connect API and Google Play Developer API pull reviews automatically on a schedule
AI analysis layer: Claude API classifies each review by category, scores sentiment, and extracts themes
Scoring layer: A weighted formula combining frequency, sentiment intensity, and low-rating rates produces priority scores
Visualization layer: A dashboard inside the Rork app surfaces the results

The backend runs on Node.js with Cloudflare Workers. The Rork app fetches data via REST API. Review collection runs as a daily batch job, and results are cached in Cloudflare KV.

Step 1: Pulling Reviews from App Store Connect API

App Store Connect API uses JWT authentication. You can generate API keys directly from the App Store Connect website — no Xcode needed.

Here's a Cloudflare Worker that handles token generation and review fetching:

// workers/fetch-reviews.js
import jwt from '@tsndr/cloudflare-worker-jwt';
 
async function generateAppStoreToken(privateKey, keyId, issuerId) {
  const now = Math.floor(Date.now() / 1000);
  const payload = {
    iss: issuerId,
    iat: now,
    exp: now + 1200, // valid for 20 minutes
    aud: 'appstoreconnect-v1',
  };
 
  const token = await jwt.sign(payload, privateKey, {
    algorithm: 'ES256',
    header: { kid: keyId, typ: 'JWT' }
  });
 
  return token;
}
 
async function fetchAppReviews(appId, token, limit = 200) {
  const url =
    `https://api.appstoreconnect.apple.com/v1/apps/${appId}/customerReviews` +
    `?sort=-createdDate&limit=${limit}&filter[rating]=1,2,3,4,5`;
 
  const response = await fetch(url, {
    headers: {
      Authorization: `Bearer ${token}`,
      'Content-Type': 'application/json',
    },
  });
 
  if (!response.ok) {
    throw new Error(`App Store API error: ${response.status}`);
  }
 
  const data = await response.json();
 
  return data.data.map(review => ({
    id: review.id,
    rating: review.attributes.rating,
    title: review.attributes.title || '',
    body: review.attributes.body,
    date: review.attributes.createdDate,
    territory: review.attributes.territory,
  }));
}
 
export default {
  async fetch(request, env) {
    try {
      const token = await generateAppStoreToken(
        env.APP_STORE_PRIVATE_KEY,
        env.APP_STORE_KEY_ID,
        env.APP_STORE_ISSUER_ID
      );
 
      const reviews = await fetchAppReviews(env.APP_ID, token);
 
      // Cache in KV for 24 hours
      await env.REVIEW_CACHE.put(
        `reviews_${new Date().toISOString().split('T')[0]}`,
        JSON.stringify(reviews),
        { expirationTtl: 86400 }
      );
 
      return new Response(
        JSON.stringify({ count: reviews.length, reviews }),
        { headers: { 'Content-Type': 'application/json' } }
      );
    } catch (error) {
      return new Response(
        JSON.stringify({ error: error.message }),
        { status: 500, headers: { 'Content-Type': 'application/json' } }
      );
    }
  },
};

One important gotcha: App Store Connect API has rate limits. When I first implemented pagination to backfill older reviews, I forgot to check data.links.next before firing the next request and hit a temporary block. Always respect the pagination links and add a short delay between pages when backfilling.

Step 2: Analyzing Reviews with Claude API

This is the heart of the system. The key design choice: don't send reviews one at a time. Batch 20–30 reviews per request to keep costs under control.

// workers/analyze-reviews.js
const CLAUDE_API_URL = 'https://api.anthropic.com/v1/messages';
 
const CATEGORIES = [
  'UI/UX',
  'Performance',
  'Feature Request',
  'Crash/Bug',
  'Pricing',
  'Content Quality',
  'Competitor Comparison',
  'Other',
];
 
async function analyzeReviewsBatch(reviews, apiKey) {
  const batchSize = 20;
  const results = [];
 
  for (let i = 0; i < reviews.length; i += batchSize) {
    const batch = reviews.slice(i, i + batchSize);
    const batchText = batch
      .map(
        (r, idx) =>
          `[${idx}] Rating: ${r.rating} stars\nTitle: ${r.title}\nBody: ${r.body}`
      )
      .join('\n---\n');
 
    const response = await fetch(CLAUDE_API_URL, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'x-api-key': apiKey,
        'anthropic-version': '2023-06-01',
      },
      body: JSON.stringify({
        model: 'claude-haiku-4-5-20251001', // Haiku for cost efficiency
        max_tokens: 2000,
        messages: [
          {
            role: 'user',
            content: `Analyze the following app reviews. For each, return a JSON array with:
 
Categories: ${CATEGORIES.join(', ')}
 
Fields to return:
- index: review number from [N]
- category: single best-fit category
- sentiment: score from -1.0 (most negative) to +1.0 (most positive)
- intensity: emotional intensity from 0.0 to 1.0 (strength of feeling)
- feature_request: one-sentence summary if a specific feature is requested, null otherwise
- key_phrase: 3–5 word phrase capturing the review's core point
 
Return only the JSON array. No explanation text.
 
Reviews:
${batchText}`,
          },
        ],
      }),
    });
 
    const data = await response.json();
    const analysisText = data.content[0].text;
 
    try {
      const batchResults = JSON.parse(analysisText);
      results.push(
        ...batchResults.map(r => ({
          ...r,
          review_id: batch[r.index].id,
          rating: batch[r.index].rating,
          date: batch[r.index].date,
        }))
      );
    } catch (e) {
      console.error('JSON parse error for batch:', e);
    }
 
    // Rate limit buffer: 0.5s between batches
    await new Promise(resolve => setTimeout(resolve, 500));
  }
 
  return results;
}

Why Haiku, not Sonnet? Sentiment classification and category labeling are instruction-following tasks, not reasoning tasks. Haiku handles them with near-identical accuracy at roughly one-tenth the cost. For 3,000 reviews per month, the total Claude API cost with Haiku runs about $0.30–0.50 USD. That's a cost that survives any indie app budget.

You Sent 20 Reviews and Got 17 Back

A few weeks into running this, I noticed something odd: feeding the same month of reviews through the pipeline twice produced different rankings. "Feature requests" and "Performance" kept trading places. Nothing in the code is random. The input was identical.

The cause is at the bottom of the function above.

} catch (e) {
  console.error('JSON parse error for batch:', e);
}

When a batch fails to parse, twenty reviews disappear in exchange for one log line. The caller sees a successful return. The scoring function computes percentages against whatever arrived, so nothing downstream can tell that anything went missing.

What makes this worse than a plain bug is that the loss isn't random.

With max_tokens: 2000 and twenty reviews times six fields, each review costs roughly 90–120 output tokens. Twenty reviews lands at 1,800–2,400 — right on the boundary. What pushes a batch over is the length of feature_request, and that field gets long when someone took the time to describe exactly what they wanted. The most informative reviews are the ones most likely to truncate the batch they're sitting in.

Truncation also always hits the tail of the array. If you queue reviews newest-first, the tail is the oldest reviews, so the loss skews in time as well.

There's a second failure mode that survives a count check:

review_id: batch[r.index].id

If the model returns an index one position off, batch[r.index] is undefined and reading .id throws. That map sits inside the try, so the exception lands in the same catch — one misnumbered item takes all twenty reviews with it.

Symptom	What's actually happening	How to confirm it
Rankings shift between identical runs	Whole batches drop, and a different batch drops each time	Log input count against `results.length` on every run
"Feature requests" is suspiciously underrepresented	Long `feature_request` values inflate output and trigger truncation	Count batches where `stop_reason` is `max_tokens`
Older reviews vanish consistently	Truncation cuts the tail of the array, which is the oldest half	Plot the dates of the missing `review_id` values
Twenty reviews disappear at once	A bad `index` makes `batch[r.index]` undefined and the map throws	Temporarily remove the range check and let the exception surface
Counts match but results look shuffled	The same `index` came back twice and the later one overwrote the earlier	Count duplicate `index` values per batch

The fix isn't a smarter prompt. It's reconciliation: compare what you sent against what came back, and re-fetch only the gap.

// workers/analyze-reviews.js (with reconciliation)
 
// Size the budget from measured output, on the high side
const TOKENS_PER_REVIEW = 140;
 
async function callClaude(batch, apiKey) {
  const batchText = batch.map((r, idx) =>
    `[${idx}] Rating:${r.rating}\nTitle:${r.title}\nBody:${r.body}`
  ).join('\n---\n');
 
  const response = await fetch(CLAUDE_API_URL, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'x-api-key': apiKey,
      'anthropic-version': '2023-06-01'
    },
    body: JSON.stringify({
      model: 'claude-haiku-4-5-20251001',
      // Never a fixed number. Scale it with the batch
      max_tokens: Math.min(8000, batch.length * TOKENS_PER_REVIEW + 200),
      messages: [{ role: 'user', content: buildPrompt(batchText) }]
    })
  });
 
  if (!response.ok) throw new Error('HTTP ' + response.status);
 
  const data = await response.json();
 
  // Check this before attempting to parse. Truncation is visible here
  if (data.stop_reason === 'max_tokens') throw new Error('truncated');
 
  const parsed = JSON.parse(data.content[0].text);
  if (!Array.isArray(parsed)) throw new Error('not an array');
  return parsed;
}
 
// Resolve one batch until every review is accounted for
async function resolveBatch(batch, apiKey, depth = 0) {
  const resolved = new Map(); // batch-local index -> analysis
 
  try {
    for (const item of await callClaude(batch, apiKey)) {
      const i = Number(item.index);
      // Drop out-of-range and duplicate indices here, before they can throw
      if (!Number.isInteger(i) || i < 0 || i >= batch.length) continue;
      if (resolved.has(i)) continue;
      resolved.set(i, {
        ...item,
        review_id: batch[i].id,
        rating: batch[i].rating,
        date: batch[i].date
      });
    }
  } catch (e) {
    // Don't swallow it. Fall through to the retry below
  }
 
  const missing = batch.map((_, i) => i).filter(i => !resolved.has(i));
  if (missing.length === 0) {
    return { results: [...resolved.values()], failed: [] };
  }
 
  // If a single review still fails, record it and move on
  if (batch.length === 1 || depth >= 4) {
    return {
      results: [...resolved.values()],
      failed: missing.map(i => batch[i])
    };
  }
 
  const targets = missing.map(i => batch[i]);
  const mid = Math.ceil(targets.length / 2);
  const head = await resolveBatch(targets.slice(0, mid), apiKey, depth + 1);
  const tail = await resolveBatch(targets.slice(mid), apiKey, depth + 1);
 
  return {
    results: [...resolved.values(), ...head.results, ...tail.results],
    failed: [...head.failed, ...tail.failed]
  };
}
 
async function analyzeReviewsBatch(reviews, apiKey) {
  const batchSize = 20;
  const results = [];
  const failed = [];
 
  for (let i = 0; i < reviews.length; i += batchSize) {
    const r = await resolveBatch(reviews.slice(i, i + batchSize), apiKey);
    results.push(...r.results);
    failed.push(...r.failed);
    await new Promise(resolve => setTimeout(resolve, 500));
  }
 
  return {
    results,
    failed,
    lossRate: reviews.length === 0 ? 0 : failed.length / reviews.length
  };
}

Changing the return type from an array to { results, failed, lossRate } is deliberate. It makes the loss rate impossible for the caller to ignore.

const { results, failed, lossRate } = await analyzeReviewsBatch(reviews, apiKey);
 
if (lossRate > 0.03) {
  // Don't publish a ranking. Past 3%, the lower categories reorder
  return { status: 'degraded', lossRate, analyzed: results.length, dropped: failed.length };
}

Three percent is a practical threshold, not a derived one — it's roughly where the gap between my top two categories stopped being wider than the noise. With a smaller review volume, tighten it.

Splitting and re-fetching does mean more API calls, but only for the parts that failed, and it's still Haiku. My monthly bill barely moved. That's a poor trade to refuse when the alternative is a ranking you can't trust.

The dashboard now prints the loss rate next to the scores. Building this taught me that a ranked list arrives looking authoritative no matter how carelessly it was assembled, and I'd rather see the caveat than remember it.

Step 3: Priority Scoring Algorithm

Raw AI output isn't enough on its own. Adding a scoring layer separates "act now" from "consider later."

// utils/scoring.js
 
function calculatePriorityScore(analysisResults) {
  const categoryStats = {};
 
  for (const result of analysisResults) {
    const cat = result.category;
    if (!categoryStats[cat]) {
      categoryStats[cat] = {
        count: 0,
        sentiment_sum: 0,
        intensity_sum: 0,
        feature_requests: [],
        low_rating_count: 0,
      };
    }
 
    categoryStats[cat].count++;
    categoryStats[cat].sentiment_sum += result.sentiment;
    categoryStats[cat].intensity_sum += result.intensity;
    if (result.rating <= 3) categoryStats[cat].low_rating_count++;
    if (result.feature_request) {
      categoryStats[cat].feature_requests.push(result.feature_request);
    }
  }
 
  const totalReviews = analysisResults.length;
  const scores = [];
 
  for (const [category, stats] of Object.entries(categoryStats)) {
    const avgSentiment = stats.sentiment_sum / stats.count;
    const avgIntensity = stats.intensity_sum / stats.count;
    const frequency = stats.count / totalReviews;
    const negativity = stats.low_rating_count / stats.count;
 
    // Priority = frequency (40) + negativity (30) + intensity (20) + low-rating rate (10)
    const priorityScore =
      frequency * 40 +
      (1 - (avgSentiment + 1) / 2) * 30 +
      avgIntensity * 20 +
      negativity * 10;
 
    scores.push({
      category,
      priority_score: Math.round(priorityScore * 10) / 10,
      review_count: stats.count,
      avg_sentiment: Math.round(avgSentiment * 100) / 100,
      avg_intensity: Math.round(avgIntensity * 100) / 100,
      low_rating_rate: Math.round(negativity * 100),
      top_feature_requests: deduplicateRequests(stats.feature_requests).slice(0, 5),
    });
  }
 
  return scores.sort((a, b) => b.priority_score - a.priority_score);
}
 
function deduplicateRequests(requests) {
  const unique = [];
  const seen = new Set();
  for (const req of requests) {
    const key = req.toLowerCase().replace(/[^a-z0-9]/g, '');
    if (!seen.has(key)) {
      seen.add(key);
      unique.push(req);
    }
  }
  return unique;
}

The weight order matters: frequency (40 points) outweighs negativity (30 points). A calm complaint mentioned by 23% of users is more actionable than an angry one from 3%, even if the angry one feels more urgent when you read it manually.

Here's a before/after from my own experience running this on a wallpaper app:

Before (manual impression):

Dark mode requests seem common
Some crash reports coming in
A few people unhappy about pricing

After (scoring output):

Performance: 67.4 (frequency 23%, negative intensity 0.82)
Crash/Bug: 52.1 (frequency 14%, low-rating rate 68%)
Feature Request: 38.9 (frequency 31%, negative intensity 0.31)
Pricing: 22.3 (frequency 8%, negative intensity 0.54)

I spent six months chasing dark mode. The data said fix performance first. I fixed performance — average rating went from 4.1 to 4.4 the following month.

Step 4: Dashboard in the Rork App

Expose the analysis via a Cloudflare Workers endpoint and build the dashboard inside your Rork app:

// screens/FeedbackDashboard.js
import React, { useState, useEffect } from 'react';
import {
  View, Text, ScrollView, ActivityIndicator,
  StyleSheet, TouchableOpacity
} from 'react-native';
 
const API_BASE = 'https://your-worker.your-domain.workers.dev';
 
export default function FeedbackDashboard() {
  const [scores, setScores] = useState([]);
  const [loading, setLoading] = useState(true);
  const [selectedCategory, setSelectedCategory] = useState(null);
 
  useEffect(() => { fetchAnalysis(); }, []);
 
  async function fetchAnalysis() {
    try {
      const res = await fetch(`${API_BASE}/api/feedback-summary`);
      const data = await res.json();
      setScores(data.scores);
    } catch (err) {
      console.error('Fetch error:', err);
    } finally {
      setLoading(false);
    }
  }
 
  if (loading) return <ActivityIndicator style={styles.loader} />;
 
  return (
    <ScrollView style={styles.container}>
      <Text style={styles.title}>Feedback Priority</Text>
      <Text style={styles.subtitle}>Review analysis — last 30 days</Text>
 
      {scores.map((item, index) => (
        <TouchableOpacity
          key={item.category}
          style={[styles.card, index === 0 && styles.topCard]}
          onPress={() =>
            setSelectedCategory(
              selectedCategory === item.category ? null : item.category
            )
          }
        >
          <View style={styles.cardHeader}>
            <Text style={styles.rank}>#{index + 1}</Text>
            <Text style={styles.category}>{item.category}</Text>
            <View style={[styles.badge, { backgroundColor: scoreColor(item.priority_score) }]}>
              <Text style={styles.badgeText}>{item.priority_score}</Text>
            </View>
          </View>
 
          <View style={styles.barBg}>
            <View style={[styles.bar, {
              width: `${item.priority_score}%`,
              backgroundColor: scoreColor(item.priority_score),
            }]} />
          </View>
 
          <Text style={styles.meta}>
            {item.review_count} reviews · {item.low_rating_rate}% low-rated
          </Text>
 
          {selectedCategory === item.category && (
            <View style={styles.requests}>
              <Text style={styles.requestsTitle}>Top issues / requests</Text>
              {item.top_feature_requests.map((req, i) => (
                <Text key={i} style={styles.requestItem}>• {req}</Text>
              ))}
            </View>
          )}
        </TouchableOpacity>
      ))}
    </ScrollView>
  );
}
 
function scoreColor(score) {
  if (score >= 60) return '#EF4444';
  if (score >= 40) return '#F97316';
  if (score >= 20) return '#EAB308';
  return '#22C55E';
}
 
const styles = StyleSheet.create({
  container: { flex: 1, backgroundColor: '#F8F9FA', padding: 16 },
  loader: { flex: 1, justifyContent: 'center' },
  title: { fontSize: 22, fontWeight: '700', color: '#1A1A2E', marginBottom: 4 },
  subtitle: { fontSize: 13, color: '#6B7280', marginBottom: 20 },
  card: {
    backgroundColor: '#FFF', borderRadius: 12, padding: 16,
    marginBottom: 12, shadowColor: '#000',
    shadowOffset: { width: 0, height: 1 }, shadowOpacity: 0.08,
    shadowRadius: 4, elevation: 2,
  },
  topCard: { borderLeftWidth: 4, borderLeftColor: '#EF4444' },
  cardHeader: { flexDirection: 'row', alignItems: 'center', marginBottom: 8 },
  rank: { fontSize: 14, color: '#9CA3AF', width: 28 },
  category: { flex: 1, fontSize: 16, fontWeight: '600', color: '#1A1A2E' },
  badge: { paddingHorizontal: 10, paddingVertical: 4, borderRadius: 12 },
  badgeText: { color: '#FFF', fontSize: 13, fontWeight: '700' },
  barBg: { height: 6, backgroundColor: '#F3F4F6', borderRadius: 3, marginBottom: 8, overflow: 'hidden' },
  bar: { height: '100%', borderRadius: 3 },
  meta: { fontSize: 12, color: '#6B7280' },
  requests: { marginTop: 12, paddingTop: 12, borderTopWidth: 1, borderTopColor: '#F3F4F6' },
  requestsTitle: { fontSize: 12, fontWeight: '600', color: '#374151', marginBottom: 8 },
  requestItem: { fontSize: 13, color: '#4B5563', marginBottom: 4, lineHeight: 20 },
});

Three Pitfalls I Hit in Production

Pitfall 1: API costs spiking unexpectedly

My first version used Claude Sonnet, one review per API call. Costs hit the equivalent of several hundred dollars a month for a reasonably active app. The fix: switch to Haiku and batch 20–30 reviews per request. Same accuracy, one-tenth the cost.

Pitfall 2: Negative reviews dominating the scores

Early in the implementation, a single -0.9 sentiment review carried the same weight as ten +0.7 reviews. The fix was in the scoring formula: frequency (40 points) has to outweigh negativity (30 points). Ten calm complaints at 0.5 intensity matter more than one furious complaint at 1.0 intensity.

Pitfall 3: Classification accuracy dropping on multilingual reviews

App Store reviews for apps with Japanese and international users often mix Japanese, English, and Korean. Claude handles multilingual input well, but category definitions written only in Japanese reduced accuracy on English reviews.

The fix: add English keyword hints to category names in the prompt. "Performance (slow, lag, loading, crash, freeze)" resolves the language ambiguity quickly.

What Changes After You Have the Numbers

The thing that took me longest to accept is that listening to users and implementing what users literally asked for are two different activities. Intuition about what people want does sharpen with time, but intuition and data don't always agree — and when they disagree, I've learned to take the data seriously.

A score is a starting question, not an answer. "Launch performance, 23%" is a good reason to go find out why the app feels slow; it doesn't tell you what to change. From there I go back and read ten raw reviews in that category by hand. What this system reduced wasn't the amount of reading — it was the hesitation about which reviews to read. The final call about what to build is still mine. What's gone is the part where I read an emotionally charged review at midnight and convinced myself that whatever bothered the loudest user this week was the most urgent problem in the app.

The feedback loop gets tighter too. After shipping a performance fix, you can run the analysis again two weeks later and see whether the scores actually moved. That feedback-to-data loop is what makes the difference between shipping improvements that matter and shipping improvements that feel good to ship.

Get your App Store Connect API key set up, pull your last month of reviews, and run the analysis. The first time you see the scored categories, there's usually a moment of "oh — that's what was actually going on." Just check that the input count matches the analyzed count before you let that moment land. The ranking can wait until the arithmetic holds.

If you're running into App Store Connect API authentication issues specifically, the App Store Connect API authentication guide from Apple covers the JWT format in detail.