Building an AI-Powered Photo Organizer App with Rork — A Complete Implementation Guide Using Vision, CLIP, and pgvector

Step 2: Extracting Objects, Text, and Faces with Vision

The Vision framework got a major upgrade in iOS 17 and again in iOS 18. The relevant pieces for photo organizers are VNGenerateImageFeaturePrintRequest (visual feature embedding), VNRecognizeTextRequest (OCR), and VNGenerateObjectnessBasedSaliencyImageRequest (subject saliency).

In my production app I run four requests per image. Bundling them into a single VNImageRequestHandler.perform call lets all of them share the image decode, which buys you a 2–3× speedup over running them sequentially.

import Vision
 
struct PhotoAnalysisResult {
  let assetIdentifier: String
  let labels: [String]               // Object labels (e.g. "dog", "beach")
  let texts: [String]                // OCR'd strings
  let faceCount: Int                 // Number of faces detected
  let saliencyBox: CGRect?           // Bounding box of the subject
  let visionFeaturePrint: Data       // Vision feature embedding
  let aestheticScore: Float          // Aesthetic score (iOS 18+)
}
 
func analyzeWithVision(image: CGImage, identifier: String) async throws -> PhotoAnalysisResult {
  let handler = VNImageRequestHandler(cgImage: image, options: [:])
 
  // 1. Object classification
  let classifyRequest = VNClassifyImageRequest()
  classifyRequest.usesCPUOnly = false  // Let it use the Neural Engine
 
  // 2. OCR
  let ocrRequest = VNRecognizeTextRequest()
  ocrRequest.recognitionLevel = .accurate
  ocrRequest.recognitionLanguages = ["en-US", "ja-JP"]
 
  // 3. Face detection
  let faceRequest = VNDetectFaceRectanglesRequest()
 
  // 4. Feature print
  let featurePrintRequest = VNGenerateImageFeaturePrintRequest()
 
  try handler.perform([classifyRequest, ocrRequest, faceRequest, featurePrintRequest])
 
  let labels = (classifyRequest.results ?? [])
    .filter { $0.confidence > 0.5 }
    .prefix(10)
    .map { $0.identifier }
 
  let texts = (ocrRequest.results ?? [])
    .compactMap { $0.topCandidates(1).first?.string }
    .filter { !$0.isEmpty }
 
  let faceCount = faceRequest.results?.count ?? 0
 
  guard let featurePrint = featurePrintRequest.results?.first as? VNFeaturePrintObservation else {
    throw NSError(domain: "Vision", code: -1)
  }
 
  return PhotoAnalysisResult(
    assetIdentifier: identifier,
    labels: Array(labels),
    texts: texts,
    faceCount: faceCount,
    saliencyBox: nil,
    visionFeaturePrint: featurePrint.data,
    aestheticScore: 0.0
  )
}

Expected output: a beach-with-dog photo returns something like labels: ["dog", "beach", "ocean", "outdoor"]; a restaurant signage shot returns labels: ["restaurant", "sign"] plus texts: ["Hours", "11am-10pm"].

Why Run VNGenerateImageFeaturePrintRequest On Top of CLIP

VNFeaturePrintObservation is tuned for same-subject distance — measuring whether two photos depict the same scene from slightly different angles. CLIP measures semantic similarity — whether two photos are about the same idea. They're complementary. In my app I use Vision feature prints for "duplicate / burst detection" and CLIP for "thematic search."

This is a real tradeoff to weigh. If duplicate detection is your only goal, Vision feature prints alone are fine. If natural-language search is the focus, CLIP alone works. Carrying both doubles the embedding storage on device, so factor in the storage profile of your target users.

Step 3: Generating CLIP Embeddings via CoreML

For CoreML CLIP models, Apple ML Research's MobileCLIP family is the current pragmatic choice. MobileCLIP-S0 is the lightest, MobileCLIP-S2 the most accurate. On iPhone 14 and newer, S2 still hits ~50ms per image, so I default to MobileCLIP-S1 for the balance.

Loading and running it looks like this:

import CoreML
 
class CLIPEmbeddingService {
  private let model: MobileCLIPS1
  private let imageProcessor: VNCoreMLRequest
 
  init() throws {
    let config = MLModelConfiguration()
    config.computeUnits = .cpuAndNeuralEngine  // Prefer the Neural Engine
    self.model = try MobileCLIPS1(configuration: config)
    let coreMLModel = try VNCoreMLModel(for: model.model)
    self.imageProcessor = VNCoreMLRequest(model: coreMLModel)
    // Do not use .centerCrop for photo search — see the measurements below
    self.imageProcessor.imageCropAndScaleOption = .scaleFill
  }
 
  func embedImage(_ cgImage: CGImage) async throws -> [Float] {
    let handler = VNImageRequestHandler(cgImage: cgImage, options: [:])
    try handler.perform([imageProcessor])
    guard let result = imageProcessor.results?.first as? VNCoreMLFeatureValueObservation,
          let array = result.featureValue.multiArrayValue else {
      throw NSError(domain: "CLIP", code: -1)
    }
    // Convert to a 512-dim Float32 vector
    let count = array.count
    var embedding = [Float](repeating: 0, count: count)
    for i in 0..<count {
      embedding[i] = array[i].floatValue
    }
    // L2 normalize so pgvector's cosine distance behaves correctly
    let norm = sqrt(embedding.reduce(0) { $0 + $1 * $1 })
    guard norm > 1e-6 else { throw NSError(domain: "CLIP", code: -2) }  // all-black frames land here
    return embedding.map { $0 / norm }
  }
 
  func embedText(_ text: String) async throws -> [Float] {
    // CLIP BPE into a fixed 77-token window: SOT, tokens, EOT, then zero padding
    let ids = tokenizer.encode(text, contextLength: 77)
    let input = try MLMultiArray(shape: [1, 77], dataType: .int32)
    for (i, id) in ids.enumerated() { input[i] = NSNumber(value: Int32(id)) }
 
    let out = try textModel.prediction(input_text: input)
    guard let array = out.featureValue(for: "text_embeddings")?.multiArrayValue else {
      throw NSError(domain: "CLIP", code: -3)
    }
    var embedding = [Float](repeating: 0, count: array.count)
    for i in 0..<array.count { embedding[i] = array[i].floatValue }
 
    // Exactly the same normalization as the image side, or the dot product stops meaning anything
    let norm = sqrt(embedding.reduce(0) { $0 + $1 * $1 })
    guard norm > 1e-6 else { throw NSError(domain: "CLIP", code: -2) }
    return embedding.map { $0 / norm }
  }
}

Expected output: a 512-dimensional Float32 array, L2-normalized (norm = 1.0). That goes straight into pgvector.

I originally left the text side as a stub and moved on, which cost me a day. If you are searching with natural language, something has to project the query into the same space — without it the pipeline simply does not close. I only noticed when I sat down to build the search UI and had nothing to hand it.

The good news is that you don't have to write it. Apple publishes both the image and text encoders in Core ML form at apple/coreml-mobileclip. Two things to watch:

• The token window is fixed at 77. That's inherited from CLIP, and long queries get silently truncated. Showing a character hint next to the search field saves your users from wondering why the tail of their query had no effect.
• Keep the image and text encoders on the same variant. Pairing a MobileCLIP-S1 image encoder with an S2 text encoder throws no error — the shapes line up — and quietly returns meaningless results.

Living in the same vector space is guaranteed by the pairing, not by the types. That's the one thing here I re-check even when everything looks fine.

Why .centerCrop Is the Wrong Choice for Photo Search

.centerCrop hands the model only the centre square of the input, so on a library full of mixed aspect ratios anything near the edges disappears from the embedding entirely. How much disappears is pure geometry:

Screenshots lose more than half. Panoramas lose four fifths. If the sunset you're hunting for sits at the left edge, that photo will never surface. I spent a while blaming CLIP's accuracy for this, and finding the cause in my own preprocessing was a humbling afternoon.

Check what reaches the model before you question the model. I run that check first now, every time.

One caveat: .scaleFill ignores aspect ratio, so very elongated images get stretched. If panoramas are common in your domain, consider slicing wide images into two or three overlapping tiles, embedding each, and taking the maximum similarity across them. It's more code, but for wide-format libraries the trade is worth it.

Why L2 Normalization Is Non-Negotiable

If you use pgvector's cosine distance operator (<=>) on un-normalized inputs, the distance no longer means what you think it means. CLIP outputs are not normalized by default, so you must normalize on-device before saving. I shipped a month of incorrect search results because I'd skipped this step — don't repeat my mistake.

Step 4: A Hybrid Search Backend on Supabase pgvector

Here's the schema. Each image gets both Vision-derived structured metadata and a CLIP embedding, with a pgvector HNSW index for sub-50ms top-K queries even at a million rows.

-- Enable pgvector
create extension if not exists vector;
 
-- Photo metadata table
create table photo_assets (
  id uuid primary key default gen_random_uuid(),
  user_id uuid references auth.users(id) on delete cascade,
  asset_identifier text not null,         -- iOS PHAsset.localIdentifier
  taken_at timestamptz,
  labels text[],                          -- Vision object labels
  ocr_text text,                          -- OCR text (for GIN index)
  face_count int default 0,
  embedding vector(512),                  -- CLIP embedding
  created_at timestamptz default now(),
  unique(user_id, asset_identifier)
);
 
-- HNSW index for cosine similarity
create index on photo_assets
  using hnsw (embedding vector_cosine_ops)
  with (m = 16, ef_construction = 64);
 
-- GIN index for label search
create index on photo_assets using gin (labels);
 
-- Full-text search on OCR
create index on photo_assets using gin (to_tsvector('simple', ocr_text));
 
-- RLS
alter table photo_assets enable row level security;
create policy "users can read own assets"
  on photo_assets for select
  using (auth.uid() = user_id);
create policy "users can insert own assets"
  on photo_assets for insert
  with check (auth.uid() = user_id);

The hybrid query function combines a natural-language query ("sunset at the beach") with a metadata filter (labels contains beach):

-- Hybrid search: NL embedding + metadata filter
create or replace function search_photos_hybrid(
  query_embedding vector(512),
  filter_labels text[],
  match_count int default 30
)
returns table (
  id uuid,
  asset_identifier text,
  labels text[],
  similarity float
)
language sql stable as $$
  select
    p.id,
    p.asset_identifier,
    p.labels,
    1 - (p.embedding <=> query_embedding) as similarity
  from photo_assets p
  where p.user_id = auth.uid()
    and (filter_labels is null or p.labels && filter_labels)
  order by p.embedding <=> query_embedding
  limit match_count;
$$;

Pass the text-encoded "sunset at the beach" embedding as query_embedding and ARRAY['beach', 'ocean'] as filter_labels, and you'll get the 30 most semantically similar photos that also carry a beach or ocean tag, ordered by similarity descending.

The Tighter You Filter, the Fewer Results You Get

Except that it often doesn't behave that way. The more you narrow by tag, the fewer rows come back — sometimes 3 rows when you asked for 30. And nothing errors.

The reason is that the where clause runs after the HNSW index has already chosen its candidates. The index pulls a batch of near neighbours, then labels && filter_labels filters that batch. If most candidates fail the filter, what's left never reaches your limit. Your library might hold 800 beach photos, but only 3 of them made the vector-nearest shortlist.

pgvector 0.8.0 added iterative index scans for exactly this. When the filtered batch comes up short, the index keeps walking the graph until the limit is satisfied or the scan budget runs out.

create or replace function search_photos_hybrid(
  query_embedding vector(512),
  filter_labels text[],
  match_count int default 30
)
returns table (
  id uuid,
  asset_identifier text,
  labels text[],
  similarity float
)
language plpgsql stable as $$
begin
  -- Keep walking the graph until match_count rows survive the filter
  set local hnsw.iterative_scan = 'relaxed_order';
  set local hnsw.max_scan_tuples = 20000;   -- scan budget (default)
  set local hnsw.ef_search = 100;           -- candidates per pass; default is 40
 
  return query
    select
      p.id,
      p.asset_identifier,
      p.labels,
      1 - (p.embedding <=> query_embedding) as similarity
    from photo_assets p
    where p.user_id = auth.uid()
      and (filter_labels is null or p.labels && filter_labels)
    order by p.embedding <=> query_embedding
    limit match_count;
end;
$$;

Choosing between strict_order and relaxed_order comes down to what the UI shows. Photo search renders a grid of thumbnails, and nobody notices when rank 1 and rank 3 swap places. Surfacing the photo that exists matters more here than ordering it perfectly, so I use relaxed_order.

The set local matters: session-level set would leave iterative scanning switched on for every other query in the connection, including the ones that don't need it.

Three Performance Pitfalls You Will Hit

In the order I hit them:

1. Reaching for ivfflat instead of HNSW. Older articles still recommend ivfflat. With pgvector 0.5+, HNSW is dramatically faster and easier to tune. Start at m = 16, ef_construction = 64 and bump m only if recall needs it.

2. Storing embedding as text. Some client libraries make it tempting to keep the JSON-encoded vector as a text column. HNSW won't kick in and queries become full scans. Always type the column as vector(512).

3. Upserting with on conflict ... do update that touches every column. Vision results get re-computed on the device fairly often; a full upsert against embedding every time is wasteful (CLIP outputs are deterministic — there's no point recomputing). Lock the embedding column to "compute once" semantics.

Step 5: Background Jobs That Tame 50,000 Images

This is the layer where production apps separate from prototypes. If you process the whole library in the foreground, users will close the app — my first prototype lost 70% of users to that single decision. With BGProcessingTask, indexing happens overnight while the phone is charging, and users wake up to a fully indexed library.

import BackgroundTasks
 
let backgroundTaskIdentifier = "com.example.photoorganizer.indexing"
 
func registerBackgroundTask() {
  BGTaskScheduler.shared.register(
    forTaskWithIdentifier: backgroundTaskIdentifier,
    using: nil
  ) { task in
    handleBackgroundIndexing(task: task as! BGProcessingTask)
  }
}
 
func scheduleBackgroundIndexing() {
  let request = BGProcessingTaskRequest(identifier: backgroundTaskIdentifier)
  request.requiresNetworkConnectivity = true   // For pgvector sync
  request.requiresExternalPower = true          // CLIP draws real power
  request.earliestBeginDate = Date(timeIntervalSinceNow: 60 * 60)  // 1 hour from now
 
  do {
    try BGTaskScheduler.shared.submit(request)
  } catch {
    print("BGTask scheduling failed: \(error)")
  }
}
 
func handleBackgroundIndexing(task: BGProcessingTask) {
  scheduleBackgroundIndexing()  // Schedule the next run
 
  let queue = OperationQueue()
  queue.maxConcurrentOperationCount = 2  // Avoid Neural Engine contention
 
  task.expirationHandler = {
    queue.cancelAllOperations()
  }
 
  Task {
    do {
      let pending = try await fetchPendingAssets(limit: 500)
      for asset in pending {
        guard !task.isCancelled else { break }
        try await processAsset(asset)
      }
      task.setTaskCompleted(success: true)
    } catch {
      task.setTaskCompleted(success: false)
    }
  }
}

With requiresExternalPower = true, the job runs while the device is plugged in (typically overnight) and processes 500 images per run. Pushing earliestBeginDate out an hour means even bursty users finish their library in 1–2 days.

Giving Power Users an Opt-Out

Some users want results now. Offer a foreground "Index now" mode, but batch in groups of 20, not one-by-one, and surface granular progress in the UI ("23,400 / 50,000 photos processed"). The biggest retention bump in my A/B tests came from adding an "about 12 minutes remaining" estimate alongside the progress bar.

Step 6: The Search UI and AI-Generated Smart Albums

Search itself is mechanical: type "sunset at the beach", encode the query through the CLIP text encoder, hit search_photos_hybrid from Step 4, render results. The more interesting feature is automatic album generation.

Run k-means clustering in CLIP embedding space and natural thematic albums emerge without users defining anything. My production app uses 50 clusters; the image closest to each cluster centroid becomes the cover; then I ship the top five label/OCR/timestamp summaries to Claude or Gemini and ask for a 10-character album title. The result is human-quality album names like "Summer at the Beach," "Coffee Shops Around Town," and "Family Holidays."

// React Native side — call a Supabase Edge Function
async function generateSmartAlbums() {
  const response = await fetch(
    `${SUPABASE_URL}/functions/v1/generate-albums`,
    {
      method: 'POST',
      headers: {
        Authorization: `Bearer ${session.access_token}`,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({ cluster_count: 50 }),
    }
  );
  const albums = await response.json();
  // [{ id, title, cover_asset_identifier, asset_count }, ...]
  return albums;
}

Expected response: an array of 50 album records with the Claude-generated title, cover asset_identifier, and asset count. In my testing, feeding Claude Sonnet the top-5 metadata snippets per cluster (labels, ocr_text, taken_at month) yields names that pass human-review 90%+ of the time.

Five Production Pitfalls You Will Definitely Hit

These are the ones I ran into post-launch. Knowing them up front saves real hours.

1. More users have "Optimize iPhone Storage" enabled than you'd guess. ~60% of mine did, meaning full-resolution originals weren't on-device. Make isNetworkAccessAllowed conditional on Wi-Fi connectivity; otherwise skip the asset and queue it for later.

2. CLIP embedding versions can quietly destroy search quality overnight. When I upgraded from MobileCLIP-S1 to S2, the new and old embeddings inhabited different vector spaces — and the search index started returning nonsense. Always carry an embedding_model_version column and either filter mismatches out of search or schedule a re-embedding job.

3. HNSW index construction takes longer than you'd think. Building an HNSW index over 50,000 × 512-dim took ~8 minutes on Supabase's Free plan and 2–3 minutes on Pro. Discover this in load testing, not at launch — otherwise you'll have a blocked release window.

4. App Tracking Transparency rules aren't strictly about PhotoKit, but reviewers ask anyway. PhotoKit isn't an ATT-gated API, but if you analyze EXIF (especially location data), have a one-paragraph reviewer-facing rationale ready. I got asked once.

5. Loading 50,000 PHAssets into a Swift array eats 100+ MB of RAM. PHFetchResult is lazy; the moment you enumerateObjects into an array you load it all. Page in chunks of 1,000 or use PHFetchResult.objectAt(_:) directly.

Beyond Photo Organizers — Where This Pipeline Travels Well

Everything above is reusable far beyond a photo organizer:

• E-commerce visual search: Embed your inventory photos with CLIP; let users upload a "something like this" image and get back semantically similar products from your catalog.
• Real estate listings: Embed property interior photos and let users search with phrases like "bright modern kitchen" or "cozy living room with a fireplace." MobileCLIP handles English and Japanese both, so multilingual UX comes nearly free.
• Education / note apps: OCR handwritten notes via Vision, embed the page contents with CLIP, and let students search "the page where I drew the calculus graph" semantically.

I personally reused the same pipeline I built for my photo organizer on a completely different category app a few months later. Once the embedding-based search foundation is in place, it travels well across product categories — that's the real long-term payoff of investing in this design.

Operational Patterns That Took Me Months to Get Right

Beyond the per-image pipeline, three operational decisions had outsized effects on whether the app felt fast and trustworthy at scale. I'm sharing them because none of them are obvious from documentation, and all three quietly determined whether the experience held together.

Throttling the Neural Engine So the Phone Stays Usable

Running CLIP back-to-back saturates the Neural Engine within seconds, and on iPhones older than the 14, the OS will start preempting your work and the device gets noticeably warm. After several user reports of "my phone is hot when this app is running," I introduced a soft throttle that pauses the embedding queue when CPU temperature reaches a threshold reported via ProcessInfo.thermalState. The trigger is simple:

import Foundation
 
func shouldPauseProcessing() -> Bool {
  switch ProcessInfo.processInfo.thermalState {
  case .nominal, .fair:
    return false
  case .serious:
    return true   // Pause for 30 seconds, then re-check
  case .critical:
    return true   // Suspend the queue entirely
  @unknown default:
    return false
  }
}

I poll this every 100 images and pause the queue when the state escalates. The result was zero "phone is hot" complaints in the next release, and indexing throughput dropped only ~15% on average — a fair trade for a cooler device and longer battery life.

Versioning the Vision and CLIP Models Together

Steps 2 and 3 each produce data that looks like normal columns, but the underlying numerical interpretation depends on the model version. When iOS 18 shipped a refined VNClassifyImageRequest, the same image started producing slightly different label distributions, and my "duplicate detection" logic flagged half the library as new. Track both versions explicitly:

struct AnalysisVersion: Codable {
  let visionOSBuild: String           // ProcessInfo.osVersionString
  let clipModel: String               // "MobileCLIP-S1@v1.0.0"
  let analyzedAt: Date
}

Persist this struct alongside the row in SQLite and Postgres. When you migrate a user to a new version, decide deliberately whether to invalidate prior rows or just keep them with a "needs reprocessing" flag.

Privacy Pages and Reviewer Notes That Pre-Empt Rejections

App Store reviewers ask about photo access more often than any other capability I've shipped. Two artifacts saved me a rejection cycle each:

• A screen recording that demonstrates the app not uploading raw photo bytes to a server — only embeddings (which are mathematically not reversible to images). I attach this as a video to the App Store Connect notes.
• A short reviewer note in the submission form: "Photo Library access is required to compute on-device CLIP embeddings (numerical vectors). Original photos and image bytes never leave the device. Server-side data is limited to 512-float vectors and Vision-derived metadata (object labels, OCR text). See attached video for confirmation."

Both add maybe ten minutes of work per submission and have shaved review-cycle time considerably in my experience.

Cost Modeling at 50,000 Images per User

A question I get asked often is "what does this actually cost to run?" Cost shapes pricing, which shapes whether the app survives. Here's the math I run for a hypothetical 1,000-active-user app where each user has 50,000 photos.

On-device cost (free to me, paid in user time and battery): Roughly 50ms per image for CLIP plus 80ms for the Vision bundle = 130ms × 50,000 = ~108 minutes of compute per user, spread across the background scheduler over 1–3 days. Battery impact is real but bounded by requiresExternalPower = true.

Server storage: 50,000 × 512 floats × 4 bytes = 100 MB of vector data per user, plus another 50 MB for OCR text and labels. At 1,000 users, that's 150 GB total. Supabase Pro at $25/month handles this comfortably without an addon.

Server compute (search queries): HNSW search hits the index in 20–50ms regardless of dataset size, and Supabase doesn't charge per query. The natural limit is connection count, not query count, so as long as you reuse connections via a pooler, this scales for nearly free.

Outbound bandwidth: Search results return only asset_identifier strings (the actual images stay on-device), so a typical search response is ~5 KB. Even at 100,000 searches per month total, that's 500 MB of egress — within free-tier limits on most providers.

External AI model calls: The smart album naming step in Step 6 uses Claude Sonnet at roughly $0.003 per 1,000 input tokens. Each cluster naming call sends about 200 tokens and gets a 30-token response, so 50 clusters per user ≈ $0.0015. Even daily re-clustering costs less than $0.05 per user per month.

Bottom line: The all-in monthly cost per user lands somewhere around $0.10–$0.15 on infrastructure plus token spend. A $4.99/month subscription clears that comfortably with margin for App Store fees, churn, and growth investments. This is the kind of unit economics that lets a single indie developer ship and operate this category profitably.

If your numbers don't pencil out at this granularity, something is wrong with the architecture — usually you're storing images server-side when you don't need to, or rebuilding the index too aggressively. Walk through the math before launch, not after.

Wrapping Up — The Smallest First Step You Can Take Today

You don't need to ship the whole pipeline at once. Start with Steps 1 and 2 only and confirm you can extract object labels from a user's library. Vision alone is enough to ship "list all dog photos" or "search photos containing text" — release that, listen to your users, then layer CLIP and pgvector in based on actual demand instead of guessing.

My own first release was Vision-only. CLIP came later, after reviews and support mail kept asking for freer, more natural search. I'm glad the order wasn't reversed. Design at your desk, prioritise from how it gets used. That's the sequence I hold onto even when a deadline is close.

The same logic applies to your server side. You don't need pgvector on day one — start with a fully on-device experience using SQLite for vector storage, and migrate to Supabase pgvector only when you need cross-device sync or when local search at scale starts to feel slow. Each step you defer is a step that doesn't risk killing momentum during the riskiest period of a new app: the first hundred days. The architecture in this article is the destination, not the starting line. Plot a path through it that matches your tolerance for risk and the time you have available.

For deeper context, the theory side of the search backend lives in Building a Semantic Search App with Rork × Supabase pgvector, the on-device model side is covered in Rork × Core ML Custom Models and On-Device AI, and the cross-modal design work in Rork × Multimodal AI Design Patterns. Reading them alongside this article gives you a fuller picture of the moving pieces.

Thank you for staying with a long one.