◉ AI Models/2026-03-28Advanced

Rork × Core ML Custom Model Development × On-Device AI

Notes from training a model in Create ML, shipping it inside a Rork Max app with Core ML, and deciding per request whether inference runs on device or in the cloud — with measured routing, quantization, Vision and Natural Language integration, and OTA model swaps.

rork-max⁴⁰core-mlmachine-learningon-device-aivision-frameworkapp-development¹²

✦ Premium Article

The Flight Where Half My AI Features Stopped Working

I tried the auto-tagging feature I'd just shipped while sitting on a plane. The spinner kept turning after I finished typing, and nothing came back before we landed. Obvious in hindsight — the request was going to a cloud API — but staring at that screen is when it hit me that the feature had only ever existed on top of a network connection.

That flight is why I started moving inference on device. Privacy and cost became reasons later. The first motivation was simply that it should work with no signal.

Rork Max works with Apple's Core ML directly: train in Create ML, quantize, ship the model inside the app, and keep inference local. What follows is that whole path, including the places where I got stuck.

Moving everything on device didn't solve everything, though. So we start with where the line actually sits.

On-Device AI Architecture and Core ML's Role

Understanding On-Device AI

On-device AI executes ML inference directly on the device's processors (CPU, GPU, Neural Engine) rather than sending data to cloud APIs. This approach delivers compelling advantages.

Key Benefits

Privacy by Default: Data never leaves the device
Sub-Second Latency: Millisecond-level inference without network roundtrips
Offline Operation: Works without internet connectivity
Cost Efficiency: No API costs or infrastructure overhead
Battery Optimization: Modern chips (Neural Engine) are power-efficient for inference

Core ML: The Unified Interface

Apple's Core ML is a unified framework that converts models trained in popular frameworks—TensorFlow, PyTorch, Scikit-learn, and others—into a single format (.mlmodel) that runs efficiently on iOS devices.

Core ML Strengths

Multi-Framework Support: TensorFlow, PyTorch, ONNX all convert to .mlmodel
Automatic Optimization: Intelligently dispatches to CPU, GPU, or Neural Engine
Tight Integration: Works seamlessly with Vision, Natural Language, and other frameworks
Memory Efficiency: Built-in quantization reduces model footprint significantly

Rork Max provides native support for Core ML, enabling you to integrate models without touching complex Swift code.

✦

Thank you for reading this far.

Continue Reading

What follows includes implementation code, benchmarks, and practical content we hope you'll find useful. This site runs without ads — server and development costs are supported entirely by members like you. If it's been helpful, we'd be truly grateful for your support.

WHAT YOU'LL LEARN

✦Custom model training in Create ML through Core ML quantization, step by step

✦End-to-end Vision and Natural Language integration inside a Rork Max app

✦An inference router that splits on-device and cloud work — measured at 67% local and 58% lower API spend

Secure payment via Stripe · Cancel anytime

✦

Unlock This Article

Get full access to the rest of this article. Buy once, read anytime. This site is ad-free — your support goes directly toward keeping it running.

Unlock all articles with Membership →

Knowing When Not to Run On-Device: Routing Inference Decisions

Read this far and on-device inference starts to look like the answer to everything.

I thought so too. My plan was to push every inference in my app onto Core ML and drive the API bill to zero. That plan fell apart when a user wrote in to say the app "only knows old things."

A model baked into the device knows exactly the world it was trained on. You get low latency and privacy, and you hand back freshness and reasoning depth in exchange.

The practical answer isn't Core ML or cloud. It's choosing a target per request.

What Actually Decides the Target

The nature of the request settles it almost every time. Here are the axes I use in production.

Request characteristic	Target	Why
Carries personal data, location, or photos	Core ML (on-device)	The data never leaves the device — treat this as a promise, not a preference
Must work offline	Core ML (on-device)	Anything network-dependent breaks the moment signal drops
Short and high-frequency (completion, classification, tagging)	Core ML (on-device)	The round trip costs more than the inference itself
Needs knowledge newer than the training cutoff	Cloud LLM	An on-device model cannot answer this in principle
Needs multi-step reasoning or long-form generation	Cloud LLM	A quantized small model won't hold up

"When in doubt, stay local" is a fine default. Making the outbound decision carry the burden of proof — an asymmetric rule — is what keeps a privacy design from eroding over time.

Keep the Router in One Module

Scatter this decision across your screens and it will collapse within six months. Put it in one place.

// src/ai/inferenceTarget.ts
 
export type InferenceTarget = 'core-ml' | 'cloud';
 
export interface InferenceInput {
  text: string;
  /** Photos, location, contacts — anything that shouldn't leave the device */
  hasSensitivePayload: boolean;
  /** Asking for knowledge newer than the model's training cutoff */
  needsFreshKnowledge: boolean;
  /** Multi-step reasoning or long-form generation */
  needsDeepReasoning: boolean;
  isOffline: boolean;
}
 
export interface TargetDecision {
  target: InferenceTarget;
  /** For logs and debugging only — never surfaced in the UI */
  rationale: string;
}
 
const SHORT_INPUT_LIMIT = 50;
 
export function selectInferenceTarget(input: InferenceInput): TargetDecision {
  // Conditions that forbid leaving the device are evaluated before anything else
  if (input.isOffline) {
    return { target: 'core-ml', rationale: 'offline' };
  }
  if (input.hasSensitivePayload) {
    return { target: 'core-ml', rationale: 'sensitive-payload' };
  }
 
  // Only send out what the local model genuinely cannot answer
  if (input.needsFreshKnowledge) {
    return { target: 'cloud', rationale: 'knowledge-cutoff' };
  }
  if (input.needsDeepReasoning) {
    return { target: 'cloud', rationale: 'reasoning-depth' };
  }
 
  // Short inputs don't justify a round trip
  if (input.text.length <= SHORT_INPUT_LIMIT) {
    return { target: 'core-ml', rationale: 'short-input' };
  }
 
  return { target: 'core-ml', rationale: 'default-local-first' };
}

The character of this function lives in that last branch returning 'core-ml' rather than 'cloud'.

Anything ambiguous falls local. A request only leaves the device when it matches one of the two explicit reasons above. With the fallback pointed at the safe side, you can keep adding conditions over the years without quietly springing a leak.

Sending rationale to logs but never to the UI is deliberate too. A "processing in the cloud" indicator is handy while you're building and just offloads a decision onto your user once you ship.

Measure the Split

Judge the routing by ratios, not by feel. Three numbers are enough.

// src/ai/inferenceStats.ts
import AsyncStorage from '@react-native-async-storage/async-storage';
 
const KEY = 'inference_stats_v1';
 
type Bucket = { coreML: number; cloud: number; cloudSpendUsd: number };
 
export async function recordInference(
  target: 'core-ml' | 'cloud',
  spendUsd: number
): Promise<void> {
  const raw = await AsyncStorage.getItem(KEY);
  const bucket: Bucket = raw
    ? JSON.parse(raw)
    : { coreML: 0, cloud: 0, cloudSpendUsd: 0 };
 
  if (target === 'core-ml') {
    bucket.coreML += 1;
  } else {
    bucket.cloud += 1;
    bucket.cloudSpendUsd += spendUsd;
  }
 
  await AsyncStorage.setItem(KEY, JSON.stringify(bucket));
}
 
/** Share handled on-device. When this slips, suspect the routing rules. */
export async function localHitRate(): Promise<number> {
  const raw = await AsyncStorage.getItem(KEY);
  if (!raw) return 0;
  const b: Bucket = JSON.parse(raw);
  const total = b.coreML + b.cloud;
  return total === 0 ? 0 : b.coreML / total;
}

Here's a week of measurements from my own assistant-style app.

Metric	Measured
Handled entirely on-device	67%
Average latency (Core ML)	28ms
Average latency (cloud LLM)	1,240ms
Monthly API spend	$45 → $19 (down 58%)

When localHitRate() starts falling, look at the routing rules before you look at model accuracy. In my case the cause was always the same: I'd been too casual about adding cloud-bound conditions.

What Surfaces Once Routing Is In

Three things I only noticed after shipping this.

Model initialization is slow exactly once. A request arriving right after launch turns the MLModel load into felt wait time. Warm it in the background at startup, accept input while it loads, and queue internally until it's ready. In Rork Max, adding "initialize the Core ML model in the background at app launch" to the prompt is enough to get this.

Nothing releases the model on a memory warning. An on-device model stays resident. Without a release path at the didReceiveMemoryWarning equivalent — and a reload on the next inference — the app dies the moment someone uses it alongside a photo app.

On iOS 26 you can pull some cloud work back. Apple's FoundationModels runs an LLM on device, so a slice of what selectInferenceTarget() was sending to 'cloud' — summaries, rewrites — can move back inside. I wrote up that design separately in using Apple FoundationModels in a Rork app.

Core ML is iOS-only. To give Android the same experience, put TensorFlow Lite or MediaPipe's inference API behind the same seam, leave the return type of selectInferenceTarget() untouched, and swap only the implementation. This is where keeping the decision in one module pays for itself.

Training Custom Models with Create ML

Create ML Essentials

Create ML is Apple's macOS application that integrates with Xcode and lets you train ML models through an intuitive visual interface.

Supported Task Types

Image Classification: Distinguish between categories (dog breeds, plant species, defects)
Object Detection: Locate and label objects within images
Sound Classification: Recognize audio patterns (ambient sounds, machinery noise)
Text Classification: Categorize text (sentiment, spam detection, intent)
Tabular Data: Predict numerical values or categories from structured data

Practical Walkthrough: Training a Defect Detection Model

Let's train a real-world example: an automated quality control model for manufacturing.

Step 1: Prepare Your Dataset

Create ML uses folder structure for organization. Each category becomes a folder:

dataset/
├── OK/
│   ├── good_001.jpg
│   ├── good_002.jpg
│   └── ... (500+ images)
└── DEFECT/
    ├── bad_001.jpg
    ├── bad_002.jpg
    └── ... (500+ images)

Best practice: 500-5,000 images per class. Data quality directly influences accuracy.

Step 2: Train in Create ML

// In Xcode:
// 1. File > New > ML Model
// 2. Select "Image Classification"
// 3. Point to your dataset/ folder
// 4. Reserve 10-20% for validation
// 5. Click Train

Step 3: Evaluate and Iterate

Create ML displays accuracy and loss curves in real-time:

Accuracy ≥ 0.95: Production-ready
0.85-0.94: Add more data or improve preprocessing
< 0.85: Reconsider model architecture

Healthy training shows Train and Validation loss decreasing together. If Validation loss increases while Training loss drops, your model is overfitting—add more validation data.

Accelerate with Transfer Learning

Training from scratch takes hours. Apple provides pre-trained models (VGG, ResNet) that you can fine-tune in minutes.

In the Create ML interface, select a pre-trained architecture under "Architecture":

Benefits

90% faster training: Hours → seconds
5-10x less data needed: Maintains accuracy with smaller datasets
Better generalization: Retains useful image features

Core ML Model Optimization and Size Reduction

Understanding Model File Size

Create ML outputs .mlmodel files that can be surprisingly large.

The App Store limit is 4 GB uncompressed for the whole iOS or iPadOS app, and 75 MB for watchOS (Maximum build file sizes).

4 GB sounds like plenty of room. What actually bit me first wasn't the ceiling — it was the first-download completion rate. The same thing happened in my wallpaper app when I let the bundle grow. People stop waiting long before Apple stops you.

Typical Sizes

ResNet50 (image classification): ~100MB
YOLOv5 (object detection): 50-150MB (varies by precision)
BERT (text): ~500MB (full version)

Quantization: Shrink Without Losing Accuracy

Quantization converts floating-point numbers (Float32) to integer formats (Int8), reducing memory and computation.

Create ML's Quantization Options

After training, you'll see a Quantization section:

- Full Precision (Float32): Best accuracy, largest file
- Float16: ~50% smaller, minimal accuracy loss
- Int8 (Quantized): ~75% smaller, typically 1-3% accuracy drop

Inspecting the bundled model:

import CoreML
 
// The generated class is the most reliable route
// (defect-detection.mlmodel becomes defect_detection)
let wrapper = try defect_detection(configuration: MLModelConfiguration())
print(wrapper.model.modelDescription)
 
// If you load by URL, the bundle contains the .mlmodelc, not the .mlmodel
guard let url = Bundle.main.url(forResource: "defect-detection", withExtension: "mlmodelc") else {
    throw MLError.modelNotFound
}
let model = try MLModel(contentsOf: url)
print(model.modelDescription)

I wrote withExtension: "mlmodel" here first, got nil back, and force-unwrapped it into a crash. Xcode compiles .mlmodel at build time, so what ships inside the bundle is a .mlmodelc directory — and MLModel(contentsOf:) only accepts a compiled model (init(contentsOf:)).

Advanced Optimization with Python

For fine-grained control, use Apple's coremltools library:

# pip install coremltools tensorflow
 
import coremltools as ct
import coremltools.optimize as cto
import tensorflow as tf
 
tf_model = tf.keras.models.load_model('defect_model.h5')
 
# Class labels go through classifier_config, not outputs
class_labels = ["ok", "scratch", "dent"]
 
mlmodel = ct.convert(
    tf_model,
    inputs=[ct.ImageType(name="image", shape=(1, 224, 224, 3))],
    classifier_config=ct.ClassifierConfig(class_labels),
    convert_to="mlprogram",
    minimum_deployment_target=ct.target.iOS17,
    compute_units=ct.ComputeUnit.CPU_AND_NE,
)
 
# Weight quantization for an ML Program goes through this API
op_config = cto.coreml.OpLinearQuantizerConfig(
    mode="linear_symmetric", weight_threshold=512
)
config = cto.coreml.OptimizationConfig(global_config=op_config)
quantized_mlmodel = cto.coreml.linear_quantize_weights(mlmodel, config=config)
 
# Save the quantized model — an ML Program is a .mlpackage
quantized_mlmodel.save('DefectDetection.mlpackage')

Twenty lines, and I tripped over two of them.

The first was the quantization call. quantization_utils.quantize_weights, the one you'll find in older write-ups, applies to the legacy neural network format — not to the ML Program that ct.convert produces by default. ML Programs go through coremltools.optimize.coreml.linear_quantize_weights with an OptimizationConfig (API Overview). Passing class labels to outputs= failed on the same afternoon; they belong in classifier_config (Classifiers).

The second one was sillier. I assigned the quantized model to a new variable and then saved the pre-quantization one. No error, a perfectly valid file, and a size that never went down — I didn't notice until I opened it in Xcode. These compression APIs return a new model rather than mutating the one you handed them. Forget to save what you were given, and all you're left with is the result of having done nothing.

Memory Optimization at Runtime

Minimize memory footprint during inference:

// Load once, and keep a way to let go of it
actor ModelCache {
    static let shared = ModelCache()
    private var cached: MLModel?
 
    func model() throws -> MLModel {
        if let cached { return cached }
        guard let url = Bundle.main.url(forResource: "defect-detection",
                                        withExtension: "mlmodelc") else {
            throw MLError.modelNotFound
        }
        let loaded = try MLModel(contentsOf: url)
        cached = loaded
        return loaded
    }
 
    /// Call on a memory warning; the next inference reloads it
    func evict() {
        cached = nil
    }
}
 
// Use autoreleasepool to free temporary allocations
autoreleasepool {
    let prediction = try model.prediction(input: input)
    // Process prediction
}

Leave the try! in and the app dies the moment the load fails. A bundled model feels like something you can count on being there, right up until the build where someone renames the file.

Integrating Core ML Models into Rork Max

Model Management in Rork Max

Rork Max treats Core ML models as first-class assets with automatic optimization.

Integration Flow

Upload .mlmodel to Rork Max's Assets panel
Rork automatically optimizes (quantization, platform tuning)
Reference the model in your visual layout
Deploy to App Store—model is bundled automatically

No-Code Integration Example: Image Classification

Build this flow visually in Rork Max:

[Camera View] → [Core ML Model] → [Classification] → [Result Display]

The Generated Code

import CoreML
import Vision
 
struct QualityCheckView: View {
    @State private var cameraImage: UIImage?
    @State private var result: String = ""
 
    var body: some View {
        VStack {
            Image(uiImage: cameraImage ?? UIImage())
                .resizable()
                .scaledToFit()
 
            Button("Analyze") {
                analyzeImage()
            }
 
            Text(result)
                .font(.headline)
                .padding()
        }
    }
 
    private func analyzeImage() {
        guard let image = cameraImage else { return }
 
        let model = try? defect_detection(configuration: MLModelConfiguration())
        guard let buffer = image.pixelBuffer(size: CGSize(width: 224, height: 224)) else {
            return
        }
 
        let input = defect_detectionInput(imageAt: buffer)
        guard let output = try? model?.prediction(input: input) else {
            result = "Analysis failed"
            return
        }
 
        let confidence = output.classLabelProbs.max(by: { $0.value < $1.value })
        result = "Status: \(confidence?.key ?? "Unknown") (\(String(format: "%.1f%%", (confidence?.value ?? 0) * 100)))"
    }
}

Adding Custom Logic

Rork Max supports "Custom Swift" sections for advanced processing:

// Batch inference across multiple frames
func processMultipleFrames(_ frames: [UIImage]) -> [String] {
    let model = try! defect_detection(configuration: MLModelConfiguration())
    var results: [String] = []
 
    for frame in frames {
        guard let buffer = frame.pixelBuffer(size: CGSize(width: 224, height: 224)) else {
            continue
        }
 
        let input = defect_detectionInput(imageAt: buffer)
        let output = try! model.prediction(input: input)
        results.append(output.classLabel)
    }
 
    return results
}

Vision Framework Integration: Image Recognition and Object Detection

Vision Framework's Role

Vision works hand-in-hand with Core ML to preprocess images and post-process detections efficiently.

If you're heading toward camera, voice, and text living on the same screen, how you shape the input surface matters before any of this does — I collected those patterns in multimodal AI app design patterns for Rork.

Key Capabilities

Face Detection & Recognition: NSFW detection, emotion analysis
Object Detection: Fast, accurate region-based detection
Barcode & QR Recognition: Read codes in real-time
Text Recognition (OCR): Extract text from images
Homography Estimation: Perspective correction

Practical Example: Object Detection

Integrate a YOLO model converted to Core ML:

import Vision
import CoreML
 
class ObjectDetector {
    let model: defect_yolo
 
    init() throws {
        self.model = try defect_yolo(configuration: MLModelConfiguration())
    }
 
    func detect(in image: UIImage) -> [DetectionResult] {
        guard let ciImage = CIImage(image: image) else { return [] }
 
        let request = VNCoreMLRequest(model: try! model.model) { request, error in
            guard let observations = request.results as? [VNRecognizedObjectObservation] else {
                return
            }
 
            for observation in observations {
                let boundingBox = observation.boundingBox
                let confidence = observation.confidence
                let label = observation.labels.first?.identifier ?? "Unknown"
 
                print("Detected: \(label) at \(boundingBox) with \(confidence * 100)% confidence")
            }
        }
 
        let handler = VNImageRequestHandler(ciImage: ciImage, options: [:])
        try? handler.perform([request])
 
        return []
    }
}
 
struct DetectionResult {
    let label: String
    let boundingBox: CGRect
    let confidence: Float
}

Vision in Rork Max: Annotation Overlays

Rork Max provides visual components to annotate detection results:

func drawAnnotations(on image: CGImage, detections: [DetectionResult]) {
    let size = CGSize(width: image.width, height: image.height)
 
    UIGraphicsBeginImageContextWithOptions(size, false, 1.0)
    UIImage(cgImage: image).draw(at: .zero)
 
    let context = UIGraphicsGetCurrentContext()!
    context.setStrokeColor(UIColor.red.cgColor)
    context.setLineWidth(2.0)
 
    for detection in detections {
        let rect = CGRect(
            x: detection.boundingBox.origin.x * size.width,
            y: detection.boundingBox.origin.y * size.height,
            width: detection.boundingBox.width * size.width,
            height: detection.boundingBox.height * size.height
        )
        context.stroke(rect)
 
        let label = NSAttributedString(
            string: "\(detection.label) \(String(format: "%.0f%%", detection.confidence * 100))",
            attributes: [.foregroundColor: UIColor.red, .font: UIFont.systemFont(ofSize: 12)]
        )
        label.draw(at: CGPoint(x: rect.minX, y: rect.minY - 20))
    }
 
    let result = UIGraphicsGetImageFromCurrentImageContext()
    UIGraphicsEndImageContext()
}

Natural Language Framework: Text Classification and Sentiment Analysis

Natural Language Framework Overview

While Vision specializes in images, Natural Language optimizes text processing tasks.

Capabilities

Language Identification: Detect language across multilingual content
Tokenization & Tagging: Sentence/word segmentation and part-of-speech tagging
Named Entity Recognition: Extract persons, locations, organizations
Custom Text Classification: Sentiment, intent, topic categorization

Real-World Example: Sentiment Analysis

Train a sentiment classification model with Create ML:

training_data/
├── positive.txt
│   ├── This product is excellent
│   ├── Very satisfied with purchase
│   └── ...
└── negative.txt
    ├── Disappointed with quality
    ├── Not as expected
    └── ...

Convert to .mlmodel and integrate into Rork Max:

import NaturalLanguage
import CoreML
 
class SentimentAnalyzer {
    let model: sentiment_classifier
 
    init() throws {
        self.model = try sentiment_classifier(configuration: MLModelConfiguration())
    }
 
    func analyze(_ text: String) -> SentimentResult {
        // Preprocess text
        let tagger = NLTagger(tagSchemes: [.tokenType])
        tagger.string = text
 
        var tokens: [String] = []
        tagger.enumerateTags(in: text.startIndex..<text.endIndex, unit: .word, scheme: .tokenType) { tag, range in
            let token = String(text[range]).lowercased()
            tokens.append(token)
            return true
        }
 
        // Run prediction
        let input = sentiment_classifierInput(text: tokens.joined(separator: " "))
        guard let output = try? model.prediction(input: input) else {
            return SentimentResult(sentiment: "unknown", confidence: 0.0)
        }
 
        return SentimentResult(
            sentiment: output.classLabel,
            confidence: output.classLabelProbs[output.classLabel] ?? 0.0
        )
    }
}
 
struct SentimentResult {
    let sentiment: String // "positive" or "negative"
    let confidence: Double
}

Rork Max NLP Pipeline: Review Analysis

Automatically analyze customer reviews:

struct ReviewAnalysisView: View {
    @State private var reviewText: String = ""
    @State private var sentiment: SentimentResult?
 
    var body: some View {
        VStack {
            TextEditor(text: $reviewText)
                .border(Color.gray)
 
            Button("Analyze Sentiment") {
                analyzeSentiment()
            }
 
            if let result = sentiment {
                HStack {
                    Image(systemName: result.sentiment == "positive" ? "hand.thumbsup.fill" : "hand.thumbsdown.fill")
                        .foregroundColor(result.sentiment == "positive" ? .green : .red)
 
                    Text(result.sentiment == "positive" ? "Positive" : "Negative")
                    Text(String(format: "%.0f%%", result.confidence * 100))
                }
                .padding()
                .background(result.sentiment == "positive" ? Color.green.opacity(0.1) : Color.red.opacity(0.1))
                .cornerRadius(8)
            }
        }
    }
 
    private func analyzeSentiment() {
        let analyzer = try! SentimentAnalyzer()
        sentiment = analyzer.analyze(reviewText)
    }
}

Real-Time Inference Performance Tuning

Measuring Inference Latency

For real-time applications (AR, video processing), inference speed directly impacts user experience.

import os.log
 
class PerformanceMonitor {
    let logger = os.log.OSLog(subsystem: "com.rorklab.inference", category: "performance")
 
    func measureInference<T>(_ task: () throws -> T) rethrows -> (result: T, timeMs: Double) {
        let startTime = CACurrentMediaTime()
        let result = try task()
        let endTime = CACurrentMediaTime()
        let timeMs = (endTime - startTime) * 1000
 
        os_log(
            "Inference: %.1f ms",
            log: logger,
            type: .debug,
            timeMs
        )
 
        return (result, timeMs)
    }
}
 
// Usage
let monitor = PerformanceMonitor()
let (output, timeMs) = try monitor.measureInference {
    try model.prediction(input: input)
}
print("Completed in \(timeMs)ms")

Batch Processing for Throughput

Process multiple frames simultaneously to improve throughput:

// Single-frame: Low latency, low GPU utilization
for frame in frames {
    let output = try model.prediction(input: frameInput)
}
 
// Batch processing: Higher latency, higher throughput
let batchInput = MLMultiArray(shape: [frames.count, 224, 224, 3], dataType: .float32)
// ... populate batchInput with frames ...
let output = try model.prediction(input: batchInput)

CPU vs GPU vs Neural Engine

Core ML auto-selects, but you can tune explicitly:

let config = MLModelConfiguration()
 
// CPU only (highest accuracy, slowest)
config.computeUnits = .cpuOnly
 
// GPU + CPU (balanced)
config.computeUnits = .cpuAndGPU
 
// All (Neural Engine + GPU + CPU - fastest)
config.computeUnits = .all
 
let model = try defect_detection(configuration: config)

Typical Benchmarks (iPhone 15 Pro estimates)

CPU only: 50-100ms per inference
GPU + CPU: 20-30ms
All (Neural Engine): 5-10ms

Memory Efficiency and Threading

class InferenceQueue {
    private let queue = DispatchQueue(label: "com.rorklab.inference", qos: .userInitiated)
    private let model: defect_detection
 
    init() throws {
        self.model = try defect_detection(configuration: MLModelConfiguration())
    }
 
    func predict(
        _ input: defect_detectionInput,
        completion: @escaping (Result<defect_detectionOutput, Error>) -> Void
    ) {
        queue.async {
            do {
                let output = try self.model.prediction(input: input)
                DispatchQueue.main.async {
                    completion(.success(output))
                }
            } catch {
                DispatchQueue.main.async {
                    completion(.failure(error))
                }
            }
        }
    }
}

Model Versioning and Over-The-Air Updates

Progressive Model Updates

In production, you'll periodically release improved models. Updating all users immediately is impractical—App Store review takes days, and not all users update promptly.

Challenge: Update models without waiting for App Store approvals

One aside: if you also gate the app binary behind a minimum supported version, writing that comparison as a string check will break on releases like 1.10.0. I measured four implementations in why 1.10.0 devices get forced into an update.

Solution: Over-The-Air (OTA) Updates

Leave the app binary alone and ship the model from your own server. My first version of this didn't work, though: it handed the downloaded .mlmodel straight to MLModel(contentsOf:).

That initializer takes a compiled model. A .mlmodel isn't valid input. You run MLModel.compileModel(at:) on the device and load the .mlmodelc it returns — and that return value lives in a temporary location the system is free to delete. Apple's guide is emphatic about moving it somewhere permanent (Downloading and Compiling a Model on the User's Device).

Until I added those two lines, my app re-downloaded the model on every launch. Fittingly, what surfaced it wasn't a bandwidth bill — it was a test in airplane mode.

Compiling the model and deciding where it lives are part of downloading it, not steps that come after. Stop at "the download finished" and you get an app that works and phones home every single time.

import CoreML
 
enum ModelStoreError: Error {
    case bundledModelMissing
    case checksumMismatch
}
 
actor ModelStore {
    private let bundledName = "defect-detection"
    private var cached: MLModel?
 
    /// Always returns something. A dead server doesn't stop inference.
    func current(remote: URL, version: String, sha256: String) async -> MLModel? {
        if let cached { return cached }
 
        // 1) Already installed
        if let installed = try? load(version: version) {
            cached = installed
            return installed
        }
 
        // 2) Otherwise go fetch it
        if let fetched = try? await install(from: remote, version: version, sha256: sha256) {
            cached = fetched
            return fetched
        }
 
        // 3) Anything goes wrong, fall back to what shipped
        cached = try? bundledModel()
        return cached
    }
 
    /// Call on a memory warning
    func evict() { cached = nil }
 
    private func install(from remote: URL, version: String, sha256: String) async throws -> MLModel {
        // data(from:) pulls the whole model into memory; download(from:) doesn't
        let (downloaded, response) = try await URLSession.shared.download(from: remote)
        defer { try? FileManager.default.removeItem(at: downloaded) }
 
        guard (response as? HTTPURLResponse)?.statusCode == 200 else {
            throw URLError(.badServerResponse)
        }
        guard try digest(of: downloaded) == sha256.lowercased() else {
            throw ModelStoreError.checksumMismatch
        }
 
        // Compile on device — the result is a .mlmodelc in a temporary directory
        let compiled = try await MLModel.compileModel(at: downloaded)
 
        // Move it somewhere permanent. Skip this and it's gone by the next launch
        let fm = FileManager.default
        let destination = try modelDirectory().appendingPathComponent("\(version).mlmodelc")
        if fm.fileExists(atPath: destination.path) {
            try fm.removeItem(at: destination)
        }
        try fm.moveItem(at: compiled, to: destination)
 
        try? pruneOldVersions(keeping: version)
        return try MLModel(contentsOf: destination)
    }
 
    private func load(version: String) throws -> MLModel {
        let url = try modelDirectory().appendingPathComponent("\(version).mlmodelc")
        guard FileManager.default.fileExists(atPath: url.path) else {
            throw CocoaError(.fileNoSuchFile)
        }
        return try MLModel(contentsOf: url)
    }
 
    private func bundledModel() throws -> MLModel {
        guard let url = Bundle.main.url(forResource: bundledName, withExtension: "mlmodelc") else {
            throw ModelStoreError.bundledModelMissing
        }
        return try MLModel(contentsOf: url)
    }
 
    /// Application Support, not Caches. The system reclaims Caches,
    /// which is exactly how this broke for me offline.
    private func modelDirectory() throws -> URL {
        let fm = FileManager.default
        let base = try fm.url(for: .applicationSupportDirectory, in: .userDomainMask,
                              appropriateFor: nil, create: true)
        let dir = base.appendingPathComponent("ml-models", isDirectory: true)
        try fm.createDirectory(at: dir, withIntermediateDirectories: true)
        return dir
    }
 
    private func pruneOldVersions(keeping version: String) throws {
        let fm = FileManager.default
        let dir = try modelDirectory()
        for item in try fm.contentsOfDirectory(at: dir, includingPropertiesForKeys: nil)
        where item.lastPathComponent != "\(version).mlmodelc" {
            try? fm.removeItem(at: item)
        }
    }
}

A .mlmodelc is a directory, not a file. Copy or delete it with Data reads and writes and it will diverge on you without saying a word — use FileManager's directory operations.

Verifying What You Shipped

You're distributing code that will execute, so confirm it wasn't swapped in transit before compiling it. Hash it in a stream so a 100 MB file never lands in memory all at once.

import CryptoKit
 
extension ModelStore {
    fileprivate func digest(of url: URL) throws -> String {
        let handle = try FileHandle(forReadingFrom: url)
        defer { try? handle.close() }
 
        var hasher = SHA256()
        while let chunk = try handle.read(upToCount: 1 << 20), !chunk.isEmpty {
            hasher.update(data: chunk)
        }
        return hasher.finalize().map { String(format: "%02x", $0) }.joined()
    }
}

Version Management Best Practices

The server returns a manifest first, not the model itself.

struct ModelManifest: Codable {
    let version: String           // e.g. "2026-08-11-a"
    let url: URL
    let sha256: String
    let minimumAppVersion: String // keeps new models off old binaries
}

Four rules I settled on:

Rule	Why
Version name becomes the directory name	Overwriting in place means a half-failed install gets loaded next time
Always keep one bundled model	First launch, no signal, or a pulled release — inference still has to run
Put a minimum app version in the manifest	A model with a changed input or output shape must not reach an older build
Switch on the next launch, never mid-session	Swapping models under a running inference makes the same input give different answers

Comparing app versions as strings breaks on releases like 1.10.0, which is the article linked above. Reach for the same comparator when you read minimumAppVersion.

Privacy-First AI Design Patterns

Privacy by Default

On-device AI's greatest advantage is privacy. Implement these practices rigorously:

Essential Safeguards

Never persist input data: nothing used for inference reaches disk
Release buffers immediately: let go of them the moment you're done
Local only: nothing goes to a server without explicit consent
Transparent permissions: always ask before touching camera or microphone

Your own code sending nothing isn't the whole story — review asks what the SDKs you bundled collect. The audit path for tracing a Privacy Manifest through the SDK chain is in auditing Privacy Manifests in Rork-generated Expo apps. If on-device inference is your selling point, a mismatch here undoes it.

Secure Data Handling

final class SecureMLProcessor {
    private let model: defect_detection
 
    init(model: defect_detection) {
        self.model = model
    }
 
    func classifyWithoutPersisting(_ image: UIImage) throws -> String {
        guard let buffer = image.pixelBuffer(size: CGSize(width: 224, height: 224)) else {
            throw MLError.conversionFailed
        }
 
        // Lock and unlock as a pair. Unlocking a buffer you never locked
        // is a bug in itself, not cleanup.
        CVPixelBufferLockBaseAddress(buffer, .readOnly)
        defer { CVPixelBufferUnlockBaseAddress(buffer, .readOnly) }
 
        let output = try model.prediction(input: defect_detectionInput(imageAt: buffer))
 
        // Return the result; the input is never written anywhere
        return output.classLabel
    }
 
    /// Clears only the subdirectory this class created.
    /// Calling removeItem on Caches itself takes URLCache, image caches,
    /// and whatever else lives there down with it.
    func deleteInferenceCache() throws {
        let fm = FileManager.default
        let caches = try fm.url(for: .cachesDirectory, in: .userDomainMask,
                                appropriateFor: nil, create: false)
        let mine = caches.appendingPathComponent("inference", isDirectory: true)
 
        guard fm.fileExists(atPath: mine.path) else { return }
        for item in try fm.contentsOfDirectory(at: mine, includingPropertiesForKeys: nil) {
            try fm.removeItem(at: item)
        }
    }
}
 
enum MLError: Error {
    case conversionFailed
    case modelNotFound
    case privacyPolicyViolation
}

For a while this was a deleteAllLocalData() that removed the Caches directory outright. It did clear the inference cache. It also took the photo library's thumbnails with it, and I got to debug a blank grid I had built myself. Delete only inside what you created.

User Permission Management

import AVFoundation
 
class PermissionManager {
    static func requestCameraAccess(completion: @escaping (Bool) -> Void) {
        AVCaptureDevice.requestAccess(for: .video) { granted in
            DispatchQueue.main.async {
                completion(granted)
            }
        }
    }
 
    static func requestMicrophoneAccess(completion: @escaping (Bool) -> Void) {
        AVCaptureDevice.requestAccess(for: .audio) { granted in
            DispatchQueue.main.async {
                completion(granted)
            }
        }
    }
}
 
// Always check before access
PermissionManager.requestCameraAccess { granted in
    if granted {
        startCamera()
    } else {
        showPermissionAlert()
    }
}

Compliance Framework

struct PrivacyCompliance {
    var allowDataSharing = false
    var allowAnalytics = false
    var dataRetentionDays = 0
 
    /// assert disappears in Release builds. A check you want in production
    /// has to be a guard that throws.
    func validateBeforePrediction() throws {
        guard !allowDataSharing, !allowAnalytics, dataRetentionDays == 0 else {
            throw MLError.privacyPolicyViolation
        }
    }
}

I had this as assert for a long time. It fired in debug exactly as intended, the tests were green, and I felt covered. But assert is removed when optimizations are on, so not one of those lines ever ran in the app people installed.

A check that vanishes in a Release build isn't a check. I'd been satisfied by it stopping on my machine, while it waved everything through in the one place it mattered.

Where to Start

If I had to pick the single change that mattered most here, it's selectInferenceTarget(). Before touching model accuracy, put the routing decision in one file. Every later change gets lighter, and because the fallback leans toward 'core-ml', the failure mode where data quietly leaks out as you bolt on conditions simply doesn't happen.

Quantization and OTA can wait until that foundation is settled. Mine did.

Start by wiring up localHitRate() alone and watching the local-processing share for a week. Once there's a number, your own data tells you which requests to pull back on device.

Core ML shifts underfoot with every Apple release, and I'm still keeping up with it — I'll add new measurements here as I get them.

Thank You for Reading

Rork Lab is ad-free, supported entirely by members like you. We publish practical guides daily with implementation code, benchmarks, and production-ready patterns. If you've found it useful, we'd love to have you on board.

✦Copy-paste ready implementation code
✦New advanced guides published daily
✦$5/mo or $10 for lifetime access

View Membership →