Because Rork Max can generate native Swift apps, on-device Core ML inference — long out of easy reach in React Native — is now within reach even for indie development. But put it on a real device and there are stumbling points before you ever get to accuracy. The first inference is oddly slow. After a while the device warms up and everything feels sluggish. Both are design problems about when and how much you run inference, not about whether the model is good.
As an indie developer at Dolice, when I built on-device inference into an app I run, I struggled with a roughly one-second freeze on the very first call. The cause was not model accuracy; it was running the first load and inference on the main thread right at launch.
This article lays out how to design Core ML around two constraints — cold start and a thermal budget — with Swift code.
Why the first inference is slow
A Core ML model runs two heavy operations the first time you use it. One is loading and compiling the model (optimized for the device's Neural Engine); the other is the first inference, which allocates internal buffers. It is normal for the first call to be an order of magnitude slower than later ones.
| Stage | What mainly happens | Felt impact |
|---|---|---|
| First load | Model compile and placement | Hundreds of ms to 1 s |
| First inference | Buffer allocation, warmup | Tens to hundreds of ms |
| Subsequent | Run on the allocated path | Often around 10-30 ms |
The problem is throwing that heavy first call at the moment the user is waiting for a result. The design goal is simple: move the heavy first call earlier, to a time when the user is not waiting.
Pull model load and warmup out of the launch flow
First, do not synchronously load the model at app launch. Defer with lazy, and do the load and warmup (one inference on dummy input) on a background queue.
import CoreML
actor InferenceEngine {
private var model: MyModel?
// warm up in the background; call after the first screen appears, not at launch
func warmUp() async {
guard model == nil else { return }
let config = MLModelConfiguration()
config.computeUnits = .all // let it use the Neural Engine too
do {
let loaded = try MyModel(configuration: config)
// run the first inference on dummy input to allocate buffers
_ = try? loaded.prediction(input: .dummy)
model = loaded
} catch {
model = nil // do not block launch even on failure
}
}
func predict(_ input: MyModelInput) async throws -> MyModelOutput {
if model == nil { await warmUp() }
guard let model else { throw InferenceError.unavailable }
return try model.prediction(input: input)
}
}The caller invokes warmUp() in the brief gap after the first screen renders and before the user starts interacting.
.task {
// warm up during idle time after the screen shows
await engine.warmUp()
}The actor is what pays off here. Even if predict is called from several places at once, the language guarantees the load does not run twice. Early on I forgot this protection and multi-loaded the model on every screen transition, needlessly bloating memory. Concurrent access is a quiet pitfall in on-device inference.