Adding Read-Aloud to a Rork Max App: AVSpeechSynthesizer Voice Selection and Live Word Highlighting

Highlight the word being spoken — reflect the delegate's range in UI

What actually decided satisfaction in a reading app was less the voice itself and more the visualization of "where are we reading right now." AVSpeechSynthesizerDelegate's willSpeakRangeOfSpeechString hands you the character range about to be spoken as an NSRange. Reflect it into an AttributedString's background color and the words flow like karaoke.

final class Reader: NSObject, ObservableObject, AVSpeechSynthesizerDelegate {
    @Published var highlightedRange: NSRange? = nil
    @Published var fullText: String = ""
    private let synth = AVSpeechSynthesizer()
 
    override init() {
        super.init()
        synth.delegate = self
    }
 
    func speak(_ text: String) {
        fullText = text
        let u = AVSpeechUtterance(string: text)
        u.voice = AVSpeechSynthesisVoice.bestVoice(for: "ja")
        synth.speak(u)
    }
 
    // The range about to be spoken (called on the main thread)
    func speechSynthesizer(_ s: AVSpeechSynthesizer,
                           willSpeakRangeOfSpeechString characterRange: NSRange,
                           utterance: AVSpeechUtterance) {
        highlightedRange = characterRange
    }
 
    func speechSynthesizer(_ s: AVSpeechSynthesizer,
                           didFinish utterance: AVSpeechUtterance) {
        highlightedRange = nil
    }
}

On the SwiftUI side, convert the NSRange into AttributedString indices to highlight. This is where Japanese trips you up most, so run the conversion through the canonical NSRange → Range<String.Index> path rather than naive Int addition.

struct ReadingView: View {
    @ObservedObject var reader: Reader
 
    var attributed: AttributedString {
        var s = AttributedString(reader.fullText)
        guard let r = reader.highlightedRange,
              let swiftRange = Range(r, in: reader.fullText),
              let lower = AttributedString.Index(swiftRange.lowerBound, within: s),
              let upper = AttributedString.Index(swiftRange.upperBound, within: s)
        else { return s }
        s[lower..<upper].backgroundColor = .yellow.opacity(0.5)
        return s
    }
 
    var body: some View {
        ScrollView { Text(attributed).font(.title3).padding() }
    }
}

Why the highlight drifts in Japanese — the UTF-16 and emoji pitfall

This is where I burned the most time. The NSRange returned by willSpeakRangeOfSpeechString is an offset in UTF-16 code units. Swift's String is measured in grapheme clusters, so passing the raw location into String.index(offsetBy:) drifts on any sentence with emoji or combining characters. That's exactly why the code above uses Range(nsRange, in: string) — that initializer interprets UTF-16 offsets correctly. Write it with your own integer math instead, and on a review sentence with emoji the highlight lands a few characters late — the classic "the spot it reads and the spot it lights don't match" bug.

There's a second point: because word boundaries in Japanese aren't as clean as in English, the ranges willSpeakRange returns vary between a phrase and one-to-a-few characters. I settled on "light up only the range that arrives, and clear it on the next one." Trying to keep prior ranges lit and accumulate them made the display messy at the wobbly boundaries. Reflecting only the incoming range each time ended up looking more natural.

Don't stop the music, speak on the lock screen — the audio session

Left at defaults, starting read-aloud can silence other apps' music or go mute with the ringer switch. Plenty of readers keep ambient sound playing, so I chose "keep other audio going, just duck it slightly while speaking."

import AVFoundation
 
func configureAudioSession() {
    let session = AVAudioSession.sharedInstance()
    do {
        // .playback = plays even with the ringer switch off;
        // .duckOthers = temporarily lowers other apps' volume
        try session.setCategory(.playback,
                                mode: .spokenAudio,
                                options: [.duckOthers])
        try session.setActive(true)
    } catch {
        // Read-aloud still works even if this fails; just log it
        print("audio session error:", error)
    }
}

The .spokenAudio mode is optimized for speech content, and .duckOthers lowers other apps' volume only while speaking and restores it afterward. To fully stop the music, drop .duckOthers; to simply mix with ambient sound, switch to .mixWithOthers. To keep reading on the lock screen or over other apps, you need the Audio Background Mode enabled in addition to the .playback category. If you go as far as building lock-screen controls (play/pause), the same MPNowPlayingInfoCenter approach from a separate article on background audio and lock-screen controls carries over directly.

The prompt I hand to Rork Max

When generating from scratch with Rork Max, spelling the feature out in detail before handing it over gave me stable results. Specifying the boundaries cuts down rework compared to a vague "add read-aloud."

Add read-aloud to a SwiftUI reading screen. Requirements:
- Hold AVSpeechSynthesizer as a property on an ObservableObject (not a local var)
- For Japanese, pick the best voice by quality: premium > enhanced > default
- Use willSpeakRangeOfSpeechString to @Publish the range being read, and
  highlight it via the background color of a Text's AttributedString
  (convert NSRange -> Range correctly)
- Add play / pause (pauseSpeaking(at: .word)) / stop buttons
- Audio session: .playback / .spokenAudio / .duckOthers
After generating, verify the highlight doesn't drift on sentences with emoji.

The output usually leaves one of two things: "works but the synthesizer is a local variable," or "converts NSRange with Int addition." So I check exactly those two spots every time. Rather than trusting generated code wholesale, re-reading only the pitfall areas listed in this article keeps the cleanup short.

What I learned after shipping it

Read-aloud isn't a feature everyone uses just because it's there. In my reading prototypes, about one in ten people used it even once. But that tenth had a clearly higher week-two retention — there really is a segment that "reads with their ears" during a commute or between chores. Small in the numbers, yet a path that lets people keep touching content without stopping their hands seems to help them stick.

If I tackle the next piece, auto-scroll tied to the word highlight (keeping the line being read centered) has the most room to improve the experience. For accessibility work overall, thinking about it alongside VoiceOver and Dynamic Type widens the reach beyond read-aloud alone. For healing apps that combine it with ambient sound, the ambient audio loop design notes should help too.

Start by holding the synthesizer as a property and speaking one sentence with the default voice. Clear that step and you can add highlighting and voice quality one at a time.