Open-Source ML Toolkit for Speech and Structured Decisions on Apple Devices

Language models, while incredibly versatile for tasks like writing, summarizing, and translating, represent only one facet of the vast capabilities within the field of machine learning. Real-world enterprise applications often require a sequence of specialized ML steps, such as first understanding spoken language and then making a structured decision based on that understanding. While it might seem intuitive to pass both tasks to a single large language model (LLM), the developers of AnySpeechRecognizer and AnyDecisionModel argue that dedicated, specialized models are often superior—offering better performance, faster processing speeds, and lower operational costs. Building upon their previous work with AnyLanguageModel and swift-huggingface, these new packages extend the principle of providing familiar Swift APIs while offering developers a crucial choice of underlying machine learning engines for both speech processing and structured decision-making.

AnySpeechRecognizer is designed to interface with Apple's modern speech-to-text capabilities, specifically mirroring the functionality introduced in macOS 26 and iOS 26 via the new SpeechAnalyzer API. This package maintains the familiar structure of Apple's framework, allowing developers to integrate it seamlessly into existing applications. However, its key innovation is decoupling the core API from the recognition engine itself. Instead of being locked into Apple's proprietary engine, developers can choose from several powerful, open-source alternatives. These alternatives include NVIDIA's Parakeet and Alibaba's Qwen3-ASR, both of which are designed to run locally on the device using Core ML, ensuring privacy and low latency.

When examining the available engines, NVIDIA's Parakeet offers a highly capable, polyglot solution, meaning it can recognize speech in 25 different European languages without the developer needing to specify the locale beforehand. Parakeet is available in several versions—including the highly accurate Parakeet Ultra and the general Parakeet Unified—all running on a custom Core ML runtime. This runtime is robust, supporting both file-based transcription and live, real-time input. Furthermore, the package enhances the live input experience by including an 8-second encoder, which significantly reduces the analysis time for live streams, although this comes with a minor trade-off in word error rate. Another critical feature added by AnySpeechRecognizer is speaker diarization. Since Apple's native framework lacks a dedicated module for this, AnySpeechRecognizer integrates NVIDIA's Streaming Sortformer. This module is vital for conversational analysis, as it can accurately track and attribute speech to up to four distinct speakers, even in complex scenarios where multiple people are speaking simultaneously or overlapping. The results are collected in a SpeakerTimeline, allowing the application to present a highly detailed transcript organized by speaker turn.

In contrast to the speech recognition component, AnyDecisionModel addresses the need for structured, programmatic decision-making. Instead of relying on the generative nature of LLMs to produce free-form text, which can be unpredictable, AnyDecisionModel focuses on deriving precise, quantifiable answers. It is designed to answer specific types of questions—such as yes/no questions, multiple-choice selections, or rating-scale assessments—and outputs these results as typed data accompanied by associated probabilities. This probability-based approach provides developers with a measure of confidence for each answer, which is crucial for mission-critical applications. The model can be powered by various backends, including MLX, Core AI, or llama.cpp for on-device execution, or it can integrate with external decision models provided by platforms like Jev or Ollama. This focus on structured, probabilistic output, rather than pure text generation, represents a significant shift toward making AI more reliable and predictable for enterprise use cases.

Technically, both packages are designed for maximum flexibility and accessibility. They are capable of downloading and utilizing open-source models directly from the Hugging Face Hub, enabling developers to run state-of-the-art models entirely on the user's device. The implementation details are highly sophisticated. For instance, the conversion of these powerful models into the Core ML format, necessary for seamless integration into Apple applications, is managed meticulously. Each repository tracks its source checkpoint and checksum, and the package pins these assets to fixed revisions, ensuring reproducibility. The resource requirements are substantial; for example, the Qwen3-ASR model, which supports a vast array of languages and 22 Chinese dialects (including Mandarin, Cantonese, Japanese, and Korean) that Parakeet does not cover, requires a large download size (0.9 to 2.4 GB). Conversely, the Parakeet models are smaller, around 600 MB each, but the overall package structure ensures that downloads only occur when the application explicitly requests them, optimizing the user experience and bandwidth usage. The ability to run complex, multi-lingual, and highly specialized models like Qwen3-ASR or Parakeet entirely on-device, without continuous cloud connectivity, is a major architectural breakthrough for privacy-sensitive and offline-first applications.

Furthermore, the architecture is designed to be modular and extensible. For developers wishing to integrate a third speech engine, the package provides clear interfaces: implementing the SpeechRecognitionModel for configuration and the SpeechRecognitionExecutor for session management. This modularity ensures that the system remains adaptable as the ML landscape evolves. The entire system is built around mirroring the established, robust APIs of the Apple ecosystem, minimizing the learning curve for developers while maximizing their choice of underlying, cutting-edge open-source technology. In summary, AnySpeechRecognizer and AnyDecisionModel provide a comprehensive, open-source toolkit that empowers developers to build sophisticated, reliable, and privacy-preserving AI applications for both speech understanding and structured decision-making across Apple platforms.