Apple Core AI Framework

Apple has introduced the Core AI framework, which lets you run, optimize, and deploy AI models within apps using the CPU, GPU, and Neural Engine. Beyond a simple execution API, this framework provides an integrated toolchain that converts PyTorch models into the .aimodel format and supports compression and debugging as well.

Key features and workflow:

• PyTorch integration: Using torch.export and TorchConverter, PyTorch models can be converted into a form optimized for Apple silicon.
• Performance optimization: To prevent inference slowdowns in Transformer models, it provides State and Cache management features that reuse Key/Value from previous steps, maintaining consistent inference speed.
• Model deployment and specialization: When a model is loaded, it goes through a Specialization process tailored to the device architecture, and coreai-build allows precompilation during development to reduce user wait times.
• Quantization and debugging: It supports various forms of Quantization including INT4, INT8, FP4, and FP8, and the Core AI Debugger allows comparison of intermediate tensor values between the converted model and the original PyTorch model to trace errors.

This framework is supported across all of Apple's major OSes, including iOS, iPadOS, and macOS, and is optimized for implementing on-device AI that keeps user data from leaving the device.