Running Mistral 7B with Core ML

Using the latest features of Apple Intelligence and Core ML unveiled at WWDC 24, the 7B-parameter Mistral 7B model can be run on-device on Mac.

The key technical features are as follows:

• Introduction of MLTensor: A new Swift type introduced to improve upon the complex manipulation methods of the existing MLMultiArray. It provides a high-level interface similar to Python's numpy or torch, simplifying tensor data manipulation.
• Memory Efficiency: Through optimized model conversion, the 7B model can be run using under 4GB of memory.

Developers can perform inference directly in a Swift environment using the converted Core ML model provided on Hugging Face and the preview branch of swift-transformers.