Music Generation on Apple Silicon

This is a hands-on note about running ACE-Step 1.5 with MLX on Apple Silicon.

The model has a 2-stage structure. First, a language model plans the song's tempo, key, structure, arrangement, and lyrics, and then a diffusion transformer generates the actual audio.

• Memory: Active usage during inference stayed under 4GB. It goes higher when loading the model, but 16GB gives you headroom and 8GB is possible though tight.
• Speed: On an M2 Air, generating a 60-90 second track took about 2-4 minutes. The M3 Pro is roughly 2x faster. The M1 also works, but feels slow for longer outputs.
• Precision: The default is fp16, and bf16 also works, with reportedly little perceptible difference in audio quality.
• Quality: Compared to the PyTorch/CUDA-based implementation, it was hard to tell the difference in a blind test, and even results that seemed slightly worse looked like the level of seed variance.

The areas where it performs well are relatively clear.

• Instrumental genres like lo-fi, cinematic, hip-hop, trap, ambient, and electronic
• English/Spanish vocals, with Japanese/Hindi also usable
• Timbre descriptors actually show up in the results: expressions like warm analog tape saturation, dry stadium kick, and vinyl crackle influence the sound
• BPM and key instructions are followed fairly accurately

There are also limitations.

• Complex prompts may require 5-10 retries
• Fast vocals have weaker pronunciation clarity than cloud commercial models
• There's no direct control feature for specific instruments or arrangement placement
• Transitions within a song can sound somewhat abrupt

Additionally, the repo also supports LoRA adapters, noting that this enables a workflow of learning a style from a small number of references. The author also bundled this into a Mac app called LoopMaker, but recommends first trying it directly with MLX and Python.