CAJAL-4B released for distributed systems

CAJAL-4B-P2PCLAW is a 4.21B parameter Qwen3.5-4B-based model created by the P2PCLAW research team in Zurich, aiming to assist research in distributed systems and cryptographic protocols. It features a 262,144 token context and support for English and Spanish, and while it is distributed under the MIT license, it also separately notes that the base model's original license should be checked.

• Architecture uses a hybrid structure mixing linear attention and full self-attention, with full attention inserted every 4 blocks, as explained.
• Training data/setup consists of 10,000 scientific examples curated by P2PCLAW, using the LoRA/QLoRA method with rank 16, alpha 32, 4-bit NF4, learning rate 2e-4, batch size 8, and a maximum sequence length of 2048.
• Results show training took about 13 hours on a single RTX 3090, with a final loss of 0.03192 and training accuracy of 98.95%, as reported.
• Inference performance is presented as about 25 tok/s with about 6.5GB usage on Ollama GPU, 20 tok/s with about 7.8GB usage on Transformers GPU, and 5 tok/s with about 4.2GB usage on CPU.
• Use cases proposed as primary applications include P2P network architecture, crypto-legal frameworks, game-theoretic consensus, ZKP/cryptography, distributed systems analysis, paper writing, and smart contract review.

Integration with Ollama, Transformers, VS Code, Cursor, Zed, Open WebUI, LM Studio, Jan, OpenAI-compatible REST API, and cajal-cli is also provided.