AI Briefing#benchmark
The latest AI and developer news about #benchmark, with the original source and a short summary.
The latest AI and developer news about #benchmark, with the original source and a short summary.
A unified leaderboard for evaluating Arabic LLM performance has launched, along with updates to the AraGen and Arabic IFEval benchmarks.
DeepSeek has released DeepSeek-V3-0324, a model with improved performance and licensing.
OpenAI has released **SWE-Lancer**, a software engineering benchmark based on real Upwork freelance jobs.
Hugging Face resolved math evaluation errors on the Open LLM Leaderboard by introducing Math-Verify and recalculated the rankings.
Adyen and Hugging Face have released the DABstep benchmark, which evaluates the capabilities of data analysis agents.
A new dataset called Big Bench Audio, designed to evaluate the reasoning capabilities of audio language models, has been released.
HuggingFace has released AraGen, an Arabic LLM benchmark based on the 3C3H metric that evaluates factuality and helpfulness simultaneously.
OpenAI has open-sourced SimpleQA, a new benchmark for measuring the factuality of language models.
Confirmed that increased cache miss rates due to higher VM counts are the primary cause of CPU performance degradation.
BenCzechMark, a benchmark that comprehensively evaluates LLMs' Czech language understanding and reasoning abilities, has been released.
xAI has released Grok-2 and Grok-2 mini in beta, with significantly improved performance.
BigCodeBench, a new LLM code generation benchmark for evaluating practical programming ability, has been released.
Qwen has released Qwen2, its next-generation model with significantly enhanced performance and multilingual capabilities.
The Open Medical-LLM Leaderboard has been released to evaluate the performance of medical-domain LLMs in a standardized way.
The LiveCodeBench leaderboard has launched, addressing data contamination issues and evaluating code generation ability from multiple angles.
A new dataset and leaderboard called 'ConTextual' has been released to evaluate multimodal models' ability to jointly reason over text and images.
Haize Labs has released a red teaming resistance benchmark that measures LLM vulnerability to sophisticated human attacks.
Upstage has launched the Open Ko-LLM Leaderboard to fairly evaluate the performance of Korean LLMs.
The NPHardEval leaderboard, which dynamically evaluates LLMs' logical reasoning ability using complexity classes, has been released.
An open-source leaderboard for evaluating factuality and faithfulness hallucinations in LLMs has been released.
Ktor offers faster boot times, lower resource usage, and superior Coroutine-based performance compared to Spring.
Hugging Face has released benchmark results for over 60 configurations to optimally deploy Llama 2 on Amazon SageMaker.
This analyzes the technical causes behind the differences in MMLU benchmark scores between the Open LLM Leaderboard and model papers.
Habana Gaudi2 recorded roughly 2x faster performance than Nvidia A100 80GB in BERT training and Stable Diffusion inference, among others.