Utilyze - A Tool That Measures How Efficiently a GPU Actually Performs Useful Work

nvidia-smi or nvtop only show whether a kernel is running on the GPU, so it can appear as 100% utilization even when the actual hardware isn't being fully used.

Utilyze reads GPU performance counters directly, displaying actual resource usage live rather than mere occupancy. The overhead is presented as negligible.

The core feature is calculating the Attainable SOL upper bound. It estimates the realistically achievable maximum utilization for a given workload, model, and hardware combination, letting you determine how much further the current setup can be pushed.

It also automatically detects running inference servers and identifies which model is loaded on each GPU. Currently, only vLLM is supported, with SGLang and others planned for future addition.

Supported hardware is NVIDIA Ampere and above. Examples mentioned include A100, H100, H200, B200, RTX 3000+, and it currently supports some models on H100-80G and A100-80G with configurations of up to 8 GPUs per node.

In terms of operation, it runs a profiling server on Linux, while macOS/Windows connect via a WebSocket remote client. A single device ID can only monitor one instance, and using it without sudo requires setting NVreg_RestrictProfilingToAdminUsers=0 followed by a reboot.