Overcoming NumPy Limitations: 25% Speedup for 200-Million-Element Array Operations with Cython Multithreading

To address Python's performance bottlenecks, we compared the performance of standard deviation calculations using NumPy, C++ Extension, Cython, and pybind11. While NumPy is optimized for large arrays, it has single-core limitations, necessitating C++-based implementations to leverage multi-core capabilities.

Performance Comparison and Bottleneck Analysis

For small arrays of 10,000 elements or fewer, NumPy can be slower than pure Python, whereas direct C++ implementations or Cython demonstrate better performance. pybind11 is convenient to use but exhibits noticeable performance degradation due to automatic conversion. For large arrays (50,000 elements or more), NumPy offers the best performance, but this is under conditions that exclude conversion overhead.

Multithreading Optimization Results

By combining Cython with std::thread to apply multithreading, we achieved a 25% performance improvement (reducing execution time from 2.49 seconds to 1.84 seconds) for 200-million-element array operations compared to single-threaded execution (Cython w/ class). Although 8 threads were used, vector loading alone took 1.6 seconds; despite reducing the computation itself to 76ms, the overall improvement was limited.

Conclusion and Implications

When processing large data, type conversion between Python and C++ types and memory copying are the primary bottlenecks. Overhead must be minimized by appropriately using pointers and references. Considering the balance between development convenience and performance, Cython is evaluated as a more favorable choice than pybind11.