Skip to content
BSWEN
Blog

Why Does Python Dominate Machine Learning Despite Being Slow?

Cowrie
Cowrie
Dev @ Bswen

The Paradox

When I first started learning machine learning, I was confused. Everyone told me Python is slow. “Interpreted language,” they said. “Dynamic typing adds overhead,” they said. “10-100x slower than C,” they said.

But then I looked at the ML landscape: PyTorch, TensorFlow, JAX, scikit-learn, NumPy - all Python-first. GPT, LLaMA, Stable Diffusion - all ship Python SDKs first. The most compute-intensive field in software is dominated by the “slowest” popular language.

I had to understand why.

The Misconception

I thought Python was doing the heavy lifting. I was wrong.

The key insight I found on Reddit captures it perfectly: “The language itself doesn’t need to be fast when you’re just orchestrating C/CUDA underneath.”

Let me show you what’s actually happening.

The Architecture

Python in ML uses a three-tier architecture:

┌─────────────────────────────────────────────────────┐
│              Python Layer (You Write Here)          │
│   High-level API, easy syntax, rapid prototyping   │
└────────────────────────┬────────────────────────────┘
                         │ Function calls
                         ▼
┌─────────────────────────────────────────────────────┐
│              C++ Backend (Compiled)                 │
│   ATen (PyTorch), XLA (TensorFlow), optimized loops │
└────────────────────────┬────────────────────────────┘
                         │ CUDA kernels
                         ▼
┌─────────────────────────────────────────────────────┐
│              GPU (NVIDIA/AMD)                       │
│   Massive parallel matrix operations               │
└─────────────────────────────────────────────────────┘

When you write c = np.dot(a, b) in Python, Python spends microseconds validating inputs and making the function call. The actual computation runs in optimized C loops for milliseconds. Python’s overhead is often less than 0.1% of total execution time.

NumPy: The Foundation

I wanted to see this in action. Let me compare what I write versus what runs:

numpy_example.py
import numpy as np


# What I write in Python
a = np.random.randn(10000, 10000)  # Allocates C array
b = np.random.randn(10000, 10000)  # Allocates C array


# This looks "slow" - matrix multiply in Python?
c = np.dot(a, b)  # Actually runs in optimized C

What happens under the hood:

Python: "I need to multiply these matrices"
   │
   ▼
NumPy C: Validates shapes, allocates result buffer
   │
   ▼
BLAS: Calls optimized SGEMM/DGEMM routines
   │
   ▼
Result: Returned to Python with minimal overhead


Timing breakdown (typical):
- Python overhead:    ~50 microseconds
- C computation:      ~500 milliseconds
- Python is ~0.01% of total time

NumPy arrays are thin wrappers around contiguous C memory blocks. The actual data never “lives” in Python’s slow object system.

PyTorch: From Python to CUDA

PyTorch takes this further. I write Python, GPUs run CUDA:

pytorch_gpu.py
import torch


# Python creates tensor metadata, GPU holds the data
x = torch.randn(1000, 1000, device='cuda')


# Python call, CUDA execution
y = torch.matmul(x, x)

The execution flow:

Step 1: Python validates tensor shapes
        Time: ~1 microsecond


Step 2: PyTorch C++ backend (ATen) dispatches operation
        Time: ~10 microseconds


Step 3: CUDA kernel runs matrix multiplication on GPU
        Time: ~100+ microseconds to milliseconds


Step 4: Result metadata returned to Python
        Time: ~1 microsecond


Total Python time: ~12 microseconds
Total GPU time: ~100,000+ microseconds (100ms)
Python overhead: < 0.01%

The actual CUDA kernel running on the GPU looks nothing like Python:

cuda_kernel_pseudocode.txt
__global__ void matrixMultiply(float* A, float* B, float* C, int N) {
    int row = blockIdx.y * blockDim.y + threadIdx.y;
    int col = blockIdx.x * blockDim.x + threadIdx.x;
    float sum = 0.0f;
    for (int k = 0; k < N; k++) {
        sum += A[row * N + k] * B[k * N + col];
    }
    C[row * N + col] = sum;
}

I never write this CUDA code. PyTorch generates it from my Python. This is the power of the glue language approach.

The Network Effect

I initially thought Python’s dominance was about technical merit. But reading discussions, I found a more powerful force: ecosystem self-reinforcement.

Python easy to learn
       │
       ▼
More data scientists use Python
       │
       ▼
More ML libraries built for Python
       │
       ▼
Better tooling and documentation
       │
       ▼
More companies adopt Python ML
       │
       ▼
New models ship Python SDKs first
       │
       ▼
Even more users pick Python
       │
       └──────► Cycle repeats

A Reddit comment captured this well: “Python’s not going anywhere because the ML ecosystem picked it and that’s self-reinforcing. Every new model release ships with a Python SDK first.”

Common Misconceptions I Had

Misconception 1: “Python is too slow for production ML”

I thought rewriting in C++ or Rust would speed things up. Then I looked at actual profiling data:

Model inference: 100ms total
├── Python orchestration: 0.1ms (0.1%)
├── Data transfer: 0.5ms (0.5%)
└── GPU computation: 99.4ms (99.4%)


Rewriting Python in Rust: 0.05ms orchestration
Total gain: 0.05ms (0.05% improvement)

The math doesn’t work. Rewriting the 0.1% that’s slow to gain 0.05% is not worth the development complexity.

Misconception 2: “Python will be replaced by Julia/Mojo/Rust”

I looked at this too. But the competition isn’t Python vs. other languages - it’s PyTorch vs. TensorFlow vs. JAX.

Current landscape:
- PyTorch (Python frontend, C++/CUDA backend)
- TensorFlow (Python frontend, XLA compiler backend)
- JAX (Python frontend, XLA backend)


The frontend language is already decided: Python.
The competition is backend compilers and frameworks.

Like SQL in databases, Python has become the standard query language for ML. SQL hasn’t been replaced in 50 years because the interface isn’t the bottleneck.

Misconception 3: “I need to worry about Python performance for ML”

I spent time optimizing Python loops before I understood the architecture. Now I know:

What matters for ML performance:
- Choosing the right framework (PyTorch/TF/JAX)
- Using batched operations
- GPU selection and memory management
- Model architecture choices


What doesn't matter:
- Python loop optimization
- Using faster Python variants
- Rewriting in compiled languages

When Python Speed Actually Matters

To be fair, there are cases where Python’s speed matters:

Python is slow for:
- Data preprocessing (use polars, not pandas)
- Custom algorithms not in numpy/scipy
- String manipulation at scale
- Control flow with many iterations


But for ML:
- Matrix operations: C/CUDA (fast)
- Neural network layers: C++ (fast)
- GPU kernels: CUDA (fast)
- You're just the conductor, not the orchestra

Summary

In this post, I explored why Python dominates machine learning despite being slow. The key insight is that Python isn’t doing the heavy lifting - it’s a glue language that orchestrates fast C/CUDA backends.

The three-tier architecture makes this work:

  1. Python layer: Easy to learn, rapid prototyping
  2. C++ backend: Optimized CPU operations
  3. CUDA kernels: GPU-accelerated computation

For developers starting ML: don’t worry about Python’s speed. Focus on understanding frameworks and ML concepts. The performance is already handled.

For the future: Python won’t be replaced by a faster language. It will be replaced when ML no longer needs a programming interface - or when code itself becomes obsolete.

Final Words + More Resources

My intention with this article was to help others share my knowledge and experience. If you want to contact me, you can contact by email: Email me

Here are also the most important links from this article along with some further resources that will help you in this scope:

Oh, and if you found these resources useful, don’t forget to support me by starring the repo on GitHub!

Comments

Tags: