Profiling and optimization

The performance fundamentals lesson's rule — measure, don't guess — has concrete tools in Python. Before you rewrite anything, find out where the time actually goes.

Time small snippets with timeit

For comparing two ways of doing one thing, timeit runs it many times and reports a reliable average, avoiding hand-rolled timing mistakes:

import timeit
timeit.timeit('"-".join(str(n) for n in range(100))', number=10000)

Profile whole programs with cProfile

To find where a program spends its time, run it under cProfile:

python -m cProfile -s cumtime my_script.py

It reports each function's call count and time, sorted by cumulative time. The function at the top is your hotspot — and it's usually not where you'd have guessed. Optimise that; ignore the rest.

Python-specific speedups

Once you've found the hotspot, the biggest wins are usually:

• Better algorithms and data structures first — a set membership test instead of scanning a list, the right structure for the operation (the fundamentals lessons). This dwarfs micro-optimisation. Switching from an O(n²) approach to an O(n) one (see Big-O) is always bigger than any constant-factor tweak.
• Built-ins and the standard library — sum(), sorted(), collections, itertools are implemented in C and far faster than equivalent Python loops.
• Generators to avoid building large intermediate lists (the generators lesson) — saves memory and often time.
• Push hot loops into libraries — for heavy numeric work, vectorised libraries do in C what a Python loop does slowly.

Where to go next

That completes Concurrency & Performance. The next module covers the tooling that makes Python projects reproducible and shippable: Tooling & Packaging.