Pagination and rate limiting

A public API rarely returns all of its data in one response. Ask for a hundred thousand records and the server will send you the first hundred and a way to ask for the next hundred. Similarly, if you fire requests as fast as Python can loop, the server will eventually tell you to slow down. Both limits are by design — they protect the API from being overwhelmed.

Offset-based pagination

The simpler model. The server accepts page and per_page (or limit and offset) as query parameters. You increment the page number on each request until you get back fewer items than you asked for — that signals the last page:

page = 1
per_page = 100
all_items = []

while True:
    response = requests.get(
        "https://api.example.com/items",
        params={"page": page, "per_page": per_page},
    )
    response.raise_for_status()
    items = response.json()

    all_items.extend(items)

    if len(items) < per_page:
        break   # last page — got fewer items than the page size

    page += 1

The termination condition — receiving fewer items than requested — is the idiomatic signal. Some APIs include a total count in the response body; you can also use that to calculate when you are done.

Cursor-based pagination

More common in modern APIs. Instead of a page number, the server returns an opaque cursor (sometimes called next_cursor, after, or embedded in a next URL) that encodes exactly where to start the next page. You pass it back on the next request:

cursor = None
all_items = []

while True:
    params = {"per_page": 100}
    if cursor:
        params["after"] = cursor

    response = requests.get("https://api.example.com/items", params=params)
    response.raise_for_status()
    body = response.json()

    all_items.extend(body["items"])

    cursor = body.get("next_cursor")
    if not cursor:
        break   # no next cursor means no more pages

Cursors are more robust than offset pagination for large, fast-changing datasets because they point at a position in the data rather than a page number that can shift as items are added or deleted.

Rate limiting

APIs limit how many requests you can make in a time window — typically expressed as requests per minute or per hour. When you exceed the limit, the server responds with 429 Too Many Requests.

Two common response headers tell you what to do next:

• X-RateLimit-Remaining — how many requests are left in the current window. Check this on every response; if it reaches zero, wait before the next request.
• Retry-After — the number of seconds to wait before retrying. Sent with 429 responses. The most direct signal: the server tells you exactly how long to wait.

A polite request loop

Combining pagination with rate limit awareness:

import time

def fetch_all(base_url, params=None):
    params = params or {}
    params["per_page"] = 100
    all_items = []
    page = 1

    while True:
        params["page"] = page
        response = requests.get(base_url, params=params)

        if response.status_code == 429:
            wait = int(response.headers.get("Retry-After", 5))
            print(f"Rate limited — waiting {wait}s")
            time.sleep(wait)
            continue   # retry the same page

        response.raise_for_status()
        items = response.json()
        all_items.extend(items)

        remaining = response.headers.get("X-RateLimit-Remaining")
        if remaining and int(remaining) < 5:
            time.sleep(1)   # slow down proactively

        if len(items) < 100:
            break
        page += 1

    return all_items

The continue on a 429 retries the same page rather than advancing — important detail. Without it, a rate-limited page would be silently skipped.

Where to go next

Next: lab — API report — combine everything from this module into a script that fetches all pages of a public API, aggregates the data, and writes a summary report.