Python Async: Where It Finally Earned Its Complexity for Me

I spent most of my Python career avoiding async. The complexity rarely paid for itself. This year a specific workload changed my mind. Not async everywhere. Async in the right place.

The Workload That Demanded It

A content pipeline that fetches from 40 external sources, runs transformations, and publishes. Synchronous version: 25 minutes per run. Async version: 4 minutes per run. The speedup paid for the complexity ten times over.

The key: the workload was I/O bound and embarrassingly parallel. Forty independent HTTP fetches is exactly what async is for.

What I Did Not Do

Make my whole codebase async. Turn my FastAPI routes async by default. Use async for CPU-bound work. Use async for sequential work. Each of these is a common anti-pattern that adds complexity without benefit.

The Pattern That Worked

Keep the outer code sync. Wrap the one I/O-bound section in an async function. Call it with asyncio.run. The async is a surgical tool, not a lifestyle.

import asyncio
import httpx
 
async def fetch_all(urls):
    async with httpx.AsyncClient() as client:
        tasks = [client.get(u) for u in urls]
        return await asyncio.gather(*tasks)
 
def pipeline():
    urls = load_sources()
    responses = asyncio.run(fetch_all(urls))
    transform_and_publish(responses)

The Gotchas That Cost Me Hours

asyncio.gather raises on the first exception and cancels the rest. I want return_exceptions=True 90 percent of the time.

Mixing sync and async HTTP clients in the same pipeline creates weird deadlocks. Pick one per pipeline.

Not every library has an async version. When you need to call a sync library from async code, use asyncio.to_thread. Do not pretend it is async when it is not.

When To Reach For Async

• I/O bound work with clear parallelism potential
• Latency-sensitive endpoints handling many concurrent clients
• Fan-out fan-in patterns over many external services

When To Avoid It

• Scripts that run once
• Sequential logic with no parallelism
• CPU-bound work (use processes)
• Beginner codebases where the team cannot yet debug it

Reading

Real Python async guide is still the best tutorial. Finish it before reaching for the feature.