HoRain云--Python asyncio 异步编程实战：从协程到高并发爬虫

1. 同步 vs 异步同步爬虫按顺序请求耗时累加。异步爬虫在等待网络响应时切换任务大幅提升吞吐量。2. 协程基础python复制下载import asyncio async def hello(): print(Hello) await asyncio.sleep(1) print(World) asyncio.run(hello())async def定义协程await挂起当前协程让出事件循环。3. 事件循环事件循环是 asyncio 的核心负责调度协程、处理 I/O 和回调。asyncio.run()会创建并关闭事件循环。4. Task 与并发python复制下载async def fetch(i): await asyncio.sleep(1) return i async def main(): tasks [asyncio.create_task(fetch(i)) for i in range(5)] results await asyncio.gather(*tasks) print(results) asyncio.run(main())gather并发执行多个任务并收集结果。5. FutureFuture 是低层对象表示未来结果。通常使用 Task 即可不需要手动创建 Future。6. 超时与取消python复制下载try: await asyncio.wait_for(fetch(1), timeout0.5) except asyncio.TimeoutError: print(超时)任务取消python复制下载task asyncio.create_task(fetch(1)) task.cancel()7. 信号量控制并发python复制下载sem asyncio.Semaphore(10) async def limited_fetch(url): async with sem: return await fetch(url)避免一次性发起过多请求。8. 异步 HTTP 请求使用aiohttp或httpxpython复制下载import aiohttp async def fetch_url(session, url): async with session.get(url) as resp: return await resp.text() async def main(): async with aiohttp.ClientSession() as session: tasks [fetch_url(session, fhttps://example.com/{i}) for i in range(100)] results await asyncio.gather(*tasks)9. 异步爬虫完整示例python复制下载import asyncio import aiohttp async def crawl(url, session, sem): async with sem: try: async with session.get(url, timeout10) as resp: return url, resp.status, len(await resp.text()) except Exception as e: return url, error, str(e) async def main(urls): sem asyncio.Semaphore(20) async with aiohttp.ClientSession() as session: tasks [crawl(url, session, sem) for url in urls] for result in asyncio.as_completed(tasks): print(await result) urls [fhttps://httpbin.org/delay/1?i{i} for i in range(50)] asyncio.run(main(urls))10. 常见坑在协程中调用requests会阻塞事件循环。忘记await导致协程未执行。异常未捕获会导致任务静默失败。大量任务需要限流否则可能压垮目标站点。11. 总结asyncio 适合 I/O 密集型场景。掌握协程、Task、Semaphore 和 aiohttp 后可以写出高并发、低资源占用的 Python 程序。