HoRain云--Python 爬虫进阶：Playwright + 异步 + 反爬策略实战

1. 为什么需要 Playwright传统 requests 无法执行 JavaScript。Playwright 支持 Chromium、Firefox、WebKit能模拟真实浏览器。安装bash复制下载pip install playwright playwright install2. 基本用法python复制下载from playwright.sync_api import sync_playwright with sync_playwright() as p: browser p.chromium.launch(headlessTrue) page browser.new_page() page.goto(https://example.com) print(page.title()) browser.close()3. 异步 Playwrightpython复制下载import asyncio from playwright.async_api import async_playwright async def main(): async with async_playwright() as p: browser await p.chromium.launch(headlessTrue) page await browser.new_page() await page.goto(https://example.com) title await page.title() print(title) await browser.close() asyncio.run(main())4. 等待元素与提取数据python复制下载await page.wait_for_selector(.item) items await page.query_selector_all(.item) for item in items: text await item.inner_text() print(text)5. 处理登录与 Cookiepython复制下载await page.fill(#username, user) await page.fill(#password, pass) await page.click(#login) await page.wait_for_url(**/dashboard) cookies await page.context.cookies()6. 反爬策略设置 User-Agentpython复制下载await browser.new_page(user_agentMozilla/5.0 ...)使用代理python复制下载browser await p.chromium.launch(proxy{server: http://127.0.0.1:7890})随机延迟python复制下载await asyncio.sleep(random.uniform(1, 3))模拟人类操作随机滚动、鼠标移动。7. 验证码处理简单验证码可使用 OCR如ddddocrpython复制下载import ddddocr ocr ddddocr.DdddOcr() code ocr.classification(image_bytes)复杂验证码建议使用打码平台或人工处理。8. 并发爬虫架构使用 asyncio Playwrightpython复制下载async def crawl(url, browser, sem): async with sem: page await browser.new_page() try: await page.goto(url, timeout30000) return await page.title() finally: await page.close() async def main(urls): async with async_playwright() as p: browser await p.chromium.launch() sem asyncio.Semaphore(5) tasks [crawl(url, browser, sem) for url in urls] results await asyncio.gather(*tasks) await browser.close() return results9. 常见坑无头模式被检测可尝试headlessFalse或 stealth 插件。页面未加载完就提取需使用wait_for_selector。代理不稳定导致超时。频繁请求触发 IP 封禁。10. 总结Playwright 是动态网页爬虫的利器。结合异步、代理和限流可以构建稳定高效的数据采集系统。务必遵守目标网站 robots.txt 和法律法规。