Python爬虫怎样实现分布式爬取_Python爬虫分布式架构设计与实现方法

import redis
import requests
from urllib.parse import urljoin
<h1>连接Redis</h1><p>r = redis.StrictRedis(host='master_ip', port=6379, db=0)</p><p><span>立即学习</span>“<a href="https://pan.quark.cn/s/00968c3c2c15" style="text-decoration: underline !important; color: blue; font-weight: bolder;" rel="nofollow" target="_blank">Python免费学习笔记（深入）</a>”；</p><p>def fetch_url():</p><h1>从待爬队列中弹出一个URL</h1><pre class='brush:python;toolbar:false;'>task = r.lpop('pending_urls')
if not task:
    return
url = task.decode('utf-8')

try:
    response = requests.get(url, timeout=5)
    # 解析页面，提取新链接
    new_urls = extract_links(response.text, base=url)
    for link in new_urls:
        # 使用集合去重后加入队列
        if r.sadd('seen_urls', link):  # 布隆过滤器更优
            r.rpush('pending_urls', link)

    # 存储抓取内容
    r.rpush('crawled_data', response.text[:500])
except Exception as e:
    print(f"Error fetching {url}: {e}")

# 启用Redis调度
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
SCHEDULER_PERSIST = True  # 持久化任务队列
<h1>Redis地址</h1><p>REDIS_URL = 'redis://master_ip:6379/0'</p>

from scrapy_redis.spiders import RedisSpider
<p>class MySpider(RedisSpider):
name = 'myspider'</p><h1>不设置start_urls，由Redis中的键触发</h1><pre class='brush:python;toolbar:false;'>redis_key = 'myspider:start_urls'

def parse(self, response):
    # 正常解析逻辑
    for href in response.css('a::attr(href)').getall():
        yield {'url': response.urljoin(href)}

rpush myspider:start_urls "https://example.com"