Spider Middleware

The spider middleware is a framework of hooks into Scrapy’s spider processing mechanism where you can plug custom functionality to process the responses that are sent to Spiders for processing and to process the requests and items that are generated from spiders.

激活spider中间件

要启用spider中间件,您可以将其加入到 SPIDER_MIDDLEWARES 设置中,该设置是一个字典,键为中间件的路径,值为中间件的顺序(order)。

下面是一个示例︰

SPIDER_MIDDLEWARES = {
    'myproject.middlewares.CustomSpiderMiddleware': 543,
}

The SPIDER_MIDDLEWARES setting is merged with the SPIDER_MIDDLEWARES_BASE setting defined in Scrapy (and not meant to be overridden) and then sorted by order to get the final sorted list of enabled middlewares: the first middleware is the one closer to the engine and the last is the one closer to the spider.

To decide which order to assign to your middleware see the SPIDER_MIDDLEWARES_BASE setting and pick a value according to where you want to insert the middleware. The order does matter because each middleware performs a different action and your middleware could depend on some previous (or subsequent) middleware being applied.

If you want to disable a builtin middleware (the ones defined in SPIDER_MIDDLEWARES_BASE, and enabled by default) you must define it in your project SPIDER_MIDDLEWARES setting and assign None as its value. For example, if you want to disable the off-site middleware:

SPIDER_MIDDLEWARES = {
    'myproject.middlewares.CustomSpiderMiddleware': 543,
    'scrapy.spidermiddlewares.offsite.OffsiteMiddleware': None,
}

Finally, keep in mind that some middlewares may need to be enabled through a particular setting. See each middleware documentation for more info.

编写您自己的spider中间件

每个中间件组件是一个定义了以下一个或多个方法的Python类:

class scrapy.spidermiddlewares.SpiderMiddleware
process_spider_input(response, spider)

This method is called for each response that goes through the spider middleware and into the spider, for processing.

process_spider_input()应该返回None或抛出一个异常。

If it returns None, Scrapy will continue processing this response, executing all other middlewares until, finally, the response is handed to the spider for processing.

如果其跑出一个异常(exception),Scrapy将不会调用任何其他中间件的process_spider_input()方法,并调用request的errback。errback的输出将会以另一个方向被重新输入到中间件链中,使用process_spider_output()方法来处理,当其抛出异常时则带调用process_spider_exception()

Parameters:
  • response (Response object) – the response being processed
  • spider (Spider object) – the spider for which this response is intended
process_spider_output(response, result, spider)

This method is called with the results returned from the Spider, after it has processed the response.

process_spider_output()必须返回Request、字典或Item对象的一个可迭代对象。

Parameters:
  • response (Response object) – the response which generated this output from the spider
  • result (an iterable of Request, dict or Item objects) – the result returned by the spider
  • spider (Spider object) – the spider whose result is being processed
process_spider_exception(response, exception, spider)

当spider或(其他spider中间件的)process_spider_input()跑出异常时, 该方法被调用。

process_spider_exception()应该返回None或者Response、字典或Item对象的一个可迭代对象。

如果其返回None,Scrapy将继续处理该异常,调用中间件链中的其他中间件的process_spider_exception()方法,直到所有中间件都被调用,该异常到达引擎(异常将被记录并被忽略)。

如果其返回一个可迭代对象,则中间件链的process_spider_output()方法被调用, 其他的process_spider_exception()将不会被调用。

Parameters:
  • response (Response object) – the response being processed when the exception was raised
  • exceptionException对象)—— 引发的异常
  • spiderSpider对象)—— 引发异常的Spider
process_start_requests(start_requests, spider)

New in version 0.15.

该方法以spider 启动的request为参数被调用,执行的过程类似于process_spider_output() ,只不过其没有相关联的response并且必须返回request(不是item)。

It receives an iterable (in the start_requests parameter) and must return another iterable of Request objects.

Note

当在您的spider中间件实现该方法时, 您必须返回一个可迭代对象(类似于参数start_requests)且不要遍历所有的 start_requests,因为该迭代器会很大(甚至是无限),进而导致内存溢出。The Scrapy engine is designed to pull start requests while it has capacity to process them, so the start requests iterator can be effectively endless where there is some other condition for stopping the spider (like a time limit or item/page count).

Parameters:
  • start_requests (an iterable of Request) – the start requests
  • spider (Spider object) – the spider to whom the start requests belong

内置Spider中间件参考

This page describes all spider middleware components that come with Scrapy. 关于如何使用及编写您自己的中间件,请参考Spider中间件使用指南

For a list of the components enabled by default (and their orders) see the SPIDER_MIDDLEWARES_BASE setting.

DepthMiddleware

class scrapy.spidermiddlewares.depth.DepthMiddleware

DepthMiddleware is a scrape middleware used for tracking the depth of each Request inside the site being scraped. It can be used to limit the maximum depth to scrape or things like that.

DepthMiddleware可以通过下列设置进行配置(更多内容请参考设置文档):

  • DEPTH_LIMIT - 爬取所允许的最大深度。如果为0,则没有限制。
  • DEPTH_STATS - Whether to collect depth stats.
  • DEPTH_PRIORITY - Whether to prioritize the requests based on their depth.

HttpErrorMiddleware

class scrapy.spidermiddlewares.httperror.HttpErrorMiddleware

过滤出所有失败(错误)的HTTP response,因此spider不需要处理这些request。 处理这些request意味着消耗更多资源,并且使得spider逻辑更为复杂。

根据HTTP标准,返回值为200-300之间的值为成功的Response。

If you still want to process response codes outside that range, you can specify which response codes the spider is able to handle using the handle_httpstatus_list spider attribute or HTTPERROR_ALLOWED_CODES setting.

For example, if you want your spider to handle 404 responses you can do this:

class MySpider(CrawlSpider):
    handle_httpstatus_list = [404]

The handle_httpstatus_list key of Request.meta can also be used to specify which response codes to allow on a per-request basis. 如果你想允许请求的任何响应代码,你也可以设置元键handle_httpstatus_allTrue

Keep in mind, however, that it’s usually a bad idea to handle non-200 responses, unless you really know what you’re doing.

更多详细信息请参阅︰HTTP 状态代码定义

HttpErrorMiddleware settings

HTTPERROR_ALLOWED_CODES

Default: []

Pass all responses with non-200 status codes contained in this list.

HTTPERROR_ALLOW_ALL

Default: False

Pass all responses, regardless of its status code.

OffsiteMiddleware

class scrapy.spidermiddlewares.offsite.OffsiteMiddleware

Filters out Requests for URLs outside the domains covered by the spider.

该中间件过滤出所有主机名不在spider属性allowed_domains的request。列表中任何域的所有子域也都会允许。如,规 www.example.org还将允许bob.www.example.org,但不允许www2.example.comexample.com

When your spider returns a request for a domain not belonging to those covered by the spider, this middleware will log a debug message similar to this one:

DEBUG: Filtered offsite request to 'www.othersite.com': <GET http://www.othersite.com/some/page.html>

To avoid filling the log with too much noise, it will only print one of these messages for each new domain filtered. So, for example, if another request for www.othersite.com is filtered, no log message will be printed. But if a request for someothersite.com is filtered, a message will be printed (but only for the first request filtered).

如果spider没有定义allowed_domains属性,或该属性为空, 则offsite 中间件将会允许所有request。

If the request has the dont_filter attribute set, the offsite middleware will allow the request even if its domain is not listed in allowed domains.

RefererMiddleware

class scrapy.spidermiddlewares.referer.RefererMiddleware

Populates Request Referer header, based on the URL of the Response which generated it.

RefererMiddleware settings

REFERER_ENABLED

New in version 0.15.

Default: True

Whether to enable referer middleware.

UrlLengthMiddleware

class scrapy.spidermiddlewares.urllength.UrlLengthMiddleware

Filters out requests with URLs longer than URLLENGTH_LIMIT

UrlLengthMiddleware可以通过下列设置进行配置(更多内容请参考设置文档):