Signals¶

Scrapy uses signals extensively to notify when certain events occur. You can catch some of those signals in your Scrapy project (using an extension, for example) to perform additional tasks or extend Scrapy to add functionality not provided out of the box.

Even though signals provide several arguments, the handlers that catch them don’t need to accept all of them - the signal dispatching mechanism will only deliver the arguments that the handler receives.

You can connect to signals (or send your own) through the Signals API.

Here is a simple example showing how you can catch signals and perform some action:

from scrapy import signals
from scrapy import Spider


class DmozSpider(Spider):
    name = "dmoz"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/",
    ]


    @classmethod
    def from_crawler(cls, crawler, *args, **kwargs):
        spider = super(DmozSpider, cls).from_crawler(crawler, *args, **kwargs)
        crawler.signals.connect(spider.spider_closed, signal=signals.spider_closed)
        return spider


    def spider_closed(self, spider):
        spider.logger.info('Spider closed: %s', spider.name)


    def parse(self, response):
        pass

Deferred signal handlers¶

有些信号支持从处理器返回Twisted deferreds，参考下边的内置信号参考来了解哪些支持。

Built-in signals reference¶

Here’s the list of Scrapy built-in signals and their meaning.

engine_started¶

scrapy.signals.engine_started()¶

Sent when the Scrapy engine has started crawling.

This signal supports returning deferreds from their handlers.

Note

该信号可能会在信号spider_opened之后被发送，取决于spider的启动方式。所以不要依赖该信号会比spider-opened更早被发送。

engine_stopped¶

scrapy.signals.engine_stopped()¶

Sent when the Scrapy engine is stopped (for example, when a crawling process has finished).

This signal supports returning deferreds from their handlers.

item_scraped¶

scrapy.signals.item_scraped(item, response, spider)¶

Sent when an item has been scraped, after it has passed all the Item Pipeline stages (without being dropped).

This signal supports returning deferreds from their handlers.

Parameters:	item (dict or `Item` object) – the item scraped spider (`Spider` object) – the spider which scraped the item response (`Response` object) – the response from where the item was scraped

item_dropped¶

scrapy.signals.item_dropped(item, response, exception, spider)¶

Sent after an item has been dropped from the Item Pipeline when some stage raised a DropItem exception.

This signal supports returning deferreds from their handlers.

Parameters:	item (dict or `Item` object) – the item dropped from the Item Pipeline spider (`Spider` object) – the spider which scraped the item response (`Response` object) – the response from where the item was dropped exception (`DropItem` exception) – the exception (which must be a `DropItem` subclass) which caused the item to be dropped

spider_closed¶

scrapy.signals.spider_closed(spider, reason)¶

Sent after a spider has been closed. This can be used to release per-spider resources reserved on spider_opened.

This signal supports returning deferreds from their handlers.

Parameters:

spider (Spider object) – the spider which has been closed
reason (str) – a string which describes the reason why the spider was closed. If it was closed because the spider has completed scraping, the reason is 'finished'. Otherwise, if the spider was manually closed by calling the close_spider engine method, then the reason is the one passed in the reason argument of that method (which defaults to 'cancelled'). If the engine was shutdown (for example, by hitting Ctrl-C to stop it) the reason will be 'shutdown'.

spider_opened¶

scrapy.signals.spider_opened(spider)¶

Sent after a spider has been opened for crawling. This is typically used to reserve per-spider resources, but can be used for any task that needs to be performed when a spider is opened.

This signal supports returning deferreds from their handlers.

Parameters:	spider (`Spider` object) – the spider which has been opened

spider_idle¶

scrapy.signals.spider_idle(spider)¶

当spider进入空闲(idle)状态时该信号被发送，空闲意味着:

requests waiting to be downloaded
requests被调度
items正在item pipeline中被处理

If the idle state persists after all handlers of this signal have finished, the engine starts closing the spider. After the spider has finished closing, the spider_closed signal is sent.

You can, for example, schedule some requests in your spider_idle handler to prevent the spider from being closed.

该信号不支持从它们的处理器返回deferreds。

Parameters:	spider (`Spider` object) – the spider which has gone idle

spider_error¶

scrapy.signals.spider_error(failure, response, spider)¶

当spider的回调函数产生错误时，该信号被发送(例如，抛出异常)。

This signal does not support returning deferreds from their handlers.

Parameters:	failure (Failure object) – the exception raised as a Twisted Failure object response (`Response` object) – the response being processed when the exception was raised spider (`Spider` object) – the spider which raised the exception

request_scheduled¶

scrapy.signals.request_scheduled(request, spider)¶

Sent when the engine schedules a Request, to be downloaded later.

该信号不支持返回deferreds。

Parameters:	request (`Request` object) – the request that reached the scheduler spider (`Spider` object) – the spider that yielded the request

request_dropped¶

scrapy.signals.request_dropped(request, spider)¶

Sent when a Request, scheduled by the engine to be downloaded later, is rejected by the scheduler.

The signal does not support returning deferreds from their handlers.

Parameters:	request (`Request` object) – the request that reached the scheduler spider (`Spider` object) – the spider that yielded the request

response_received¶

scrapy.signals.response_received(response, request, spider)¶

Sent when the engine receives a new Response from the downloader.

该信号不支持返回deferreds。

Parameters:	response (`Response` object) – the response received request (`Request` object) – the request that generated the response spider (`Spider` object) – the spider for which the response is intended

response_downloaded¶

scrapy.signals.response_downloaded(response, request, spider)¶

Sent by the downloader right after a HTTPResponse is downloaded.

该信号不支持返回deferreds。

Parameters:	response (`Response` object) – the response downloaded request (`Request` object) – the request that generated the response spider (`Spider` object) – the spider for which the response is intended