自动限速扩展¶
This is an extension for automatically throttling crawling speed based on load of both the Scrapy server and the website you are crawling.
设计目标¶
- be nicer to sites instead of using default download delay of zero
- automatically adjust scrapy to the optimum crawling speed, so the user doesn’t have to tune the download delays to find the optimum one. The user only needs to specify the maximum concurrent requests it allows, and the extension does the rest.
How it works¶
AutoThrottle extension adjusts download delays dynamically to make spider send AUTOTHROTTLE_TARGET_CONCURRENCY concurrent requests on average to each remote website.
It uses download latency to compute the delays.The main idea is the following: if a server needs latency seconds to respond, a client should send a request each latency/N seconds to have N requests processed in parallel.
Instead of adjusting the delays one can just set a small fixed download delay and impose hard limits on concurrency using CONCURRENT_REQUESTS_PER_DOMAIN or CONCURRENT_REQUESTS_PER_IP options. It will provide a similar effect, but there are some important differences:
- because the download delay is small there will be occasional bursts of requests;
- often non-200 (error) responses can be returned faster than regular responses, so with a small download delay and a hard concurrency limit crawler will be sending requests to server faster when server starts to return errors. But this is an opposite of what crawler should do - in case of errors it makes more sense to slow down: these errors may be caused by the high request rate.
AutoThrottle doesn’t have these issues.
限速算法¶
限速算法基于以下规则调整下载延迟：
- spiders always start with a download delay of
AUTOTHROTTLE_START_DELAY; - when a response is received, the target download delay is calculated as
latency / Nwherelatencyis a latency of the response, andNisAUTOTHROTTLE_TARGET_CONCURRENCY. - download delay for next requests is set to the average of previous download delay and the target download delay;
- latencies of non-200 responses are not allowed to decrease the delay;
- download delay can’t become less than
DOWNLOAD_DELAYor greater thanAUTOTHROTTLE_MAX_DELAY
Note
The AutoThrottle extension honours the standard Scrapy settings for concurrency and delay. This means that it will respect CONCURRENT_REQUESTS_PER_DOMAIN and CONCURRENT_REQUESTS_PER_IP options and never set a download delay lower than DOWNLOAD_DELAY.
In Scrapy, the download latency is measured as the time elapsed between establishing the TCP connection and receiving the HTTP headers.
注意，由于Scrapy可能在忙着处理spider的回调函数或者无法下载，因此在合作的多任务环境下准确测量这些延迟是十分苦难的。不过，这些延迟仍然是对Scrapy(甚至是服务器)繁忙程度的合理测量，而这扩展就是以此为前提进行编写的。
设置¶
The settings used to control the AutoThrottle extension are:
AUTOTHROTTLE_ENABLEDAUTOTHROTTLE_START_DELAYAUTOTHROTTLE_MAX_DELAYAUTOTHROTTLE_DEBUGCONCURRENT_REQUESTS_PER_DOMAINCONCURRENT_REQUESTS_PER_IPDOWNLOAD_DELAY
For more information see How it works.
AUTOTHROTTLE_MAX_DELAY¶
Default: 60.0
The maximum download delay (in seconds) to be set in case of high latencies.
AUTOTHROTTLE_TARGET_CONCURRENCY¶
New in version 1.1.
默认值：1.0
Average number of requests Scrapy should be sending in parallel to remote websites.
By default, AutoThrottle adjusts the delay to send a single concurrent request to each of the remote websites. Set this option to a higher value (e.g. 2.0) to increase the throughput and the load on remote servers. A lower AUTOTHROTTLE_TARGET_CONCURRENCY value (e.g. 0.5) makes the crawler more conservative and polite.
Note that CONCURRENT_REQUESTS_PER_DOMAIN and CONCURRENT_REQUESTS_PER_IP options are still respected when AutoThrottle extension is enabled. This means that if AUTOTHROTTLE_TARGET_CONCURRENCY is set to a value higher than CONCURRENT_REQUESTS_PER_DOMAIN or CONCURRENT_REQUESTS_PER_IP, the crawler won’t reach this number of concurrent requests.
At every given time point Scrapy can be sending more or less concurrent requests than AUTOTHROTTLE_TARGET_CONCURRENCY; it is a suggested value the crawler tries to approach, not a hard limit.