修正部分错误，并增加专用蜘蛛与中间件。

Author: DotNetAge

.gitignore

@@ -176,5 +176,5 @@ pyvenv.cfg
 pip-selfcheck.json
 
 build
-.vscode
-.idea
+.vscode/
+.idea/

README.md

@@ -1,175 +1,30 @@
 # Scrapy+
 
-Scrapy扩展工具包。具体使用方法与配置方法可以参考《虫术——Python绝技》一书。
-
-## 过滤器
-
-### Redis 去重过滤器 `scrapy_plus.dupefilters.RedisDupeFilter`
-
-基于Redis使用`Set`存储曾访问过的URL。
-
-**使用方法**
-
-在`settings`文件内引入以下的内容:
-
-```py
-# 覆盖原有的去重过滤器
-DUPEFILTER_CLASS = 'scrapy_plus.dupefilters.RedisDupeFilter' 
-REDIS_PORT = 6379                       # REDIS服务器端口
-REDIS_HOST = '127.0.0.1'                # REDIS服务器地址
-REDIS_DB = 0                            # 数据库名
-```
-
-**默认配置**
-
-```py
-REDIS_PORT = 6379                       # REDIS服务器端口
-REDIS_HOST = '127.0.0.1'                # REDIS服务器地址
-REDIS_DB = 0                            # 数据库名
-```
-
-### Redis 布隆去重过滤器 `scrapy_plus.dupefilters.RedisBloomDupeFilter`
-
-基于Redis采用布隆算法对URL进行去重处理
-
-**使用方法**
-
-在`settings`文件内引入以下的内容:
-
-```py
-# 覆盖原有的去重过滤器
-DUPEFILTER_CLASS = 'scrapy_plus.dupefilters.RedisBloomDupeFilter' 
-REDIS_PORT = 6379                       # REDIS服务器端口
-REDIS_HOST = '127.0.0.1'                # REDIS服务器地址
-REDIS_DB = 0                            # 数据库名
-```
-
-**默认配置**
-
-```
-REDIS_PORT = 6379                       # REDIS服务器端口
-REDIS_HOST = '127.0.0.1'                # REDIS服务器地址
-REDIS_DB = 0                            # 数据库名
-BLOOMFILTER_REDIS_KEY = 'bloomfilter'   # 去重键名
-BLOOMFILTER_BLOCK_NUMBER = 1            # 块大小
-```
-
-## 中间件
-
-### 自登录中间件 `scrapy_plus.middlewares.LoginMiddleWare`
-
-```py
-LOGIN_URL = '网站登录地址'
-LOGIN_USR = '用户名'
-LOGIN_PWD = '密码'
-LOGIN_USR_ELE = '用户名input元素名称(name)'
-LOGIN_PWD_ELE = '密码input元素名称(name)'
-DOWNLOADER_MIDDLEWARES = {
-    'scrapyplus.middlewares.LoginMiddleWare': 330
-}
-```
-
-### Chrome 浏览器仿真中间件 `scrapy_plus.middlewares.ChromeMiddleware`
-
-Chrome 无头浏览器仿真中间件。让爬虫用Chrome来访问目标URL，完美解决富JS页面的问题。
-
-```py
-SELENIUM_TIMEOUT = 30 # 设置页面打开的超时秒数
-CHROMEDRIVER = "/path/to/chrome" # Chrome浏览器驱动地址
-DOWNLOADER_MIDDLEWARES = {
-    'scrapyplus.middlewares.ChromeMiddleware': 800
-}
-
-```
-
-
-### Splash `scrapy_plus.middlewares.SplashSpiderMiddleware`
-
-Splash 中间件，可将请求转发至指定的Splash服务，使蜘蛛具有浏览器仿真功能。
-
-```py
-WAIT_FOR_ELEMENT = "选择器" # 等待该元素被加载成功才认为页面加载完成
-DOWNLOADER_MIDDLEWARES = {
-    'scrapyplus.middlewares.SplashSpiderMiddleware': 800
-}
-```
-
-### 随机UA `scrapyplus.middlewares.RandomUserAgentMiddleware`
-
-随机模拟User Agent
-
-```python
-DOWNLOADER_MIDDLEWARES = {
-    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
-    'scrapyplus.middlewares.RandomUserAgentMiddleware': 500
-}
-## 可随机增加更多的UA，中间件会进行自动随机选择
-USER_AGENTS = [
-    'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:51.0) Gecko/20100101 Firefox/51.0',
-    'Mozilla/5.0 (Linux; U; Android 2.2) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1',
-    'Mozilla/5.0 (Windows NT 5.1; rv:7.0.1) Gecko/20100101 Firefox/7.0.1',
-    'Mozilla/5.0 (Linux; Android 6.0.1; SM-G532G Build/MMB29T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.83 Mobile Safari/537.36',
-    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/604.5.6 (KHTML, like Gecko) Version/11.0.3 Safari/604.5.6'
-]
-```
-
-### Tor 中间件 `scrapyplus.middlewares.TorProxyMiddleware`
-
-洋葱头代理中间件,让你的蜘蛛不停地更换IP地址,化身万千。
-
-需要先安装 tor 与 privoxy 具体配置方法请参考《虫术——python绝技》
-
-```py
-# Tor代理
-TOR_PROXY = 'http://127.0.0.1:8118'  # 8118是Privoxy的默认代理端口
-TOR_CTRL_PORT = 9051
-TOR_PASSWORD = 'mypassword'
-TOR_CHANGE_AFTER_TIMES = 50 # 在发出多少次请求之后更换IP地址。
-```
-
-## 管道
-
-### MongoDB数据存储管道 `scrapy_plus.piplines.MongoDBPipeline`
-
-可以将Item直接写入MongoDB数据库中。
-
-**默认配置**
-
-```py
-ITEM_PIPELINES = {'scrapy_plus.pipelines.MongoDBPipeline':2}
-
-MONGODB_SERVER = "localhost"    # mongodb服务器地址
-MONGODB_PORT = 27017            # mongodb服务端口
-MONGODB_DB = "数据库名"          # 数据库名
-MONGODB_COLLECTION = "表名"     # 表名
-```
-
-## 存储后端
-
-### SQL数据库存储后端 `scrapy_plus.extensions.SQLFeedStorage`
-
-```py
-# 数据存储
-ORM_MODULE = 'movies.entities'
-ORM_METABASE = 'Base'
-ORM_ENTITY = 'Movie'
-
-FEED_FORMAT = 'entity' # 
-FEED_EXPORTERS = {
-    'entity': 'scrapyplus.extensions.SQLItemExporter'
-}
-
-FEED_URI = 'dialect+driver://username:password@host:port/database'  # 默认后端存储文件的名称
-FEED_STORAGES = {
-    'sqlite': 'scrapyplus.extensions.SQLFeedStorage',
-    'postgresql': 'scrapyplus.extensions.SQLFeedStorage',
-    'mysql': 'scrapyplus.extensions.SQLFeedStorage'
-}
-```
-
-### 阿里云OSS存储后端 `scrapy_plus.extensions.OSSFeedStorage`
-
-```py
-OSS_ACCESS_KEY_ID = ''
-OSS_SECRET_ACCESS_KEY = ''
-```
+Scrapy扩展工具包。为[《从0学爬虫专栏》](https://www.imooc.com/read/34) 提供，详细的使用方法请到专栏内参考。
+
+```
+$ pip install scrapy_plus
+```
+
+Scrapy+提供以下的内容
+
+- 过滤器
+  - Redis 去重过滤器
+  - Redis 布隆去重过滤器
+- 中间件
+  - 自登录中间件
+  - 花瓣网专用中间件
+  - Chrome通用中间件
+  - Splash渲染中间件
+  - Tor中间件
+  - 随机UA中间件
+  - 随机代理中间件
+- 管道
+  - MongoDB数据存储管道
+  - 可支持阿里云的OSS图片管道
+- SQL存储端
+- 输入/输出处理器
+- 蜘蛛
+  - `BookSpider`
+  - `NeteaseSpider`
+  - `TaobaoSpider`

requirements.txt

@@ -44,7 +44,7 @@ python-dateutil==2.8.0
 pytz==2018.9
 qt5reactor==0.5
 queuelib==1.5.0
-redis==3.2.0
+redis==3.2.1
 regex==2019.2.21
 requests==2.21.0
 Scrapy==1.6.0

scrapy_plus/__init__.py

@@ -1,3 +1,4 @@
 from .redis import RedisDupeFilter
-#from .bloom import FileBloomDupeFilter
 from .redisbloom import RedisBloomDupeFilter
+
+__all__ = ["RedisBloomDupeFilter", "RedisDupeFilter"]

scrapy_plus/dupefilters/__init__.py

@@ -1,37 +1,36 @@
 # -*- coding: utf-8 -*-
 import logging
-from scrapy.utils.request import request_fingerprint
-from redis import StrictRedis
+from redis import Redis
 from scrapy.dupefilters import BaseDupeFilter
 
-BLOOMFILTER_HASH_NUMBER = 6
-BLOOMFILTER_BIT = 30
 
 
 class RedisDupeFilter(BaseDupeFilter):
     """
-    Redis去重过滤器
+    Redis 去重过滤器
     """
+    def __init__(self, host='localhost', port=6379, db=0):
+        self.redis = Redis(host=host, port=port, db=db)
+        self.logger = logging.getLogger(__name__)
 
     @classmethod
     def from_settings(cls, settings):
-        return cls(host=settings.get('REDIS_HOST'),
-                   port=settings.getint('REDIS_PORT'),
-                   db=settings.get('REDIS_DB'))
-
-    def __init__(self, host, port, db):
-        self.redis = StrictRedis(host=host, port=port, db=db)
-        self.logger = logging.getLogger(__name__)
+        host = settings.get('REDIS_HOST', 'localhost')
+        redis_port = settings.getint('REDIS_PORT')
+        redis_db = settings.get('REDIS_DUP_DB')
+        return cls(host, redis_port, redis_db)
 
     def request_seen(self, request):
-        fp = request_fingerprint(request)
-        key = 'UriFingerprints'
-        if self.redis.sismember(key, fp) is None:
+        fp = request.url
+        key = 'UrlFingerprints'
+        if not self.redis.sismember(key, fp):
             self.redis.sadd(key, fp)
             return False
         return True
 
     def log(self, request, spider):
-        msg = ("已过滤的重复请求:%(request)s")
+        msg = ("已过滤的重复请求: %(request)s")
         self.logger.debug(msg, {'request': request}, extra={'spider': spider})
         spider.crawler.stats.inc_value('dupefilter/filtered', spider=spider)
+
+

scrapy_plus/dupefilters/redis.py

@@ -1,10 +1,13 @@
 # -*- coding: utf-8 -*-
 import logging
 from scrapy.utils.request import request_fingerprint
-from redis import StrictRedis
+from redis import Redis
 from hashlib import md5
 from scrapy.dupefilters import BaseDupeFilter
 
+BLOOMFILTER_HASH_NUMBER = 6
+BLOOMFILTER_BIT = 30
+
 
 class SimpleHash(object):
     def __init__(self, cap, seed):
@@ -21,14 +24,7 @@ def hash(self, value):
 class RedisBloomDupeFilter(BaseDupeFilter):
 
     def __init__(self, host='localhost', port=6379, db=0, blockNum=1, key='bloomfilter'):
-        """
-        :param host: the host of Redis
-        :param port: the port of Redis
-        :param db: witch db in Redis
-        :param blockNum: one blockNum for about 90,000,000; if you have more strings for filtering, increase it.
-        :param key: the key's name in Redis
-        """
-        self.redis = StrictRedis(host=host, port=port, db=db)
+        self.redis = Redis(host=host, port=port, db=db)
 
         self.bit_size = 1 << 31  # Redis的String类型最大容量为512M，现使用256M
         self.seeds = [5, 7, 11, 13, 31, 37, 61]
@@ -44,7 +40,7 @@ def __init__(self, host='localhost', port=6379, db=0, blockNum=1, key='bloomfilt
     def from_settings(cls, settings):
         _port = settings.getint('REDIS_PORT', 6379)
         _host = settings.get('REDIS_HOST', '127.0.0.1')
-        _db = settings.get('REDIS_DB', 0)
+        _db = settings.get('REDIS_DUP_DB', 0)
         key = settings.get('BLOOMFILTER_REDIS_KEY', 'bloomfilter')
         block_number = settings.getint(
             'BLOOMFILTER_BLOCK_NUMBER', 1)
@@ -85,4 +81,4 @@ def log(self, request, spider):
         msg = ("已过滤的重复请求: %(request)s")
         self.logger.debug(msg, {'request': request}, extra={'spider': spider})
         spider.crawler.stats.inc_value(
-            'redisbloomfilter/filtered', spider=spider)
+            'redisbloomfilter/filtered', spider=spider)