Scrape Amazon product listings utilising Scrapy & residential proxies
To get started with Scrapy, you will first need to install it using the methods provided in its documentation. Check here for more information
Once you have an active subscription, you can find your credentials & proxy addresses in Dashboard > Residential > Proxy Setup
Navigate to settings.py in /amazon/amazon/ folder and modify the following lines to authenticate.
DECODO_USER = 'username' ## Decodo Username (Sub-user)
DECODO_PASSWORD = 'password' ## Password for your user
DECODO_ENDPOINT = 'gate.decodo.com' ## Endpoint you'd like to use
DECODO_PORT = '7000' ## Port of the endpoint you are using.
Navigate to the project folder and run the following command:
scrapy crawl amazon_search
Amazon search results will be saved in /amazon/data folder in a .csv format