scrapy-selenium-cdp

Scrapy downloader middleware that uses SeleniumBase's pure CDP mode to make requests, allowing to bypass most anti-bot protections (e.g. CloudFlare).

Using Selenium's pure CDP mode also makes the middle more platform independent as no WebDriver is required.

🚧 Work in progress, see working example here. 🚧

Installation

pip install scrapy-seleniumbase-cdp

Configuration

1. Add the SeleniumBaseAsyncCDPMiddleware to the downloader middlewares:

   DOWNLOADER_MIDDLEWARES = {
       'scrapy_seleniumbase_cdp.SeleniumBaseAsyncCDPMiddleware': 800
   }

2. If needed, Driver configuration can be provided:

   SELENIUMBASE_DRIVER_KWARGS = {
       # …
   }

Usage

Use the scrapy_seleniumbase_cdp.SeleniumBaseRequest instead of the scrapy built-in Request like below:

from scrapy_seleniumbase_cdp import SeleniumBaseRequest

yield SeleniumBaseRequest(url=url, callback=self.parse_result)

The request will be handled by SeleniumBase, and the request will have an additional meta key, named driver containing the SeleniumBase driver with the request processed.

def parse_result(self, response):
    print(response.request.meta['driver'].title)

For more information about the available driver methods and attributes, refer to the selenium python documentation (all vanilla selenium driver methods are available) and seleniumbase documentation (look for "driver" specific methods, located at the end of the page).

The selector response attribute work as usual (but contains the html processed by the selenium driver).

def parse_result(self, response):
    print(response.selector.xpath('//title/@text'))

Additional arguments

The scrapy_selenium.SeleniumBaseRequest accept 5 additional arguments:

wait_time / wait_until

When used, webdriver will perform an explicit wait before returning the response to the spider.

from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

yield SeleniumBaseRequest(
    url=url,
    callback=self.parse_result,
    wait_time=10,
    wait_until=EC.element_to_be_clickable((By.ID, 'someid'))
)

screenshot

When used, webdriver will take a screenshot of the page and the binary data of the .png captured will be added to the response meta:

yield SeleniumBaseRequest(
    url=url,
    callback=self.parse_result,
    screenshot=True
)


def parse_result(self, response):
    with open('image.png', 'wb') as image_file:
        image_file.write(response.meta['screenshot'])

script

When used, webdriver will execute custom JavaScript code.

yield SeleniumBaseRequest(
    url=url,
    callback=self.parse_result,
    script='window.scrollTo(0, document.body.scrollHeight);',
)

driver_methods

When used, seleniumbase webdriver will execute methods, provided as strings in a list, before returning page's html.

def start_requests(self):
    for url in self.start_urls:
        yield SeleniumRequest(
            url=url,
            driver_methods=['''.find_element("xpath","some_xpath").click()'''])

)

License

This project is licensed under the MIT License. It is a fork of Quartz-Core/scrapy-seleniumbase which was originally released under the WTFPL.