Scrapy

To be added

Python web scraper

Table of contents

Installation

pip install Scrapy
# or
poetry add Scrapy

cd <proj-root>
scrapy startproject <proj-name> .

First navigate to a specific Scrapy project:

cd <proj-name>

Check that you are indeed in the right project by:

$ scrapy
Scrapy x.x.x - project: <proj-name>

$ scrapy genspider -l
  basic
  crawl
  csvfeed
  xmlfeed

scrapy genspider -t crawl <spider-name> <allowed-domain>

scrapy list