Web scraping tutorial project

Scrapy project built following Zyte’s web scraping tutorial.

Note

If you are looking for a template for a new Scrapy project pre-configured to make the most out of Zyte services, see zyte-spider-templates-project instead.

Requirements

Python 3.8 or higher.

Setup

To be able to use this project, you must first:

1. Create a Python virtual environment.

   • On Windows:

     python3 -m venv tutorial-env
     tutorial-env\Scripts\activate.bat

   • On macOS and Linux:

     python3 -m venv tutorial-env
     . tutorial-env/bin/activate

2. Install the project requirements:

   pip install --upgrade -r requirements.txt

3. To be able to deploy to Scrapy Cloud, copy your Scrapy Cloud API key, run shub login and, when prompted, paste your API key and press Enter.

4. To be able to use Zyte API, append the following line to tutorial/settings.py, replacing YOUR_API_KEY with your Zyte API key:

   ZYTE_API_KEY = "YOUR_API_KEY"

   Tip

   For local development, you can alternatively use an environment variable with that name. In Scrapy Cloud, the ZYTE_API_KEY setting will be automatically defined with your Zyte API key.