Skip to content
@scrapy

Scrapy project

An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.

Pinned Loading

  1. scrapy scrapy Public

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    Python 64.6k 12k

  2. scrapyd scrapyd Public

    A service daemon to run Scrapy spiders

    Python 3.1k 581

  3. parsel parsel Public

    Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

    Python 1.4k 175

  4. w3lib w3lib Public

    Python library of web-related functions

    HTML 419 129

  5. protego protego Public

    A pure-Python robots.txt parser with support for modern conventions.

    DIGITAL Command Language 94 35

  6. itemadapter itemadapter Public

    Common interface for data container classes

    Python 69 19

Repositories

Showing 10 of 35 repositories
  • w3lib Public

    Python library of web-related functions

    scrapy/w3lib's past year of commit activity
    HTML 419 BSD-3-Clause 129 3 6 Updated Oct 3, 2026
  • unattended-pr-guard Public

    A GitHub action to close pull requests that an LLM is opening unattended.

    scrapy/unattended-pr-guard's past year of commit activity
    JavaScript 0 Apache-2.0 0 0 0 Updated Oct 3, 2026
  • scrapy Public

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    scrapy/scrapy's past year of commit activity
    Python 64,566 BSD-3-Clause 11,989 159 (7 issues need help) 154 Updated Oct 3, 2026
  • sphinx-llm-friendly Public

    Sphinx extension to make documentation LLM-friendly

    scrapy/sphinx-llm-friendly's past year of commit activity
    Python 2 MIT 1 1 0 Updated Oct 3, 2026
  • cssselect Public

    CSS Selectors for Python

    scrapy/cssselect's past year of commit activity
    Python 311 70 9 7 Updated Oct 2, 2026
  • queuelib Public

    Collection of persistent (disk-based) and non-persistent (memory-based) queues for Python

    scrapy/queuelib's past year of commit activity
    Python 300 BSD-3-Clause 58 0 1 Updated Oct 2, 2026
  • scrapy-lint Public

    A linter for Scrapy projects.

    scrapy/scrapy-lint's past year of commit activity
    Python 22 MIT 8 20 23 Updated Oct 1, 2026
  • protego Public

    A pure-Python robots.txt parser with support for modern conventions.

    scrapy/protego's past year of commit activity
    DIGITAL Command Language 94 BSD-3-Clause 35 0 1 Updated Oct 1, 2026
  • itemadapter Public

    Common interface for data container classes

    scrapy/itemadapter's past year of commit activity
    Python 69 BSD-3-Clause 19 4 6 Updated Oct 1, 2026
  • itemloaders Public

    Library to populate items using XPath and CSS with a convenient API

    scrapy/itemloaders's past year of commit activity
    Python 49 BSD-3-Clause 23 8 2 Updated Oct 1, 2026
View all repositories