Skip to content

README: lead with what the SDK does #72

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
YahiaBakour merged 3 commits into from
Oct 8, 2026
Merged

README: lead with what the SDK does #72

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
fb0da87
docs: lead README with Context.dev capabilities
YahiaBakour Oct 8, 2026
a11a64a
docs: remove Stainless README attribution
YahiaBakour Oct 8, 2026
78920ab
docs: demonstrate scraping formats throughout README
YahiaBakour Oct 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
157 changes: 116 additions & 41 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,7 @@
<!-- prettier-ignore -->
[![PyPI version](https://img.shields.io/pypi/v/context.dev.svg?label=pypi%20(stable))](https://pypi.org/project/context.dev/)

The Context Dev Python library provides convenient access to the Context Dev REST API from any Python 3.9+
application. The library includes type definitions for all request params and response fields,
and offers both synchronous and asynchronous clients powered by [httpx](https://github.com/encode/httpx).

It is generated with [Stainless](https://www.stainless.com/).
Context.dev is a web scraping API for AI agents and LLMs. This SDK turns any URL into clean, LLM-ready markdown, crawls whole sites, searches the web, takes screenshots and extracts structured JSON against a schema you define, all with one API key. Proxies, JavaScript rendering and anti-bot handling run on Context.dev's side, so there is no headless browser to host.

## Documentation

Expand All @@ -22,23 +18,101 @@ pip install context.dev

## Usage

Set `CONTEXT_DEV_API_KEY` to your API key; the client reads it automatically.

### Scrape markdown and HTML

The full API of this library can be found in [api.md](api.md).

```python
import os
from context.dev import ContextDev

client = ContextDev(
api_key=os.environ.get("CONTEXT_DEV_API_KEY"), # This is the default and can be omitted
client = ContextDev()

page = client.web.scrape(
url="https://example.com",
formats={"markdown": True, "html": True},
)
print(page.markdown.data)
print(page.html.data)
```

### Extract structured JSON

```python
from context.dev import ContextDev

client = ContextDev()

page = client.web.scrape(
url="https://example.com",
formats={"json": True},
json_params={
"schema": {
"type": "object",
"properties": {
"title": {"type": ["string", "null"]},
"description": {"type": ["string", "null"]},
},
"required": ["title", "description"],
"additionalProperties": False,
},
},
)
print(page.json_.data)
```

### Extract relevant highlights

Return the passages that answer a question about the page.

```python
from context.dev import ContextDev

client = ContextDev()

page = client.web.scrape(
url="https://example.com",
formats={"highlights": True},
highlights_params={"query": "What is this domain used for?"},
)
print(page.highlights.data)
```

### Take a screenshot

The screenshot is returned as a base64 image data URL.

brand = client.brand.retrieve(
domain="REPLACE_ME",
type="by_domain",
```python
from context.dev import ContextDev

client = ContextDev()

page = client.web.scrape(
url="https://example.com",
formats={"screenshot": True},
)
print(brand.request_id)
print(page.screenshot.data)
```

## What you can do

| Task | Method |
| --- | --- |
| Scrape a URL to markdown, HTML, JSON, highlights or a screenshot | `client.web.scrape` |
| Crawl a site and get every page as markdown | `client.web.web_crawl_md` |
| Map every URL on a domain | `client.web.map_urls` |
| Search the web | `client.web.search` |
| Take a screenshot of a page | `client.web.screenshot` |
| Parse PDFs and documents | `client.parse.handle` |
| Run thousands of URLs as a batch | `client.batch.submit` |
| Watch a page for changes | `client.monitors.create` |
| Look up a company's logo, colors and brand data | `client.brand.retrieve` |

## Use it from an AI agent

Context.dev also ships as a plugin for [Claude](https://github.com/context-dot-dev/claude-plugin), [Cursor](https://github.com/context-dot-dev/cursor-plugin) and [Gemini CLI](https://github.com/context-dot-dev/gemini-cli-context), and as tools for [LangChain](https://github.com/context-dot-dev/langchain-context) and [Haystack](https://github.com/context-dot-dev/context-haystack).

While you can provide an `api_key` keyword argument,
we recommend using [python-dotenv](https://pypi.org/project/python-dotenv/)
to add `CONTEXT_DEV_API_KEY="My API Key"` to your `.env` file
Expand All @@ -59,11 +133,11 @@ client = AsyncContextDev(


async def main() -> None:
brand = await client.brand.retrieve(
domain="REPLACE_ME",
type="by_domain",
page = await client.web.scrape(
url="https://example.com",
formats={"markdown": True},
)
print(brand.request_id)
print(page.markdown.data)


asyncio.run(main())
Expand Down Expand Up @@ -96,11 +170,11 @@ async def main() -> None:
api_key=os.environ.get("CONTEXT_DEV_API_KEY"), # This is the default and can be omitted
http_client=DefaultAioHttpClient(),
) as client:
brand = await client.brand.retrieve(
domain="REPLACE_ME",
type="by_domain",
page = await client.web.scrape(
url="https://example.com",
formats={"markdown": True},
)
print(brand.request_id)
print(page.markdown.data)


asyncio.run(main())
Expand All @@ -124,12 +198,12 @@ from context.dev import ContextDev

client = ContextDev()

brand = client.brand.retrieve(
domain="xxx",
type="by_domain",
page = client.web.scrape(
url="https://example.com",
formats={"markdown": True},
timeout_opts={"milliseconds": 1000},
)
print(brand.timeout_opts)
print(page.markdown.data)
```

## Handling errors
Expand All @@ -148,9 +222,9 @@ from context.dev import ContextDev
client = ContextDev()

try:
client.brand.retrieve(
domain="REPLACE_ME",
type="by_domain",
client.web.scrape(
url="https://example.com",
formats={"markdown": True},
)
except context.dev.APIConnectionError as e:
print("The server could not be reached")
Expand Down Expand Up @@ -194,9 +268,9 @@ client = ContextDev(
)

# Or, configure per-request:
client.with_options(max_retries=5).brand.retrieve(
domain="REPLACE_ME",
type="by_domain",
client.with_options(max_retries=5).web.scrape(
url="https://example.com",
formats={"markdown": True},
)
```

Expand All @@ -206,6 +280,7 @@ By default requests time out after 1 minute. You can configure this with a `time
which accepts a float or an [`httpx.Timeout`](https://www.python-httpx.org/advanced/timeouts/#fine-tuning-the-configuration) object:

```python
import httpx
from context.dev import ContextDev

# Configure the default for all requests:
Expand All @@ -220,9 +295,9 @@ client = ContextDev(
)

# Override per-request:
client.with_options(timeout=5.0).brand.retrieve(
domain="REPLACE_ME",
type="by_domain",
client.with_options(timeout=5.0).web.scrape(
url="https://example.com",
formats={"markdown": True},
)
```

Expand Down Expand Up @@ -264,14 +339,14 @@ The "raw" Response object can be accessed by prefixing `.with_raw_response.` to
from context.dev import ContextDev

client = ContextDev()
response = client.brand.with_raw_response.retrieve(
domain="REPLACE_ME",
type="by_domain",
response = client.web.with_raw_response.scrape(
url="https://example.com",
formats={"markdown": True},
)
print(response.headers.get('X-My-Header'))

brand = response.parse() # get the object that `brand.retrieve()` would have returned
print(brand.request_id)
page = response.parse()
print(page.markdown.data)
```

These methods return an [`APIResponse`](https://github.com/context-dot-dev/context-python-sdk/tree/main/src/context/dev/_response.py) object.
Expand All @@ -285,9 +360,9 @@ The above interface eagerly reads the full response body when you make the reque
To stream the response body, use `.with_streaming_response` instead, which requires a context manager and only reads the response body once you call `.read()`, `.text()`, `.json()`, `.iter_bytes()`, `.iter_text()`, `.iter_lines()` or `.parse()`. In the async client, these are async methods.

```python
with client.brand.with_streaming_response.retrieve(
domain="REPLACE_ME",
type="by_domain",
with client.web.with_streaming_response.scrape(
url="https://example.com",
formats={"markdown": True},
) as response:
print(response.headers.get("X-My-Header"))

Expand Down
Loading