Playground Sign in Start free
Developer and reference · stackoverflow.com

How to scrape Stack Overflow

Stack Overflow (stackoverflow.com) is a developer and reference site. This guide is built from our own checks of its public pages: what the pages contain, whether our requests get through right now, and the request settings that worked.

Ujeebu scrapes any public website. Not listed here? Try any URL in the Playground or ask us about a site.

Mostly working 5 credits a page, about $0.50 per 1,000 on Pro

15 of 21 of our checks passed in the last 7 days (71.4%).

Last check 2026-11-22 20:31 UTC · last pass 2026-11-22 13:33 UTC · checked with our own requests, never customer data.

Try it in the Playground

Recipe one call

Use the Stack Overflow recipe: send the URL, get JSON

5 credits (JS) a page
curl -G 'https://api.ujeebu.com/scrape' \
  -H "ApiKey: $UJEEBU_API_KEY" \
  --data-urlencode 'url=https://stackoverflow.com/questions/tagged/web-scraping' \
  --data-urlencode 'recipe=stackoverflow'

The answer is {"result": {...}} with the fields of the recipe. Billed like any /scrape call: successful responses cost credits; blocked, timed-out and failed requests are free.

Run the code on this page with your own key: 5,000 free credits for 7 days, no card. Get a free API key →

Recipe

Call the recipe by name with the recipe parameter: the Scrape API applies its extract rules and the settings that work on Stack Overflow. Each recipe was run on two real Stack Overflow pages and keeps only the fields that came back on both; it is checked again every week, and hidden if it stops working.

Search and listing pages

Validated 2026-11-14

One call: recipe=stackoverflow 5 credits (JS) a page

curl -G 'https://api.ujeebu.com/scrape' \
  -H "ApiKey: $UJEEBU_API_KEY" \
  --data-urlencode 'url=https://stackoverflow.com/questions/tagged/web-scraping' \
  --data-urlencode 'recipe=stackoverflow'

The answer is {"result": {...}} with the fields below; the Ujb-Recipe header names the version (for example stackoverflow/listing@v1) and Ujb-Credits the cost.

What this recipe extracts

FieldTypeExample
resultslist of 3+ items
results[].urlURLhttps://stackoverflow.com/questions/80008831/merge-two-celery-tasks
results[].titletextMerge two Celery tasks
results[].imageimage URLhttps://i.sstatic.net/zlTU5.png?s=32
results[].scoretext-5
results[].commentsnumber (as text)6
results[].descriptiontextLet's say I have a web scraping pipeline. For some reason, I use Ce...
results[].updateddate2026-10-14 02:52:47Z

Examples from stackoverflow.com/questions/tagged/web-scraping on 2026-11-14, shortened.

Sample output (real, trimmed)

{
    "results": [
        {
            "url": "https://stackoverflow.com/questions/80008831/merge-two-celery-tasks",
            "title": "Merge two Celery tasks",
            "image": "https://i.sstatic.net/zlTU5.png?s=32",
            "score": "-5",
            "comments": "6",
            "description": "Let's say I have a web scraping pipeline. For some reason, I use Celery to write sanitized payloads to the database. Each payload is a list of rows. Due to t...",
            "updated": "2026-10-14 02:52:47Z"
        },
        {
            "url": "https://stackoverflow.com/questions/79995776/why-does-selenium-playwright-keep-getting-detected-by-cloudflare-even-with-steal",
            "title": "Why does Selenium/Playwright keep getting detected by Cloudflare even with stealth options? [duplicate]",
            "image": "https://www.gravatar.com/avatar/ac856fbd26442684a89f853d1a6c349f?s=32&d=identicon&r=PG",
            "score": "-8",
            "comments": "5",
            "description": "I'm working on a web scraping project where I need to extract data from several e-commerce sites. I started with Puppeteer, then tried Playwright with stealt...",
            "updated": "2026-08-17 09:32:27Z"
        },
        {
            "url": "https://stackoverflow.com/questions/79962099/my-restuarant-web-scraping-program-for-google-maps-only-returns-an-empty-string",
            "title": "My restuarant web-scraping program for google maps only returns an empty string array. How can I fix this issue?",
            "image": "https://www.gravatar.com/avatar/f45b69a9d91d0943e955ba79ea3be9ce?s=32&d=identicon&r=PG&f=y&so-version=2",
            "score": "3",
            "comments": "1",
            "description": "I'm working on a project where my program take restaurant details (name, location, prices, rating, etc) from google maps then add them to SQL database on pyt...",
            "updated": "2026-06-17 09:45:30Z"
        }
    ]
}
extract_rules (1 rule): the rules behind the recipe

The API applies these rules and the settings js=true for you when you send recipe=stackoverflow. To change what comes back, send your own extract_rules (they replace the recipe's) or any setting (yours always win). Sites without a recipe work the same way with extract rules.

{
    "results": {
        "selector": "div[itemprop=\"itemListElement\"]",
        "type": "obj",
        "multiple": true,
        "children": {
            "url": {
                "selector": "a[itemprop=\"url\"]",
                "type": "link"
            },
            "title": {
                "selector": "h3.s-post-summary--content-title a span[itemprop='name']",
                "type": "text"
            },
            "image": {
                "selector": "img",
                "type": "image"
            },
            "score": {
                "selector": "span[itemprop='upvoteCount']",
                "type": "text"
            },
            "comments": {
                "selector": "meta[itemprop='commentCount']",
                "type": "attr",
                "attribute": "content"
            },
            "description": {
                "selector": "div.s-post-summary--content-excerpt[itemprop='text']",
                "type": "text"
            },
            "updated": {
                "selector": "meta[itemprop='dateModified']",
                "type": "attr",
                "attribute": "content"
            }
        }
    }
}

Code

Complete scripts: set the UJEEBU_API_KEY environment variable and run them as they are. Each one calls the recipe by name (the Go SDK has no recipe field, so the Go script sends the recipe's rules and settings itself), prints the extracted JSON, and prints the credits the request cost (the Ujb-Credits response header) on stderr. A non-200 answer stops the script with the API's error message.

Search and listing pages

Pages through 3 result pages, one at a time, and stops early when a page has no results.

stackoverflow_listing_sdk.py pip install ujeebu-python==0.1.5
# Scrape Stack Overflow search and listing pages with the Ujeebu Scrape API.
# Calls the recipe stackoverflow: the API applies its extract rules and validated settings (validated 2026-11-14). Send extract_rules or any setting to override them.
# Uses the official SDK ujeebu-python 0.1.5: pip install ujeebu-python==0.1.5
# Run: UJEEBU_API_KEY=your_key python3 stackoverflow_listing_sdk.py

import json
import os
import sys
import time
from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit

import requests
from ujeebu_python import UjeebuClient

API_KEY = os.environ.get("UJEEBU_API_KEY")
START_URL = "https://stackoverflow.com/questions/tagged/web-scraping"
PAGES = 3
RECIPE = "stackoverflow"  # rules and settings come from the recipe; add your own to override them


def page_url(page):
    if page == 1:
        return START_URL
    parts = urlsplit(START_URL)
    query = dict(parse_qsl(parts.query))
    query["page"] = str(page)
    return urlunsplit(parts._replace(query=urlencode(query)))


def scrape(client, url):
    # a GET with recipe=: the API applies the recipe and answers {"result": {...}}
    try:
        response = client.scrape(url, params={"recipe": RECIPE})
    except requests.RequestException as error:
        sys.exit(f"Request failed for {url}: {error}")
    if response.status_code != 200:
        try:
            message = response.json().get("message", response.text)
        except ValueError:
            message = response.text
        sys.exit(f"HTTP {response.status_code} for {url}: {str(message)[:300]}")
    print(f"Ujb-Credits: {response.headers.get('Ujb-Credits')}, Ujb-Recipe: {response.headers.get('Ujb-Recipe')}", file=sys.stderr)
    return response.json()["result"]


if not API_KEY:
    sys.exit("Set the UJEEBU_API_KEY environment variable.")

client = UjeebuClient(api_key=API_KEY)

for page in range(1, PAGES + 1):
    result = scrape(client, page_url(page))
    for item in result.get("results") or []:
        print(item.get("title"), item.get("updated"), item.get("score"))
    # Example output:
    # Merge two Celery tasks 2026-10-14 02:52:47Z -5
    # Why does Selenium/Playwright keep get... 2026-08-17 09:32:27Z -8
    # every field: print(json.dumps(result, indent=2, ensure_ascii=False))
    if not result.get("results"):
        break
    time.sleep(2)  # be polite: one page at a time

SDK tabs use the official Ujeebu SDKs (ujeebu-python 0.1.5, @ujeebu-org/ujeebu-sdk 0.1.2, github.com/ujeebu/ujeebu-go v0.1.8); there is no PHP SDK, so the PHP tab calls the API with curl. For agents and scripts, the same status and settings are at /scrape/stackoverflow.json.

Try it yourself

Run this on Stack Overflow with your own key

Get a free API key, set UJEEBU_API_KEY and run any script above as it is. 5,000 free credits for 7 days, no card.

URL patterns and pagination

What the URLs look like, read from the pages our checks fetched.

Search and listing pages

Search or listing URL
https://stackoverflow.com/questions/tagged/{query}
Next pages
add page=2, page=3 and so on. Page 2: https://stackoverflow.com/questions/tagged/web-scraping?page=2
Validated on
https://stackoverflow.com/questions/tagged/web-scrapinghttps://stackoverflow.com/questions/tagged/php

Settings that work today

From our health checks over the last 7 days: the cheapest settings that return the real page most of the time.

Endpoint/scrape
jstrue
Costup to 5 credits a request

Each public page is one request: change the path in the URL to fetch another one. Collect detail page URLs from listing pages first, then fetch each detail page. Our checks run with JavaScript rendering on (js=true): keep it on.

Try it in the Playground

Troubleshooting

The failures our own checks of stackoverflow.com ran into over the last 7 days, and what got the page in the same check.

What happenedTimesWhat fixed it
The site blocked the request (HTTP 403 or a block page) 4 None of the settings we tried got past it in 7 days.

What people build with Stack Overflow data

  • Research datasetsBuild datasets of public pages for analysis, search or LLM pipelines.More on this use case
  • Trend trackingFollow what gets attention over time: scores, comments, stars.

Limits and etiquette

  • Public pages only. Pages behind a login, and data shown only to signed-in users, are out of scope.
  • Keep the rate modest: fetch pages one after another or with low concurrency, and cache what you already have.
  • Read Stack Overflow's terms of service before you collect data at scale, and do not collect personal data you have no basis to process. This guide is not legal advice.

Questions

Can you scrape Stack Overflow with Ujeebu?

Partly. 15 of 21 checks of public Stack Overflow pages passed in the last 7 days; most failures were the site blocked the request (HTTP 403 or a block page). Retry failed pages later, and use the settings on this page.

Which settings work for Stack Overflow?

Call /scrape with js=true.

How many credits does a Stack Overflow request cost?

With these settings a request costs up to 5 credits. Only responses with HTTP 200 or 404 are billed: a blocked or timed-out request is free.

Which fields can I extract from Stack Overflow?

With the extraction recipes on this page, validated on real Stack Overflow pages through our API: search and listing pages: url, title, image, score, comments, description, updated. Pass the recipe as extract_rules in a POST to /scrape and the API returns those fields as JSON.

How fresh is this page?

The status and settings come from our own health checks of stackoverflow.com, which run around the clock. Last check: 2026-11-22 20:31:54 UTC; last pass: 2026-11-22 13:33:49 UTC.