Playground Sign in Start free
News and finance · bbc.com

How to scrape BBC News

BBC News is a news and finance site offering public access to articles and reporting. Developers typically collect article metadata, publication dates, headlines, and publisher information from its news pages.

Ujeebu scrapes any public website. Not listed here? Try any URL in the Playground or ask us about a site.

Working 5 credits a page, about $0.50 per 1,000 on Pro

41 of 41 of our checks passed in the last 7 days (100%).

Last check 2026-11-22 20:02 UTC · last pass 2026-11-22 20:02 UTC · checked with our own requests, never customer data.

Try it in the Playground

Recipe one call

Use the BBC News recipe: send the URL, get JSON

5 credits (JS) a page
curl -G 'https://api.ujeebu.com/scrape' \
  -H "ApiKey: $UJEEBU_API_KEY" \
  --data-urlencode 'url=https://www.bbc.com/news/technology' \
  --data-urlencode 'recipe=bbc/listing'

The answer is {"result": {...}} with the fields of the recipe. Billed like any /scrape call: successful responses cost credits; blocked, timed-out and failed requests are free.

Run the code on this page with your own key: 5,000 free credits for 7 days, no card. Get a free API key →

Recipes

Call a recipe by name with the recipe parameter: the Scrape API applies its extract rules and the settings that work on BBC News. Each recipe was run on two real BBC News pages and keeps only the fields that came back on both; it is checked again every week, and hidden if it stops working.

bbc.com news and articles: Search and listing pages

Validated 2026-11-21

One call: recipe=bbc/listing 5 credits (JS) a page

curl -G 'https://api.ujeebu.com/scrape' \
  -H "ApiKey: $UJEEBU_API_KEY" \
  --data-urlencode 'url=https://www.bbc.com/news/technology' \
  --data-urlencode 'recipe=bbc/listing'

The answer is {"result": {...}} with the fields below; the Ujb-Recipe header names the version (for example bbc/listing@v1) and Ujb-Credits the cost.

What this recipe extracts

FieldTypeExample
articleslist of 3+ items
articles[].urlURLhttps://www.bbc.com/news/articles/ckreyqvqe3gyo
articles[].titletextCouncillor sanctioned over offensive Facebook posts
articles[].imageimage URLhttps://ichef.bbci.co.uk/news/480/cpsprodpb/749e/live/4b2e5830-c6fd...
articles[].summarytextSub-committee members recommend that Carl Miller be removed from al...
articles[].publishedtext36 mins ago

Examples from www.bbc.com/news/technology on 2026-11-21, shortened.

Sample output (real, trimmed)

{
    "articles": [
        {
            "url": "https://www.bbc.com/news/articles/c6j9m2rg42n2o",
            "title": "MP opposing data centre joins government committee",
            "image": "https://ichef.bbci.co.uk/news/480/cpsprodpb/8889/live/952c2ae0-e4fb-11f1-a8e4-b1e12a49db15.jpg.webp",
            "summary": "He admits data centres are needed, but he is opposing plans for one in his constituency.",
            "published": "14 mins ago"
        },
        {
            "url": "https://www.bbc.com/news/articles/cjx2ry0d9913o",
            "title": "What is being done to stop disruptive cable theft?",
            "image": "https://ichef.bbci.co.uk/news/480/cpsprodpb/4d87/live/f9f8dd90-cc5c-11f1-b1a3-5bda642ff934.jpg.webp",
            "summary": "The removal of copper cables wipes out telecommunications and costs the UK millions.",
            "published": "27 mins ago"
        },
        {
            "url": "https://www.bbc.com/news/articles/cq7821d0pv49o",
            "title": "Phoebe Bridgers banned phones at her gigs, so fans are using art to remember them",
            "image": "https://ichef.bbci.co.uk/news/480/cpsprodpb/d17c/live/c4671de0-e507-11f1-b40b-5f46e8cf687c.jpg.webp",
            "summary": "The American singer has brought in a no-phone policy for the shows, inspiring fans to explore their artistic talent.",
            "published": "7 hrs ago"
        }
    ]
}
extract_rules (1 rule): the rules behind the recipe

The API applies these rules and the settings js=true for you when you send recipe=bbc/listing. To change what comes back, send your own extract_rules (they replace the recipe's) or any setting (yours always win). Sites without a recipe work the same way with extract rules.

{
    "articles": {
        "selector": "div[data-testid=\"alaska\"] > div",
        "type": "obj",
        "multiple": true,
        "children": {
            "url": {
                "selector": "a[data-testid=\"internal-link\"]",
                "type": "link"
            },
            "title": {
                "selector": "h2",
                "type": "text"
            },
            "image": {
                "selector": "img.Image-styles__ImageStyled-sc-8c99a12b-0:not(.hide-when-no-script)",
                "type": "image"
            },
            "summary": {
                "selector": "p[data-testid='card-description']",
                "type": "text"
            },
            "published": {
                "selector": "span[data-testid='card-metadata-lastupdated']",
                "type": "text"
            }
        }
    }
}

bbc.com news and articles: Articles

Validated 2026-11-21

One call: recipe=bbc/detail 5 credits (JS) a page

curl -G 'https://api.ujeebu.com/scrape' \
  -H "ApiKey: $UJEEBU_API_KEY" \
  --data-urlencode 'url=https://www.bbc.com/news/articles/cmr4y2yl17x7o' \
  --data-urlencode 'recipe=bbc/detail'

The answer is {"result": {...}} with the fields below; the Ujb-Recipe header names the version (for example bbc/detail@v1) and Ujb-Credits the cost.

What this recipe extracts

FieldTypeExample
titletextBlue plaque honour for poet's childhood home
descriptiontextTony Harrison, who died last year aged 88, grew up in Leeds and is...
imageimage URLhttps://ichef.bbci.co.uk/news/1024/branded_news/ac34/live/226f37f0-...
urlURLhttps://www.bbc.com/news/articles/cmr4y2yl17x7o
publisheddate2026-10-12T19:06:02.949Z
summarytextThe man who wrote Prime Minister Andy Burnham's favourite poem has...
sectiontextLeeds

Examples from www.bbc.com/news/articles/cmr4y2yl17x7o on 2026-11-21, shortened.

Sample output (real, trimmed)

{
    "title": "Blue plaque honour for poet's childhood home",
    "description": "Tony Harrison, who died last year aged 88, grew up in Leeds and is best known for his poem V.",
    "image": "https://ichef.bbci.co.uk/news/1024/branded_news/ac34/live/226f37f0-c655-11f1-8600-efb62b613ab5.jpg",
    "url": "https://www.bbc.com/news/articles/cmr4y2yl17x7o",
    "published": "2026-10-12T19:06:02.949Z",
    "summary": "The man who wrote Prime Minister Andy Burnham's favourite poem has been honoured with a blue plaque in his home city.",
    "section": "Leeds"
}
extract_rules (7 rules): the rules behind the recipe

The API applies these rules and the settings js=true for you when you send recipe=bbc/detail. To change what comes back, send your own extract_rules (they replace the recipe's) or any setting (yours always win). Sites without a recipe work the same way with extract rules.

{
    "title": {
        "selector": "h1",
        "type": "text"
    },
    "description": {
        "selector": "meta[property=\"og:description\"]",
        "type": "attr",
        "attribute": "content"
    },
    "image": {
        "selector": "meta[property=\"og:image\"]",
        "type": "attr",
        "attribute": "content"
    },
    "url": {
        "selector": "link[rel=\"canonical\"]",
        "type": "attr",
        "attribute": "href"
    },
    "published": {
        "selector": "time[datetime]",
        "type": "attr",
        "attribute": "datetime"
    },
    "summary": {
        "selector": "article p:first-of-type",
        "type": "text"
    },
    "section": {
        "selector": "a[data-testid='internal-link'][href*='/news/topics/']",
        "type": "text"
    }
}

Code

Complete scripts: set the UJEEBU_API_KEY environment variable and run them as they are. Each one calls the recipe by name (the Go SDK has no recipe field, so the Go script sends the recipe's rules and settings itself), prints the extracted JSON, and prints the credits the request cost (the Ujb-Credits response header) on stderr. A non-200 answer stops the script with the API's error message.

bbc.com news and articles: Search and listing pages

Runs the recipe on the two pages it was validated on, one after the other.

bbc_listing_sdk.py pip install ujeebu-python==0.1.5
# Scrape BBC News bbc.com news and articles: search and listing pages with the Ujeebu Scrape API.
# Calls the recipe bbc/listing: the API applies its extract rules and validated settings (validated 2026-11-21). Send extract_rules or any setting to override them.
# Uses the official SDK ujeebu-python 0.1.5: pip install ujeebu-python==0.1.5
# Run: UJEEBU_API_KEY=your_key python3 bbc_listing_sdk.py

import json
import os
import sys
import time

import requests
from ujeebu_python import UjeebuClient

API_KEY = os.environ.get("UJEEBU_API_KEY")
URLS = [
    "https://www.bbc.com/news/technology",
    "https://www.bbc.com/business",
]
RECIPE = "bbc/listing"  # rules and settings come from the recipe; add your own to override them


def scrape(client, url):
    # a GET with recipe=: the API applies the recipe and answers {"result": {...}}
    try:
        response = client.scrape(url, params={"recipe": RECIPE})
    except requests.RequestException as error:
        sys.exit(f"Request failed for {url}: {error}")
    if response.status_code != 200:
        try:
            message = response.json().get("message", response.text)
        except ValueError:
            message = response.text
        sys.exit(f"HTTP {response.status_code} for {url}: {str(message)[:300]}")
    print(f"Ujb-Credits: {response.headers.get('Ujb-Credits')}, Ujb-Recipe: {response.headers.get('Ujb-Recipe')}", file=sys.stderr)
    return response.json()["result"]


if not API_KEY:
    sys.exit("Set the UJEEBU_API_KEY environment variable.")

client = UjeebuClient(api_key=API_KEY)

for i, url in enumerate(URLS):
    if i:
        time.sleep(2)  # be polite: one page at a time
    result = scrape(client, url)
    for item in result.get("articles") or []:
        print(item.get("title"), item.get("published"))
    # Example output:
    # MP opposing data centre joins governm... 14 mins ago
    # What is being done to stop disruptive... 27 mins ago
    # every field: print(json.dumps(result, indent=2, ensure_ascii=False))

bbc.com news and articles: Articles

Runs the recipe on the two pages it was validated on, one after the other.

bbc_detail_sdk.py pip install ujeebu-python==0.1.5
# Scrape BBC News bbc.com news and articles: articles with the Ujeebu Scrape API.
# Calls the recipe bbc/detail: the API applies its extract rules and validated settings (validated 2026-11-21). Send extract_rules or any setting to override them.
# Uses the official SDK ujeebu-python 0.1.5: pip install ujeebu-python==0.1.5
# Run: UJEEBU_API_KEY=your_key python3 bbc_detail_sdk.py

import json
import os
import sys
import time

import requests
from ujeebu_python import UjeebuClient

API_KEY = os.environ.get("UJEEBU_API_KEY")
URLS = [
    "https://www.bbc.com/news/articles/cmr4y2yl17x7o",
    "https://www.bbc.com/news/articles/cq74pkelv9rxo",
]
RECIPE = "bbc/detail"  # rules and settings come from the recipe; add your own to override them


def scrape(client, url):
    # a GET with recipe=: the API applies the recipe and answers {"result": {...}}
    try:
        response = client.scrape(url, params={"recipe": RECIPE})
    except requests.RequestException as error:
        sys.exit(f"Request failed for {url}: {error}")
    if response.status_code != 200:
        try:
            message = response.json().get("message", response.text)
        except ValueError:
            message = response.text
        sys.exit(f"HTTP {response.status_code} for {url}: {str(message)[:300]}")
    print(f"Ujb-Credits: {response.headers.get('Ujb-Credits')}, Ujb-Recipe: {response.headers.get('Ujb-Recipe')}", file=sys.stderr)
    return response.json()["result"]


if not API_KEY:
    sys.exit("Set the UJEEBU_API_KEY environment variable.")

client = UjeebuClient(api_key=API_KEY)

for i, url in enumerate(URLS):
    if i:
        time.sleep(2)  # be polite: one page at a time
    result = scrape(client, url)
    print(result.get("title"), result.get("published"), result.get("section"))
    # Example output: Blue plaque honour for poet's childho... 2026-10-12T19:06:02.949Z Leeds
    # every field: print(json.dumps(result, indent=2, ensure_ascii=False))

bbc.co.uk news and sport

No extraction recipe for these pages yet: the script fetches the page with the working settings. Add extract rules to get JSON.

bbc_page_sdk.py pip install ujeebu-python==0.1.5
# Fetch a BBC News page with the Ujeebu Scrape API.
# Settings from our health checks (Ujeebu site guide). Prints the start of the page HTML.
# Uses the official SDK ujeebu-python 0.1.5: pip install ujeebu-python==0.1.5
# Run: UJEEBU_API_KEY=your_key python3 bbc_page_sdk.py

import json
import os
import sys
import time

import requests
from ujeebu_python import UjeebuClient

API_KEY = os.environ.get("UJEEBU_API_KEY")
URLS = [
    "https://www.bbc.co.uk/news",
]
SETTINGS = {
    "js": True,
}


def scrape(client, url):
    try:
        response = client.scrape(url, params=SETTINGS)
    except requests.RequestException as error:
        sys.exit(f"Request failed for {url}: {error}")
    if response.status_code != 200:
        try:
            message = response.json().get("message", response.text)
        except ValueError:
            message = response.text
        sys.exit(f"HTTP {response.status_code} for {url}: {str(message)[:300]}")
    print(f"Ujb-Credits: {response.headers.get('Ujb-Credits')}", file=sys.stderr)
    return response.text


if not API_KEY:
    sys.exit("Set the UJEEBU_API_KEY environment variable.")

client = UjeebuClient(api_key=API_KEY)

for i, url in enumerate(URLS):
    if i:
        time.sleep(2)  # be polite: one page at a time
    html = scrape(client, url)
    print(len(html), "bytes")
    print(html[:500])

SDK tabs use the official Ujeebu SDKs (ujeebu-python 0.1.5, @ujeebu-org/ujeebu-sdk 0.1.2, github.com/ujeebu/ujeebu-go v0.1.8); there is no PHP SDK, so the PHP tab calls the API with curl. For agents and scripts, the same status and settings are at /scrape/bbc.json.

Try it yourself

Run this on BBC News with your own key

Get a free API key, set UJEEBU_API_KEY and run any script above as it is. 5,000 free credits for 7 days, no card.

URL patterns and pagination

What the URLs look like, read from the pages our checks fetched.

bbc.com news and articles: Search and listing pages

Next pages
We did not see a next-page link on the pages we fetched. Results may load as you scroll: try scroll_down.
Validated on
https://www.bbc.com/news/technologyhttps://www.bbc.com/business

bbc.com news and articles: Articles

Pattern
https://www.bbc.com/news/articles/{id}
Validated on
https://www.bbc.com/news/articles/cmr4y2yl17x7ohttps://www.bbc.com/news/articles/cq74pkelv9rxo

Settings that work today

From our health checks over the last 7 days: the cheapest settings that return the real page most of the time.

bbc.com news and articles

Endpoint/scrape
jstrue
Costup to 5 credits a request

Collect detail page URLs from listing pages first, then fetch each detail page. Our checks run with JavaScript rendering on (js=true): keep it on.

Try it in the Playground

bbc.co.uk news and sport

Endpoint/scrape
jstrue
Costup to 5 credits a request

Collect detail page URLs from listing pages first, then fetch each detail page. Our checks run with JavaScript rendering on (js=true): keep it on.

Try it in the Playground

What people build with BBC News data

  • Media monitoringTrack the articles published on the topics and companies you follow.More on this use case
  • Content for AI and RAGCollect headlines, dates and summaries as structured input for search and LLM pipelines.More on this use case
  • Publishing timelineMeasure when and how often a site publishes, by section.

Limits and etiquette

  • Public pages only. Pages behind a login, and data shown only to signed-in users, are out of scope.
  • Keep the rate modest: fetch pages one after another or with low concurrency, and cache what you already have.
  • robots.txt of www.bbc.com (read 2026-11-17): bbc.com news and articles allowed for generic crawlers; no crawl delay set.
  • Read BBC News's terms of service before you collect data at scale, and do not collect personal data you have no basis to process. This guide is not legal advice.

Questions

Can you scrape BBC News with Ujeebu?

Yes. 41 of 41 checks of public BBC News pages passed in the last 7 days with the settings on this page.

Which settings work for BBC News?

Call /scrape with js=true.

How many credits does a BBC News request cost?

With these settings a request costs up to 5 credits. Only responses with HTTP 200 or 404 are billed: a blocked or timed-out request is free.

Which fields can I extract from BBC News?

With the extraction recipes on this page, validated on real BBC News pages through our API: bbc.com news and articles: search and listing pages: url, title, image, summary, published; bbc.com news and articles: articles: title, description, image, url, published, summary, section. Pass the recipe as extract_rules in a POST to /scrape and the API returns those fields as JSON.

Is it allowed to scrape BBC News?

Its robots.txt allows generic crawlers on the page types in this guide. Collect public pages only, read BBC News's terms of service, keep your request rate modest and do not collect personal data you have no basis to process. This is not legal advice.

How fresh is this page?

The status and settings come from our own health checks of bbc.com, which run around the clock. Last check: 2026-11-22 20:02:10 UTC; last pass: 2026-11-22 20:02:10 UTC.