How to Scrape Google News Results with a SERP API: Python Tutorial

To scrape Google News reliably, call a SERP API from Python and get structured JSON in one request, no proxy management, no broken selectors, no headless browser to maintain. Imagine you are building a daily competitor-mention monitor: you need headlines, publisher names, article URLs, and publish timestamps, refreshed every morning. By the end of this tutorial you will be able to make your first request, parse the response, collect up to 50 results per call, and feed the output into a vector store for a RAG pipeline, with exact credit costs so you can plan your spend before writing a line of production code.

Is There an API for Google News?

Google offered an official News API for developers, but it was discontinued and is no longer available. Searching for it today leads to archived documentation and community workarounds, not a working endpoint.

Three categories of alternatives exist, and they have meaningfully different trade-offs:

• RSS feed wrappers like the unofficial python-googlenews library parse Google News RSS feeds. They are free and easy to start with, but RSS feeds return a limited number of items, omit many fields, and break when Google adjusts the feed structure.
• Raw HTML scraping with requests and BeautifulSoup or Scrapy gives you full control but requires constant maintenance. Google News layouts change without notice, so selectors that worked last week may return nothing today. More on this below.
• SERP APIs return pre-parsed JSON with stable field names and handle the infrastructure layer on your behalf. You pay per request, but you eliminate the maintenance burden entirely.

For media monitoring, brand mention tracking, news aggregation pipelines, and RAG data ingestion, a SERP API is the practical replacement for the discontinued official API. The cost is predictable, the output is structured, and you are not racing against Google's next layout change.

Why Parsing Google News HTML Directly Breaks

Direct HTML parsing of Google News is brittle for two reasons that compound each other.

First, layouts change. Google redesigns its result pages without notice, so a CSS selector or XPath expression that matched article titles yesterday may match nothing today. Every layout change means a debugging session before your pipeline produces data again.

Second, headless-browser scrapers often hit blocks and require ongoing maintenance: proxy rotation, session management, and CAPTCHA handling each add engineering surface area. The result is a scraper that works at small scale but demands constant attention as volume grows.

A SERP API returns structured results from the Google News surface with a stable parameter interface, so you code against an API rather than changing page HTML. When Google updates its layout, the API provider absorbs the change; your code stays the same.

One distinction worth making before you write any code: search_type=news queries the Google News surface, which surfaces aggregated articles, clustered stories, and publisher attribution. search_type=search targets the main Google SERP, which may include a news box but is a different surface with different result shapes. If you want news results specifically, use search_type=news. See the SERP API search types documentation for the full list of supported values.

Try it yourself

Run this on your own pages

Get a free API key and make your first call in minutes. 5,000 free credits for 7 days, no card.

Get a free API key or try the playground first →

How to Scrape Google News with Python and Ujeebu's SERP API

The Ujeebu SERP API accepts a search string and a search_type parameter. Authentication is a single header: ApiKey: YOUR_API_KEY. Every successful SERP request costs 5 credits regardless of how many results it returns, which makes the unit to optimise results per request rather than requests per query.

A minimal first request

curl -X GET \
  'https://api.ujeebu.com/serp?search=OpenAI+funding&search_type=news&lang=en&location=US&results_count=10' \
  -H "ApiKey: YOUR_API_KEY"

The same request in Python:

import requests

API_KEY = "YOUR_API_KEY"

params = {
    "search": "OpenAI funding",
    "search_type": "news",
    "lang": "en",
    "location": "US",
    "results_count": 10,
    "batch": False,
}

response = requests.get(
    "https://api.ujeebu.com/serp",
    params=params,
    headers={"ApiKey": API_KEY},
)

data = response.json()

# Inspect the top-level keys to locate the news results array
print(list(data.keys()))

The response is JSON. Inspect the top-level keys to locate the news results array and the fields for title, source, URL, and published time, as defined in the SERP API docs. Each result typically includes the article headline, the publisher name, a URL, and a timestamp indicating when the article was last updated. Some results also include a snippet and an image source. Use the URL field from each item as the input to your downstream pipeline.

To monitor your credit balance and usage programmatically, call the Account API:

account = requests.get(
    "https://api.ujeebu.com/account",
    headers={"ApiKey": API_KEY},
)
print(account.json())
# Returns: plan, quota, used, used_percent, balance, and more

Filtering by language and location

The lang parameter accepts a language code (en, fr, de, and so on). The location parameter targets a geographic market (default US). See the SERP API docs for accepted values. To pull French-language news from France:

params = {
    "search": "intelligence artificielle",
    "search_type": "news",
    "lang": "fr",
    "location": "France",
    "results_count": 10,
    "batch": False,
}

response = requests.get(
    "https://api.ujeebu.com/serp",
    params=params,
    headers={"ApiKey": API_KEY},
)
print(response.json())

This is more reliable than running separate scrapers per market. One API, one parameter change, one consistent response shape.

Collecting up to 50 results per page

results_count defaults to 10 and batch defaults to false. Enable batch=true to request more results per page, up to 50 SERP results at 5 credits per page. This is the most credit-efficient way to collect volume.

params = {
    "search": "EU AI Act",
    "search_type": "news",
    "lang": "en",
    "location": "US",
    "results_count": 50,
    "batch": True,
}

response = requests.get(
    "https://api.ujeebu.com/serp",
    params=params,
    headers={"ApiKey": API_KEY},
)

data = response.json()
# Locate the news results array in the response
news_results = data.get("news_results", [])
print(f"Collected {len(news_results)} articles")

To go beyond 50 results, increment the page parameter. Each page costs 5 credits regardless of how many results it returns, so always set results_count=50 with batch=true before paginating to get the most results per credit spent.

For a deeper look at the available parameters, see the SERP API docs and the Google News SERP tutorial.

Takeaway: With batch=true and results_count=50 you get up to 50 results at the same cost as a 10-result request. Set this from the start and paginate only when you need more than 50 results.

How to Scrape Data from Google News: Step-by-Step

Once you have the response JSON, the processing steps are straightforward.

Step 1: Load results into a DataFrame. Use pd.json_normalize on the news results list. Print the column names to confirm which fields the API returned for your query, since the exact field names are defined in the SERP API docs.

Step 2: Deduplicate by URL. Google News sometimes surfaces the same article across multiple queries or consecutive runs. Drop duplicates on the URL column before storing anything.

Step 3: Write to line-delimited JSON. Append each run to a .jsonl file so runs accumulate cleanly without rewriting the whole dataset.

Step 4: Schedule the request. For a daily news monitor, a cron job or a workflow tool like Make or n8n is enough. Run the SERP call first, compare returned URLs against your stored set, and only fetch article content for URLs you have not seen before.

Step 5: Choose specific queries. Terms like "Anthropic Series E" or "EU AI Act enforcement" return tighter, more relevant clusters than broad terms like "AI". Use lang and location to segment by market rather than running one query per region.

Here is a complete example that calls the API, inspects the response, and writes results to a file:

import requests
import json

API_KEY = "YOUR_API_KEY"

def fetch_google_news(query, lang="en", location="US", results_count=50):
    params = {
        "search": query,
        "search_type": "news",
        "lang": lang,
        "location": location,
        "results_count": results_count,
        "batch": True,
    }
    response = requests.get(
        "https://api.ujeebu.com/serp",
        params=params,
        headers={"ApiKey": API_KEY},
    )
    response.raise_for_status()
    return response.json()

data = fetch_google_news("EU AI Act enforcement")

# Inspect the response structure before assuming field names
print("Top-level keys:", list(data.keys()))

# Write the full response for inspection and storage
with open("news_results.jsonl", "a") as f:
    f.write(json.dumps(data) + "\n")

Takeaway: Inspect the response keys before hardcoding field names. The SERP API docs define the exact schema; your code should reference those names rather than guessing.

Feeding Google News Results into a RAG Pipeline

The SERP response gives you headlines and URLs, but not article body text. For a RAG pipeline you need the full content of each article, not just the headline. This is where the Ujeebu Markdown API fits in.

The Markdown API converts a URL to markdown-formatted content for downstream NLP. Think of the SERP API as the table of contents and the Markdown API as the full text of each chapter. You need both to build a searchable library.

import requests
import time

API_KEY = "YOUR_API_KEY"

def fetch_article_markdown(url):
    response = requests.post(
        "https://api.ujeebu.com/markdown",
        json={"url": url},
        headers={"ApiKey": API_KEY},
    )
    response.raise_for_status()
    # Inspect the returned JSON for the content field; see /docs/apis/markdown
    return response.json()

# Assume news_urls is a list of article URLs from your SERP results
news_urls = [
    "https://scrape.li/quotes/",  # replace with real article URLs from SERP results
]

articles = []
for url in news_urls:
    content = fetch_article_markdown(url)
    articles.append({"url": url, "content": content})
    time.sleep(0.5)  # be polite to origin servers

Once you have the article content, split each piece into overlapping chunks and attach the source URL and timestamp as metadata before embedding. The exact chunking and embedding logic depends on your vector store and embedding model; the pattern below is pseudocode to illustrate the structure:

# Pseudocode: swap in your text splitter and embedding model
for article in articles:
    # Inspect article["content"] keys from /docs/apis/markdown to locate the text field
    text = article["content"].get("text", "")
    chunks = split_into_chunks(text, size=512, overlap=64)
    for chunk in chunks:
        vector = embed(chunk)
        vector_store.upsert(vector, metadata={"url": article["url"]})

To keep the pipeline fresh, schedule the SERP call on a cron, compare the returned URLs against your stored set, and call /markdown only for URLs you have not seen before. This keeps credit spend proportional to the rate of new content rather than the total corpus size. Check your remaining quota with GET /account before each scheduled run so you can alert on low balance before a run fails mid-way.

Takeaway: Diff against your URL store before calling /markdown. Fetching article content for URLs you already have wastes credits and adds nothing to the vector store.

Start with 5,000 free credits and no credit card required at ujeebu.com/signup.

What Google News Scraping Costs at Scale

Every SERP request costs 5 credits, whether you fetch 10 results or 50 with batch=true. The unit to optimise is results per credit, not requests per credit, which is why batching matters from the start.

Example calculation: a daily news monitor running 10 queries a day, each fetching 50 results with batch=true, uses 10 requests multiplied by 5 credits, which is 50 credits per day and roughly 1,500 credits per month.

If you also fetch article content via the Markdown API, each article costs 10 credits. For N new articles per day, add 10 multiplied by N credits per day. For example, 10 new articles add 100 credits per day, which is roughly 3,000 credits per month on top of the SERP cost. Adjust N to match the actual new-article rate for your queries.

The cost per 1,000 SERP requests ranges from $0.82 on the Starter plan to $0.33 on the Business plan. The chart below shows how that scales across plans.

The free trial includes 5,000 credits with no credit card required, enough to run a meaningful batch of news queries before spending anything. Failed requests are not billed, so a transient error does not consume credits.

For a fuller comparison of how credit-pool pricing works across providers, see scraping API pricing: per-request vs credit pool.

Takeaway: Compute queries_per_day × 5 credits × 30 to estimate your monthly SERP spend, then add new_articles_per_day × 10 credits × 30 for Markdown API calls. See Ujeebu pricing for current plan details.

Is Scraping Google Allowed?

Many sites restrict automated access in their terms of service, and Google is no exception. Review Google's current terms and seek legal advice for your specific use case before scraping at scale.

The risk profile varies by use case. Internal research or monitoring for your own organisation sits in a different category from reselling raw data or building a product that competes directly with Google's own services. Commercial applications at scale warrant a closer look at the legal picture.

Practical steps that reduce risk regardless of use case: do not store more data than you need, do not republish raw article text without a licence from the publisher, and check whether a licensed news data provider fits your compliance requirements better than scraping.

For a thorough treatment of the legal landscape, including robots.txt, terms of service, and relevant case law, see Is web scraping legal?.

Takeaway: Technical feasibility and legal permissibility are separate questions. If your use case is commercial or involves redistribution, get legal advice before scraping at scale.

---

FAQ

How do I scrape data from Google News?

The most reliable approach is a SERP API. Send a request with search_type=news and your query string; the API returns structured JSON with headlines, publisher names, URLs, and timestamps. Raw HTML scraping with BeautifulSoup is brittle because Google News layouts change without notice, breaking selectors and requiring constant maintenance.

Is there an API for Google News?

Google's official News API was discontinued and is no longer available. The practical alternative is a SERP API that queries the public Google News surface and returns structured JSON. RSS feed wrappers exist but return limited data and break when Google adjusts the feed format.

Is scraping Google allowed?

Many sites, including Google, restrict automated access in their terms of service. Enforcement is primarily technical (blocks and rate limiting) rather than legal for most use cases, but commercial applications or data redistribution carry higher risk. See Is web scraping legal? for a fuller treatment.

Is AI scraping illegal?

There is no law that specifically prohibits AI-related scraping. The legal questions that apply are the same as for any automated data collection: terms of service, copyright, and data protection regulations. The answer depends on what you scrape, how you use it, and where you operate. Legal advice is warranted for commercial or large-scale use.

How do I filter Google News results by language or country?

Use the lang parameter to set the language (for example lang=fr for French) and the location parameter to target a geographic market (for example location=France). Both parameters are supported by the Ujeebu SERP API and can be changed per request without running separate scrapers per market.

How much does it cost to scrape Google News at scale?

Every SERP request costs 5 credits. With batch=true and results_count=50 you collect up to 50 results at that same cost, making it the most efficient option for volume collection. The cost per 1,000 SERP requests ranges from $0.82 on the Starter plan to $0.33 on the Business plan. The free trial includes 5,000 credits with no credit card required. See Ujeebu pricing for current plan details.