Playground Sign in Start free

The Structured Data Challenge

Extracting structured data from the web should be straightforward, but every site presents unique obstacles. Traditional scraping approaches break down when faced with the diversity and complexity of modern websites.

Every Site is Different

Each website uses its own unique HTML structure, class names, and layout patterns, making universal extraction nearly impossible with a single approach

Fragile CSS Selectors

Hand-crafted CSS selectors break whenever sites redesign or update their markup, creating a constant maintenance burden for scraping pipelines

Dynamic Content

JavaScript-rendered pages, lazy-loaded elements, and single-page applications require headless browsers and careful wait strategies to extract correctly

Data Validation

Raw extracted data often needs normalization, type checking, and schema validation before it can reliably feed into downstream applications

Three Ways to Extract Structured Data

Choose the extraction approach that fits your use case, from zero-config article extraction to precise CSS rules

Zero-Config Article Extraction

Send a URL to the Extract API and get clean, structured article fields back - title, author, publish date, main text, and images - with no selectors or configuration. Purpose-built for news, blogs, and editorial pages and tuned across thousands of publisher layouts.

Best For:

  • News, blog, and editorial pages where you want the article, not the chrome
  • Rapid prototyping and quick content ingestion
  • Clean text ready for search, RAG, or summarisation
  • Getting structured content with a single API call

Rule-Based Precision with Scrape API

Define exact CSS selectors with extract_rules for deterministic, repeatable extraction. Perfect when you know the target site structure and need pixel-perfect control over what gets extracted. Rules support nested objects, arrays, multiple output types (text, HTML, attributes), and pagination across list pages.

Best For:

  • Production pipelines requiring 100% deterministic results
  • High-volume scraping where cost efficiency matters
  • Sites with stable, well-known HTML structures
  • Developers comfortable writing CSS selectors

Auto-Generated Rules with Guess-er

Get the best of both worlds: Guess-er analyzes the page and generates CSS extract_rules you can use with the Scrape API. Provide a target schema or example fields, and Guess-er returns reusable selectors. Generate rules once, then reuse them at Scrape API cost for all subsequent requests to similar pages.

Best For:

  • Automating selector creation for new site types
  • Reducing ongoing AI costs with reusable rules
  • Building extraction templates for similar page layouts
  • Teams scaling across hundreds of target sites

Extract Structured Data from Any Site Type

Our extraction tools work across every category of website, automatically adapting to each page structure

E-commerce Pages

Extract product names, prices, descriptions, images, ratings, reviews, and availability from any online store or marketplace

Business Directories

Pull company names, addresses, phone numbers, websites, ratings, and categories from directories like Yelp or Yellow Pages

Real Estate Listings

Capture property details, prices, square footage, bedrooms, locations, and agent information from real estate platforms

Job Boards

Extract job titles, companies, locations, salaries, requirements, and posting dates from job listing sites and career pages

News Articles

Pull article titles, authors, publication dates, content, tags, and featured images from news sites and blogs

Social Profiles

Extract profile names, bios, follower counts, contact information, and recent activity from social media and professional networks

Government Databases

Collect public records, permit data, business registrations, and regulatory filings from government and municipal portals

Review Sites

Scrape reviewer names, ratings, review text, dates, and verified purchase status from review platforms and aggregators

Try Now
No Credit Card Required.

How Structured Data Extraction Works

Three simple steps from raw web page to clean, structured data

1

Choose Your Extraction Approach

Pick the method that fits your workflow. Use the Extract API for zero-config article extraction, the Scrape API with extract_rules for CSS selector precision, or Guess-er to auto-generate reusable rules. All approaches handle JavaScript rendering, anti-bot protection, and proxy rotation automatically.

2

Send Your Request via API

Make a single API call with the target URL and your extraction parameters. Provide extract_rules for structured JSON, or just the URL for article extraction. Our system fetches the page, renders JavaScript if needed, bypasses anti-bot measures, and extracts your data in one request.

3

Receive Clean, Structured JSON

Get back clean, structured JSON data ready for your application. Results include the extracted fields with proper data types plus metadata like extraction time. Integrate directly into your data pipeline, database, or analytics workflow via webhooks or polling.

Try Structured Extraction Now

Enter any URL and see rules-based extraction results instantly

No API key required for testing

Powerful Structured Extraction Features

Zero-Config Article Extraction

The Extract API returns structured article fields - title, author, date, body, images - from news and blog pages with no selectors. Purpose-built and tuned across thousands of publisher layouts, so you get clean content ready for search, RAG, or summarisation.

Nested & Repeated Fields

extract_rules support nested objects and arrays, so a single Scrape API call returns structured records - not just flat key-value pairs - across list and detail pages. Shape the output to match your exact data model.

CSS Extract Rules

Define precise CSS selectors with the Scrape API extract_rules parameter for deterministic, repeatable extraction. Support for nested objects, arrays, multiple output types (text, HTML, attributes), and pagination across listing pages.

JSON Schema Support

Guess-er accepts a JSON schema - types, required fields, enums, nested objects, arrays, $ref/$defs (Pydantic-compatible), and anyOf/oneOf combinators - when generating rules. And because Scrape API output is deterministic JSON, you can validate every response against your own schema on your side.

Guess-er Rule Generation

Let Guess-er analyze any page and generate reusable CSS extract_rules automatically. Provide a target schema or example fields, get back selectors you can use with the Scrape API. Generate once, reuse thousands of times at lower cost.

JS Rendering & Anti-Bot

All extraction approaches include full headless browser rendering, automatic proxy rotation, stealth mode, and CAPTCHA solving. Handle JavaScript-heavy SPAs, infinite scroll pages, and bot-protected sites without extra configuration.

FAQ

Frequently Asked Questions

  • What is the Scrape API and how do extract_rules work?

    The Scrape API fetches a page (rendering JavaScript when needed) and returns structured JSON. With the extract_rules parameter you map field names to CSS selectors - for example {"title":"h1","price":".price"} - and get back exactly those fields. Rules support nested objects, arrays, per-field output types (text, HTML, attributes), and pagination across list pages. Because it is selector-based, results are deterministic and repeatable, so the same rules return the same JSON on every run.

  • Should I write extract_rules by hand or generate them?

    Hand-written extract_rules give you full control and are ideal once you know a site's HTML structure - production pipelines that need 100% deterministic results, maximum speed, and the lowest cost per request. When you are scaling across many new site types, Guess-er can generate CSS rules for you to review and reuse instead of writing them from scratch. Either way you run the same Scrape API at the same low cost per request, and the selectors return identical results every time with no model variability.

  • How does Guess-er work and when should I use it?

    Guess-er analyzes a page's HTML structure and generates CSS extract_rules that you can reuse with the Scrape API. Provide a target JSON schema or an example of the fields you want, and it returns selectors. The key advantage is cost efficiency: you generate the rules once, then reuse them with the cheaper Scrape API for all subsequent requests to similar pages. Use Guess-er when you are scaling extraction across many pages with similar structures, when you want the reliability of CSS selectors without writing them manually, or when you need to quickly bootstrap extraction templates for new site types.

  • Can I validate the extracted data against a specific schema?

    Yes. Guess-er accepts a standard JSON Schema when generating rules - type definitions, required fields, enums, and nested structures - so the selectors it returns target the fields your schema expects. The schema format supports $ref and $defs for Pydantic-compatible schemas, anyOf/oneOf/allOf combinators, and all standard JSON Schema types including nested objects and arrays. And because Scrape API output is deterministic JSON, you can validate every response against the same schema (or your Pydantic model) on your side.

  • How do you handle JavaScript-rendered and bot-protected pages?

    The Scrape and Extract APIs include full headless browser rendering with Chromium. JavaScript is executed, SPAs are rendered, and dynamic content is loaded before extraction begins. For bot-protected sites, our system includes stealth mode with realistic browser fingerprints, automatic proxy rotation across residential and datacenter pools, and automatic CAPTCHA solving. You can configure wait_for to pause extraction until a specific CSS selector appears, or set timeouts for pages with complex loading sequences. Set js=true to render before extraction.

  • What output formats does structured extraction support?

    Both endpoints return clean JSON by default with properly typed fields. The Scrape API with extract_rules returns data matching your defined rule hierarchy - nested objects, arrays, and per-field output types. The Extract API returns article fields with consistent naming. You can integrate results directly into databases, data warehouses, or downstream applications via our REST API. For batch processing, results can be delivered via webhooks or exported as CSV through our dashboard. All responses include metadata such as extraction time for monitoring and optimization.

Get started with 5000 free credits
No Credit Card Required.

Start Extracting Structured Data Today

Turn any web page into clean, structured JSON with a single API call. Reusable rules, deterministic results, minimal maintenance.

Start Using
Start Free Trial
No Credit Card Required.
Talk to a Data Extraction Expert