The Structured Data Challenge
Extracting structured data from the web should be straightforward, but every site presents unique obstacles. Traditional scraping approaches break down when faced with the diversity and complexity of modern websites.
Every Site is Different
Each website uses its own unique HTML structure, class names, and layout patterns, making universal extraction nearly impossible with a single approach
Fragile CSS Selectors
Hand-crafted CSS selectors break whenever sites redesign or update their markup, creating a constant maintenance burden for scraping pipelines
Dynamic Content
JavaScript-rendered pages, lazy-loaded elements, and single-page applications require headless browsers and careful wait strategies to extract correctly
Data Validation
Raw extracted data often needs normalization, type checking, and schema validation before it can reliably feed into downstream applications
Three Ways to Extract Structured Data
Choose the extraction approach that fits your use case, from zero-config article extraction to precise CSS rules
Zero-Config Article Extraction
Send a URL to the Extract API and get clean, structured article fields back - title, author, publish date, main text, and images - with no selectors or configuration. Purpose-built for news, blogs, and editorial pages and tuned across thousands of publisher layouts.
Best For:
- News, blog, and editorial pages where you want the article, not the chrome
- Rapid prototyping and quick content ingestion
- Clean text ready for search, RAG, or summarisation
- Getting structured content with a single API call
Rule-Based Precision with Scrape API
Define exact CSS selectors with extract_rules for deterministic, repeatable extraction. Perfect when you know the target site structure and need pixel-perfect control over what gets extracted. Rules support nested objects, arrays, multiple output types (text, HTML, attributes), and pagination across list pages.
Best For:
- Production pipelines requiring 100% deterministic results
- High-volume scraping where cost efficiency matters
- Sites with stable, well-known HTML structures
- Developers comfortable writing CSS selectors
Auto-Generated Rules with Guess-er
Get the best of both worlds: Guess-er analyzes the page and generates CSS extract_rules you can use with the Scrape API. Provide a target schema or example fields, and Guess-er returns reusable selectors. Generate rules once, then reuse them at Scrape API cost for all subsequent requests to similar pages.
Best For:
- Automating selector creation for new site types
- Reducing ongoing AI costs with reusable rules
- Building extraction templates for similar page layouts
- Teams scaling across hundreds of target sites
Extract Structured Data from Any Site Type
Our extraction tools work across every category of website, automatically adapting to each page structure
E-commerce Pages
Extract product names, prices, descriptions, images, ratings, reviews, and availability from any online store or marketplace
Business Directories
Pull company names, addresses, phone numbers, websites, ratings, and categories from directories like Yelp or Yellow Pages
Real Estate Listings
Capture property details, prices, square footage, bedrooms, locations, and agent information from real estate platforms
Job Boards
Extract job titles, companies, locations, salaries, requirements, and posting dates from job listing sites and career pages
News Articles
Pull article titles, authors, publication dates, content, tags, and featured images from news sites and blogs
Social Profiles
Extract profile names, bios, follower counts, contact information, and recent activity from social media and professional networks
Government Databases
Collect public records, permit data, business registrations, and regulatory filings from government and municipal portals
Review Sites
Scrape reviewer names, ratings, review text, dates, and verified purchase status from review platforms and aggregators
How Structured Data Extraction Works
Three simple steps from raw web page to clean, structured data
Choose Your Extraction Approach
Pick the method that fits your workflow. Use the Extract API for zero-config article extraction, the Scrape API with extract_rules for CSS selector precision, or Guess-er to auto-generate reusable rules. All approaches handle JavaScript rendering, anti-bot protection, and proxy rotation automatically.
Send Your Request via API
Make a single API call with the target URL and your extraction parameters. Provide extract_rules for structured JSON, or just the URL for article extraction. Our system fetches the page, renders JavaScript if needed, bypasses anti-bot measures, and extracts your data in one request.
Receive Clean, Structured JSON
Get back clean, structured JSON data ready for your application. Results include the extracted fields with proper data types plus metadata like extraction time. Integrate directly into your data pipeline, database, or analytics workflow via webhooks or polling.
Try Structured Extraction Now
Enter any URL and see rules-based extraction results instantly
No API key required for testing
Powerful Structured Extraction Features
Zero-Config Article Extraction
The Extract API returns structured article fields - title, author, date, body, images - from news and blog pages with no selectors. Purpose-built and tuned across thousands of publisher layouts, so you get clean content ready for search, RAG, or summarisation.
Nested & Repeated Fields
extract_rules support nested objects and arrays, so a single Scrape API call returns structured records - not just flat key-value pairs - across list and detail pages. Shape the output to match your exact data model.
CSS Extract Rules
Define precise CSS selectors with the Scrape API extract_rules parameter for deterministic, repeatable extraction. Support for nested objects, arrays, multiple output types (text, HTML, attributes), and pagination across listing pages.
JSON Schema Support
Guess-er accepts a JSON schema - types, required fields, enums, nested objects, arrays, $ref/$defs (Pydantic-compatible), and anyOf/oneOf combinators - when generating rules. And because Scrape API output is deterministic JSON, you can validate every response against your own schema on your side.
Guess-er Rule Generation
Let Guess-er analyze any page and generate reusable CSS extract_rules automatically. Provide a target schema or example fields, get back selectors you can use with the Scrape API. Generate once, reuse thousands of times at lower cost.
JS Rendering & Anti-Bot
All extraction approaches include full headless browser rendering, automatic proxy rotation, stealth mode, and CAPTCHA solving. Handle JavaScript-heavy SPAs, infinite scroll pages, and bot-protected sites without extra configuration.
Powered By
Frequently Asked Questions
-
What is the Scrape API and how do extract_rules work?
The Scrape API fetches a page (rendering JavaScript when needed) and returns structured JSON. With the extract_rules parameter you map field names to CSS selectors - for example {"title":"h1","price":".price"} - and get back exactly those fields. Rules support nested objects, arrays, per-field output types (text, HTML, attributes), and pagination across list pages. Because it is selector-based, results are deterministic and repeatable, so the same rules return the same JSON on every run.
-
Should I write extract_rules by hand or generate them?
Hand-written extract_rules give you full control and are ideal once you know a site's HTML structure - production pipelines that need 100% deterministic results, maximum speed, and the lowest cost per request. When you are scaling across many new site types, Guess-er can generate CSS rules for you to review and reuse instead of writing them from scratch. Either way you run the same Scrape API at the same low cost per request, and the selectors return identical results every time with no model variability.
-
How does Guess-er work and when should I use it?
Guess-er analyzes a page's HTML structure and generates CSS extract_rules that you can reuse with the Scrape API. Provide a target JSON schema or an example of the fields you want, and it returns selectors. The key advantage is cost efficiency: you generate the rules once, then reuse them with the cheaper Scrape API for all subsequent requests to similar pages. Use Guess-er when you are scaling extraction across many pages with similar structures, when you want the reliability of CSS selectors without writing them manually, or when you need to quickly bootstrap extraction templates for new site types.
-
Can I validate the extracted data against a specific schema?
Yes. Guess-er accepts a standard JSON Schema when generating rules - type definitions, required fields, enums, and nested structures - so the selectors it returns target the fields your schema expects. The schema format supports $ref and $defs for Pydantic-compatible schemas, anyOf/oneOf/allOf combinators, and all standard JSON Schema types including nested objects and arrays. And because Scrape API output is deterministic JSON, you can validate every response against the same schema (or your Pydantic model) on your side.
-
How do you handle JavaScript-rendered and bot-protected pages?
The Scrape and Extract APIs include full headless browser rendering with Chromium. JavaScript is executed, SPAs are rendered, and dynamic content is loaded before extraction begins. For bot-protected sites, our system includes stealth mode with realistic browser fingerprints, automatic proxy rotation across residential and datacenter pools, and automatic CAPTCHA solving. You can configure wait_for to pause extraction until a specific CSS selector appears, or set timeouts for pages with complex loading sequences. Set js=true to render before extraction.
-
What output formats does structured extraction support?
Both endpoints return clean JSON by default with properly typed fields. The Scrape API with extract_rules returns data matching your defined rule hierarchy - nested objects, arrays, and per-field output types. The Extract API returns article fields with consistent naming. You can integrate results directly into databases, data warehouses, or downstream applications via our REST API. For batch processing, results can be delivered via webhooks or exported as CSV through our dashboard. All responses include metadata such as extraction time for monitoring and optimization.
Start Extracting Structured Data Today
Turn any web page into clean, structured JSON with a single API call. Reusable rules, deterministic results, minimal maintenance.