Beautiful Soup Python: Web Scraping Tutorial + When to Use an API

Beautiful Soup is a Python library that parses HTML and XML documents into a navigable tree, so you can search and extract data with a few lines of code. Imagine you need to pull product prices from a retailer's catalogue every morning: Beautiful Soup lets you point at the right HTML elements and read them out, without writing a parser from scratch. By the end of this guide you will know how to run a complete python web scraping beautifulsoup script, understand where the library hits its limits, and know exactly when to swap in a managed API instead.

What is Beautiful Soup in Python?

Beautiful Soup (imported as bs4) is a parsing library, not a scraping framework. It takes an HTML or XML string, builds a tree of Python objects that mirrors the document structure, and gives you methods to search and navigate that tree. It does not fetch pages, execute JavaScript, or manage proxies. Those jobs belong to other tools, and understanding that boundary is the key to using Beautiful Soup well.

The current release is Beautiful Soup 4 (package name beautifulsoup4 on PyPI, import name bs4). Python 2 support was dropped after version 4.9.3, so all new work targets Python 3.

What is Web Scraping?

Web scraping is the automated process of extracting data from websites. It's essentially like copying information from a webpage manually, but it's done programmatically, allowing for large amounts of data to be collected quickly and efficiently.

Web Scraping has two parts to it:

Sending HTTP GET Requests:

Using libraries like requests in Python, you can send a GET request to a website's server. The server then responds with the HTML content of the webpage.

HTML Parsing:

After retrieving the HTML content, you can use libraries like BeautifulSoup to parse the document. Parsing converts the HTML into a structured format that can be manipulated and used to extract specific data points.

Here is the exact 7-step flow you will use to go from a URL to clean JSON on disk.

Try it yourself

Run this on your own pages

Get a free API key and make your first call in minutes. 5,000 free credits for 7 days, no card.

Get a free API key or try the playground first →

Steps Involved in Web Scraping

Step 1: Identify Your Target

• Determine the website from which you want to scrape data.
• Check if it provides APIs(Application Programming Interface) to extract the information you need.
• Ensure the website allows web scraping by checking its terms of service and robots.txt file.

Step 2: Inspect the Website Structure

• Use a browser's developer tools (F12 or Ctrl + Shift + I) to view the website's HTML code.
• Understand how the data you want is organized on the webpage. If it is loaded dynamically through JavaScript, you would need to integrate Headless browser libraries like Playwright, Puppeteer, or Selenium.

Step 3: Crawl the website.

• Navigate through multiple web pages to find the data you need.
• Use a web crawler or a spider to traverse the website's pages.

Step 4: Download

• Download the HTML content of the web pages using a library like Requests in Python.

Step 5: Extract

• Extract the specific data points from the HTML content using parsing techniques.
• Use a library like Beautiful Soup in Python to parse the HTML code and pinpoint the data elements.

Step 6: Store

Save the extracted data into a structured format like:

• CSV (comma-separated values)
• JSON (JavaScript Object Notation)
• Database

These formats allow for easy import into spreadsheets or databases for further analysis.

Run the final script and you should find quotes.json on disk with every quote and author from the page. The next section covers the Beautiful Soup methods that make the extraction step more precise.

Why use Python for Web Scraping?

With requests and bs4, a 20-line script can fetch and parse a static page in under a second. That combination handles the two core jobs cleanly: requests sends the HTTP request and gives you the raw HTML; Beautiful Soup turns that HTML into a searchable tree. Add json from the standard library and you have a complete extract-and-store pipeline with no extra dependencies. When you need to go further, the ecosystem scales with you: Scrapy for crawling thousands of pages, Playwright or Selenium for JavaScript-heavy sites, and a managed scraping API when bot protection blocks plain requests.

Web scraping with Python using Beautiful Soup

Let's follow the above steps in web scraping using the BeautifulSoup and Requests library.

Before that, let's install the Beautiful Soup and Requests library using pip:

pip install beautifulsoup4 requests

Step 1: Identify Your Target Website/Websites

We will use https://scrape.li/quotes/ as a target website for this tutorial. It serves static HTML quotes with the same structure as the classic scraping sandbox, so the code runs reliably.

Step 2: Inspect the Web Page

• Open the web page you want to scrape in a browser.
• Inspect the HTML structure of the page using the browser's developer tools (F12 or Ctrl + Shift + I).
• Identify the HTML elements that contain the data you want to extract. In this tutorial, we will extract the quotes and their author names.

Step 3: Send an HTTP Request to the Web Page

• Use the Requests library to send an HTTP request to the web page:

import requests

url = 'https://scrape.li/quotes/'
response = requests.get(url)

Step 4: Parse the HTML Content

• Use Beautiful Soup to parse the HTML content of the web page:

from bs4 import BeautifulSoup

soup = BeautifulSoup(response.content, 'html.parser')

Step 5: Find the Desired HTML Elements

• Use Beautiful Soup's methods (e.g., find, find_all) to locate the HTML elements that contain the data you want to extract:

quotes = soup.find_all('div', {'class': 'quote'})

Step 6: Extract the Data

• Extract the data from the HTML elements using Beautiful Soup's methods

quotes_data = []

for quote in quotes:
    author = quote.find('small', {'class': 'author'}).text
    quote_text = quote.find('span', {'class': 'text'}).text
    quotes_data.append({
        'quote': quote_text,
        'author': author
    })

Step 7: Store the Data

• Store the extracted data in a format of your choice (e.g., CSV, JSON, database)

import json

with open('quotes.json', 'w') as f:
    json.dump(quotes_data, f, indent=4)

print("Data stored in quotes.json")

The final code looks like this:

import json
import requests
from bs4 import BeautifulSoup

url = 'https://scrape.li/quotes/'
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
quotes = soup.find_all('div', {'class': 'quote'})

quotes_data = []
for quote in quotes:
    author = quote.find('small', {'class': 'author'}).text
    quote_text = quote.find('span', {'class': 'text'}).text
    quotes_data.append({
        'quote': quote_text,
        'author': author
    })

with open('quotes.json', 'w') as f:
    json.dump(quotes_data, f, indent=4)

print("Data stored in quotes.json")

The output looks like this:

[
    {
        "quote": "“The world as we have created it is a process of our thinking.”",
        "author": "Albert Einstein"
    },
    {
        "quote": "“It is our choices, Harry, that show what we truly are, far more than our abilities.”",
        "author": "J.K. Rowling"
    },
    {
        "quote": "“There are only two ways to live your life. One is as though nothing is a miracle.”",
        "author": "Albert Einstein"
    }
]

If your file matches this structure, the scraper is working correctly. Each object holds exactly two keys, quote and author, which is the shape the rest of the pipeline expects.

Key Beautiful Soup methods every scraper uses

Once you have a soup object, three methods cover the vast majority of real scraping work.

• find(tag, attrs): Returns the first matching element, or None if nothing matches. Use it when you expect exactly one result: a page title, a price, a login form.
• find_all(tag, attrs): Returns a list of every matching element. Use it for repeating structures: product cards, table rows, search results. Iterate the list to extract each item.
• select(css_selector): Accepts a CSS selector string and returns a list, the same way find_all does. Useful when the element you want is easier to describe as .product-card > h2 than as a tag-plus-attribute pair.

A quick rule of thumb: use find for unique elements, find_all or select for collections. If find returns None when you expected a result, the element is either named differently than you think or it is loaded by JavaScript after the initial HTML response, which Beautiful Soup never sees.

Drawbacks and limitations of using BeautifulSoup

Beautiful Soup shines for static pages, but it stops at the first JavaScript request or bot check. The moment a site loads its prices via a React component, or Cloudflare intercepts your request, Beautiful Soup has nothing to work with because it only sees the HTML string you hand it. Here are the specific limits that matter when you scale.

BeautifulSoup is not suitable for Large-Scale Web Scraping because

1. It only parses the initial HTML response and does not execute JavaScript code. This means it cannot extract content that is generated dynamically by JavaScript.

2. Have built-in support for handling anti-scraping measures such as CAPTCHAs, rate limiting, or IP blocking.

3. It is not thread-safe, which means it's not suitable for use in multi-threaded or multi-process environments.

4. It can be slow, particularly when handling large documents or complex HTML structures. This is due to the need to parse the entire document and construct a tree-like data structure in memory. As a result, for large-scale web scraping involving numerous pages, BeautifulSoup can become both slow and memory-intensive

Beautiful Soup vs a managed scraping API: where each one fits

Beautiful Soup is a parser, not a scraping stack. It reads HTML you hand it and helps you find things inside it. Everything else, fetching the page, rotating proxies, solving CAPTCHAs, executing JavaScript, managing concurrency, is your problem. That distinction matters a lot once you move past a personal project.

Here is how the two approaches compare across the dimensions that cause the most pain at scale:

Dimension	Beautiful Soup (DIY)	Managed scraping API
JavaScript rendering	Not supported. You must add Playwright or Selenium separately, which adds complexity and resource cost.	Built in. Pass js=true and the API runs a real browser for you.
Anti-bot and CAPTCHA	None. You handle headers, fingerprinting, and CAPTCHA solving yourself.	Handled automatically. Residential proxy rotation, browser fingerprinting, and optional CAPTCHA solving are included.
Proxy management	You source, rotate, and pay for proxies separately.	Included in the request. Rotating residential or datacenter proxies are selected per request.
Concurrency	Limited by your machine and network. Scaling requires infrastructure work.	Handled by the API. Plans include defined concurrency limits so you do not have to manage a thread pool.
Maintenance	You own every breakage: site layout changes, new bot detection, IP bans.	The provider absorbs most breakage. Your parsing logic may still need updates, but the fetch layer is maintained for you.
Cost model	Low marginal cost per request, but real engineering time to build and maintain the stack.	Pay per successful request. Blocks, timeouts and server errors are not billed (a 404 is).

The right choice depends on what you are scraping. Beautiful Soup is genuinely excellent for:

• Static, permissive sites: public data sources, your own test pages, sites with no bot protection.
• One-off or low-volume tasks: a script you run once a week against a site that never changes.
• Learning and prototyping: understanding HTML structure before committing to a production pipeline.

A managed API earns its place when:

• The target renders content with JavaScript: single-page apps, infinite scroll, lazy-loaded prices.
• The site actively blocks scrapers: Cloudflare, Akamai, DataDome, or aggressive rate limiting.
• You need reliable uptime at scale: hundreds of concurrent requests without building your own proxy infrastructure.
• Engineering time is the bottleneck: maintaining a DIY stack costs more than the API fee.

Replacing the fetch layer: Beautiful Soup plus a scraping API

The two tools are not mutually exclusive. A common production pattern is to use a managed API for the fetch step (getting clean HTML reliably) and Beautiful Soup for the parse step (extracting the fields you need). This keeps your parsing logic familiar while offloading the hard parts.

Here is what that looks like with Ujeebu's Scrape API. The js=true parameter tells the API to render JavaScript before returning HTML, and proxy_type=rotating rotates IPs automatically:

curl -X GET 'https://api.ujeebu.com/scrape?url=https://scrape.li/quotes/&js=true&proxy_type=rotating' \
  -H "ApiKey: YOUR_API_KEY"

import requests
from bs4 import BeautifulSoup
import json

# Fetch the rendered HTML via Ujeebu
response = requests.get(
    'https://api.ujeebu.com/scrape',
    params={
        'url': 'https://scrape.li/quotes/',
        'js': True,
        'proxy_type': 'rotating',
    },
    headers={'ApiKey': 'YOUR_API_KEY'}
)

html = response.json()['html']

# Parse with Beautiful Soup as normal
soup = BeautifulSoup(html, 'html.parser')
quotes_data = []

for quote in soup.find_all('div', {'class': 'quote'}):
    quotes_data.append({
        'author': quote.find('small', {'class': 'author'}).text,
        'quote': quote.find('span', {'class': 'text'}).text,
    })

with open('quotes.json', 'w') as f:
    json.dump(quotes_data, f, indent=4)

print(f"Extracted {len(quotes_data)} quotes")

Your Beautiful Soup parsing code stays exactly as it was. The only change is where the HTML comes from: instead of a plain requests.get, you call the Scrape API, which handles JavaScript rendering, proxy rotation, and anti-bot evasion before handing you the HTML. For sites that block scrapers or load content dynamically, this single swap is often all you need. See the Scrape API docs for the full list of parameters, including wait_for (wait for a CSS selector before returning), auto_captcha_solve, and proxy_country.

If you want to skip HTML parsing entirely and get structured data directly, the Ujeebu Article Extract API extracts article content and the Markdown API converts any page to clean markdown, which is useful for feeding scraped content into an LLM pipeline. You can try both on the playground without writing any code.

By following these steps, you can set up a basic web scraping script to gather data from a target website. Libraries like Beautiful Soup make web scraping manageable and efficient. However, it's important to be aware of the limitations of using Beautiful Soup, especially when scaling up your projects. Challenges such as handling JavaScript-generated content, anti-scraping measures, and performance issues with large-scale scraping need to be addressed.

For those looking to scale their web scraping efforts without getting bogged down by the complexities of handling proxies, JavaScript execution, and other technical hurdles, consider using Ujeebu. Ujeebu offers a robust API that simplifies the web scraping process, managing all the behind-the-scenes difficulties for you. The free trial includes 5,000 credits with no credit card required, so you can test it against your actual target before committing. Start for free.

FAQ

What is Beautiful Soup in Python?

Beautiful Soup is a Python library for parsing HTML and XML documents. It takes raw HTML, builds a parse tree, and gives you methods like find() and find_all() to locate elements by tag, class, attribute, or CSS selector. It does not fetch pages or execute JavaScript: you pair it with requests for fetching and a headless browser or scraping API when JavaScript rendering is needed.

How do I install and import Beautiful Soup?

Install with pip: pip install beautifulsoup4. Then import it in your script with from bs4 import BeautifulSoup. Note that the package name on PyPI is beautifulsoup4, but the import name is bs4. You will also need pip install requests for the HTTP fetch step.

Can Beautiful Soup scrape JavaScript-rendered pages?

Not on its own. Beautiful Soup only parses the HTML string you give it. If a page loads its content via JavaScript after the initial response, the HTML you fetch with requests will be empty or incomplete. You need either a headless browser (Playwright, Selenium) or a scraping API with js=true to get the rendered HTML first, then pass that to Beautiful Soup.

What is the difference between find and find_all in Beautiful Soup?

find() returns the first matching element (or None if nothing matches). find_all() returns a list of all matching elements, which you then iterate over. Use find when you expect exactly one result (a page title, a price field), and find_all when you need every instance of a repeating element (all product cards, all table rows).

When should I use a scraping API instead of Beautiful Soup?

Use a scraping API when the site renders content with JavaScript, actively blocks scrapers with tools like Cloudflare or CAPTCHA challenges, or when you need to run many concurrent requests without building your own proxy infrastructure. Beautiful Soup remains the right choice for parsing the HTML once you have it, and the two tools work well together: the API handles the fetch, Beautiful Soup handles the parse. See the web scraping tools overview for a broader comparison of the ecosystem.

How do I scrape web data to JSON with Beautiful Soup?

Parse the page with Beautiful Soup, collect your extracted fields into a list of dictionaries, then write it out with Python's built-in json module: json.dump(data, f, indent=4). The complete example in the tutorial above does exactly this, saving each quote and author as a JSON object. If the target site blocks plain requests calls, fetch the HTML through a scraping API first and pass the returned HTML string to BeautifulSoup() as shown in the API integration example.

How long does it take to learn web scraping with Python?

The time it takes to learn web scraping with Python depends on several factors, such as:

• Your prior experience with Python and programming in general.
• The complexity of the web scraping tasks you want to accomplish.
• The amount of time you dedicate to learning and practicing.

However, with consistent effort, you can learn the basics of web scraping with Python in a few weeks to a few months. Here's a rough estimate of the time it may take to reach different levels of proficiency:

• Basic understanding of web scraping concepts and tools: 1-2 weeks
• Ability to scrape simple websites: 2-4 weeks
• Ability to scrape complex websites with JavaScript rendering: 2-6 months
• Mastery of advanced web scraping techniques and tools: 6-12 months

What are the benefits of using Beautiful Soup to scrape web data?

Beautiful Soup is a popular Python library for web scraping, and its benefits include:

• Easy to learn and use, even for those without extensive programming experience
• Supports various parsing engines, including lxml and html5lib
• Allows for easy navigation and searching of the parsed document tree
• Supports handling of broken or malformed HTML documents
• Extensive community support and documentation

What is the best way to web scrape in Python?

The best way to web scrape in Python follows a layered approach, where each layer handles one job:

• HTTP client (requests or Scrapy): Fetches the raw HTML. Start here; swap in a managed scraping API if the site blocks plain requests, as covered in the Scrape API section above.
• HTML parser (Beautiful Soup or lxml): Extracts the fields you need from the HTML string. Beautiful Soup is the readable choice; lxml is faster for very large documents.
• JavaScript rendering (Playwright, Selenium, or js=true on a scraping API): Required when the page loads content after the initial response. Add this layer only when you confirm the data is missing from the plain HTML.
• Anti-bot handling (proxy rotation, CAPTCHA solving): Needed for any site with bot protection. A managed API absorbs this layer entirely; a DIY stack requires sourcing and rotating proxies yourself.
• Storage (JSON, CSV, or a database): Write extracted data into a structured format immediately. JSON works for exploration; a database is better for anything you query repeatedly.
• Error handling and logging: Wrap each request in a try/except, log HTTP status codes, and retry on transient failures. Silent failures are the most expensive kind.
• Respect robots.txt and terms of service: Check both before you start. The web scraping legality guide covers what the rules actually mean in practice.

What is the purpose of BeautifulSoup in Python?

Beautiful Soup is a Python library used for web scraping, specifically for parsing and navigating HTML and XML documents. Its primary purpose is to:

• Parse HTML and XML documents into a tree-like data structure
• Allow for easy navigation and searching of the parsed document tree
• Extract specific data from the document, such as text, links, or images

What is the Python library for scraping data?

There are several Python libraries for web scraping, including:

• Beautiful Soup (BS4)
• Scrapy
• Selenium or Playwright (for JavaScript-heavy pages)
• Requests
• LXML
• PyQuery

Each library has its strengths and weaknesses, and the choice of the library depends on the specific web scraping task at hand.

Can Python scrape a website?

Yes, Python can be used to scrape websites. Python's versatility, combined with its extensive libraries and tools, make it an ideal language for web scraping. With Python, you can:

• Send HTTP requests to websites using libraries like requests or urllib
• Parse HTML and XML documents using libraries like Beautiful Soup or lxml
• Extract dynamic data from websites using libraries like Selenium or Playwright.
• Store scraped data in structured formats like CSV, JSON, or databases.

However, it's essential to respect website terms of service and robots.txt files and to avoid scraping websites that explicitly prohibit web scraping.

For a deeper look at Python scraping techniques beyond Beautiful Soup, see Mastering HTML Text Extraction in Python and How to Scrape Stock Market Data using Python.