Amazon web scraping is the practice of programmatically extracting product data, prices, and reviews from Amazon's pages, and the reason most scrapers fail is not a network error but a silent substitution: Amazon's WAF returns a bot-check page with HTTP 200, your parser finds none of its selectors, and your output CSV fills with empty fields. You have seen this: the scraper worked fine in testing, you ran it against a few hundred product pages, and the data came back blank. No exception, no 4xx, nothing in the logs. By the end of this guide you will know exactly what triggers that substitution, how to detect it, and how to scrape Amazon data reliably without maintaining your own proxy infrastructure.
Why Amazon web scraping is harder than it looks
On Amazon, a bot-check or CAPTCHA page returns HTTP 200. Your HTTP client sees a successful response, your parser runs, and because none of your CSS selectors match the bot-check page's markup, every field comes back empty. No error is raised. The scraper finishes, writes the CSV, and you only notice the problem when you open the file.
This is the core difficulty of Amazon web scraping. It is not a firewall that rejects you outright; it is a substitution that serves fake-looking success. As hasdata.com's guide to scraping Amazon puts it: "Your parser runs, finds none of its selectors, and writes empty fields without a single error in the log."
The practical fix is to treat a missing title field as a hard error, not a missing value. Before you parse anything, check that the page you received actually contains product data. If the element you expect is absent, raise an exception and retry rather than writing a blank row.
Takeaway: Add a guard at the top of your parser that checks for a known product selector. An empty title is a signal to retry, not a value to record.
What Amazon checks before it serves you a page
Amazon's detection works in layers, and each layer can independently trigger the substitution. Understanding the mechanism tells you why each mitigation works.
Request headers. A bare requests.get() sends a python-requests/2.x user agent and almost nothing else. Switching to a browser user agent helps on low-traffic pages, but it fails as soon as the rest of the headers disagree with it. A Chrome user agent arriving without Accept-Language, without Sec-Fetch-* headers, and without the Accept string Chrome actually sends is a combination no real browser ever produces. Amazon's WAF recognizes that combination immediately.
IP concentration. Sending many requests from the same IP address is a reliable detection signal. Rotating proxies spread requests across multiple IPs to avoid that concentration, so no single address accumulates enough requests to trigger a block. The goal is to look like many different visitors rather than one automated client making thousands of calls.
JavaScript rendering. Parts of an Amazon product page are rendered after the initial HTML arrives, so even a request that is served correctly can come back missing fields because the JavaScript that populates them never ran. You need a real browser engine to execute the page and return the fully rendered DOM.
The combination problem is what makes Amazon hard at scale. Each layer is addressable individually, but all three must be handled together for reliable results. A scraper with realistic headers but a concentrated IP still gets flagged. A scraper with rotating proxies but no JS rendering still misses dynamically loaded fields.
Takeaway: Address all three layers from the start: realistic headers, proxy rotation, and JavaScript rendering. Fixing one and ignoring the others produces inconsistent results that get worse as volume grows.
What data you can scrape from Amazon
Before writing a selector, it helps to know what is actually extractable. Amazon product pages expose a well-defined set of fields that both prebuilt scrapers and custom parsers target.
- Product identity: ASIN, product name, brand, and product URL.
- Pricing: current price, original price, discount amount, and currency.
- Availability and seller: stock status and seller name.
- Social proof: star rating and review count.
- Content: product description, bullet-point features, and product specifications.
- Images: main product image and additional image URLs.
Search and listing pages typically expose a subset of these fields in product cards, which you can follow to the full product page for complete details. These cards are the starting point for catalogue extraction and competitor monitoring: collect ASINs from search results, then fetch each product page individually.
One practical constraint worth knowing: Amazon sometimes restricts search result pagination to 7 or 20 pages for certain queries, which caps the number of product URLs you can reach from a single search at around 306 results. If you need more than that, run multiple searches with different keywords or category filters rather than paginating a single query indefinitely.
Takeaway: Decide upfront whether you need product pages, listing pages, or both. They have different URL structures, different selectors, and different detection profiles. A clear field list prevents scope creep mid-build.
Scraping an Amazon product page with the Ujeebu Scrape API
The fastest way to handle all three detection layers without building your own infrastructure is to route requests through a scraping API that manages headers, proxies, and JS rendering for you. Your code then focuses entirely on parsing.
Here is a minimal example using Ujeebu's Scrape API. The key parameters are js=true to render the page with a real browser engine, proxy_type=rotating to distribute requests across multiple IPs, proxy_country=US to match the Amazon locale, and auto_captcha_solve=true to handle CAPTCHA pages automatically.
curl -X GET \
'https://api.ujeebu.com/scrape?url=https://scrape.li/quotes/&js=true&proxy_type=rotating&proxy_country=US&auto_captcha_solve=true' \
-H "ApiKey: YOUR_API_KEY"
import requests
from bs4 import BeautifulSoup
API_KEY = "YOUR_API_KEY"
TARGET_URL = "https://scrape.li/quotes/" # replace with your target Amazon product URL
response = requests.get(
"https://api.ujeebu.com/scrape",
params={
"url": TARGET_URL,
"js": True,
"proxy_type": "rotating",
"proxy_country": "US",
"auto_captcha_solve": True,
},
headers={"ApiKey": API_KEY},
)
html = response.text
soup = BeautifulSoup(html, "html.parser")
# Guard: check the page is a real product page, not a bot-check page
title_el = soup.select_one("#productTitle")
if not title_el:
raise RuntimeError("Bot-check page detected: no product title found. Retry this request.")
title = title_el.get_text(strip=True)
price_el = soup.select_one(".a-price .a-offscreen")
price = price_el.get_text(strip=True) if price_el else None
rating_el = soup.select_one("span[data-hook='rating-out-of-text']")
rating = rating_el.get_text(strip=True) if rating_el else None
print({"title": title, "price": price, "rating": rating})
The guard on #productTitle is the key addition. If Amazon served a bot-check page, that selector returns nothing and the script raises an error immediately rather than writing blank fields. Wrap this in a retry loop with a short delay between attempts.
Each JS-rendered scrape costs 5 credits. On the Starter plan (300,000 credits for $49 a month), that works out to 60,000 JS-rendered product pages per month. Blocked, timed-out and server-error requests are not billed, so you pay for pages that come back (plus any 404 Not Found responses).
Takeaway: Set js=true and proxy_type=rotating from the first request, even if a plain request works in local testing. Amazon's detection is inconsistent at low volume and aggressive once you cross a few hundred requests from the same IP.
Paginating through Amazon search results
Listing pages are the more common starting point for price monitoring and catalogue extraction. You fetch a search results page, extract the product cards, then follow each product URL to the detail page.
When paginating, use proxy_session to keep the same IP across all pages in a sequence. Without it, each page request arrives from a different IP, which looks like a different visitor arriving at exactly the same search query in rapid succession. With a session, the requests look like one person clicking through pages.
import requests
from bs4 import BeautifulSoup
API_KEY = "YOUR_API_KEY"
SESSION_ID = "amzsession01" # alphanumeric, reused across pages in this crawl
def scrape_listing_page(page_url: str) -> list[dict]:
response = requests.get(
"https://api.ujeebu.com/scrape",
params={
"url": page_url,
"js": True,
"proxy_type": "rotating",
"proxy_country": "US",
"auto_captcha_solve": True,
"proxy_session": SESSION_ID,
},
headers={"ApiKey": API_KEY},
)
soup = BeautifulSoup(response.text, "html.parser")
# Guard: a real results page has product cards
cards = soup.select("[data-component-type='s-search-result']")
if not cards:
raise RuntimeError(f"No product cards found on {page_url}. Possible bot-check page.")
products = []
for card in cards:
asin = card.get("data-asin")
title_el = card.select_one("h2 a span")
price_el = card.select_one(".a-price .a-offscreen")
products.append({
"asin": asin,
"title": title_el.get_text(strip=True) if title_el else None,
"price": price_el.get_text(strip=True) if price_el else None,
})
return products
# Example: paginate up to 5 pages
base_url = "https://www.amazon.com/s?k=wireless+headphones&page={}"
all_products = []
for page_num in range(1, 6):
page_products = scrape_listing_page(base_url.format(page_num))
if not page_products:
break
all_products.extend(page_products)
print(f"Page {page_num}: {len(page_products)} products")
Stop paginating when the cards list is empty or when you reach Amazon's pagination cap. Trying to fetch pages past that cap can return a bot-check page or an empty results page, and your guard will catch both.
Takeaway: Use proxy_session when paginating so each page in a sequence shares the same IP. It makes the traffic pattern look like a human browsing rather than a new visitor on every request.
Handling CAPTCHAs and bot detection at scale
At low volume, the parameters above are usually enough. As volume grows, CAPTCHAs appear more frequently, and the right response is to slow down rather than speed up.
auto_captcha_solve=true delegates solving to the API. The auto_captcha_solve_timeout parameter controls how long the API waits before giving up, defaulting to 120 seconds. For high-volume runs, that timeout is worth keeping at its default: a solved CAPTCHA costs one retry, but a failed request costs nothing because Ujeebu does not bill for failed requests.
You can also set auto_proxy=true if you prefer not to specify proxy settings explicitly. Adding a short delay between requests can reduce blocks in practice. Maximizing request speed is the fastest way to trigger a ban. The goal is throughput over time, not requests per second.
Takeaway: For sustained high-volume runs, combine auto_captcha_solve=true with a delay between requests. Speed is the enemy of reliability on Amazon.
Choosing the right approach to scrape Amazon data
Three broad approaches exist, and the right one depends on how much engineering time you want to spend on infrastructure versus parsing.
- Prebuilt scrapers like WebScraper.io's Amazon listings scraper can be a fit for non-technical users and one-off extractions. They handle navigation and export to CSV or JSON without writing code. The trade-off is limited flexibility: when Amazon changes its layout or you need a custom field, you are waiting on the tool vendor.
- DIY with requests and BeautifulSoup is fast to start. The ongoing cost is maintaining header sets, proxy pools, and CAPTCHA handling yourself, which becomes a significant engineering burden as volume grows and Amazon's detection evolves.
- Scraping APIs handle the infrastructure layer so your code only does parsing. This is the right choice when you need reliability at scale without building your own stack.
Other scraping APIs exist; compare pricing and credit models against your expected volume before committing. The key question is what counts as a failed request: some providers bill for attempts that return a bot-check page, which adds up quickly on a site like Amazon.
For a deeper walkthrough of the full extraction pipeline, the step-by-step guide to scraping Amazon product data covers field-by-field parsing in more detail. The tutorials on Amazon product scraping and Amazon reviews scraping show complete end-to-end examples. If you want to understand the proxy and IP classification mechanics behind Amazon's detection in more depth, overcoming web scraping blocks goes further into how servers classify IP addresses and what that means for your strategy. For the legal picture, see our post on web scraping legality.
The free trial includes 5,000 credits with no credit card required. At 5 credits per JS-rendered request, that is 1,000 product pages to test your parser before committing to a plan.
FAQ
Does Amazon block web scrapers?
Yes, but not with a simple error response. Amazon's WAF serves bot-check or CAPTCHA pages that return HTTP 200, so your scraper appears to succeed while actually receiving no product data. The reliable fix is to check for a known product selector before parsing and treat its absence as a hard error requiring a retry.
What data can I scrape from an Amazon product page?
A product page exposes: ASIN, title, brand, price, original price, discount, currency, availability, star rating, review count, description, product specifications, seller name, main image, and additional images. Search and listing pages typically expose a subset of these fields in product cards, which you follow to the full product page for complete details.
Do I need JavaScript rendering to scrape Amazon?
For many pages, yes. Parts of an Amazon product page are rendered after the initial HTML arrives, which means a plain HTTP request can return the page missing fields even when it is not blocked. Setting js=true in the Scrape API ensures a real browser engine executes the page before returning the HTML.
How do I handle pagination when scraping Amazon search results?
Increment the page parameter in the search URL and stop when the product cards list is empty. Amazon sometimes caps search result pagination at 7 or 20 pages for certain queries, so do not assume you can paginate indefinitely. Use proxy_session to keep the same IP across all pages in a sequence.
Is it legal to scrape Amazon product data?
The legal picture for publicly visible data is more nuanced than a terms of service clause alone. For a thorough treatment of the relevant case law and considerations, see our dedicated post on web scraping legality. For commercial-scale use cases, consult a lawyer.