How to scrape popular websites
One guide per site, built from our own health checks of public pages around the clock: whether requests get through right now, the proxy and rendering settings that worked, code to copy and the data the pages carry. Statuses change with every check.
Sites
110
with enough recent checks to publish
Working or partly working
48
over the last 7 days
Developer and reference 5
- GitHub Working Extraction recipe · github.com · 18/20 checks passed
- Hacker News Working Extraction recipe · ycombinator.com · 19/21 checks passed
- IMDb Intermittent Extraction recipe · imdb.com · 7/20 checks passed
- Wikipedia Working Extraction recipe · wikipedia.org · 18/20 checks passed
- Stack Overflow Currently blocked stackoverflow.com · 0/19 checks passed
E-commerce 16
- Amazon Partly working Extraction recipe · amazon.com · 11/22 checks passed
- Etsy Currently blocked etsy.com · 1/21 checks passed
- Walmart Currently blocked walmart.com · 1/23 checks passed
- Best Buy Partly working bestbuy.com · 13/21 checks passed
- Costco Currently blocked costco.com · 0/15 checks passed
- Target Intermittent Extraction recipe · target.com · 7/23 checks passed
- eBay Intermittent ebay.com · 3/23 checks passed
- AliExpress Partly working Extraction recipe · aliexpress.com · 14/21 checks passed
- Flipkart Partly working Extraction recipe · flipkart.com · 14/21 checks passed
- Home Depot Currently blocked homedepot.com · 0/21 checks passed
- Lowe's Currently blocked lowes.com · 0/21 checks passed
- Mercado Libre Currently blocked mercadolibre.com.mx · 0/21 checks passed
- Newegg Partly working Extraction recipe · newegg.com · 13/21 checks passed
- Noon Currently blocked noon.com · 0/21 checks passed
- Temu Partly working temu.com · 12/21 checks passed
- Wayfair Currently blocked wayfair.com · 1/21 checks passed
Food delivery 7
- Deliveroo Partly working deliveroo.co.uk · 12/21 checks passed
- DoorDash Intermittent doordash.com · 4/20 checks passed
- Glovo Currently blocked glovoapp.com · 0/10 checks passed
- Grubhub Partly working Extraction recipe · grubhub.com · 12/20 checks passed
- Just Eat Currently blocked just-eat.co.uk · 0/20 checks passed
- Talabat Partly working Extraction recipe · talabat.com · 13/20 checks passed
- Uber Eats Intermittent Extraction recipe · ubereats.com · 4/20 checks passed
Freelance and agencies 7
- Clutch Currently blocked clutch.co · 1/21 checks passed
- Fiverr Intermittent fiverr.com · 7/21 checks passed
- Freelancer Intermittent Extraction recipe · freelancer.com · 3/21 checks passed
- Houzz Intermittent houzz.com · 7/21 checks passed
- PeoplePerHour Intermittent Extraction recipe · peopleperhour.com · 4/21 checks passed
- Toptal Intermittent toptal.com · 7/19 checks passed
- Upwork Intermittent upwork.com · 4/20 checks passed
Jobs 12
- Indeed Currently blocked indeed.com · 1/21 checks passed
- Ashby Working ashbyhq.com · 19/21 checks passed
- Dice Intermittent Extraction recipe · dice.com · 7/23 checks passed
- Greenhouse Working Extraction recipe · greenhouse.io · 20/21 checks passed
- Lever Working Extraction recipe · lever.co · 17/19 checks passed
- Workday Partly working Extraction recipe · myworkdayjobs.com · 13/22 checks passed
- BambooHR Partly working bamboohr.com · 14/21 checks passed
- Glassdoor Currently blocked glassdoor.com · 0/20 checks passed
- Remote OK Working Extraction recipe · remoteok.com · 18/21 checks passed
- SmartRecruiters Working Extraction recipe · smartrecruiters.com · 18/21 checks passed
- Wellfound Currently blocked wellfound.com · 2/21 checks passed
- ZipRecruiter Currently blocked ziprecruiter.com · 0/20 checks passed
Local and reviews 6
- TripAdvisor Currently blocked tripadvisor.com · 1/21 checks passed
- Yelp Currently blocked yelp.com · 0/21 checks passed
- Foursquare Behind a login wall foursquare.com · 0/16 checks passed
- OpenTable Currently blocked opentable.com · 0/21 checks passed
- Yellow Pages Currently blocked yellowpages.com · 0/20 checks passed
- Zomato Currently blocked zomato.com · 0/22 checks passed
Marketplaces and classifieds 6
- Avito Intermittent Extraction recipe · avito.ma · 14/42 checks passed
- Craigslist Working Extraction recipe · craigslist.org · 19/21 checks passed
- Gumtree Partly working Extraction recipe · gumtree.com · 12/20 checks passed
- OLX Currently blocked olx.pl · 0/20 checks passed
- OfferUp Intermittent Extraction recipe · offerup.com · 5/20 checks passed
- Vinted Partly working Extraction recipe · vinted.com · 14/21 checks passed
News and finance 20
- AFP Fact Check Partly working Extraction recipe · afp.com · 9/20 checks passed
- BBC News Working Extraction recipe · bbc.com · 18/20 checks passed
- CNBC Working Extraction recipe · cnbc.com · 18/20 checks passed
- GlobeNewswire Intermittent Extraction recipe · globenewswire.com · 7/20 checks passed
- Medium Currently blocked medium.com · 0/19 checks passed
- Nature Partly working Extraction recipe · nature.com · 10/20 checks passed
- PR Newswire Working Extraction recipe · prnewswire.com · 18/20 checks passed
- TechRadar Partly working Extraction recipe · techradar.com · 12/20 checks passed
- The Verge Partly working Extraction recipe · theverge.com · 13/20 checks passed
- TradingView Working Extraction recipe · tradingview.com · 18/20 checks passed
- Google News Partly working news.google.com · 9/21 checks passed
- Yahoo Finance Partly working Extraction recipe · finance.yahoo.com · 15/22 checks passed
- CNN Intermittent Extraction recipe · cnn.com · 5/20 checks passed
- CoinMarketCap Working Extraction recipe · coinmarketcap.com · 18/20 checks passed
- Morningstar Intermittent Extraction recipe · morningstar.com · 6/20 checks passed
- New York Post Partly working Extraction recipe · nypost.com · 12/20 checks passed
- Reuters Currently blocked reuters.com · 1/20 checks passed
- Substack Working Extraction recipe · substack.com · 18/20 checks passed
- The Guardian Working Extraction recipe · theguardian.com · 17/19 checks passed
- The Motley Fool Working Extraction recipe · fool.com · 18/20 checks passed
Real estate 11
- Zillow Intermittent Extraction recipe · zillow.com · 8/22 checks passed
- Bayut Currently blocked bayut.com · 0/21 checks passed
- Homes.com Currently blocked homes.com · 0/20 checks passed
- Idealista Currently blocked idealista.com · 0/21 checks passed
- Immobiliare.it Currently blocked immobiliare.it · 0/21 checks passed
- Immowelt Currently blocked Extraction recipe · immowelt.de · 2/21 checks passed
- Mubawab Partly working Extraction recipe · mubawab.ma · 14/21 checks passed
- Realtor.com Currently blocked realtor.com · 0/20 checks passed
- Redfin Intermittent Extraction recipe · redfin.com · 6/21 checks passed
- Rightmove Partly working rightmove.co.uk · 9/21 checks passed
- SeLoger Currently blocked seloger.com · 0/21 checks passed
Software reviews and apps 6
- Google Play Working Extraction recipe · play.google.com · 19/22 checks passed
- Trustpilot Currently blocked trustpilot.com · 0/21 checks passed
- Apple App Store Working Extraction recipe · apps.apple.com · 18/22 checks passed
- Capterra Currently blocked capterra.com · 0/20 checks passed
- G2 Currently blocked g2.com · 0/20 checks passed
- Product Hunt Currently blocked producthunt.com · 0/20 checks passed
Travel 7
- Airbnb Partly working airbnb.com · 14/22 checks passed
- Booking.com Intermittent Extraction recipe · booking.com · 8/21 checks passed
- Agoda Partly working Extraction recipe · agoda.com · 14/21 checks passed
- Expedia Currently blocked expedia.com · 0/20 checks passed
- Hotels.com Currently blocked hotels.com · 0/20 checks passed
- Trip.com Partly working trip.com · 12/21 checks passed
- Vrbo Currently blocked vrbo.com · 0/20 checks passed
No site matches. Ask us for a guide.
Working: at least 80% of checks passed in 7 days. Partly working: 40% to 80%. Intermittent: 10% to 40%. Currently blocked: under 10%. Each check uses our own API key on public pages and climbs from the cheapest proxy to the most expensive one.
Social and video 7