Your Scrapy spider crawled ten thousand pages yesterday. Today it returns 403s on the first request. That is not a Scrapy bug, it is the point where Scrapy's design ends and a scraping API's design begins. By the end of this post you will know exactly which wall you have hit, whether to route around it with an API, and how to wire the two together in a working spider without rewriting your existing logic.
What Scrapy is
Scrapy is a free and open-source web-crawling framework written in Python, originally designed for web scraping but also usable for extracting data via APIs or as a general-purpose web crawler. It is licensed under the BSD License and was first released publicly in August 2008, developed at Mydeco, a London-based web-aggregation and e-commerce company, with contributions from employees of Insophia, a web-consulting firm based in Montevideo, Uruguay. Zyte, formerly called Scrapinghub, became the official maintainer in 2011 and continues to maintain it today. The stable release as of September 2026 is version 2.19.0, and the project requires Python 3.11 or newer.
The framework follows the don't-repeat-yourself principle, in the spirit of Django, which means common crawling patterns are handled by the framework so you write only the logic specific to your target. That design makes it straightforward to build and scale large crawling projects while reusing code across spiders and projects. Real-world users include Lyst, Parse.ly, Sayone Technologies, Sciences Po Medialab, and the UK government's World Government Data site.
That distinction between framework and library matters. BeautifulSoup and lxml are libraries for parsing HTML and XML; comparing them to Scrapy is like comparing a template engine to a full web framework. BeautifulSoup parses a string of HTML you hand it. Scrapy manages the entire pipeline: the request queue, concurrency, retries, middleware, and output. You can use BeautifulSoup inside a Scrapy spider by feeding response.body into a BeautifulSoup object inside your callback, because it is just a parsing library importable from any Python code.
BeautifulSoup is widely used for its simplicity, but it is a parser, not a crawler. It has no concept of following links, managing concurrency, or retrying failed requests. When you need to crawl an entire domain and store structured results, you need a framework, not a parser.
Scrapy runs on Windows, macOS, and Linux, which means your spider code is portable across development and production environments without modification.
How Scrapy's architecture works
The core unit in Scrapy is the spider class. Spiders are self-contained crawlers: you define start URLs, link-following rules, and extraction callbacks, and Scrapy's engine drives the rest. Each spider receives a set of instructions and processes responses as they arrive asynchronously. The built-in generic spiders, CrawlSpider and SitemapSpider, cover the most common crawl patterns so you do not have to write link-following logic from scratch.
Scrapy's Downloader Middleware sits between the spider and the network, which makes it the natural place to inject proxy rotation or custom headers. The built-in selectors wrap the parsel library, which uses lxml under the hood, so response.xpath() and response.css() are fast and accurate. Both are shortcuts on the Response object; response.selector.css() is the longer form, but the shortcuts are more convenient and parse the body only once.
Items and Item Pipelines let you clean, validate, and route data before it lands in storage. Scrapy can export scraped items to JSON, CSV, or XML, which makes it straightforward to feed downstream systems without writing custom serialization code. Version 1.0, released in June 2015, marked the point where the framework's architecture stabilised into the form most teams use today.
What Scrapy does not include is a headless browser or a JavaScript execution engine. It sends HTTP requests and parses the responses the server returns. If the server returns a skeleton that a browser would fill in with JavaScript, Scrapy gets the skeleton.
Takeaway: Scrapy is the right tool for crawling logic, link discovery, deduplication, and pipeline management. It is not the right tool for executing JavaScript or evading bot detection on its own.
Try it yourself
Run this on your own pages
Get a free API key and make your first call in minutes. 5,000 free credits for 7 days, no card.
Where Scrapy hits walls
Three failure modes account for most production breakdowns.
JavaScript-rendered content. Many modern sites build their DOM in the browser. Scrapy fetches what the server sends over the wire. If the data you need is injected by a React or Vue application after page load, response.css("div.product-price") returns nothing because that element does not exist in the raw HTML. Scrapy alone cannot execute JavaScript, so the fix requires either adding a headless browser layer to your infrastructure or routing requests through a scraping API that returns fully rendered HTML.
Throttling and non-standard status codes. Some sites use status code 999 to signal throttling rather than returning a standard 429. Non-standard codes like this require custom handling in your retry logic, and retrying from the same IP address is exactly what the site is blocking. The right response is a different exit IP, not just another attempt.
Proxy management. Scrapy supports HTTP proxies and SOCKS proxies through its middleware system, but it does not provide a managed rotating pool. A rotating proxy pool works like a call center rotating its outgoing lines: each call appears to come from a different number, so the recipient cannot block a single source. The problem is that maintaining that pool, buying residential bandwidth, detecting dead routes, refreshing sessions, and handling per-domain rate limits is a separate engineering project that has nothing to do with your spider logic.
The gap is not just convenience. Self-managed middleware gives you retry counts. A managed API gives you sticky sessions across multi-page flows, geo-targeted exit IPs, and automatic proxy selection based on the target. You send one request; the API handles the pool.
User-agent and header rotation. Sites fingerprint scrapers partly through request headers. Rotating user-agents and setting realistic browser headers is a standard technique for reducing detection. Scrapy lets you set a custom User-Agent via settings or per-request, but maintaining a realistic, up-to-date set of browser fingerprints is ongoing work. A scraping API handles this at the infrastructure level.
Takeaway: If you see consistent 403s, 429s, or non-standard throttling codes after a few hundred requests, the problem is IP reputation or content rendering, not your spider logic. Fixing it inside Scrapy means building what a scraping API already provides.
Using the Ujeebu Scrapy plugin
The fastest way to connect Scrapy to Ujeebu is the official ujeebu-scrapy plugin on PyPI. It ships as a pre-built DownloaderMiddleware so you do not need to write the integration yourself. Install it, add two lines to settings.py, and every spider in your project can opt into API-backed rendering and proxy management on a per-request basis.
Installation
pip install ujeebu-scrapy
Configuration in settings.py
Add the middleware to your project's DOWNLOADER_MIDDLEWARES dict and set your API key:
# settings.py
DOWNLOADER_MIDDLEWARES = {
"ujeebu_scrapy.UjeebuMiddleware": 543,
}
UJEEBU_API_KEY = "YOUR_API_KEY"
The priority value 543 places the middleware after Scrapy's built-in retry and redirect middlewares, which is the correct position for a downloader that replaces the network layer.
Writing a spider that uses the plugin
Once the middleware is active, you control its behaviour through request.meta flags. A request with no meta flags passes through the middleware and hits the Ujeebu /scrape endpoint with plain-HTML settings. Add ujeebu_js=True for JavaScript rendering, ujeebu_auto_proxy=True for automatic proxy selection on bot-protected targets, or ujeebu_wait_for to wait for a CSS selector before the page is returned.
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = ["https://scrape.li/quotes/"]
def start_requests(self):
for url in self.start_urls:
yield scrapy.Request(
url,
meta={
"ujeebu_js": True, # render JavaScript
"ujeebu_wait_for": "div.quote", # wait for quotes to appear
"ujeebu_auto_proxy": True, # let the API pick the right proxy
},
)
def parse(self, response):
for quote in response.css("div.quote"):
yield {
"text": quote.css("span.text::text").get(),
"author": quote.css("small.author::text").get(),
"tags": quote.css("a.tag::text").getall(),
}
next_page = response.css("li.next a::attr(href)").get()
if next_page:
yield response.follow(next_page, self.parse)
Run it with the standard Scrapy command:
scrapy crawl quotes -o quotes.json
Every callback, pipeline, and feed exporter works exactly as before. The plugin is invisible to the rest of the stack: your spider code does not change, and you pay API credits only for the requests the middleware intercepts.
Checking your credit balance mid-crawl
To see how many credits a crawl has consumed, query the Account API:
curl -X GET 'https://api.ujeebu.com/account' \
-H "ApiKey: YOUR_API_KEY"
The response includes used, quota, and used_percent fields, giving you a clear picture of consumption without instrumenting individual requests.
Takeaway: The plugin is the lowest-friction path. Install it, set the API key, and add meta flags to the requests that need rendering or proxy management. Everything else stays the same.
How to route Scrapy requests through Ujeebu's Scrape API manually
If you prefer to write the middleware yourself, or need behaviour the plugin does not expose, the pattern is straightforward: a custom DownloaderMiddleware intercepts each request, forwards the URL to the Ujeebu /scrape endpoint, and returns the HTML body as a Scrapy HtmlResponse. Every spider callback, selector, and pipeline works unchanged.
Here is a plain HTML request to see what the API returns:
curl -X GET 'https://api.ujeebu.com/scrape?url=https://scrape.li/quotes/' \
-H "ApiKey: YOUR_API_KEY"
And here is the middleware class that plugs into any Scrapy project:
import requests
from scrapy.http import HtmlResponse
class UjeebuMiddleware:
API_URL = "https://api.ujeebu.com/scrape"
API_KEY = "YOUR_API_KEY"
def process_request(self, request, spider):
params = {"url": request.url}
# Enable JS rendering for pages that need it
if request.meta.get("ujeebu_js"):
params["js"] = True
selector = request.meta.get("ujeebu_wait_for")
if selector:
params["wait_for"] = selector
# Let the API manage proxy selection for bot-protected targets
if request.meta.get("ujeebu_auto_proxy"):
params["auto_proxy"] = True
resp = requests.get(
self.API_URL,
params=params,
headers={"ApiKey": self.API_KEY},
timeout=90,
)
resp.raise_for_status()
return HtmlResponse(
url=request.url,
body=resp.text,
encoding="utf-8",
request=request,
)
Enable it in settings.py by adding it to DOWNLOADER_MIDDLEWARES with a priority value. For a page that builds its content in the browser, pass ujeebu_js=True in the request meta:
curl -X GET 'https://api.ujeebu.com/scrape?url=https://scrape.li/quotes/&js=true&wait_for=div.quote' \
-H "ApiKey: YOUR_API_KEY"
For a site that blocks datacenter traffic, set auto_proxy=true. You can also pin a session across paginated requests using proxy_session with an alphanumeric identifier, which keeps the same exit IP for the duration of that session. For geo-restricted content, proxy_country lets you specify the exit country.
Takeaway: The manual middleware gives you full control over every parameter. Use it when you need behaviour the plugin does not expose, or when you want to keep the dependency count low.
A practical parameter map: Scrapy intent to Ujeebu options
When you decide a request needs API backing, the choice of parameters depends on what is blocking you. This table maps the common Scrapy failure modes to the relevant API options.
| Problem | Ujeebu parameter | Notes |
|---|---|---|
| JS-rendered content | js=true |
Runs a headless browser before returning HTML |
| Wait for a specific element | wait_for=<css selector> |
Useful for lazy-loaded content |
| Datacenter IP blocked | auto_proxy=true |
API selects proxy type for the target |
| Geo-restricted content | proxy_country=<code> |
Two-letter country code |
| Multi-step login or pagination | proxy_session=<id> |
Sticky session across requests |
| Realistic browser fingerprint | useragent=<string> |
Override the default user-agent |
| CAPTCHA on target | auto_captcha_solve=true |
Automated solving, billed on success |
You do not need all of these on every request. The ujeebu_js meta flag in the middleware above lets you opt individual requests into the more expensive rendering path while leaving the rest at the cheaper plain-HTML rate.
Performance and cost: self-hosted Scrapy vs managed API at scale
Self-hosted Scrapy has real costs that are easy to undercount: server or container compute, residential proxy bandwidth billed per gigabyte, a headless browser fleet for JS rendering, and the engineering time to keep all of it running. The proxy bandwidth cost dominates at scale and is hard to predict because it depends on page size, not request count.
Managed API costs are predictable because they are per successful request. Failed requests are not billed. The trade-off is less control over the exact exit IP and request timing.
On Ujeebu's Starter plan, a plain HTML scrape costs 1 credit per request and a JavaScript-rendered scrape costs 5 credits. That works out to $0.16 per 1,000 plain requests and $0.82 per 1,000 JS-rendered requests at the Starter tier, falling to $0.07 and $0.33 respectively at the Business tier. The free trial gives you 5,000 credits with no credit card required, which is enough to run a meaningful test crawl against your actual targets.
The practical implication: if most of your pages are plain HTML and only a subset need a browser, a tiered credit model is cheaper than a flat model that charges the same for both. Multiply your request mix by the credits per request shown on the pricing page to estimate cost before choosing a plan.
The build-vs-buy decision comes down to whether your proxy bandwidth and infrastructure costs, plus engineering maintenance time, exceed the API credit cost for your request volume. For most teams running more than a few thousand requests a day against bot-protected or JS-heavy targets, the API wins on total cost of ownership.
Takeaway: Estimate your credit cost from your actual request mix before committing to a plan. The tiered credit model rewards workloads where most pages are plain HTML.
When to keep Scrapy, when to replace it, and when to combine both
| Scenario | Scrapy alone | Scrapy + API | API only |
|---|---|---|---|
| Plain HTML at scale | Good fit | Works, adds cost | Overkill |
| JS-heavy SPA | Poor fit | Good fit | Good fit |
| Bot-protected site | Poor fit | Good fit | Good fit |
| Mixed (some JS, some plain) | Poor fit | Best fit | Expensive |
| Quick one-off extraction | Works with effort | Works | Good fit |
Keep Scrapy alone when your targets serve plain HTML, have no meaningful bot detection, and you need fine-grained control over crawl order, link following, and pipeline logic. Scrapy's asynchronous architecture handles high concurrency efficiently with low overhead. Our guide to building a crawler with Scrapy covers the setup in detail.
Replace Scrapy with an API or managed crawler when the target is a single-page application, you have no existing spider logic to preserve, and you want to reach data quickly. Ujeebu's Markdown API is a good fit when the output feeds an LLM pipeline rather than a structured database, because it returns clean, structured text rather than raw HTML.
Combine both for production crawls with mixed content. Keep Scrapy for crawl orchestration, link discovery, deduplication, and pipeline logic. Route individual requests through the Scrape API only for pages that need JS rendering or hit bot detection. You pay JS-rendering credits only where you need them, and your spider logic stays portable. This is exactly what the plugin and the middleware above implement: the ujeebu_js meta flag lets you opt individual requests into the rendering path while leaving the rest at the plain-HTML rate.
The combination pattern also keeps your data pipeline intact. Scrapy's Item Pipelines, feed exporters, and logging all work exactly as before. The API is invisible to the rest of the stack.
For a deeper look at how proxy types affect scraping success rates, see our post on residential proxies vs datacenter proxies and the broader guide to mastering web scraping proxies.
Takeaway: For most production crawls, the combination pattern is the right answer. Scrapy handles the graph traversal; the API handles the hard requests. You pay for the API only where you need it.
Start with 5,000 free credits
To test the plugin or the middleware pattern against your own targets, see the pricing page for plan details and the free 5,000-credit trial, which requires no credit card and does not bill failed requests. You can also try the playground to test individual requests before wiring them into your spider.
FAQ
What is Scrapy and what is it used for?
Scrapy is a free, open-source web crawling framework written in Python, first released in August 2008 under the BSD license and currently maintained by Zyte. It is used primarily for web scraping, but also for extracting data via APIs and for general-purpose web crawling. Its spider-based architecture manages the request queue, concurrency, retries, middleware, and output pipelines, so you write only the extraction logic specific to your target.
Does Scrapy work with JavaScript-rendered pages?
Not by default. Scrapy sends HTTP requests and parses the HTML the server returns. If the page builds its content in the browser after load, Scrapy receives the pre-render skeleton. The practical fix is to route those requests through a scraping API with js=true, which returns the fully rendered HTML that your spider callbacks can then parse normally.
How do I use proxies with Scrapy?
Scrapy supports HTTP proxies and SOCKS proxies through its middleware system. For rotating pools, you write a custom DownloaderMiddleware that picks an exit IP from a list, or you route requests through a managed API that handles rotation, session stickiness, and dead-route detection automatically. The latter removes the operational burden of maintaining a proxy fleet.
What is the scrapy web crawler and how does it differ from a scraping library?
Scrapy is a complete web crawling framework: it manages the request queue, concurrency, retries, middleware, and output pipelines. A library like BeautifulSoup only parses HTML you hand it. The distinction matters when you need to crawl multiple pages, follow links automatically, and store structured results. Scrapy handles all of that; BeautifulSoup handles none of it.
Can I use Scrapy and a scraping API at the same time?
Yes, and this is the most common production pattern. Scrapy handles crawl orchestration, link discovery, deduplication, and pipelines. The ujeebu-scrapy plugin or a custom DownloaderMiddleware forwards individual requests to the scraping API for pages that need JS rendering or hit bot detection. The rest of the spider is unchanged, and you pay API credits only for the requests that need them.
How much does it cost to route Scrapy through a scraping API?
On Ujeebu's Starter plan, a plain HTML scrape costs 1 credit and a JS-rendered scrape costs 5 credits, working out to $0.16 and $0.82 per 1,000 requests respectively. Failed requests are not billed. The free trial includes 5,000 credits with no credit card required, which is enough to benchmark your actual workload before committing to a plan. See the full credit table on the pricing page.
Try it yourself
Run this on your own pages
Get a free API key and make your first call in minutes. 5,000 free credits for 7 days, no card.