To scrape Twitter (X) with Python today, you need JavaScript rendering, not just an HTTP client. A plain requests.get() call to an X profile returns an HTML shell with almost no content. The posts load later, triggered by JavaScript, so BeautifulSoup finds nothing to parse. The tutorial you followed in 2022 does not fail because of a bug in your code; it fails because the page you are trying to parse does not exist in the initial HTTP response.
This post shows how to scrape Twitter Python-style without browser automation: send one GET request to the Ujeebu Scrape API with js=true and scroll parameters, get back fully rendered HTML, parse it locally in Python, and write the results to CSV. No Selenium, no session management, no OAuth. By the end you have a complete script under 50 lines that you can run against any public X profile.
Before running any pipeline at scale, review x.com/robots.txt and X's developer terms. The official API is the supported programmatic route for production use cases that require guaranteed access.
Why Plain requests and BeautifulSoup Fail on X
On many modern sites, visible content is rendered by JavaScript after the initial HTTP response. When you send a GET to an X profile URL, the server returns a lightweight HTML document. The posts load asynchronously after that, driven by JavaScript that calls internal data endpoints. A requests.get() call captures the shell; the content you actually want is never in the response.
Browser automation tools can execute that JavaScript, but infinite-scrolling patterns add a second problem. Content loads as the user scrolls, not all at once, so you need extra scrolling logic on top of the rendering. That logic is brittle, slow, and resource-intensive. Running it at any meaningful volume from a single IP compounds the problem: sites detect repeated requests from the same address and start returning blocks or empty responses. A rotating proxy pool distributes requests across many IPs so no single address accumulates a block history, the same way a call center rotates its outgoing lines so no single number gets flagged.
The official API is the supported programmatic route, but it requires OAuth and a developer account. For research pulls and data pipelines that do not need the firehose, a headless-browser scraping API handles the rendering and proxy rotation as infrastructure, so your Python code stays simple.
Takeaway: If your X scraper stopped working, the root cause is JavaScript rendering combined with IP-level detection, not a parsing bug. Fix the infrastructure layer first.
How to Scrape Twitter Without the Official REST API
The alternative to the official API is to render the page the same way a browser would, then extract the content from the resulting HTML. This is not a workaround; it is how every headless-browser scraping API works. You send a URL, the API loads it in a JavaScript-capable environment with rotating proxies, and returns the rendered HTML to your script.
The Ujeebu Scrape API accepts a js=true parameter that enables JavaScript rendering before returning the response. Add scroll_down=true and a scroll_wait value to give the JavaScript time to load posts before the HTML is captured. Your Python script sends one GET request and receives rendered HTML. No browser binary to install, no WebDriver to manage, no proxy pool to configure.
This approach also sidesteps the IP blocking problem that makes scraping at scale difficult. Each request goes through a rotating proxy, so the target site sees a different IP on every call. If a request is blocked or returns a bot-check interstitial instead of the page, the API treats it as a failed request and does not charge you for it.
Try it yourself
Run this on your own pages
Get a free API key and make your first call in minutes. 5,000 free credits for 7 days, no card.
Prerequisites and Setup
You need one library:
pip install requests
Sign up for a free trial at ujeebu.com/signup (5,000 credits, no credit card required) and copy your API key from the dashboard. Store it as an environment variable:
export UJEEBU_API_KEY="your_key_here"
Check your quota before a large run:
curl -X GET 'https://api.ujeebu.com/account' \
-H "ApiKey: YOUR_API_KEY"
The response includes quota, used, and balance fields, which you can use to estimate remaining usage before committing to a large job.
Scrape Twitter Python: Fetch a Single X Profile Page
The core request is a GET to /scrape with js=true, scroll_down=true, and a scroll_wait value in milliseconds. The API renders the page with JavaScript enabled, scrolls down to trigger lazy-loaded content, waits the specified time, then returns the HTML.
Start with curl to confirm your key works and see the raw response:
curl -X GET \
'https://api.ujeebu.com/scrape?url=https://x.com/ujeebu&js=true&scroll_down=true&scroll_wait=2000' \
-H "ApiKey: YOUR_API_KEY"
If you get back a large HTML document with post content visible in the body, rendering is working. If you get back a short shell with no posts, increase scroll_wait and try again.
Here is the Python equivalent:
import os
import requests
API_KEY = os.environ.get("UJEEBU_API_KEY")
BASE_URL = "https://api.ujeebu.com/scrape"
def fetch_profile_html(profile_url: str, scroll_wait: int = 2000) -> str:
params = {
"url": profile_url,
"js": True,
"scroll_down": True,
"scroll_wait": scroll_wait,
}
response = requests.get(
BASE_URL,
params=params,
headers={"ApiKey": API_KEY},
)
response.raise_for_status()
return response.text
html = fetch_profile_html("https://x.com/ujeebu")
print(f"Fetched {len(html)} characters of rendered HTML")
Each successful call with js=true costs 5 credits. Failed or blocked requests are not billed, so a page that returns a security interstitial instead of the profile does not consume credits.
The rendered HTML now contains the post content that JavaScript loaded. From here you parse it locally in Python using the standard library or any HTML parser you prefer.
Takeaway: Run the curl example first. A large HTML response with visible post content means rendering succeeded. A short shell means you need a longer scroll_wait.
Parse the Rendered HTML Locally in Python
Once you have the rendered HTML as a string, you can extract fields with Python's built-in html.parser or any library you already use. The key difference from the old BeautifulSoup approach is that the HTML you are parsing is now fully rendered, so the content is actually there.
If you parse HTML yourself, avoid brittle generated class names and prefer stable attributes where available. Many modern sites use data-testid attributes on interactive elements precisely because they are meant to be stable identifiers. Inspect the rendered HTML from your API response to find the attributes that work for your target fields, then build your selectors around those.
A minimal parsing pass that pulls post text and timestamps from rendered HTML:
import re
from html.parser import HTMLParser
class PostParser(HTMLParser):
def __init__(self):
super().__init__()
self.posts = []
self._in_post_text = False
self._current_text = []
def handle_starttag(self, tag, attrs):
attr_dict = dict(attrs)
if attr_dict.get("data-testid") == "tweetText":
self._in_post_text = True
self._current_text = []
def handle_data(self, data):
if self._in_post_text:
self._current_text.append(data)
def handle_endtag(self, tag):
if self._in_post_text and tag in ("div", "span"):
text = "".join(self._current_text).strip()
if text:
self.posts.append({"text": text})
self._in_post_text = False
def parse_posts(html: str) -> list:
parser = PostParser()
parser.feed(html)
return parser.posts
This is a starting point, not a production parser. Real post structures nest multiple elements, and you will want to add timestamp extraction and engagement counts once you have confirmed the basic text extraction works. The Ujeebu recipes catalogue lists sites with validated extraction configurations and sample responses, so check there first before building your own parser for a supported site.
Takeaway: Parse the rendered HTML you receive, not the raw HTTP response. The content is there once JavaScript has run.
Paginate Through an X Profile to Get Older Posts
A single scroll pass captures the posts visible in one viewport. To go deeper into a profile's history, you need multiple passes with progressively more scrolling. The Scrape API's progressive_scroll parameter enables additional scrolling before capture. Combine it with a loop that deduplicates results across passes and stops when no new content appears.
Here is the complete paginating script with CSV export:
import os
import csv
import requests
from html.parser import HTMLParser
API_KEY = os.environ.get("UJEEBU_API_KEY")
BASE_URL = "https://api.ujeebu.com/scrape"
class PostParser(HTMLParser):
def __init__(self):
super().__init__()
self.posts = []
self._in_post = False
self._buf = []
def handle_starttag(self, tag, attrs):
attr_dict = dict(attrs)
if attr_dict.get("data-testid") == "tweetText":
self._in_post = True
self._buf = []
if self._in_post and tag == "time":
self.posts.append({"text": "".join(self._buf).strip(), "timestamp": attr_dict.get("datetime", "")})
self._in_post = False
def handle_data(self, data):
if self._in_post:
self._buf.append(data)
def parse_posts(html: str) -> list:
parser = PostParser()
parser.feed(html)
return [p for p in parser.posts if p.get("text")]
def fetch_all_posts(profile_url: str, max_passes: int = 5) -> list:
all_posts = []
seen: set = set()
for pass_num in range(1, max_passes + 1):
params = {
"url": profile_url,
"js": True,
"progressive_scroll": True,
"scroll_wait": 1500 * pass_num,
}
response = requests.get(BASE_URL, params=params, headers={"ApiKey": API_KEY})
if response.status_code != 200:
print(f"Pass {pass_num}: HTTP {response.status_code}, skipping")
continue
posts = parse_posts(response.text)
new_count = 0
for post in posts:
key = post.get("timestamp", "") + post.get("text", "")[:50]
if key and key not in seen:
seen.add(key)
all_posts.append(post)
new_count += 1
print(f"Pass {pass_num}: {new_count} new posts (total {len(all_posts)})")
if new_count == 0:
print("No new posts found, stopping.")
break
return all_posts
def save_to_csv(posts: list, filename: str = "x_posts.csv") -> None:
if not posts:
print("No posts to save.")
return
fieldnames = list(posts[0].keys())
with open(filename, "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=fieldnames, extrasaction="ignore")
writer.writeheader()
writer.writerows(posts)
print(f"Saved {len(posts)} posts to {filename}")
if __name__ == "__main__":
posts = fetch_all_posts("https://x.com/ujeebu", max_passes=5)
save_to_csv(posts)
A few design decisions worth noting. The deduplication key combines the timestamp and the first 50 characters of the post text. That handles the common case where the same posts appear across multiple scroll passes without requiring a database. The fieldnames list is derived from the first result object rather than hardcoded, so if your parser adds a new field, the CSV gains a column automatically. The max_passes ceiling is your budget guard: set it before you run against a large profile.
Takeaway: Set max_passes to a number you are comfortable paying for before running. Five passes at 5 credits each is 25 credits, well within the free trial's 5,000 credits.
Credit Cost and Pricing at Scale
Each /scrape call with js=true costs 5 credits. Here is what that means at different volumes, using published plan prices:
| Monthly requests | Credits needed | Plan that fits |
|---|---|---|
| 100 | 500 | Free trial (5,000 credits) |
| 1,000 | 5,000 | Free trial or Starter |
| 10,000 | 50,000 | Starter ($49/month, 300,000 credits) |
| 60,000 | 300,000 | Starter ($49/month, 300,000 credits) |
| 200,000 | 1,000,000 | Growth ($119/month, 1,000,000 credits) |
| 600,000 | 3,000,000 | Pro ($299/month, 3,000,000 credits) |
Failed or blocked requests are not billed. Your actual spend tracks successful fetches, not attempts. That matters for budget planning on sites that occasionally return blocks: you are not paying for the misses.
The Starter plan's 300,000 credits covers up to 60,000 JavaScript-rendered requests per month. For a one-off research project, the free trial's 5,000 credits covers 1,000 rendered pages, enough to pull the public post history of several dozen profiles. Annual billing saves 10% across all plans.
The pricing page shows the full plan comparison and top-up pack options for jobs that exceed a monthly quota.
Takeaway: Multiply your expected monthly request count by 5 to get your credit requirement, then pick the plan whose quota covers that number with a buffer for retries.
When to Parse HTML Locally vs. Use a Dedicated Extraction API
Parsing rendered HTML locally gives you full control but requires maintenance. When the site updates its markup, your selectors break and you update them. That is a reasonable trade-off for a one-off research project or a pipeline you own end to end.
For article-style pages, the Article Extractor returns clean structured JSON (title, author, date, body text) without any selector work on your part. For pages you want as clean Markdown for downstream processing, the Markdown API handles that in one call. Neither replaces a custom parser for a social profile, but they eliminate the maintenance burden for the content types they cover.
The Ujeebu recipes catalogue lists supported sites with validated extraction configurations, sample responses, and per-field documentation. When a recipe exists for your target site, use it: the selectors are tested and maintained, and the call costs the same credits as the equivalent page fetch with the recipe's settings.
For scraping patterns on other platforms, the TikTok scraping guide and the Amazon scraping guide show how the same API handles different site structures. The proxy and anti-bot guide explains the proxy layer in more depth if you are seeing blocks on high-volume runs.
Start with the free trial (5,000 credits, no credit card) at ujeebu.com/signup.
FAQ
How do I scrape X (Twitter) without the official API?
Send a GET request to the Ujeebu /scrape endpoint with js=true and scroll_down=true. JavaScript rendering is enabled before the HTML is returned, so the post content that loads asynchronously is present in the response. Parse the rendered HTML locally in Python to extract the fields you need.
What JSON fields does the X recipe return?
Ujeebu does not currently have a pre-built recipe for X. When you scrape X profile pages with js=true, you receive rendered HTML and parse it yourself. Common fields you can extract include post text, timestamp (from the datetime attribute on <time> elements), and engagement counts. Check the recipes catalogue for the current list of supported sites.
How many credits does scraping an X profile page cost?
A /scrape call with js=true costs 5 credits per successful request. Failed or blocked requests are not charged. The free trial includes 5,000 credits, enough for 1,000 JavaScript-rendered pages.
Can I scrape multiple X profiles in a loop with Python?
Yes. Call fetch_all_posts() in a loop over a list of profile URLs. Each profile is an independent request. Check your credit balance with GET /account before large runs, and keep a short delay between requests to avoid hammering the target site.
Do failed or blocked X requests cost credits?
No. Ujeebu only bills successful responses. If a request returns a security interstitial or a network block instead of the profile page, it is treated as a failed request and is not charged.
How do I paginate through an X profile to get older posts?
Use progressive_scroll=true in the /scrape request to enable additional scrolling before capture. Run multiple passes with a deduplication check on each pass, and stop when a pass returns no new posts. The complete loop with deduplication and CSV export is shown in the pagination section above.
Try it yourself
Run this on your own pages
Get a free API key and make your first call in minutes. 5,000 free credits for 7 days, no card.