Web scraping is the process of automatically extracting data from websites — product prices, news articles, job listings, financial data, or any publicly available information. With Python's requests and BeautifulSoup libraries, this becomes surprisingly straightforward.
In this guide, we'll build up from scratch — covering installation, HTML parsing, CSS selectors, real-world pagination, proper error handling, and exporting data to CSV, JSON, and Excel. Everything here is production-ready code you can use immediately.
Always check a website's robots.txt file and Terms of Service before scraping. Never scrape login-protected or private data. Add delays between requests to avoid overloading servers. When in doubt, look for an official API first.
1. What Is Web Scraping and When Should You Use It?
Web scraping automates the extraction of data from HTML pages. It's the right tool when:
- No official API exists for the data you need
- You need to collect data from hundreds or thousands of pages
- Data changes frequently and you need regular snapshots (prices, news, listings)
- Manual copy-paste would take hours or days
Real-world use cases for Data Analysts:
- Competitor product pricing and e-commerce market research
- Job market trend analysis across multiple job boards
- News article collection for sentiment analysis
- Real estate listing aggregation
- Financial data — stock prices, exchange rates, company filings
2. Installation
Three libraries cover everything you need. Open your terminal and run:
# Core scraping libraries
pip install requests beautifulsoup4 lxml
# Data processing and export
pip install pandas openpyxl
# Verify installation
python -c "import requests, bs4, pandas; print('All libraries ready ✅')"BeautifulSoup requires an HTML parser. lxml is the fastest and most lenient parser — it handles malformed HTML gracefully. Python's built-in html.parser works too, but lxml is recommended for production use.
3. The requests Library — Downloading Web Pages
The requests library sends HTTP requests to web servers and retrieves their responses — exactly like a browser does, but without the graphical interface.
import requests
url = "https://quotes.toscrape.com"
response = requests.get(url)
# Check HTTP status (200 = OK, 403 = Forbidden, 404 = Not Found)
print(response.status_code) # 200
print(response.encoding) # utf-8
print(len(response.text)) # HTML length in characters
# Preview the raw HTML
print(response.text[:500])Adding Headers to Avoid Bot Detection
Many websites check the User-Agent header to determine whether a request comes from a real browser or an automated script. Adding proper headers makes your scraper look like a legitimate browser visit:
import requests
import time
import random
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/120.0.0.0 Safari/537.36"
),
"Accept-Language": "en-US,en;q=0.9",
"Accept-Encoding": "gzip, deflate, br",
}
response = requests.get(url, headers=HEADERS, timeout=10)
# Always handle HTTP errors explicitly
if response.status_code == 200: print("✅ Success")
elif response.status_code == 403: print("❌ Forbidden — website blocked the request")
elif response.status_code == 429: print("⚠️ Too Many Requests — slow down")
else: print(f"Unexpected status: {response.status_code}")
# Be polite — add a random delay between requests
time.sleep(random.uniform(1.0, 2.5))4. BeautifulSoup — Parsing HTML
BeautifulSoup parses raw HTML into a navigable tree structure, making it easy to locate and extract specific elements.
from bs4 import BeautifulSoup
import requests
response = requests.get("https://quotes.toscrape.com")
soup = BeautifulSoup(response.text, "lxml")
# Access common elements directly
print(soup.title.text) # "Quotes to Scrape"
print(soup.h1.text) # First h1 element
# Pretty-print the HTML tree (useful for debugging)
print(soup.prettify()[:800])5. Finding Elements — find(), find_all(), and CSS Selectors
BeautifulSoup provides four main methods for locating elements:
| Method | Use When | Returns |
|---|---|---|
find() | You need the first matching element | Single element or None |
find_all() | You need all matching elements | List of elements |
select_one() | CSS selector — first match | Single element or None |
select() | CSS selector — all matches | List of elements |
# ── By tag ─────────────────────────────────────
soup.find("h2") # first h2
soup.find_all("p") # all paragraphs → list
soup.find_all("a", limit=5) # first 5 links
# ── By class ───────────────────────────────────
soup.find("div", class_="quote") # first div.quote
soup.find_all("span", class_="text") # all span.text
# ── By ID ──────────────────────────────────────
soup.find("div", id="main")
# ── CSS Selectors (most flexible) ──────────────
soup.select(".quote") # class="quote"
soup.select(".quote .text") # .text inside .quote
soup.select("div.quote > span") # direct child span
soup.select("a[href]") # all links with href attribute
soup.select_one("li.next a") # next page link
# ── Extracting data from an element ────────────
el = soup.find("a")
el.text # visible text (preserves whitespace)
el.get_text(strip=True) # text with leading/trailing whitespace removed
el["href"] # attribute value (raises KeyError if missing)
el.get("href", "") # safe attribute access with defaultIn Chrome or Firefox, right-click any element → Inspect. In the DevTools panel, right-click the highlighted HTML → Copy → Copy Selector. Paste that directly into soup.select(). This eliminates guesswork entirely.
6. Real Project — Scraping a Quotes Website
quotes.toscrape.com is a website built specifically for scraping practice. It contains quotes, authors, and tags across 10 pages — a perfect real-world exercise.
Step 1 — Inspect the HTML Structure
Before writing any code, open the website in your browser and press F12 to open DevTools. Identify the HTML structure of the data you want:
"""
HTML structure on quotes.toscrape.com:
<div class="quote">
<span class="text">"The world as we have created it..."</span>
<small class="author">Albert Einstein</small>
<div class="tags">
<a class="tag">change</a>
<a class="tag">deep-thoughts</a>
</div>
</div>
"""
from bs4 import BeautifulSoup
import requests
response = requests.get("https://quotes.toscrape.com", timeout=10)
soup = BeautifulSoup(response.text, "lxml")
# Extract data from the first quote
quotes = soup.select(".quote")
print(f"Quotes found on this page: {len(quotes)}") # 10
first = quotes[0]
text = first.select_one(".text").get_text(strip=True)
author = first.select_one(".author").get_text(strip=True)
tags = [t.get_text(strip=True) for t in first.select(".tag")]
print(text) # "The world as we have created it..."
print(author) # Albert Einstein
print(tags) # ['change', 'deep-thoughts', 'thinking', 'world']7. Handling Pagination — Scraping Multiple Pages
Most real websites spread data across multiple pages. The key is to detect the "Next" button and follow it automatically until no more pages remain:
from bs4 import BeautifulSoup
import requests
import time
BASE_URL = "https://quotes.toscrape.com"
HEADERS = {"User-Agent": "Mozilla/5.0 Chrome/120.0"}
all_quotes = []
page_url = BASE_URL
page_num = 1
while page_url:
print(f"Scraping page {page_num}: {page_url}")
response = requests.get(page_url, headers=HEADERS, timeout=10)
if response.status_code != 200:
print(f"Stopped — HTTP {response.status_code}")
break
soup = BeautifulSoup(response.text, "lxml")
# Extract all quotes on the current page
for q in soup.select(".quote"):
all_quotes.append({
"text" : q.select_one(".text").get_text(strip=True),
"author": q.select_one(".author").get_text(strip=True),
"tags" : ", ".join(t.get_text(strip=True) for t in q.select(".tag")),
"page" : page_num,
})
# Check for a "Next" page link
next_btn = soup.select_one("li.next a")
page_url = BASE_URL + next_btn["href"] if next_btn else None
page_num += 1
time.sleep(1.5) # Respectful delay between requests
print(f"Done. Total quotes collected: {len(all_quotes)}") # 1008. Saving Data — CSV, JSON, and Excel
Once you have a list of dictionaries, Pandas makes it trivial to export to any format:
import pandas as pd
import json
df = pd.DataFrame(all_quotes)
# ── CSV ────────────────────────────────────────
df.to_csv("quotes.csv", index=False, encoding="utf-8-sig")
# utf-8-sig → BOM encoding, opens correctly in Excel on Windows
# ── JSON ───────────────────────────────────────
with open("quotes.json", "w", encoding="utf-8") as f:
json.dump(all_quotes, f, indent=2, ensure_ascii=False)
# ── Excel ──────────────────────────────────────
df.to_excel("quotes.xlsx", index=False, sheet_name="Quotes")
# ── Quick analysis ─────────────────────────────
print(df.head())
print(f"\nTotal rows : {len(df)}")
print(f"Unique authors: {df['author'].nunique()}")
print(f"Most quoted author:\n{df['author'].value_counts().head(3)}")9. Production-Ready Scraper — Complete Script
The following is a clean, fully structured scraper with logging, error handling, configuration, and automatic export — ready to adapt to any project:
"""
quotes_scraper.py
Production-ready web scraper for quotes.toscrape.com
Author : Ankit Kumar | ETLGuru.in
Usage : python quotes_scraper.py
"""
from bs4 import BeautifulSoup
import requests
import pandas as pd
import json, time, logging
from datetime import datetime
# ── Logging ────────────────────────────────────
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s [%(levelname)s] %(message)s",
handlers=[logging.FileHandler("scraper.log"), logging.StreamHandler()],
)
log = logging.getLogger(__name__)
# ── Configuration ──────────────────────────────
CONFIG = {
"base_url" : "https://quotes.toscrape.com",
"delay" : 1.5, # seconds between pages
"timeout" : 15, # request timeout
"max_pages" : 50, # safety cap
"output_csv" : "quotes.csv",
"output_json" : "quotes.json",
"output_xlsx" : "quotes.xlsx",
}
HEADERS = {
"User-Agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/120.0.0.0",
"Accept-Language": "en-US,en;q=0.9",
}
# ── Fetch page ─────────────────────────────────
def fetch(url: str) -> BeautifulSoup | None:
try:
r = requests.get(url, headers=HEADERS, timeout=CONFIG["timeout"])
r.raise_for_status()
return BeautifulSoup(r.text, "lxml")
except requests.exceptions.Timeout:
log.error(f"Timeout: {url}")
except requests.exceptions.HTTPError as e:
log.error(f"HTTP {e.response.status_code}: {url}")
except requests.exceptions.ConnectionError:
log.error(f"Connection error: {url}")
return None
# ── Parse quotes from a page ───────────────────
def parse(soup: BeautifulSoup, page: int) -> list[dict]:
results = []
for q in soup.select(".quote"):
try:
results.append({
"text" : q.select_one(".text").get_text(strip=True),
"author" : q.select_one(".author").get_text(strip=True),
"author_url": CONFIG["base_url"] + q.select_one("a").get("href", ""),
"tags" : ", ".join(t.get_text(strip=True) for t in q.select(".tag")),
"page" : page,
"scraped_at": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
})
except Exception as exc:
log.warning(f"Parse error on page {page}: {exc}")
return results
# ── Save data ──────────────────────────────────
def save(data: list[dict]) -> None:
df = pd.DataFrame(data)
df.to_csv (CONFIG["output_csv"], index=False, encoding="utf-8-sig")
df.to_excel(CONFIG["output_xlsx"], index=False, sheet_name="Quotes")
with open(CONFIG["output_json"], "w", encoding="utf-8") as f:
json.dump(data, f, indent=2, ensure_ascii=False)
log.info(f"Saved {len(df)} rows → CSV, JSON, Excel")
# ── Main ───────────────────────────────────────
def main() -> None:
log.info("Scraper started")
all_data = []
page_url = CONFIG["base_url"]
page_num = 1
while page_url and page_num <= CONFIG["max_pages"]:
log.info(f"Page {page_num}: {page_url}")
soup = fetch(page_url)
if not soup:
log.error("Fetch failed — stopping")
break
page_data = parse(soup, page_num)
all_data.extend(page_data)
log.info(f" {len(page_data)} quotes extracted (running total: {len(all_data)})")
nxt = soup.select_one("li.next a")
page_url = CONFIG["base_url"] + nxt["href"] if nxt else None
page_num += 1
time.sleep(CONFIG["delay"])
if all_data:
save(all_data)
log.info(f"Done — {len(all_data)} records from {page_num - 1} pages")
else:
log.warning("No data was collected")
if __name__ == "__main__":
main()10. Common Errors and How to Fix Them
| Error | Cause | Fix |
|---|---|---|
AttributeError: 'NoneType' | Element not found on the page | Use el = soup.select_one(...) then el.get_text() if el else "" |
403 Forbidden | Server detected a bot | Add realistic headers; increase delay |
429 Too Many Requests | Scraping too fast | Increase time.sleep() to 3–5 seconds |
ConnectionError | Network issue or site down | Add retry logic with exponential back-off |
| Empty results on valid page | Content loaded by JavaScript | Use Selenium or Playwright instead |
| Garbled characters in CSV | Wrong encoding | Use encoding="utf-8-sig" for Excel compatibility |
11. Ethical Scraping — Best Practices
- Check
robots.txtfirst — visitexample.com/robots.txtto see what the site allows or disallows - Respect rate limits — add at least 1–2 seconds between requests; more for large scraping jobs
- Identify your scraper — include your name or contact in the User-Agent string
- Cache downloaded pages — save HTML locally during development; don't re-download unnecessarily
- Scrape during off-peak hours — server load is lower at night
- Look for an API first — if the website offers one, always prefer it over scraping
- Read the Terms of Service — some websites explicitly prohibit scraping; respect that
- Only scrape public data — never scrape behind authentication or access private user data
If a website loads data through JavaScript (React, Angular, Vue apps), the HTML BeautifulSoup sees will be empty. In those cases, use Selenium or Playwright — they run a real browser and wait for JavaScript to execute before you extract the fully rendered HTML.