Web Scraping Introduction
A web crawler is a program that simulates browser behavior to automatically fetch data from the internet. Python has become the go-to language for web scraping thanks to its rich ecosystem (requests, BeautifulSoup, Scrapy, etc.).
What Is a Web Crawler
When a browser visits a web page, it sends an HTTP request to a server and renders the returned HTML/JSON response. A crawler reproduces this process programmatically:
- Send an HTTP request (simulating a browser)
- Receive the response (HTML, JSON, images, etc.)
- Parse the content and extract target data
- Store the data (files, databases, etc.)
Types of Web Crawlers
| Type | Characteristics | Use Cases |
|---|---|---|
| General crawler | Fetches the entire content of each page | Search engine indexing (Googlebot, etc.) |
| Focused crawler | Extracts only specific data from a page | Price monitoring, news collection |
| Incremental crawler | Only fetches new or updated content | News feeds, e-commerce updates |
| Distributed crawler | Multiple machines work in parallel on massive datasets | Large-scale data collection (Scrapy-Redis) |
Focused crawlers are usually built on top of general crawlers: fetch the complete page first, then extract the required fields.
Legality of Web Scraping
When using a crawler, keep the following principles in mind:
- Respect robots.txt: Check the
robots.txtfile at the root of the target site to see which paths are disallowed. - Do not disrupt normal service: Control your request rate to avoid putting excessive load on the server (DDoS-like behavior is illegal).
- Comply with data regulations: Do not scrape or distribute infringing content (user privacy, copyrighted material, etc.).
- Obtain authorization for commercial use: If you intend to use scraped data commercially, get written permission from the site owner first.
Anti-Scraping and Counter-Measures
Websites typically deploy anti-scraping mechanisms, and crawlers need corresponding countermeasures:
| Anti-scraping Technique | Description | Countermeasure |
|---|---|---|
| User-Agent detection | Identifies non-browser request headers | Spoof the User-Agent |
| IP rate limiting / blocking | Limits requests per IP | Random delays + proxy IP pool |
| Cookie / Session | Requires a logged-in state | Simulate login, maintain session |
| CAPTCHA | Human verification challenge | CAPTCHA-solving service / AI recognition |
| JS rendering | Data generated dynamically by JavaScript | Selenium / Playwright |
| Data encryption | API parameters are signed | Reverse-engineer the encryption algorithm |
Typical Workflow
import requests
from bs4 import BeautifulSoup
url = "https://books.toscrape.com/"
headers = {"User-Agent": "Mozilla/5.0 (compatible; MyBot/1.0)"}
resp = requests.get(url, headers=headers, timeout=10)
resp.raise_for_status()
soup = BeautifulSoup(resp.text, "html.parser")
for article in soup.select("article.product_pod"):
title = article.h3.a["title"]
price = article.select_one(".price_color").text
print(f"{title}: {price}")This example demonstrates the three fundamental steps of a crawler: request → parse → extract. The following chapters will cover each step in depth.