requests Advanced Operations

Building on requests Basic Operations, this article covers advanced scenarios commonly encountered in web scraping: session persistence, cookie-based anti-scraping, proxy IPs, lazy-loaded image detection, and concurrent downloads.

Session: Simulating Login and Cookie Persistence

Some websites require you to visit the homepage first to trigger a cookie before you can access a target API. A Session handles this automatically:

import requests

headers = {
    "User-Agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"
    )
}

with requests.Session() as sess:
    # First request: server sets a cookie; Session saves it automatically
    sess.get("https://xueqiu.com/", headers=headers, timeout=10)

    # Second request: cookie is sent automatically
    api_url = "https://xueqiu.com/statuses/hot/listV2.json?since_id=-1&max_id=72813&size=15"
    data = sess.get(api_url, headers=headers, timeout=10).json()
    print(data)

Setting Cookies Manually

import requests

# Copy from browser DevTools -> Application -> Cookies
cookies = {
    "session_id": "abc123",
    "user_token": "xyz789",
}

with requests.Session() as sess:
    sess.headers.update(headers)
    sess.cookies.update(cookies)
    resp = sess.get("https://example.com/api/data", timeout=10)
    print(resp.json())

Proxy IPs

When your IP is blocked due to frequent requests to the same site, use a proxy to forward your requests:

import requests
import random

proxies_pool = [
    {"https": "http://1.2.3.4:8080"},
    {"https": "http://5.6.7.8:3128"},
]

headers = {"User-Agent": "Mozilla/5.0"}
url = "https://httpbin.org/ip"

proxy = random.choice(proxies_pool)
try:
    resp = requests.get(url, headers=headers, proxies=proxy, timeout=10)
    print(resp.json())
except requests.exceptions.ProxyError:
    print("Proxy connection failed, switching proxy")

Proxy Anonymity Levels

Transparent proxy: The target server can see both your real IP and the proxy IP.
Anonymous proxy: The server knows a proxy is in use, but cannot see your real IP.
Elite (high-anonymity) proxy: The server cannot tell whether a proxy is being used.

Elite proxies are best for scraping. You can dynamically retrieve IP lists from commercial proxy provider APIs.

Lazy-Loaded Images

Some sites use lazy loading as an anti-scraping measure: the real image URL is not stored in the src attribute but in pseudo-attributes such as src2, data-src, or original. These are only assigned to src when the user scrolls to that area.

import requests
from lxml import etree
from pathlib import Path

headers = {"User-Agent": "Mozilla/5.0"}
resp = requests.get("https://example.com/gallery", headers=headers, timeout=10)
tree = etree.HTML(resp.text)

# Normal loading: read src
normal_srcs = tree.xpath('//img/@src')

# Lazy loading: read data-src (attribute name varies by site)
lazy_srcs = tree.xpath('//img/@data-src')
lazy_srcs2 = tree.xpath('//img/@src2')

print("Lazy-loaded images:", lazy_srcs or lazy_srcs2)

Identifying lazy loading: In browser DevTools, open the Elements panel and inspect an <img> tag. If src is a blank image or a base64 placeholder, the page uses lazy loading — find the real attribute name.

Concurrent Downloads (ThreadPoolExecutor)

For I/O-bound bulk downloads, a thread pool can dramatically improve throughput:

import requests
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
import time

headers = {"User-Agent": "Mozilla/5.0"}

def download_image(url: str, save_dir: Path) -> str:
    filename = url.rsplit("/", 1)[-1]
    filepath = save_dir / filename
    if filepath.exists():
        return f"Already exists: {filename}"

    resp = requests.get(url, headers=headers, timeout=30, stream=True)
    resp.raise_for_status()

    with filepath.open("wb") as f:
        for chunk in resp.iter_content(chunk_size=8192):
            f.write(chunk)

    time.sleep(0.2)   # Polite delay to avoid triggering rate limits
    return f"Done: {filename}"

if __name__ == "__main__":
    img_urls = [
        "https://httpbin.org/image/png",
        "https://httpbin.org/image/jpeg",
    ]
    save_dir = Path("downloads")
    save_dir.mkdir(exist_ok=True)

    with ThreadPoolExecutor(max_workers=4) as executor:
        futures = {executor.submit(download_image, url, save_dir): url for url in img_urls}
        for future in as_completed(futures):
            try:
                print(future.result())
            except Exception as e:
                print(f"Download failed: {futures[future]} -> {e}")

Pagination Scraping

Build a URL template and loop through page numbers:

import requests
from bs4 import BeautifulSoup
import time

headers = {"User-Agent": "Mozilla/5.0"}

def crawl_page(url: str) -> list[dict]:
    resp = requests.get(url, headers=headers, timeout=10)
    resp.raise_for_status()
    soup = BeautifulSoup(resp.text, "lxml")

    items = []
    for article in soup.select("article.product_pod"):
        items.append({
            "title": article.h3.a["title"],
            "price": article.select_one(".price_color").get_text(strip=True),
        })
    return items

all_data = []
base_url = "https://books.toscrape.com/catalogue/page-{}.html"

for page in range(1, 6):   # Scrape the first 5 pages
    url = base_url.format(page)
    print(f"Scraping page {page}...")
    try:
        data = crawl_page(url)
        all_data.extend(data)
    except requests.RequestException as e:
        print(f"Page {page} failed: {e}")
    time.sleep(0.5)   # Reduce request frequency

print(f"Total records scraped: {len(all_data)}")

robots.txt Compliance

Before scraping a site, check its robots.txt to understand which paths are allowed or forbidden:

import urllib.robotparser

rp = urllib.robotparser.RobotFileParser()
rp.set_url("https://books.toscrape.com/robots.txt")
rp.read()

url_to_check = "https://books.toscrape.com/catalogue/page-1.html"
user_agent = "my-bot"

if rp.can_fetch(user_agent, url_to_check):
    print(f"Allowed: {url_to_check}")
else:
    print(f"Disallowed: {url_to_check}")

Last updated on 2026-06-23

Web Scraping Introduction Data Parsing