BeautifulSoup — Selecting Elements

• Install and use beautifulsoup4 (plus lxml parser).
• Pick elements with soup.find / soup.find_all.
• Use CSS selectors with soup.select / soup.select_one.
• Read text, attributes, and children of each result.

pip install beautifulsoup4 lxml requests

from bs4 import BeautifulSoup

html = """
<html><body>
  <h1 class="hero">Hello world</h1>
  <p>Paragraph one.</p>
  <p class="lead">Important paragraph.</p>
</body></html>
"""

soup = BeautifulSoup(html, "lxml")
print(soup.h1.text)               # → Hello world
print(soup.find("p").text)        # → Paragraph one.
print(soup.find("p", class_="lead").text)  # → Important paragraph.

The page we'll use

html = """
<ul class="book-list">
  <li class="book">
    <a href="/b/1">Refactoring</a>
    <span class="price">$89</span>
  </li>
  <li class="book featured">
    <a href="/b/2">The Pragmatic Programmer</a>
    <span class="price">$95</span>
  </li>
  <li class="book">
    <a href="/b/3">Clean Code</a>
    <span class="price">$79</span>
  </li>
</ul>
"""
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, "lxml")

find — first match

first = soup.find("li", class_="book")
print(first.a.text)        # → Refactoring
print(first.a["href"])     # → /b/1
print(first.find("span", class_="price").text)  # → $89

Two patterns to notice:

• tag.text — the text inside the element.
• tag["attr"] — reads an attribute. tag.get("attr") returns None if it's missing.

find_all — every match

for book in soup.find_all("li", class_="book"):
    title = book.a.text
    price = book.find("span", class_="price").text
    print(f"  {title:<30} {price}")

  Refactoring                    $89
  The Pragmatic Programmer       $95
  Clean Code                     $79

select / select_one — full CSS

# Every book that's also featured
print(soup.select_one("li.book.featured a").text)
# → The Pragmatic Programmer

# Every price span
for p in soup.select(".book .price"):
    print(p.text)

# Direct children only — > selector
for li in soup.select("ul.book-list > li"):
    print(li.a.text)

get_text(strip=True) for tidy text

Real-world HTML has lots of whitespace. get_text(strip=True) trims it; get_text(" ", strip=True) joins children with a single space.

soup.find("p").get_text(" ", strip=True)

# scrape_books.py — title + price + rating from the first page
import requests
from bs4 import BeautifulSoup

URL = "https://books.toscrape.com/"

r = requests.get(URL, timeout=10,
                 headers={"User-Agent": "advaslearning-py-l4 (demo)"})
r.raise_for_status()

soup = BeautifulSoup(r.text, "lxml")

books = []
for card in soup.select("article.product_pod"):
    title  = card.h3.a["title"]                            # full title is the attribute
    price  = card.select_one(".price_color").get_text(strip=True)
    rating = card.select_one("p.star-rating")["class"][1]  # ["star-rating", "Three"]
    books.append({"title": title, "price": price, "rating": rating})

# Top 5 by name length, just to show data shaping
for b in books[:5]:
    print(f"  {b['rating']:<6}  {b['price']:<8}  {b['title']}")

print(f"\nfound {len(books)} books on the page")

Sample output

  Three   £51.77    A Light in the Attic
  One     £53.74    Tipping the Velvet
  One     £50.10    Soumission
  Four    £47.82    Sharp Objects
  Five    £54.23    Sapiens: A Brief History of Humankind

found 20 books on the page

Read the diff

Three selectors capture everything:

• article.product_pod — the repeating book block.
• .price_color — the price text.
• p.star-rating — the rating is stored in the CSS class, not the text.

That last one is real-world: half the time the data you want is in an attribute, not text. Inspect carefully.

Three methods cover 95% of scraping: find for one element, find_all for many, select when you want a full CSS selector. Data lives in .text, in tag["attr"] or sometimes in a class name. Inspect, isolate the selector, write it once. Tomorrow we ship a real project — a price tracker.