Jan 14, 2025
2 min read
How to scrape otto.de?
The code automates a web scraping task using Selenium, BeautifulSoup, and Python libraries to collect product information (titles and prices) from the German e-commerce website Otto.de. Here's a breakdown of what each section does:
Imports
python
Copy code
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from time import sleep
import csv
from bs4 import BeautifulSoup
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import json
- selenium: Automates browser actions.
- BeautifulSoup: Parses HTML content.
- time.sleep: Adds delays for dynamic page loading.
- csv: Writes scraped data into a CSV file.
- json: Handles JSON data.
- WebDriverWait and EC: Waits for elements to load dynamically.
Initialization
python
Copy code
driver = webdriver.Chrome()
driver.get('https://www.otto.de/')
- Launches a Chrome browser instance and navigates to the Otto website.
Accepting Cookies
python
Copy code
WebDriverWait(driver, 60).until(EC.presence_of_element_located((By.ID, "onetrust-accept-btn-handler")))
cookies = driver.find_element(By.ID, "onetrust-accept-btn-handler")
cookies.click()
- Waits for the cookie consent popup and clicks "Accept."
Search for Products
python
Copy code
WebDriverWait(driver, 60).until(
EC.presence_of_element_located((By.CSS_SELECTOR, ".squirrel_searchfield.js_squirrel_searchbar__input.svelte-11jrfxz"))
)
search_bar = driver.find_element(By.CSS_SELECTOR, ".squirrel_searchfield.js_squirrel_searchbar__input.svelte-11jrfxz")
search_bar.click()
search_bar.send_keys("football" + Keys.RETURN)
sleep(8)
- Locates the search bar, enters the term "football," and submits the search query.
Write CSV File Header
python
Copy code
with open('products.csv', mode='w', newline='', encoding='utf-8') as file:
writer = csv.writer(file)
writer.writerow(['Product Number', 'Product Title', 'Product Price'])
- Opens a CSV file and writes the column headers:
Product Number,Product Title, andProduct Price.
Scrolling and Scraping Products
python
Copy code
initial_height = driver.execute_script("return document.body.scrollHeight")
scroll_position = 0
total_scrolls = 25
for _ in range(total_scrolls):
driver.execute_script(f"window.scrollTo(0, {scroll_position + initial_height / total_scrolls});")
scroll_position += initial_height / total_scrolls
sleep(7 / total_scrolls)
- Gradually scrolls down the page to load more products dynamically.
Parsing Product Data
python
Copy code
page_source = driver.page_source
soup = BeautifulSoup(page_source, 'html.parser')
product_elements = soup.find_all('article', attrs={'data-product-listing-type': 'SearchResultPage'})
- Fetches the page source and uses BeautifulSoup to locate product elements.
Extracting Product Title and Price
python
Copy code
for idx, product in enumerate(product_elements, 1):
title_element = product.find('p', class_='find_tile__name pl_copy100')
title = title_element.get_text(strip=True) if title_element else 'No Title Found'
price_element = product.find('span', class_='find_tile__retailPrice pl_headline50 find_tile__priceValue')
if not price_element:
price_element = product.find('span', class_='find_tile__retailPrice pl_headline50 find_tile__priceValue find_tile__priceValue--red')
price = price_element.get_text(strip=True) if price_element else 'No Price Found'
writer.writerow([idx, title, price])
- Extracts product titles and prices, handling variations in class names for prices.
- Writes the data to the CSV file.
Pagination
python
Copy code
nextpage = driver.find_element(By.CSS_SELECTOR,'li#reptile-paging-bottom-next > button').get_attribute('data-page')
if nextpage:
nextpage = json.loads(nextpage)
url = driver.current_url.split("?")[0]
url = f"{url}?l=gq&o={nextpage.get('o')}"
driver.get(url)
else:
break
- Checks for the "Next Page" button.
- Constructs the URL for the next page using JSON data.
- Navigates to the next page or exits the loop if there are no more pages.
Closing the Browser
python
Copy code
driver.quit()
- Closes the browser instance after the scraping process is complete.
Key Points
- Dynamic Loading: Uses Selenium to handle dynamically loaded content.
- Robust Element Selection: Uses
WebDriverWaitand multiple class checks to ensure elements are located correctly. - CSV Output: Saves scraped data in a structured format.
- Pagination Handling: Scrapes multiple pages by detecting and navigating to the next page.
Comments (0)
Leave a Reply
Log in to post a comment.