Search for a command to run...
A Highly Effective Technique for Web Scraping
No comments yet. Be the first to comment.
After trying many journaling apps, I settled on Diarium for its rich, calendar-driven features like photo, audio, and mood attachments. However, I was frustrated when I switched to Linux and found it
A Self-Hosted Health Records Keeper
How the SDD Workflow plugin for Claude Code turns vague ideas into shipping software — with traceability built in. If you've ever shipped a feature that didn't match what was asked for, or merged code
Neural TTS for Linux at $0 cost
If you use Obsidian for note-taking, you know how tedious capturing content from applications on your desktop: copy, switch windows, paste, format. I wanted something faster: highlight text, hit a hot
In the fast-paced digital age, harnessing data has become crucial to understand patterns, make informed decisions, or simply collect information. Web scraping or web harvesting, is a technique that helps us do just that. This blog post aims to delve into the utility and efficiency of web scraping, leveraging Python and BeautifulSoup4, an excellent duo for the purpose.
Web scraping is a method used to extract large amounts of data from websites. This information is collected and then exported into a format that is more useful for the user, be it a.csv file or database.
Ethical and legal considerations are crucial when scraping a website, as not all websites allow it, and the typical workflow involves requesting, receiving, and extracting information.
Python has become the go-to language for developers involved in web scraping due to its simplicity and robust libraries, such as Beautiful Soup 4.
Beautiful Soup 4 is a Python library designed for parsing HTML and XML documents, converting them into parse trees that can be used to extract data easily.
BeautifulSoup4's powerful functionalities allow users to locate HTML elements quickly. It's possible to extract data by drilling down into tags or directly accessing attributes.
However, there's more to it than this: Web scraping has evolved beyond static HTML pages. It's now exceedingly common to encounter dynamic web pages driven by AJAX. To handle these, one can supplement BeautifulSoup4 with libraries like Selenium or Requests-HTML.
Be aware that the script worked fine today may not work next month as web pages are frequently updated. In that case, you will have to modify your code to get it working again.
First, write down in a note book the workflow:
what data you want?
carefully inspect the web page html code
Identify the tag, class, and div where the data you need is located.
write the code in steps, often ensuring that the code works as expected.
comment on each function as a code block so that others can understand.
Create a project directory:mkdir web_scraping
load the directory:cd web_scraping
virtualize the project:python -m venv .venv
activate the virtual environment:.venv/scripts/activate
pip install modules:
pip install requests BeautifulSoup4
import libraries:
import requests
from bs4 import BeautifulSoup
import os
import re
Here is the complete code with comments for easy understanding of each piece of code and its function. It scrapes the text content of a page from https://www.marrowmatters.com/Aplastic-Anemia.html and saves it as a markdown file.
# URL of the website
url = "https://www.marrowmatters.com/Aplastic-Anemia.html"
# Send a GET request to the URL
response = requests.get(url)
# Check if the request was successful (status code 200)
if response.status_code == 200:
# Parse the HTML content using BeautifulSoup
soup = BeautifulSoup(response.content, 'html.parser')
# Extract text within <h1>, <h2>, <h3>, and <p> tags
h1_texts = [h1.text.strip() for h1 in soup.find_all('h1')]
h2_and_p_texts = []
h3_and_p_texts = []
# Iterate through each <h2> tag
for h2 in soup.find_all('h2'):
h2_text = h2.text.strip()
p_texts = []
# Find all <p> tags following the current <h2> tag
next_sibling = h2.find_next_sibling()
while next_sibling and (next_sibling.name == 'p' or next_sibling.name == 'h3'):
if next_sibling.name == 'p':
p_texts.append(next_sibling.text.strip())
elif next_sibling.name == 'h3':
h3_text = next_sibling.text.strip()
h3_p_texts = []
# Find all <p> tags following the current <h3> tag
h3_next_sibling = next_sibling.find_next_sibling()
while h3_next_sibling and h3_next_sibling.name == 'p':
h3_p_texts.append(h3_next_sibling.text.strip())
h3_next_sibling = h3_next_sibling.find_next_sibling()
h3_and_p_texts.append((h3_text, h3_p_texts))
next_sibling = next_sibling.find_next_sibling()
h2_and_p_texts.append((h2_text, p_texts))
# Determine the filename based on the first <h1> content
filename = h1_texts[0] if h1_texts else "extracted_content"
# Replace invalid characters in the filename
filename = re.sub(r'[\\/:*?"<>|]', '_', filename)
# Create the 'data_extracted' directory if it doesn't exist
output_directory = "data_extracted"
os.makedirs(output_directory, exist_ok=True)
# Save the extracted content as a Markdown file
output_path = os.path.join(output_directory, f"{filename}.md")
with open(output_path, "w", encoding="utf-8") as file:
file.write("# " + "\n# ".join(h1_texts) + "\n\n")
for h2_text, p_texts in h2_and_p_texts:
file.write(f"## {h2_text}\n")
file.write("\n".join(f"{text}\n" for text in p_texts))
for h3_text, h3_p_texts in h3_and_p_texts:
file.write(f"### {h3_text}\n")
file.write("\n".join(f"{text}\n" for text in h3_p_texts))
print(f"Extraction and saving completed successfully. File saved as {filename}.md in {output_directory}")
else:
print(f"Failed to retrieve the webpage. Status code: {response.status_code}")
Web scraping with Python and BeautifulSoup4 is crucial for efficiently accessing needed information from unstructured data. Enjoy the power of data at your fingertips and be sure to use it responsibly.
Your web scraping journey now awaits. Remember, make sure to respect your data sources, understand what you need, and keep refining your code. With time and practice, you will become proficient at effectively harvesting data from the web. Remember, good web scraping isn’t fulfilled by having data but by having the right data. Happy web scraping!