Microproject
Write a Python program which takes a single argument — a URL. Your program will use the Unix command curl to download the file at that URL, remove everything within <script></script> and <style></style> tags, and write the resulting file to the screen. Be sure to test it on a variety of URLs. (Look at the curl documentation to figure out how to follow redirects, this will make your life better.)
Main Project
Write a Python program that does the following:
- Asks the user to enter a URL;
- Downloads the contents of that URL using curl;
- Extracts features from the page, including at least:
- Headings
- Link text
- Image alt text
- Email addresses
- Phone numbers
- Uses unix tools to determine:
- word count
- top N words, sorted alphabetically
- Builds a structured summary including the above items and show it to the user;
- Sends the summary to an LLM using a prompt of your choice to analyze the structured summary. Some ideas:
- Ask the LLM to try to determine what the website is about.
- Modify the structured summary so that it is segmented by heading section and ask the LLM to verify that the contents of each section of the page matches the heading.
- Ask the LLM to try to predict any bias on the page.
For the LLM component, I recommend using the OpenRouter Python library. OpenRouter provides API access for a few LLMs, rate limited to a higher amount than you should use in this project… Sign up for an API key on their website, and skip providing any payment info. Free models are listed here: https://openrouter.ai/collections/free-models
Demo
Here’s a demo of my version of the project’s output.
Enter a URL: http://pizzatoday.com
--- Summary ---
Words: 8161
Headings:
Pizza Today
Featured
Latest Posts
Latest Recipes
Latest Podcasts
General
Topics
About Us
Contact Us
Links:
Emerald Media Network
Advertise
International Pizza Expo
Pizza Expo Columbus
Latest
View All Posts »
News
Press Releases
Podcasts
Recipes
Resources
[... truncated for space ...]
Images:
Image without alt-text
Image without alt-text
pizzeria women of influence
Image of Mirko D'Agata, Pizza Maker of the Year 2026.
image of World Pizza Games area at Pizza Expo 2026
2026 Pizza Industry Trends Report
Pizza Today Pizza Styles Guide Featured Image
pizzas at a pizza festival
Image of a vegan ricotta and squash pizza.
[... truncated for space ...]
Emails:
Phone Numbers:
Top words:
67 list
50 pizza
24 news
14 with
13 screen
12 only
10 dough
10 april
9 about
8 view
--- LLM Summary ---
This website appears to be an industry-focused publication about pizza, featuring news, recipes, podcasts, and resources for pizza makers and pizzeria businesses. It also promotes industry events, trends reports, and professional resources related to the pizza and restaurant industry.