Reporting Automation · Workflow Improvement
A configurable Python workflow for collecting website updates and publishing them as scheduled RSS feeds.
Tools: Python, Selenium, feedgen, JSON configuration, cron
The Problem
RSS feeds provide a convenient way to follow website updates, but not every website publishes one. Without a feed or another structured notification method, keeping up with new content can require repeatedly visiting and checking the same pages.
A useful alternative is a repeatable process that collects the latest website content and converts it into a standardized feed that can be read by an RSS reader.
The Approach
I designed the project as a configurable workflow rather than a script tied to one specific website. Website details and scraping rules are stored in separate JSON configuration files, while the main Python script handles browser automation, content extraction, and feed generation.
Each configuration file can define information such as the website URL, feed title, description, output filename, and the CSS selectors used to locate article titles, links, descriptions, images, and dates.
Why configuration matters: Separating website-specific settings from the application logic makes the workflow easier to reuse and update without modifying the core Python code for every source.
The Solution
The finished workflow uses Selenium with a headless Firefox browser to
load a configured website and locate its latest content. It extracts the
relevant fields from each item and uses the Python
feedgen library to create a valid RSS feed.
The generated feed is saved as an XML file that can be hosted and added to an RSS reader. The same Python script can process different websites by receiving the appropriate configuration filename as a command-line argument.
The workflow can also be scheduled with cron so the feeds are regenerated automatically. A shell script can process every JSON configuration file in the project directory during a single scheduled run.
Project Outcomes
- Automating a recurring content-collection process
- Using browser automation to collect structured website data
- Separating configuration from application logic
- Generating standardized, reusable output files
- Scheduling a Python workflow to run without manual intervention
- Designing one process to support multiple data sources
Project Structure
The project separates the main script, website configurations, and generated feeds into distinct locations:
automated-feed-generator/
├── automated_feed_generator.py
├── config/
│ └── example-site.json
└── feeds/
└── example-site.xmlThis structure keeps source-specific settings organized and prevents generated output from being mixed with the application code.
Tools Used
- Python for the primary workflow and file handling
- Selenium for loading pages and extracting content
- Firefox WebDriver for headless browser automation
- feedgen for creating RSS XML output
- JSON for reusable website configuration
- cron for scheduled execution
Possible Extensions
A production version of this workflow could be adapted for different content sources, validation requirements, hosting environments, or delivery methods. Depending on the use case, additional features might include logging, failure notifications, duplicate detection, automated deployment, or a lightweight interface for managing configurations.
Read the full technical article →
Have a similar workflow you want to improve?
Tell me about the current process, the data involved, and what you would like the finished solution to produce.