$1,000.00 Fixed
We're building a competitive intelligence tool for e-commerce businesses and need an experienced Python Developer to create a robust web scraping system that monitors competitor pricing, product availability, and reviews across multiple online retailers. You'll develop Python scripts using BeautifulSoup and Selenium to extract structured data from websites, handle dynamic JavaScript-rendered content, implement rotation proxies to avoid blocking, and store collected data in a PostgreSQL database for analysis.
The scraper needs to monitor 50-100 products across 5-10 competitor websites, running automated daily crawls and detecting price changes, stock status updates, and new reviews. You'll implement error handling for site structure changes, create logging systems for monitoring scraper health, and build a simple dashboard (Flask or Django) where our team can view collected data. Anti-bot measures are common, so experience with proxy rotation, user-agent spoofing, and CAPTCHA handling is essential.
This is a 2-3 week project with clear deliverables and milestone payments. We're looking for a Python developer with proven web scraping experience, understanding of ethical scraping practices, and ability to build maintainable code that won't break when websites change. Post-delivery support for 2 weeks is included for adjustments and bug fixes. If your portfolio includes similar data extraction projects, we'd love to discuss the technical approach.
Key Responsibilities
Develop Python web scraper using BeautifulSoup and Selenium
Extract product data (prices, titles, descriptions, reviews, stock status)
Handle JavaScript-rendered dynamic content with Selenium
Implement proxy rotation and user-agent randomization
Build error handling for site structure changes
Store scraped data in PostgreSQL database
Create automated scheduling for daily scraping runs
Implement logging and monitoring system
Build simple Flask/Django dashboard for data viewing
Optimize scraper performance and efficiency
Document code and provide usage instructions
Handle edge cases and anti-scraping measures
Required Skills
Python Programming
Web Scraping
BeautifulSoup
Selenium WebDriver
Data Extraction
PostgreSQL
Flask or Django
Proxy Integration
HTML & CSS Parsing
Automation Scripts
Error Handling
API Development
Tools
Python 3.x
BeautifulSoup4
Selenium WebDriver
Requests library
PostgreSQL database
Flask or Django (for dashboard)
ChromeDriver or GeckoDriver
Git/GitHub (version control)
Proxy services (Bright Data, Oxylabs, etc.)
Cron or Task Scheduler (automation)
VS Code or PyCharm
Deliverables
Python web scraping scripts for 5-10 target websites
Automated data extraction for 50-100 products
PostgreSQL database schema and setup
Data storage and update mechanisms
Daily automated scraping schedule implementation
Simple web dashboard (Flask/Django) for viewing data
Proxy rotation and anti-blocking measures implemented
Error handling and logging system
Email notifications for scraping failures
Complete source code with documentation
Setup instructions and requirements.txt file
2 weeks post-delivery support for adjustments
- Proposal: 0
- Less than a month
