Price comparison website and custom scraper – a web application to compare prices in multiple USA shops and notify end users when prices go down. Script monitors 6pm.com, amazon.com, bestbuy.com, ebay.com, homedepot.com, macys.com, microsoft.com/en-us/store/b/home, newegg.com, rakuten.com, walmart.com and couple dozens more shops are scraped.
System admin loads the list of urls into csv file. The system loads them into the database and links shops and product so that one product may be assigned to multiple shops. It allows to view product prices at multiple shops at the same time. After that the system scrapes product data: price, title, category, shipping cost, product availability.
Scraping is done with multiple threads with the help of PHP PCNTL Functions. Scraper uses proxy and time delay between queries in order to avoid ban.
In backend admin can set up a schedule to run scraper and set a timeout delay for every hour of work. For instance, at night timeout delay is longer (and scraping speed is lower, accordingly) and during a day the speed is max.
The system watches price changes and notifies user when the price goes down.
Price for each product goes through the following process:
- adding shipment cost to destination specified by user
- checking the price agains coupons (added by user before)
- re-calculating price for hidden discounts (some shops give discount when product is added to shopping cart).
When price goes down to amount previously set by user (for instance, #1), this product is added to a section for the most attractive deals and prices where user can watch them. So even the price hasn’t changes but shipping cost changed, user will be notified.