Tool for data extraction from websites. Based on `Web Scraper`, now support MySQL/Anti Lazy-Loading/CMD/Terminal/JS filter and more
This tool is based on original WEB SCRAPER with some more functions: 1. Support MySQL(v5.7+) database. 2. Start scraping from CMD/Terminal. 3. Anti-Lazy Loading. 4. JS Data Filter for data pre process. 5. Remove duplicate insertion.(Only works with https://github.com/hejiheji001/ArrestDB/) 6. Custom Columns*. 7. Easy Scrape 8. Dynamic Urls 9. Progress Tracking Please check the source code: https://github.com/hejiheji001/web-scraper-chrome-extension Please check the use cases: https://github.com/hejiheji001/web-scraper-chrome-extension/wiki Feel free to post issues, requests on my github and star it if you like it. *: If you use JS Data Filter and changed the structure of your final data, you can define an array of column names to match the new format of your data.
- (2020-11-13) Dantae Hiruma: The anti lazy loading function is god-send but for some reasons the element selection was really wonky for me, sometimes it works and sometimes it just doesn't, even on the same website. Shame, would have loved it if not for that one major flaw.
- (2020-02-05) Jack García: It looked good from the description, but I wasn't able to make it work with ArrestDB and MySQL, and the github repo is mostly dead. It looks like a personal project with no support. The other features worked, but the main purpose which was using MySQL didn't. IDK how to debug chrome extensions so...