An Intelligent System for Automatic Web Data Extraction with Self Repairing and Proactive Failure Detection Capability using Machine Learning

Abstract

Data is crucial to the growth of e-commerce in today s world of highly demanding hyperpersonalized consumer experiences. Data is collected using advanced web scraping newlinetechnologies with traditional and machine learning techniques from web pages, documents newline(e.g., pdf, .doc or .txt) and stored in a database. There is significant development in newlineextracting relevant product information using machine learning techniques and tools with newlinehigh precision and recall, while the web scraping engines still rely on location and layout of the website. This dependency on the location of the website or web page or specific objects within the web site fails since the location changes dynamically in the websites and requires the web scraping engine to be fixed and hence, does not adapt to dynamic changes in websites or website layout. This leads to data quality issues and impacts developer productivity as the effort to fix the changes in the web scraping engine is manual and labor intensive.