Quality based evaluation and selection of external web data source for a data warehouse

Abstract

The Data Warehouse (DW) is a powerful technology to store and analyse huge volumes of historical operational data of the enterprise to support strategic decision-making in organizations[1]. Using only the enterprise s internal data in the DW has become insufficient to gain valuable insights into the market, competitors, products, or customers for comprehensive and unbiased decision making[2]. External data incorporation in the DW provides the organization a wide-angle lens in decision-making. To maintain their competitive edge in the business climate of intense competition, organizations need to incorporate external data to provide holistic information for decision-making in a timely and accurate manner [3]. However, acquiring external data is a very challenging task[4]. It has been reported in recent works [5], [6] that most organizations rely heavily the internal and structured data of the enterprise alone. This results in incomplete view of reality [5] and thus, biased decision-making at the severe risk of being led behind competitive forces [3]. World Wide Web (WWW) provides a viable, simpler-to-load and low-cost solution for accessing external data for the DW[7]. However, using Web as a data source for the DW presents many challenges due to the vast amount of information, dynamicity of web sources [8], heterogenous formats of data[7] and poor quality of Web data[9]. So, ironically, despite the availability of many Web data sources, the main and often only used web data source is that of geographical information systems (GIS) [3]. Since the DW plays a principal role in the decision-making process, it is crucial to maintain the quality of a Data Warehouse [10] which in turn, is dependent on... newline