Data Mining Architecture and Algorithms for Big Earth Observation Data

Abstract

Earth Observation data from satellites and ground-based observations available in the archives have many hidden insights and have immense potential for deriving new applications and knowledge. Earth Observation data is characterized by velocity, veracity, variety, and volume. These characteristics, qualify it as Big Data. This data is also called Big Earth Observation Data (BEOD). Information/Knowledge extraction from BEOD is a complex task, this research aims to address some of the associated challenges. newline newlineThis research work proposes Hadoop-based architecture using a big data technology stack and its customization for the execution of the data mining algorithms on Big Earth Observation Data. newline newlineThe major research challenges include handling data at scale for storage, access, and retrieval, spatial indexing for fast access of geo-spatial data, and addressing issues of skew and data shuffle in MapReduce for providing fault tolerance, and flexibility to users for handling volumes of data. newline newlineA Study of the existing literature and exploration of available Big-data architectures suggest that Geomesa is observed to be one of the most appropriate architectures due to its capability of handling data at scale and ease of interfacing with other big-data tools. newline newlineSome of the customizations include selected regions of interest-based processing and support for data from heterogeneous sources and formats. Data access and retrieval performance of the architecture is evaluated with spatio-temporal queries. Results reveal that, with the rise in data volume, the time required to process the query is observed to be nearly linear. newline newlineThe proposed architecture is scalable in terms of storage and processing, as adding more nodes enhances the system s processing power. Secondly, scalable data mining algorithms are developed, and their performance is tested on the proposed architecture. Traditional data mining algorithms are time inefficient and are not to scale in a distributed environment. newline newline newlineFor spatial data clustering, an efficient comp