An Efficient Big Data Analytics Framework
Loading...
Date
item.page.authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
quotThe concept of Big Data represents more than just an increase in data volume; it embodies
newlinea fundamental shift in how data is processed and analyzed. Traditional data systems,
newlinedesigned for smaller datasets, are ill-equipped to manage the diverse, voluminous mix
newlineof structured, unstructured, and semi-structured data characteristic of Big Data. The
newlinevelocity and variety at which this data is generated require not only enhanced hardware
newlinecapabilities but also sophisticated algorithms and advanced data processing techniques.
newlineThis growth represents a continuous enhancement of existing technologies, addressing technical challenges without fundamentally altering the data management landscape. There is no gradual upscaling but a technological shift from data warehouses to data lakes, relational databases to NoSQL databases, adoption of real-time data streaming over batch
newlineprocessing. Additionally, there is a shift from coding-centric methodologies to
newlinethe orchestration of complex software tools, each with a wide range of features and
newlinecompatibility considerations.
newline
newlineThis thesis addresses the challenges of designing and implementing a generic Big
newlineData Analytics Framework that effectively integrates the unique and complex features
newlineof modern Big Data tools while ensuring their compatibility and seamless orchestration.
newlineThe research focuses on developing a fuzzy suitability index and a fuzzy compatibility
newlineindex specifically tailored for Big Data tools, deliberately excluding traditional tools
newlinetypically used in smaller data systems.
newlineThe proposed framework is intended to guide Big Data engineers in selecting and
newlineintegrating tools such as Web Scrapers, ETL tools, NoSQL databases, and Data Lakes.
newlineA key focus is on addressing the multifaceted compatibility issues that arise during the
newlineorchestration of these tools. By providing a structured approach to tool selection and
newlineintegration, this work aims to streamline the development of efficient and effective Big
newlineData Analytics Frameworks.quot
newline