An Efficient Big Data Analytics Framework

Abstract

quotThe concept of Big Data represents more than just an increase in data volume; it embodies newlinea fundamental shift in how data is processed and analyzed. Traditional data systems, newlinedesigned for smaller datasets, are ill-equipped to manage the diverse, voluminous mix newlineof structured, unstructured, and semi-structured data characteristic of Big Data. The newlinevelocity and variety at which this data is generated require not only enhanced hardware newlinecapabilities but also sophisticated algorithms and advanced data processing techniques. newlineThis growth represents a continuous enhancement of existing technologies, addressing technical challenges without fundamentally altering the data management landscape. There is no gradual upscaling but a technological shift from data warehouses to data lakes, relational databases to NoSQL databases, adoption of real-time data streaming over batch newlineprocessing. Additionally, there is a shift from coding-centric methodologies to newlinethe orchestration of complex software tools, each with a wide range of features and newlinecompatibility considerations. newline newlineThis thesis addresses the challenges of designing and implementing a generic Big newlineData Analytics Framework that effectively integrates the unique and complex features newlineof modern Big Data tools while ensuring their compatibility and seamless orchestration. newlineThe research focuses on developing a fuzzy suitability index and a fuzzy compatibility newlineindex specifically tailored for Big Data tools, deliberately excluding traditional tools newlinetypically used in smaller data systems. newlineThe proposed framework is intended to guide Big Data engineers in selecting and newlineintegrating tools such as Web Scrapers, ETL tools, NoSQL databases, and Data Lakes. newlineA key focus is on addressing the multifaceted compatibility issues that arise during the newlineorchestration of these tools. By providing a structured approach to tool selection and newlineintegration, this work aims to streamline the development of efficient and effective Big newlineData Analytics Frameworks.quot newline

Description

Keywords

Citation

item.page.endorsement

item.page.review

item.page.supplemented

item.page.referenced