Consolidation and Scheduling Technique for Improving Job Execution om Hadoop Cluster

Abstract

The proposed research work focuses on developing a flexible and reliable newlinetechnology for the consolidation of small files and the scheduling technique to newlineimprove the job execution in Hadoop Cluster. Though, the Hadoop Distributed newlineFile System (HDFS) is storing data in large files, significant amount of small newlinesize files is deployed in HDFS for various operations as demanded by the newlineclients. The small files are required to load frequently as the clients are newlineperforming the file operations. The namenode is loaded heavily when the clients newlinedemand file read or file write operations. Hence, the smaller size files are newlineaffecting the scalability and performance of the HDFS system. It would increase newlinethe overload with the already loaded namenode in the HDFS. This research proposes the Small File Consolidation (SAFICON) algorithm newlinewhich consolidates the small size files into a large size file in order to reduce newlinethe storage space utilization. The SAFICON has designed and experimented newlineunder various scenarios for read and write operation to ensure the accuracy of newlinethe system. The SAFICON provides a better average throughput for the read newlineand write operation on the various numbers of DataNodes. The implementation newlineresults of SAFICON are compared and analysed with the existing methods such newlineas Distributed Dynamic Hashing (DDH), Round Robin (RR), Shortest Job First newline(SJF) and First Come First Serve (FCFS). Each result is taken from the average newlineof 100 simulation runs. The implementation result shows that the proposed newlineSAFICON provides efficient storage space utilization than the existing newlinetechniques. Also, it helps to increase the performance through throughput and newlineturnaround time which is superior to the existing techniques. The experimental evaluation has demonstrated that the proposed technique has newlinereduced the NameNode memory consumption and has improved the efficiency newlineof storing and accessing small files in HDFS. It is further observed that the newlineprocessing performance of small files has been improved up to 94.62% by the newlineproposed approach.

Description

Keywords

Citation

item.page.endorsement

item.page.review

item.page.supplemented

item.page.referenced