Consolidation and Scheduling Technique for Improving Job Execution om Hadoop Cluster
Loading...
Date
item.page.authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
The proposed research work focuses on developing a flexible and reliable
newlinetechnology for the consolidation of small files and the scheduling technique to
newlineimprove the job execution in Hadoop Cluster. Though, the Hadoop Distributed
newlineFile System (HDFS) is storing data in large files, significant amount of small
newlinesize files is deployed in HDFS for various operations as demanded by the
newlineclients. The small files are required to load frequently as the clients are
newlineperforming the file operations. The namenode is loaded heavily when the clients
newlinedemand file read or file write operations. Hence, the smaller size files are
newlineaffecting the scalability and performance of the HDFS system. It would increase
newlinethe overload with the already loaded namenode in the HDFS. This research proposes the Small File Consolidation (SAFICON) algorithm
newlinewhich consolidates the small size files into a large size file in order to reduce
newlinethe storage space utilization. The SAFICON has designed and experimented
newlineunder various scenarios for read and write operation to ensure the accuracy of
newlinethe system. The SAFICON provides a better average throughput for the read
newlineand write operation on the various numbers of DataNodes. The implementation
newlineresults of SAFICON are compared and analysed with the existing methods such
newlineas Distributed Dynamic Hashing (DDH), Round Robin (RR), Shortest Job First
newline(SJF) and First Come First Serve (FCFS). Each result is taken from the average
newlineof 100 simulation runs. The implementation result shows that the proposed
newlineSAFICON provides efficient storage space utilization than the existing
newlinetechniques. Also, it helps to increase the performance through throughput and
newlineturnaround time which is superior to the existing techniques. The experimental evaluation has demonstrated that the proposed technique has
newlinereduced the NameNode memory consumption and has improved the efficiency
newlineof storing and accessing small files in HDFS. It is further observed that the
newlineprocessing performance of small files has been improved up to 94.62% by the
newlineproposed approach.