Hadoop job schedulers typically use a single resource abstraction and resources are allocated at the level of fixed-size partition of the nodes, called slots. These job schedulers ignore the different demands of jobs and fair allocation of multiple types of resources, leading to poor performance in throughput and average job completion time. This paper studies and implements a Muti-resource Fair Scheduler (MFS) in Hadoop. MFS adopts the idea of Dominant Resource Fairness (DRF). It uses a demand vector to describe demands for resources of a job and allocates resources to the job according to the demand vector. MFS uses resources more efficiently and satisfies multiple jobs with heterogeneous demands for resources. Experiment results show that MFS has higher throughput and shorter average job completion time compared to Hadoop slot-based Fair Scheduler and Capacity Scheduler.
MA Xiao-yan, HONG Jue. A Muti-resource Fair Scheudler of Hadoop[J]. Journal of Integration Technology,2012,1(3):66-71Copy