A Muti-resource Fair Scheudler of Hadoop
CSTR:
Author:
Affiliation:

Clc Number:

Fund Project:

  • Article
  • |
  • Figures
  • |
  • Metrics
  • |
  • Reference
  • |
  • Related
  • |
  • Cited by
  • |
  • Materials
  • |
  • Comments
    Abstract:

    Hadoop job schedulers typically use a single resource abstraction and resources are allocated at the level of fixed-size partition of the nodes, called slots. These job schedulers ignore the different demands of jobs and fair allocation of multiple types of resources, leading to poor performance in throughput and average job completion time. This paper studies and implements a Muti-resource Fair Scheduler (MFS) in Hadoop. MFS adopts the idea of Dominant Resource Fairness (DRF). It uses a demand vector to describe demands for resources of a job and allocates resources to the job according to the demand vector. MFS uses resources more efficiently and satisfies multiple jobs with heterogeneous demands for resources. Experiment results show that MFS has higher throughput and shorter average job completion time compared to Hadoop slot-based Fair Scheduler and Capacity Scheduler.

    Reference
    Related
    Cited by
Get Citation

MA Xiao-yan, HONG Jue. A Muti-resource Fair Scheudler of Hadoop[J]. Journal of Integration Technology,2012,1(3):66-71

Copy
Share
Article Metrics
  • Abstract:
  • PDF:
  • HTML:
  • Cited by:
History
  • Received:
  • Revised:
  • Adopted:
  • Online: January 04,2013
  • Published:
Article QR Code