高级检索

基于LOGO计算框架的新型分布式机器学习算法库

A New Distributed Machine Learning Algorithm Library Based on LOGO Computing Framework

  • 摘要: LOGO是一种“局部-全局(LOcal-GlObal)”式的分布式计算框架,与流行的MapReduce计算框架不同,LOGO框架下的大数据分布式计算分两步完成:LO操作在节点虚拟机内运行串行算法,完成一个随机样本块的独立计算,产生局部计算结果;GO操作将所有局部结果上传到主节点,在主节点内对局部结果做集成,得到大数据的近似计算结果。LOGO计算框架在执行迭代算法时,消除了节点间的数据通信,极大地提高了分布式计算的效率,降低了内存需求,提高了数据扩展性。本文提出一种新的基于LOGO计算框架的分布式机器学习算法库,由LO操作执行的串行算法和GO操作执行的集成算法两部分组成。LO操作直接执行已有的机器学习串行算法,无须按MapReduce编程模型对算法进行重写,GO操作对串行计算结果进行集成。本文阐述了LOGO分布式计算的原理、算法库架构、串行算法封装和GO操作集成策略,展示了Spark实现、App应用开发和多种算法测试结果。

     

    Abstract: Unlike the popular MapReduce computing framework, LOGO is a new distributed computing framework using a LOcal-GlObal computing paradigm. Under the LOGO framework, big data distributed computing is completed in two steps. The LO operation runs a serial algorithm in a number of slave nodes or virtual machines to process independently the random sample data blocks, generating local results. The GO operation uploads all local results to the master node and integrate them to obtain the approximate result of the big data set. The LOGO computing framework eliminates data communication between nodes during iterations of the algorithm, greatly improving computing efficiency, reducing memory requirements, and enhancing data scalability. This article proposes a new distributed machine learning algorithm library under the LOGO computing framework. A new distributed computing is divided into two parts: the serial algorithm executed by the LO operation and the ensemble algorithm executed in the GO operation. The LO operation can directly execute existing serial machine learning algorithms without the need to rewrite them according to MapReduce. The GO operation executes ensemble algorithms of different kinds depending on the ensemble tasks. In this article, the principle of LOGO distributed computing is introduced first, followed by the algorithm library structure, the method for packaging existing serial algorithms and the ensemble strategy. Finally, implementation in Spark, App development, and the results of performance tests for various algorithms are demonstrated.

     

/

返回文章
返回