論文

査読有り
2016年12月8日

Accuracy Improvement for Backup Tasks in Hadoop Speculative Algorithm

2016 IEEE INTERNATIONAL CONFERENCE ON COMPUTER AND INFORMATION TECHNOLOGY (CIT)
  • Worachate Apichanukul
  • ,
  • Jun Kawahara
  • ,
  • Shoji Kasahara

開始ページ
500
終了ページ
507
記述言語
英語
掲載種別
研究論文(国際会議プロシーディングス)
DOI
10.1109/CIT.2016.17
出版者・発行元
IEEE

Hadoop suffers from the issue of stragglers in which some tasks with unusual long execution time delay the whole job. Hadoop relieves the straggler problem by launching backup tasks, that are cloned from original tasks and executed on server nodes different from those executing original tasks. A speculative algorithm is responsible for classifying straggler tasks and launching backup tasks. Although launching backup tasks can solve the problem, it requires additional resources to run. Existing algorithms wrongly classify and start many wasteful backup tasks, which reduces resource utilization. This paper proposes an algorithm called Accuracy Improvement for Backup Task (AIBT) in Hadoop speculative algorithm. In the AIBT algorithm, a task is divided into phases, and the execution time of the task is estimated from progress rates of the phases. Moreover, AIBT uses execution time information both from other finished tasks and its own to improve the accuracy of the estimated values by changing the weight dynamically. We implement the AIBT algorithm in Hadoop and evaluate its performance for four job types: Wordcount, KMean clustering, PageRank, and Inverted Index. Numerical results show that AIBT significantly reduces the number of unnecessary and wrong backup tasks compared with the existing algorithms.

リンク情報
DOI
https://doi.org/10.1109/CIT.2016.17
Web of Science
https://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcAuth=JSTA_CEL&SrcApp=J_Gate_JST&DestLinkType=FullRecord&KeyUT=WOS:000411239100072&DestApp=WOS_CPL
URL
https://grid.chu.edu.tw/sc2-2016/
ID情報
  • DOI : 10.1109/CIT.2016.17
  • Web of Science ID : WOS:000411239100072

エクスポート
BibTeX RIS