論文

2021年

Auto-Scaling System in Apache Spark Cluster Using Model-Based Deep Reinforcement Learning

Studies in Computational Intelligence
  • Thonglek, K.
  • ,
  • Ichikawa, K.
  • ,
  • Sangkeettrakarn, C.
  • ,
  • Piyatumrong, A.

906
開始ページ
347
終了ページ
360
記述言語
掲載種別
研究論文(学術雑誌)
DOI
10.1007/978-3-030-58930-1_23
出版者・発行元
Studies in Computational Intelligence

Real-time processing is a fast and prompt processing technology that needs to complete the execution within a limited time constraint almost equal to the input time. Executing such real-time processing needs an efficient auto-scaling system which provides sufficient resources to compute the process within the time constraint. We use Apache Spark framework to build a cluster which supports real-time processing. The major challenge of scaling Apache Spark cluster automatically for the real-time processing is how to handle the unpredictable input data size and also the unpredictable resource availability of the underlying cloud infrastructure. If the scaling-out of the cluster is too slow then the application can not be executed within the time constraint as a result of insufficient resources. If the scaling-in of the cluster is slow, the resources are wasted without being utilized, and it leads less resource utilization. This research follows the real-world scenario where the computing resources are bounded by a certain number of computing nodes due to limited budget as well as the computing time is limited due to the nature of near real-time application. We design an auto-scaling system that applies a deep reinforcement learning technique, DQN (Deep Q-Network), to improve resource utilization efficiently. Our model-based DQN allows to automatically optimize the scaling of the cluster, because the DQN can autonomously learn the given environment features so that it can take suitable actions to get the maximum reward under the limited execution time and worker nodes.

リンク情報
DOI
https://doi.org/10.1007/978-3-030-58930-1_23
URL
http://www.scopus.com/inward/record.url?eid=2-s2.0-85097946874&partnerID=MN8TOARS
Scopus Citedby
https://www.scopus.com/inward/citedby.uri?partnerID=HzOxMe3b&scp=85097946874&origin=inward

エクスポート
BibTeX RIS