論文

査読有り
2021年9月29日

Feature Selective Likelihood Ratio Estimator for Low- and Zero-frequency N-grams

The 8th International Conference on Advanced Informatics: Concepts, Theory and Applications (ICAICTA 2021)
  • Masato Kikuchi
  • ,
  • Mitsuo Yoshida
  • ,
  • Kyoji Umemura
  • ,
  • Tadachika Ozono

記述言語
英語
掲載種別
研究論文(国際会議プロシーディングス)
DOI
10.1109/icaicta53211.2021.9640293
出版者・発行元
IEEE

In natural language processing (NLP), the likelihood ratios (LRs) of N-grams are often estimated from the frequency information. However, a corpus contains only a fraction of the possible N-grams, and most of them occur infrequently. Hence, we desire an LR estimator for low- and zero-frequency N-grams. One way to achieve this is to decompose the N-grams into discrete values, such as letters and words, and take the product of the LRs for the values. However, because this method deals with a large number of discrete values, the running time and memory usage for estimation are problematic. Moreover, use of unnecessary discrete values causes deterioration of the estimation accuracy. Therefore, this paper proposes combining the aforementioned method with the feature selection method used in document classification, and shows that our estimator provides effective and efficient estimation results for low- and zero-frequency N-grams.

リンク情報
DOI
https://doi.org/10.1109/icaicta53211.2021.9640293
ID情報
  • DOI : 10.1109/icaicta53211.2021.9640293

エクスポート
BibTeX RIS