3単語共起フィルタリングによる有害文書分類手法と大規模データ処理

カテゴリ: 論文誌(論文単位)

グループ名: 【C】電子・情報・システム部門

発行日: 2014/01/01

タイトル(英語): Text Filtering for Harmful Document Classification Method Using Three words Co-occurrence and Large-scale Data Processing

著者名: 大塚　孝信（名古屋工業大学大学院情報工学専攻），Deyue Deng（名古屋工業大学大学院産業戦略工学専攻），伊藤　孝行（名古屋工業大学大学院産業戦略工学専攻）

著者名(英語): Takanobu Otsuka (Doctor of Information Science, Nagoya Institute of Technology), Deyue Deng (Master of Techno Busuiness School, Nagoya Institute of Technology), Takayuki Ito (Master of Techno Busuiness School, Nagoya Institute of Technology)

キーワード: テキストフィルタリング，3単語共起，大規模データ処理　　Text filtering，3words co-occurrence，Large-scale data processing

要約(英語): In recent years, young people are increasingly using internet. However, problem of received information to adversely affect the young people. Therefore, we propose a method to automatically classify to harmful sentences. Recently research on the information filtering have been improved the performance of the filter by introducing the co-occurrence information. Extended by two words co-occurrence information, which is commonly studied in this study we have created a training data using the co-occurrence information with three words. However, compared with the words two co-occurrence information processing time becomes a problem is increased the amount of training data. In addition, we have found that noise is caused by increase the co-occurrences, exceeds the number of double-precision floating-point calculation. We realized the processing speed by implementing a text filtering system with three-word co-occurrence using a Bayesian filter, to parallelize fast MyISAM database. In addition, by removing the noise caused by the increase in the number of co-occurrence BigDecimal, We realized the high F value.

本誌: 電気学会論文誌C（電子・情報・システム部門誌） Vol.134 No.1 （2014） 特集：スマート社会を支える電子回路技術

本誌掲載ページ: 168-175 p

原稿種別: 論文／日本語

電子版へのリンク: https://www.jstage.jst.go.jp/article/ieejeiss/134/1/134_168/_article/-char/ja/