基于Spark的组合分类器链多标签分类方法

doi:10.3969/j.issn.0253-2778.2017.04.010

中国科学技术大学学报 ›› 2017, Vol. 47 ›› Issue (4): 350-357.DOI: 10.3969/j.issn.0253-2778.2017.04.010

基于Spark的组合分类器链多标签分类方法

王进，王鸿，夏翠萍，欧阳卫华，陈乔松，邓欣

重庆邮电大学计算智能重庆市重点实验室，重庆 400065

收稿日期:2016-08-28 修回日期:2016-12-08 出版日期:2017-04-30 发布日期:2017-04-30
通讯作者: 王进
作者简介:王进（通讯作者），男，1979年生，博士/教授. 研究方向：数据挖掘、人工智能. E-mail: wangjin@cqupt.edu.cn
基金资助:
重庆市基础与前沿研究计划(cstc2014jcyjA40001，cstc2014jcyjA40022)，重庆教委科学技术研究项目（KJ1400436）资助.

Ensembles of classifier chains for multi-label classification based on Spark

WANG Jin, WANG Hong, XIA Cuiping, OUYANG Weihua, CHEN Qiaosong, DENG Xin

Chongqing Key Laboratory of Computational Intelligence, Chongqing University of Posts and Telecommunications, Chongqing 400065, China

Received:2016-08-28 Revised:2016-12-08 Online:2017-04-30 Published:2017-04-30

摘要/Abstract

摘要： 随着数据挖掘技术在现实问题中的广泛应用，多标签学习现已成为数据挖掘技术中的一个研究热点.组合分类器链(ECC)算法是一种性能较好的多标签分类方法，其分类效果好、准确度高，但该算法的时空复杂度较高，不能适应大规模多标签数据分类任务.为此提出了一种基于Spark的组合分类器链多标签分类方法，将串行组合分类器链算法的各步骤进行了并行化实现.通过单机实验和集群并行化实验，证明该方法对大规模多标签数据集具有良好的适应能力和加速比，且分类效果不输于传统的串行多标签分类方法.

关键词: 多标签学习, 组合分类器链, Apache Spark, MapReduce

Abstract: With the wide application of data mining technology, multi-label learning has become a hot topic in the data mining domain. Although ensembles of classifier chains (ECC) algorithm is a multi-label learning method which is effective and accurate, its complexity of time and space is so high that it cannot adapt to the large-scale multi-label classification tasks. A new algorithm named Spark ensembles of classifier chains(S-ECC) was proposed based on Spark platform on which a parallel implementation was conducted of each step of the sequential ECC algorithm. The test results in stand-alone and cluster environments show that S-ECC has a good adaptability to large-scale data with a high speedup, and that it is no less capable than the traditional sequential program.

Key words: multi-label learning, ECC, Apache Spark, MapReduce

中图分类号:

TP391

王进，王鸿，夏翠萍，欧阳卫华，陈乔松，邓欣. 基于Spark的组合分类器链多标签分类方法[J]. 中国科学技术大学学报, 2017, 47(4): 350-357.

WANG Jin, WANG Hong, XIA Cuiping, OUYANG Weihua, CHEN Qiaosong, DENG Xin. Ensembles of classifier chains for multi-label classification based on Spark[J]. Journal of University of Science and Technology of China, 2017, 47(4): 350-357.

参考文献

［1］
ZHANG M L, ZHOU Z H. A review on multi-label learning algorithms[J]. IEEE Transactions on Knowledge and Data Engineering, 2014, 26(8): 1819-1837.
[2] READ J, PFAHRINGER B, HOLMES G, et al. Classifier chains for multi-label classification[J]. Machine Learning, 2011, 85(3): 333-359.
[3] ZAHARIA M, CHOWDHURY M, DAS T, et al. Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing[C]// Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation. San Hose, USA: USENIX Association, 2012: 141-146.
[4] ZHU B, MARA A, MOZO A. CLUS: Parallel subspace clustering algorithm on spark[A]// New Trends in Databases and Information Systems[M]. Springer, 2015, 539:175-185.
[5] MAILLO J, RAMREZ S, TRIGUERO I, et al. kNN-IS: An iterative spark-based design of the k-nearest neighbors classifier for big data[J]. Knowledge-Based Systems, 2016, 117: 3-15； doi: 10.1016/j.knosys.2016.06.012.
[6] KIM H, PARK J, JANG J, et al. DeepSpark: Spark-based deep learning supporting asynchronous updates and Caffe compatibility[J/OL]. https://arxiv.org/abs/1602.08191v1,2016.03.08, 2016: arXiv:1602.08191v1.
[7] DUAN M X, LI K L, TANG Z, et al. Selection and replacement algorithms for memory performance improvement in Spark[J]. Concurrency & Computation Practice & Experience, 2015, 28(8): 2473-2486.
[8] TSOUMAKAS G, KATAKIS I, VLAHAVAS I. Mining multi-label data[A]// Data Mining and Knowledge Discovery Handbook[M]. Springer, 2009: 667-685.
[9] READ J, PFAHRINGER B, HOLMES G. Multi-label classification using ensembles of pruned sets[C]// Proceedings of the 8th International Conference on Data Mining. Pisa, Italy: IEEE Computer Society, 2008: 995-1000.
[10] TSOUMAKAS G, SPYROMITROS-XIOUFIS E, VILCEK J, et al. Mulan: A Java library for multi-label learning[J]. Journal of Machine Learning Research, 2011, 12(2): 2411-2414.
[11] MENCA E L, FRNKRANZ J. Efficient pairwise multilabel classification for large-scale problems in the legal domain[C]// European Conference on Machine Learning & Knowledge Discovery in Databases. Antwerp, Belgium: Springer, 2008: 50-65.
[12] SNOEK C G M, WORRING M, VAN GEMERT J C, et al. The challenge problem for automated detection of 101 semantic concepts in multimedia[C]// Proceedings of the 14th ACM International Conference on Multimedia. Santa Barbara, USA: ACM Press. 2006: 421-430.
[13] CHUA T S, TANG J, HONG R, et al. NUS-WIDE: A real-world web image database from National University of Singapore[C]// Proceedings of the ACM International Conference on Image and Video Retrieval. Santorini, Greece: ACM Press, 2009: doi:10.1145/1646396.1646452.
[14] SPYROMITROS-XIOUFIS E, PAPADOPOULOS S, KOMPATSIARIS I Y, et al. A comprehensive study over VLAD and product quantization in large-scale image retrieval[J]. IEEE Transactions on Multimedia, 2014, 16(6): 1713-1728.
[15] ZHANG M L, WU L. LIFT: Multi-label learning with label-specific features[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2015, 37(1): 107-20.
[16] FRNKRANZ J, HLLERMEIER E, MENCA E L, et al. Multilabel classification via calibrated label ranking[J]. Machine Learning, 2008, 73(2): 133-153.
[17] TSOUMAKAS G, KATAKIS I, VLAHAVAS I. Random k-labelsets for multilabel classification[J]. IEEE Transactions on Knowledge & Data Engineering, 2011, 23(7): 1079-1089.

[1]	苏浩，刘其成，牟春晓. 基于MapReduce的商品评论热点发现算法研究[J]. 中国科学技术大学学报, 2019, 49(2): 112-118.
[2]	蔡勇，陈红梅，. MapReduce环境下基于概念分层的概念格并行构造算法[J]. 中国科学技术大学学报, 2018, 48(4): 275-283.
[3]	张其龙，邓维斌，胡峰，瞿原，胡宗容. 一种基于朴素贝叶斯的校准标签排序方法[J]. 中国科学技术大学学报, 2018, 48(1): 65-74.
[4]	陈远，汪璟玢. 分布式RDF关键词近似搜索方法[J]. 中国科学技术大学学报, 2017, 47(10): 823-836.

基于Spark的组合分类器链多标签分类方法

Ensembles of classifier chains for multi-label classification based on Spark

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 4

编辑推荐

Metrics

本文评价