An incremental cost-sensitive support vector machine

doi:10.3969/j.issn.0253-2778.2016.09.003

Abstract

Abstract: Cost-sensitive learning is an important field in machine learning, which widely exists in real-world applications, such as cancer diagnosis, credit application, etc. Cost-sensitive support vector machine proposed by Masnadi et al. handles cost-sensitive problems through making the hinge loss function cost-sensitive, which has better generalization accuracy than other traditional cost-sensitive algorithms. In practice data are obtained one batch after another. Conventional batch algorithms would waste a lot of time when appending samples, because they should re-train the model from scratch. To make the cost-sensitive support vector machine more practical in on-line learning problems, an incremental cost-sensitive support vector machine algorithm was proposed, which can directly update the trained model without re-training it from scratch when appending samples. Experiment study on several datasets show that our algorithm is significantly more efficient than batch algorithms of the cost-sensitive support vector machine.

Key words: on-line learning, incremental learning, cost sensitive learning, support vector machine

CLC Number:

TP181

QUAN Xin, GU Yuanhua, ZHENG Guansheng, GU Bin. An incremental cost-sensitive support vector machine[J]. Journal of University of Science and Technology of China, 2016, 46(9): 727-735.

References

［1］
SHENG V S, LING C X. Thresholding for making classifiers cost-sensitive[C]// Proceedings of the 21st National Conference on Artificial Intelligence. Boston: AAAI Press, 2006: 476.
[2] PARK Y J, CHUN S H, KIM B C. Cost-sensitive case-based reasoning using a genetic algorithm: Application to medical diagnosis[J]. Artificial Intelligence in Medicine, 2011, 51(2): 133-145.
[3] ELKAN C. The foundations of cost-sensitive learning[C]// Proceedings of the 17th International Joint Conference on Artificial Intelligence. San Francisco: Morgan Kaufmann Publishers, 2001: 973-978.
[4] DOMINGOS P. Metacost: A general method for making classifiers cost-sensitive[C]// Proceedings of the 5th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. Lisbon: ACM Press, 1999: 155-164.
[5] CHAWLA N V, BOWYER K W, HALL L O, et al. SMOTE: Synthetic minority over-sampling technique[J]. Journal of Artificial Intelligence Research, 2011, 16(1): 321-357.
[6] DRUMMOND C, HOLTE R C. C4. 5, class imbalance, and cost sensitivity: Why under-sampling beats over-sampling[C]// Proceedings of the LCML Workshop on Learning from Imbalanced Datasets II. 2003: 1-8.
[7] MARGINEANTU D D, DIETTERICH T G. Bootstrap methods for the cost-sensitive evaluation of classifiers[R]. Corvallis, OR: Oregon State University, 2000.
[8] TING K M. A comparative study of cost-sensitive boosting algorithms[C]// Proceedings of the 17th International Conference on Machine Learning. San Francisco: Morgan Kaufmann, 2000: 983-990.
[9] TING K M. An instance-weighting method to induce cost-sensitive trees[J]. IEEE Transactions on Knowledge and Data Engineering, 2002, 14(3): 659-665.
[10] ZHOU Z H, LIU X Y. Training cost-sensitive neural networks with methods addressing the class imbalance problem[J]. IEEE Transactions on Knowledge and Data Engineering, 2006, 18(1): 63-77.
[11] LIN Y, LEE Y, WAHBA G. Support vector machines for classification in nonstandard situations[J]. Machine Learning, 2002, 46(1): 191-202.
[12] GEIBEL P, BREFELD U, WYSOTZKI F. Perceptron and SVM learning with generalized cost models[J]. Intelligent Data Analysis, 2004, 8(5): 439-455.
[13] SCHLKOPF B, SMOLA A J. Learning with Kernels: support Vector Machines, Regularization, Optimization, and Beyond[M].Cambridge:MIT Press, 2001.
[14] MASNADI-SHIRAZI H, VASCONCELOS N. Risk minimization, probability elicitation, and cost-sensitive SVMs[C]// Proceedings of the 27th International Conference on Machine Learning. Haifa, Israel: IEEE Press, 2010: 759-766.
[15] MASNADI-SHIRAZI H, VASCONCELOS N, IRANMEHR A. Cost-Sensitive Support Vector Machines[J]. arXiv preprint, 2012: arXiv:1212.0975.
[16] CORTES C, VAPNIK V. Support-vector networks[J]. Machine learning, 1995, 20(3): 273-297.
[17] BACH F R, HECKERMAN D, HORVITZ E. Considering cost asymmetry in learning classifiers[J]. Journal of Machine Learning Research, 2006, 7(4): 1713-1741.
[18] DAVENPORT M, BARANIUK R G, SCOTT C D. Controlling false alarms with support vector machines[C]// Proceedings of the International Conference on Acoustics, Speech and Signal Processing. IEEE Press, 2006: 589-592.
[19] CHANG C C, LIN C J. LIBSVM: A library for support vector machines[J]. ACM Transactions on Intelligent Systems and Technology, 2011, 2(3): 389-396.
[20] WANG J, ZHAO P, HOI S C. Cost-sensitive online classification[J]. IEEE Transactions on Knowledge and Data Engineering, 2014, 26(10): 2425-2438.
[21] ZHENG J, SHEN F, FAN H, et al. An online incremental learning support vector machine for large-scale data[J]. Neural Computing and Applications, 2013, 22(5): 1023-1035.
[22] PLATT J C. Fast training of support vector machines using sequential minimal optimization[C]// Advances in Kernel Methods. Cambridge, USA: MIT Press, 1999: 185-208.
[23] POGGIO G C T. Incremental and decremental support vector machine learning[C]// Proceedings of the 2000 Conference on Advances in Neural Information Processing Systems, MIT Press, 2001: 409.
[24] GU B, WANG J D, CHEN H. On-line off-line ranking support vector machine and analysis[C]// IEEE International Joint Conference on Neural Networks. 2008: 1364-1369.
[25] GU B, WANG J D, YU Y C, et al. Accurate on-line ν-support vector learning[J]. Neural Networks, 2012, 27: 51-59.
[26] GU B, SHENG V S, WANG Z, et al. Incremental learning for ν-support vector regression[J]. Neural Networks, 2015, 67(C): 140-150.
[27] GU B, SHENG V S, TAY K Y, et al. Incremental support vector learning for ordinal regression[J]. Neural Networks and Learning Systems, IEEE Transactions on, 2015, 26(7): 1403-1416.
[28] SCHOTT J R. Matrix analysis for statistics[J]. Exvi Print, 2005, 30(5): xvi,456.
[29] GU B, SHENG V S. Feasibility and finite convergence analysis for accurate on-line-support vector machine[J]. IEEE Transactions on Neural Networks and Learning Systems, 2013, 24(8): 1304-1315.
[30] CHANG C C, LIN C J. LIBSVM: A library for support vector machines[J]. ACM Transactions on Intelligent Systems and Technology (TIST), 2011, 2(3): 389-396.

()
()

[1]	LIU Haibo, QIAN Wei, WANG Fuzhong. Gas outburst prediction based on rough set and particle swarm optimization support vector machine [J]. Journal of University of Science and Technology of China, 2019, 49(2): 87-92.
[2]	ZHANG Li, LU Xingning, LU Conglin, WANG bangjun, LI Fanzhang. National matriculation test prediction based on support vector machines [J]. Journal of University of Science and Technology of China, 2017, 47(1): 1-9.
[3]	WANG Yujie, GUO Li, WANG Cuiping. An audio steganalysis method for echo hiding based on statistical features of power cepstrums [J]. Journal of University of Science and Technology of China, 2010, 40(10): 1074-1081.