一种基于贝叶斯后验的异常值在线检测及置信度评估算法

doi:10.3969/j.issn.0253-2778.2017.08.003

中国科学技术大学学报 ›› 2017, Vol. 47 ›› Issue (8): 644-652.DOI: 10.3969/j.issn.0253-2778.2017.08.003

一种基于贝叶斯后验的异常值在线检测及置信度评估算法

孙栓柱，宋蓓，李春岩，王皓

1. 江苏方天电力技术有限公司，江苏南京 211102；
2.南京大学计算机软件新技术国家重点实验室，江苏南京 210023

收稿日期:2017-05-26 修回日期:2017-07-14 出版日期:2017-08-31 发布日期:2017-08-31
通讯作者: 王皓
作者简介:孙栓柱, 男，1973年生, 硕士/教授级高级工程师. 研究方向：电力行业节能减排技术及工业大数据挖掘. E-mail: sunshuanzhu@sina.com
基金资助:
国家自然科学基金(61503178)，江苏省自然科学基金(BK20150587)资助.

An online outlier detection and confidence estimation algorithm based on Bayesian posterior ratio

SUN Shuanzhu, SONG Bei, LI Chunyan, WANG Hao

1. Jiangsu Frontier Electric Technology Co. Ltd., Nanjing 211102，China;
2. State Key Laboratory for Novel Software Technology at Nanjing University, Nanjing 210023, China

Received:2017-05-26 Revised:2017-07-14 Online:2017-08-31 Published:2017-08-31

摘要/Abstract

摘要： 为识别一类更新速度快、变化趋势平缓、缺少人工类标的大数据量工业时间序列中所存在的异常值，提出了一种以贝叶斯后验为基础的异常值在线检测及置信度评估算法.算法将预测检测和假设检验相结合，首先建立时间序列自回归模型，然后对预测残差作基于贝叶斯原理的后验检验，用后验概率对数比确定序列中的异常值.为减少识别过程中的误判，在检测完成后，利用自组织映射神经网络计算状态转移概率，进一步对已标记的异常值进行置信度评估.通过定期更新模型，算法各参数能动态保持与数据变化规律同步，提高了检测的准确率.实验结果表明，该算法能够对时间序列异常值准确快速地进行在线检测，同时给出可靠的置信度评估，具有较高的实用价值.

关键词: 时间序列, 异常检测, 贝叶斯后验, 置信度评估

Abstract: In order to satisfy the outlier detection requirements in one kind of high-speed, small-variance unlabeled industrial time series, an online outlier detection and confidence estimation algorithm based on Bayesian posterior ratio was proposed. The algorithm combined prediction and hypothesis testing, establishing the autoregressive model firstly and then using Bayesian posterior logarithm of residuals to identify outliers. To reduce misjudgment, the state transition probabilities were calculated by self-organizing map neural network and the reliability of detected outliers was evaluated afterwards. It updated models periodically to dynamically adapt to data changes, thus improving accuracy. Experimental results demonstrate that the online algorithm can effectively detect outliers in time series provide reliable confidence evaluation, bringing higher adaptability and practicability.

Key words: time series, outlier detection, Bayesian posterior ratio, confidence estimation

中图分类号:

TP399

孙栓柱，宋蓓，李春岩，王皓. 一种基于贝叶斯后验的异常值在线检测及置信度评估算法[J]. 中国科学技术大学学报, 2017, 47(8): 644-652.

SUN Shuanzhu, SONG Bei, LI Chunyan, WANG Hao. An online outlier detection and confidence estimation algorithm based on Bayesian posterior ratio[J]. Journal of University of Science and Technology of China, 2017, 47(8): 644-652.

参考文献

［1］
PAWAR A D, KALAVADEKAR P N, TAMBE S N. A survey on outlier detection techniques for credit card fraud detection [J]. IOSR Journal of Computer Engineering, 2014, 16(2): 44-48.
[2] GOLMOHAMMADI K, ZAIANE O R. Time series contextual anomaly detection for detecting market manipulation in stock market[C]// IEEE International Conference on Data Science and Advanced Analytics. Pairs, France: IEEE Press, 2015: 1-10.
[3] KIM G, LEE S, KIM S. A novel hybrid intrusion detection method integrating anomaly detection with misuse detection[J]. Expert Systems with Applications, 2014, 41(4): 1690-1700.
[4] SCHIFF G D, VOLK L A, VOLODARSKAYA M, et al. Screening for medication errors using an outlier detection system[J]. Journal of the American Medical Informatics Association, 2017, 24(2): 281-287.
[5] BILLOR N, HADI A S, VELLEMAN P F. BACON: Blocked adaptive computationally efficient outlier nominators[J]. Computational Statistics & Data Analysis, 2000, 34(3):279-298.
[6] KNORR E M，NG R T. Algorithms for mining distance-based outliers in large datasets[C]//Proceedings of the 24th International Conference on Very Large Data Bases. San Francisco：Morgan Kaufmann Publishers，1998: 392-403.
[7] RAMASWAMY S, RASTOGI R, SHIM K. Efficient algorithms for mining outliers from large data sets[C]//Proceedings of the ACM SIGMOD International Conference on Management of Data. Dallas, USA: ACM Press, 2000, 29(2): 427-438.
[8] BREUNIG M M, KRIEGEL H P, NG R T, et al. LOF: Identifying density-based local outliers[C]//Proceedings of the ACM SIGMOD International Conference on Management of Data. Dallas, USA: ACM Press, 2000, 29(2): 93-104.
[9] HAWKINS D M. Identification of Outliers[M]. London: Chapman and Hall, 1980.
[10] ABRAHAM B, BOX G E P. Bayesian Analysis of Some Outlier Problems in Time Series[J]. Biometrika, 1979, 66(2):229-236.
[11] KARIMIAN S H, KELARESTAGHI M, HASHEMI S. I-IncLOF: Improved incremental local outlier detection for data streams[C]// Proceedings of the 16th CSI International Symposium on Artificial Intelligence and Signal Processing. Shiraz, Fars, Iran: IEEE Press, 2012: 23-28.
[12] 潘渊洋, 李光辉, 徐勇军. 基于DBSCAN的环境传感器网络异常数据检测方法[J]. 计算机应用与软件, 2012(11): 69-72.
[13] HILL D J, MINSKER B S, AMIR E. Real-time Bayesian anomaly detection for environmental sensor data[C]// Proceedings of the 32nd Congress-International Association for Hydraulic Research. 2007, (2): 503.
[14] ERFANI S M, RAJASEGARAR S, KARUNASEKERA S, et al. High-dimensional and large-scale anomaly detection using a linear one-class SVM with deep learning[J]. Pattern Recognition, 2016, 58(C): 121-134.
[15] JADIDI Z, MUTHUKKUMARASAMY V, SITHIRASENAN E, et al. Flow-based anomaly detection using neural network optimized with GSA algorithm[C]// Proceedings of the 33rd International Conference on Distributed Computing Systems Workshops. Philadelphia, USA: IEEE Press, 2013: 76-81.
[16] MARTINS H, PALMA L, CARDOSO A, et al. A support vector machine based technique for online detection of outliers in transient time series[C]// 10th Asian Control Conference. Kota, Kinabalu: IEEE Press, 2015: 1-6.
[17] JOHANSEN S, NIELSEN B. Asymptotic theory of outlier detection algorithms for linear time series regression models[J]. Scandinavian Journal of Statistics, 2016, 43(2): 321-348.
[18] 刘芳, 毛志忠. 过程控制时间序列中异常值的动态检测[J]. 控制理论与应用, 2012, 29(4): 424-432.
[19] 杨志勇, 朱跃龙, 万定生. 基于知识粒度的时间序列异常检测研究[J]. 计算机技术与发展, 2016, 26(7): 51-54.
[20] BOX G E P, JENKINS G M, REINSEL G C, et al. Time Series Analysis: Forecasting and Control[M]. John Wiley & Sons, 2015.
[21] LACOUR C, MASSART P, RIVOIRARD V. Estimator selection: A new method with applications to kernel density estimation[J]. arXiv preprint, 2016, arXiv:1607.05091.
[22] ANDERSSON B, DAVIER A A. Improving the bandwidth selection in kernel equating[J]. Journal of Educational Measurement, 2014, 51(3): 223-238.
[23] 苏卫星, 朱云龙, 胡琨元,等. 基于模型的过程工业时间序列异常值检测方法[J]. 仪器仪表学报, 2012, 33(9): 2080-2087.
[24] GUIDO D, TEUVO K.Visual Explorations in Finance: With Self-Organizing Maps[M]. Springer Science & Business Media, 2013.
[25] TAKEUCHI J I, YAMANISHI K. A unifying framework for detecting outliers and change points from time series[J]. Journal of Taiyuan Normal University, 2006, 18(4): 482-492.

()
(

[1]	霍文君，王伟，李文. AnomalyDetect：一种基于欧式距离的在线异常检测算法[J]. 中国科学技术大学学报, 2019, 49(7): 555-563.
[2]	林钱洪，王志海，原继东，张伟. 基于趋势信息的时间序列分类方法[J]. 中国科学技术大学学报, 2019, 49(2): 138-148.
[3]	王子一，商琳. 基于电力负荷曲线的设备识别方法[J]. 中国科学技术大学学报, 2019, 49(2): 100-104.
[4]	齐方鹏，王童，周明洋，傅忠谦. 基于互信息的动态网络链路预测算法研究[J]. 中国科学技术大学学报, 2018, 48(6): 440-446.
[5]	邓波,丁鲲,蒋国权,张宾. 动态时间弯曲距离推导[J]. 中国科学技术大学学报, 2018, 48(4): 261-274.

一种基于贝叶斯后验的异常值在线检测及置信度评估算法

An online outlier detection and confidence estimation algorithm based on Bayesian posterior ratio

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 5

编辑推荐

Metrics

本文评价