Cross-media semantic retrieval with deep canonical correlation analysis

doi:10.3969/j.issn.0253-2778.2018.04.008

Abstract

Abstract: The cross-media retrieval with canonical correlation analysis (CCA) is a method to map different media features to the largest correlation isomorphism subspace through the canonical correlation analysis, and compare the similarity between cross-media data in the subspace. However CCA is a linear model and can not adequately exploit the complex correlation between cross-media data. The structure of the traditional deep canonical correlation analysis (DCCA) is improved, and the latent dirichlet allocation (LDA) is used to discover the semantic information in the text data and learns the semantic mapping. The cross-media correlation learning with deep canonical correlation analysis (CMC-DCCA) and the cross-media semantic correlation retrieval (CMSCR) are proposed. Experiments on the Wikipedia text image dataset shows that the CMC-DCCA model can mine the complex correlation between cross-media data better, and that CMSCR has better performance in cross-media retrieval.

Key words: canonical correlation analysis, deep canonical correlation analysis, semantic mapping, cross-media retrieval

CLC Number:

TP391

WANG Shu, SHI Zhongzhi. Cross-media semantic retrieval with deep canonical correlation analysis[J]. Journal of University of Science and Technology of China, 2018, 48(4): 322-330.

References

［1］
RASIWASIA N, PEREIRA J C, COVIELLO E, et al. A new approach to cross-modal multimedia retrieval[C]// Proceedings of the 18th ACM International Conference on Multimedia. Firenze, Italy: ACM Press, 2010: 251-260.
[2] PEREIRA J C, COVIELLO E, DOYLE G, et al. On the role of correlation and abstraction in cross-modal multimedia retrieval[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014, 36(3): 521-535.
[3] WANG S, LU J, GU X, et al. Unsupervised discriminant canonical correlation analysis based on spectral clustering[J]. Neurocomputing, 2016, 171(C): 425-433.
[4] ZU C, ZHANG D. Canonical sparse cross-view correlation analysis[J]. Neurocomputing, 2016, 191: 263-272.
[5] BALLAN L, URICCHIO T, SEIDENARI L, et al. A cross-media model for automatic image annotation[C]// Proceedings of International Conference on Multimedia Retrieval. New York: ACM Press, 2014: No.73(1-8).
[6] WANG S, ZHUANG F, JIANG S, et al. Cluster-sensitive structured correlation analysis for web cross-modal retrieval[J]. Neurocomputing, 2015, 168: 747-760.
[7] GONG Y, KE Q, ISARD M, et al. A multi-view embedding space for modeling internet images, tags, and their semantics[J]. International Journal of Computer Vision, 2014, 106(2): 210-233.
[8] 张博, 郝杰, 马刚, 等. 混合概率典型相关性分析[J]. 计算机研究与发展, 2015, 52(7):1463-1476.
ZHANG B, HAO J, MA G, et al. Mixture of probabilistic canonical correlation analysis[J]. Journal of Computer Research and Development, 2015, 52(7): 1463-1476.
[9] 张博, 郝杰, 马刚, 等. 基于弱匹配概率典型相关性分析的图像自动标注[J]. 软件学报, 2017, 28(2): 292-309.
ZHANG B, HAO J, MA G, et al. Automatic image annotation based on semi-paired probabilistic canonical correlation analysis [J]. Journal of Software, 2017, 28(2): 292-309.
[10] SRIVASTAVA N, SALAKHUTDINOV R. Learning representations for multimodal data with deep belief nets[C]// International Conference on Machine Learning Workshop. Edinburgh, Scotland: IMLS Press, 2012: 1-8.
[11] FENG F, WANG X, LI R. Cross-modal retrieval with correspondence autoencoder[C]// Proceedings of the 22nd ACM international conference on Multimedia. San Francisco, USA: ACM Press, 2014: 7-16.
[12] WANG C, YANG H, MEINEL C. Deep semantic mapping for cross-modal retrieval[C]// 27th International Conference on Tools with Artificial Intelligence. Vietri sul Mare, Italy: IEEE Computer Society, 2015: 234-241.
[13] ANDREW G, ARORA R, BILMES J A, et al. Deep canonical correlation analysis[C]// Proceedings of the 30th International Conference on Machine Learning . Atlanta, USA: IMLS Press, 2013: 1247-1255.
[14] BLEI D M, NG A Y, JORDAN M I. Latent dirichlet allocation[J]. Journal of Machine Learning Research, 2003, (3): 993-1022.
[15] LIU D C, NOCEDAL J. On the limited memory BFGS method for large scale optimization[J]. Mathematical programming, 1989, 45(1): 503-528.
[16] JOACHIMS T. Optimizing search engines using clickthrough data[C]// Proceedings of the 8th ACM SIGKDD International Conference on Knowledge discovery and Data Mining. Edmonton, Canada: ACM Press, 2002: 133-142.

()
(

[1]	ZHAN Yongkun, YANG Wenfei, ZHANG Tianzhu. Relation aware network for weakly-supervised temporal action localization [J]. Journal of University of Science and Technology of China, 2021, 51(10): 753-765.
[2]	Liang Qipeng, Zhu Qiaohui, Kang Yu, Zhao YunBo. Estimation based approximating control for wireless networked control systems [J]. Journal of University of Science and Technology of China, 2021, 51(4): 327-334.
[3]	Li Huicong, Fu Xiaoming∗ . Anisotropic surface meshing using locally isometric embedding [J]. Journal of University of Science and Technology of China, 2020, 50(12): 1460-1471.
[4]	YU Jun, LI Yajie, CHENG Lilei, LIAN Shun, TAN Chang, DING Decheng, Liu Qi. Research and practice of plagiarism detection in program code assignments by college students [J]. Journal of University of Science and Technology of China, 2020, 50(8): 1048-1057.
[5]	ZHANG Wenjing, BAN Zhijie. Citation network’s influence maximization algorithm based on global influence [J]. Journal of University of Science and Technology of China, 2020, 50(8): 1058-1063.
[6]	REN Min, XU Ling, WANG Feng, WU Chao. Campus encyclopedia platform based on knowledge recommendation [J]. Journal of University of Science and Technology of China, 2020, 50(8): 1072-1076.
[7]	SUN Lei, ZHANG Yining, XUE Yanfang, QIAO Lishan, ZHANG Limei. Adaptive functional connectivity network learning and application in brain disorders identification [J]. Journal of University of Science and Technology of China, 2020, 50(8): 1102-1109.
[8]	BAI Jiangbo, YANG Yang, ZHANG Wensheng. A simulation of the synthetic aperture radar image based on improved CycleGAN [J]. Journal of University of Science and Technology of China, 2020, 50(8): 1181-1186.
[9]	XIN Shouyu, ZHENG Ruirui, ZHOU Yu, LIU Wenpeng, HE Jianjun. A one-shot learning algorithm using support set information during training [J]. Journal of University of Science and Technology of China, 2020, 50(8): 1187-1192.
[10]	WANG Zhihua, KANG Hongmei. Using splines on triangular mesh to solve PDE with nonhomogeneous boundary [J]. Journal of University of Science and Technology of China, 2020, 50(7): 901-905.
[11]	GONG Lejun , ZHOU Shehai, CHENG Yifei, GAO Zhihong, LI Huakang. Identification of PBMC-related cells of single cell RNA sequence data [J]. Journal of University of Science and Technology of China, 2020, 50(7): 1013-1018.
[12]	DU Shuying, DU Peng, DING Shifei. A malicious domain name detection method based on CNN [J]. Journal of University of Science and Technology of China, 2020, 50(7): 1019-1025.
[13]	LI Yongjun, CAO Weihua, LING Qiang. A multi-target tracking algorithm based on feature point trajectories [J]. Journal of University of Science and Technology of China, 2020, 50(6): 726-732.
[14]	. A modified coherent diffraction algorithm based on the total variation algorithm for insufficient data [J]. Journal of University of Science and Technology of China, 2020, 50(4): 418-427.
[15]	WANG Xuhui, YAN Mingye, WU Meng. Suitable for the construction of degenerate smooth interpolated surfaces patches in isogeometric analysis [J]. Journal of University of Science and Technology of China, 2020, 50(3): 335-342.