Corpus selection for Uyghur-Chinese machine translation based on bilingual sentence coverage

doi:10.3969/j.issn.0253-2778.2017.04.001

Abstract

Abstract: When making the selection of corpora, information includes not only redundancy at the vocabulary level but also redundancy at the sentential level. Present methods for this purpose are mainly focused on selecting corpora at the vocabulary level of coverage. These methods can effectively reduce the redundancy of words and phrases, but does not take into account the level of sentence coverage. Aiming at selecting a smaller training corpus from large-scale bilingual corpus, in order to get a the same or better translation system than the mass training data, the corpus from sentence coverage was mainly selected, by combining unseen n-grams method and edit distance. The experimental results show that the proposed method uses less training corpus, but still achieves almost equivalent performance compared with the original training corpus.

Key words: statistical machine translation, sentence pairs, corpus selection

CLC Number:

TP391

ZHU Shaolin, YANG Yating, MI Chenggang, LI Xiao,WANG Lei,. Corpus selection for Uyghur-Chinese machine translation based on bilingual sentence coverage[J]. Journal of University of Science and Technology of China, 2017, 47(4): 283-289.

References

［1］
CHAO W H, LI Z J. A Graph-based bilingual corpus selection approach for SMT[C]// Proceedings of the 25th Pacific Asia Conference on Language, Information and Computation. Singapore: Waseda University Press, 2011: 120-129.
[2] CUI L, ZHANG D D, LIU S J, et al. Collective corpus weighting and phrase scoring for SMT using graph-based random walk[C]// The 2nd Conference on Natural Language Processing & Chinese Computing. Chongqing, China, 2013: 176-187.
[3] ECK M, VOGEL S, WAIBEL A. Low cost portability for statistical machine translation based on n-gram coverage[C]// International Workshop on Spoken Language Translation. Pittsburgh, USA: IWSLT Press, 2005: 61-67.
[4] MANDAL A, VERGYRI D, WANG W, et al. Efficient data selection for machine translation[C]// Spoken Language Technology Workshop. Goa, India: IEEE Press, 2008: 261-264.
[5] SKADIA I, BRLTIS E. English-Latvian SMT: knowledge or data? [C]// Proceedings of the 17th NODALIDA Conference Processing, http://beta.visl.sdu.dk/~eckhard/nodalida/paper_57.pdf, 2009: 242-245.
[6] HAN X W, LI H Z, ZHAO T J. Train the machine with what it can learn: Corpus selection for SMT[C]// Proceedings of the 2nd Workshop on Building and Using Comparable Corpora: From Parallel to Non-Parallel Corpora. Suntec, Singapore: ACM Press, 2009: 27-33.
[7] 王志洋,吕雅娟,刘群. 面向形态丰富语言的多粒度翻译融合[J]. 中文信息学报. 2011, 25(4): 75-81.
WANG Z Y, LV Y J, LIU Q. System combination with multiple granularities for morphologically rich language translation[J]. Journal of Chinese Information Processing, 2011, 25(4): 75-81.
[8] 米莉万·雪合来提, 刘凯，吐尔根·依布拉音. 基于维语尔语词干词缀粒度的汉维机器翻译[J]. 中文信息学报, 2015， 29(3): 201-206.
MILIWAN·XUEHELAITI, LIU KAI, TURGUN·IBRAHIM. Chinese-Uyghur machine translation based on smallest translation units of stem and suffixes[J]. Journal of Chinese Information Processing, 2015, 29(3):201-206.
[9] HAN J W, JI H, SUN Y Z. Successful data mining methods for NLP[C]// Proceedings of the Tutorials of the 53rd Annual Meeting of the ACL and the 7th International Joint Conference on Natural Language Processing. Beijing, China: ACL Press, 2015: 1-4.
[10] LIU L, HONG Y, LIU H, et al. Effective selection of translation model training data[C]// Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. Baltimore, USA: IEEE Press, 2014: 569-573.
[11] HILDEBRAND A S, ECK M, VOGEL S, et al. Adaptation of the translation model for statistical machine translation based on information retrieval[C]// Proceedings of the 10th Annual Conference on European Association for Machine Translation. San Diego, USA: ACM Press, 2005: 133-142.
[12] 黄瑾,吕雅娟,刘群. 基于信息检索方法的统计翻译系统训练数据选择与优化[J]. 中文信息学报, 2008, 22(2): 40-46.
HUANG Jin, LV Yajun, lIU Qun. The statistical translation system based on information retrieval method selection and optimization of training data[J]. Journal of Chinese Information Processing, 2008, 22(2): 40-46.
[13] 姚树杰, 肖桐, 朱靖波. 基于句对质量和覆盖度的统计机器翻译训练语料选取[J]. 中文信息学报, 2011, 25(1): 72-77.
YAO Shujie, XIAO Tong, ZHU Jingbo. Selection of SMT training data based on sentence pair quality and coverage[J]. Journal of Chinese Information Processing, 2011, 25(1): 72-77.
[14] 王星, 涂兆鹏, 谢军, 等. 一种基于分类的平行语料选取方法[J]. 中文信息学报, 2013, 27(6): 144-150.
WANG Xing, TU Zhaopeng, XIE Jun, etal. Selection of parallel corpus based on classification[J]. Journal of Chinese Information Processing, 2013, 27(6): 144-150.
[15] KIRCHHOFF K, BILMES J. Submodularity for data selection in statistical machine translation[C]// Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Doha, Qatar: ACL Press, 2014: 131-141.

[1]	ZHAN Yongkun, YANG Wenfei, ZHANG Tianzhu. Relation aware network for weakly-supervised temporal action localization [J]. Journal of University of Science and Technology of China, 2021, 51(10): 753-765.
[2]	Liang Qipeng, Zhu Qiaohui, Kang Yu, Zhao YunBo. Estimation based approximating control for wireless networked control systems [J]. Journal of University of Science and Technology of China, 2021, 51(4): 327-334.
[3]	Li Huicong, Fu Xiaoming∗ . Anisotropic surface meshing using locally isometric embedding [J]. Journal of University of Science and Technology of China, 2020, 50(12): 1460-1471.
[4]	YU Jun, LI Yajie, CHENG Lilei, LIAN Shun, TAN Chang, DING Decheng, Liu Qi. Research and practice of plagiarism detection in program code assignments by college students [J]. Journal of University of Science and Technology of China, 2020, 50(8): 1048-1057.
[5]	ZHANG Wenjing, BAN Zhijie. Citation network’s influence maximization algorithm based on global influence [J]. Journal of University of Science and Technology of China, 2020, 50(8): 1058-1063.
[6]	REN Min, XU Ling, WANG Feng, WU Chao. Campus encyclopedia platform based on knowledge recommendation [J]. Journal of University of Science and Technology of China, 2020, 50(8): 1072-1076.
[7]	SUN Lei, ZHANG Yining, XUE Yanfang, QIAO Lishan, ZHANG Limei. Adaptive functional connectivity network learning and application in brain disorders identification [J]. Journal of University of Science and Technology of China, 2020, 50(8): 1102-1109.
[8]	BAI Jiangbo, YANG Yang, ZHANG Wensheng. A simulation of the synthetic aperture radar image based on improved CycleGAN [J]. Journal of University of Science and Technology of China, 2020, 50(8): 1181-1186.
[9]	XIN Shouyu, ZHENG Ruirui, ZHOU Yu, LIU Wenpeng, HE Jianjun. A one-shot learning algorithm using support set information during training [J]. Journal of University of Science and Technology of China, 2020, 50(8): 1187-1192.
[10]	WANG Zhihua, KANG Hongmei. Using splines on triangular mesh to solve PDE with nonhomogeneous boundary [J]. Journal of University of Science and Technology of China, 2020, 50(7): 901-905.
[11]	GONG Lejun , ZHOU Shehai, CHENG Yifei, GAO Zhihong, LI Huakang. Identification of PBMC-related cells of single cell RNA sequence data [J]. Journal of University of Science and Technology of China, 2020, 50(7): 1013-1018.
[12]	DU Shuying, DU Peng, DING Shifei. A malicious domain name detection method based on CNN [J]. Journal of University of Science and Technology of China, 2020, 50(7): 1019-1025.
[13]	LI Yongjun, CAO Weihua, LING Qiang. A multi-target tracking algorithm based on feature point trajectories [J]. Journal of University of Science and Technology of China, 2020, 50(6): 726-732.
[14]	. A modified coherent diffraction algorithm based on the total variation algorithm for insufficient data [J]. Journal of University of Science and Technology of China, 2020, 50(4): 418-427.
[15]	WANG Xuhui, YAN Mingye, WU Meng. Suitable for the construction of degenerate smooth interpolated surfaces patches in isogeometric analysis [J]. Journal of University of Science and Technology of China, 2020, 50(3): 335-342.