基于网格流的视频修补网络

doi:10.52396/JUST-2020-0022

中国科学技术大学学报 ›› 2021, Vol. 51 ›› Issue (1): 1-11.DOI: 10.52396/JUST-2020-0022

• 科研论文 • 下一篇

基于网格流的视频修补网络

刘森, 张直政, 俞涛, 陈志波^*

中国科学技术大学中国科学院空间信息处理与应用重点实验室,安徽合肥 230027

出版日期:2021-01-31 发布日期:2021-05-27

MOVIE: Mesh oriented video inpainting network

LIU Sen, ZHANG Zhizheng, YU Tao, CHEN Zhibo^*

CAS Key Laboratory of Technology in Geo-spatial Information Processing and Application System, University of Science and Technology of China, Hefei 230027, China

Online:2021-01-31 Published:2021-05-27
Contact: *E-mail: chenzhibo@ustc.edu.cn
About author:Sen Liu received the B.S. degree in computer science from the Beijing University of Posts and Telecommunications, Beijing, China, in 2013. Currently, he is working towards the PhD degree at School of Information Science and Technology, University of Science and Technology of China. His area of interests includes artificial intelligence, deep learning, video coding, computer vision and pattern recognition and reinforcement learning.
Zhibo Chen (M'01-SM'11) received the B. S., and PhD degree from Department of Electrical Engineering Tsinghua University in 1998 and 2003, respectively. He is now a professor in University of Science and Technology of China. His research interests include image and video compression, visual quality of experience assessment, immersive media computing and intelligent media computing. He has more than 100 publications and more than 50 granted EU and US patent applications. He is IEEE senior member, Secretary (Chair-Elect) of IEEE Visual Signal Processing and Communications Committee. He was TPC chair of IEEE PCS 2019 and organization committee member of ICIP 2017 and ICME 2013, served as Track chair in IEEE ISCAS and Area chair in IEEE VCIP.
Zhizheng Zhang (S'19) received the B.S. degree in electronic information engineering from the University of Electronic Science and Technology of China, Chengdu, China, in 2016. He is currently pursuing the PhD degree in the University of Science and Technology of China, Hefei, China. His current research interests include reinforcement learning, few-shot learning, and intelligent media computing.
Tao Yu is currently pursuing the PhD degree with the Department of Electronic Engineering and Information Science, University of Science and Technology of China. He received the B.S. degree in Electronics and Information Engineering in Anhui University in 2018. His research interests include computer vision, image processing and reinforcement learning.
Supported by:
National Natural Science Foundation of China(61571413,61632001).

摘要/Abstract

摘要： 视频修补的目的是基于视频帧之间的时空域上下文信息修补空洞.现有的方法由于不能准确地对运动轨迹进行建模经常导致修补结果无法保持良好的时空一致性.为此引入灵活的形状自适应网格作为基本处理单元,将网格流用于运动表示,提出了一个基于网格流的视频修补网络,通过先预测网络流再添补空洞区域的方式对视频进行修补.具体地,首先设计了一个网格流预测模块用于预测视频中可见内容的网格流的预测和一个网格流修补模块用于修补视频中空洞区域的网格流,通过这种方式将网格流的预测和修补解耦以达到更容易训练优化的目的.我们进一步设计了一个混合损失函数用于同时优化可见区域、修补区域和整个视频帧范围的网格流预测结果.为修正经过网格流变换引起的失真现象,最后设计了一个修补优化网络.大量试验结果证明,本文提出的方法不仅从主观评判和客观指标得到相比于现有方法更好的修补结果,而且相比于现有最快的方法达到了4倍的速度提升.

关键词: 视频修补, 网格流, 深度神经网络

Abstract: Video inpainting aims to fill the holes across different frames upon limited spatio-temporal contexts. The existing schemes still suffer from achieving precise spatio-temporal coherence especially in hole areas due to inaccurate modeling of motion trajectories. In this paper, we introduce fexible shape-adaptive mesh as basic processing unit and mesh flow as motion representation, which has the capability of describing complex motions in hole areas more precisely and efficiently. We propose a Mesh Oriented Video Inpainting nEtwork, dubbed MOVIE, to estimate mesh flows then complete the hole region in the video. Specifically, we first design a mesh flow estimation module and a mesh flow completion module to estimate the mesh flow for visible contents and holes in a sequential way, which decouples the mesh flow estimation for visible and corrupted contents for easy optimization. A hybrid loss function is further introduced to optimize the flow estimation performance for the visible regions, the entire frames and the inpainted regions respectively. Then we design a polishing network to correct the distortion of the inpainted results caused by mesh flow transformation. Extensive experiments show that MOVIE not only achieves over four-times speed-up in completing the missing area, but also yields more promising results with much better inpainting quality in both quantitative and perceptual metrics.

Key words: mesh flow, deep neural networks, video inpainting

中图分类号:

TU459

刘森, 张直政, 俞涛, 陈志波. 基于网格流的视频修补网络[J]. 中国科学技术大学学报, 2021, 51(1): 1-11.

Liu Sen, Zhang Zhizheng, Yu Tao, Chen Zhibo. MOVIE: Mesh oriented video inpainting network[J]. Journal of University of Science and Technology of China, 2021, 51(1): 1-11.

参考文献

[1] HuoX, Tan J. A novel non-linear method of automatic video scratch removal. Fourth International Conference on Digital Home. Guangzhou, China: IEEE, 2012: 39-45.
[2] Jin X, Su Y, Zou L, et al. Video logo removal detection based on sparse representation. Multimedia Tools and Applications, 2018, 77(22): 29303-29322.
[3] Qin C, He Z, Yao H, et al. Visible watermark removal scheme based on reversible data hiding and image inpainting. Signal Processing: Image Communication, 2018, 60: 160-172.
[4] Le T T, Almansa A, Gousseau Y, et al. Object removal from complex videos using a few annotations. Computational Visual Media, 2019, 5(3): 267-291.
[5] Callico G M , Lopez S, Sosa O, et al. Analysis of fast block matching motion estimation algorithms for video super-resolution systems. IEEE Transactions on Consumer Electronics, 2008, 54(3): 1430-1438.
[6] Wang L, Guo Y, Liu L,et al. Deep video super-resolution using hr optical flow estimation. IEEE Transactions on Image Processing, 2020, 29: 4323-4336.
[7] Liu S, Yuan L, Tan P,et al. Steadyflow: Spatially smooth optical flow for video stabilization. IEEE Conference on Computer Vision and Pattern Recognition. Columbus, USA: IEEE, 2014: 4209-4216.
[8] Lim A, Ramesh B, Yang Y,et al. Real-time optical flow-based video stabilization for unmanned aerial vehicles. Journal of Real-time Image Processing, 2019, 16(6): 1975-1985.
[9] Granados M, Tompkin J, Kim K,et al. How not to be seen-object removal from videos of crowded scenes. Comput. Graph. Forum, 2012, 31( 2): 219-228.
[10] Wexler Y, Shechtman E, Irani M. Space-time completion of video. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(3): 463-476.
[11] Newson A, Almansa A, Fradet M,et al. Video inpainting of complex scenes. Siam Journal on Imaging Sciences, 2014, 7(4): 1993-2019.
[12] Huang J, Kang S B, Ahuja N,et al. Temporally coherent completion of dynamic video. ACM Transactions on Graphics,2016,35(6):196.
[13] Woo S, Kim D, Park K,et al. Align-andattend network for globally and locally coherent video inpainting. 2019, arXiv:1905.13066.
[14] Chang Y L, Liu Z Y, Hsu W. Vornet: Spatio-temporally consistent video inpainting for object removal. Proceedings of the IEEE conference on computer vision and pattern recognition workshops. Long Beach, USA: IEEE, 2019: 00229.
[15] Ding Y, Wang C, Huang H,et al. Framerecurrent video inpainting by robust optical flow inference. 2019, arXiv:1905.02882.
[16] Xu R, Li X, Zhou B,et al. Deep flow-guided video inpainting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE, 2019: 3723-3732.
[17] Lee S, Oh S W, Won D,et al. Copy-and-paste networks for deep video inpainting. Proceedings of the IEEE International Conference on Computer Vision. Seoul, South Korea: ACM, 2019: 4413-4421.
[18] Chang Y L, Liu Z Y, Lee K Y, et al. Free-form video inpainting with 3D gated convolution and temporal patchgan. Proceedings of the IEEE International Conference on Computer Vision. Seoul, South Korea: ACM, 2019: 9066-9075.
[19] Wang C, Huang H, Han X,et al. Video inpainting by jointly learning temporal structure and spatial details. Proceedings of the AAAI Conference on Artificial Intelligence. Hawaii, USA: IEEE, 2019, 33: 5232-5239.
[20] Chang Y L, Liu Z Y, Lee K Y, et al. Learnable gated temporal shift module for deep video inpainting. 2019, arXiv:1907.01131.
[21] Oh S W, Lee S, Lee J Y, et al. Onion-peel networks for deep video completion. in Proceedings of the IEEE International Conference on Computer Vision. Seoul, South Korea: ACM, 2019: 4403-4412.
[22] Kim D, Woo S, Lee J Y, et al. Deep video inpainting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE, 2019: 5792-5801.
[23] He K, Zhang X, Ren S,et al. Deep residual learning for image recognition. IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE, 2016: 770-778.
[24] Goodfellow I, Pougetabadie J, Mirza M,et al. Generative adversarial nets. Proceedings of the 27th International Conference on Neural Information Processing Systems. 2014, 2: 2672-2680.
[25] Xu N, Yang L, Fan Y,et al. Youtube-vos: A large-scale video object segmentation benchmark. Computer Vision and Pattern Recognition, 2018, arXiv:1809.03327.
[26] Liu G, Reda F A, Shih K J,et al. Image inpainting for irregular holes using partial convolutions. Computer Vision and Pattern Recognition, 2018: 89-105.
[27] Perazzi F, Ponttuset J, Mcwilliams B,et al. A benchmark dataset and evaluation methodology for video object segmentation.IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA, IEEE, 2016: 724-732.
[28] Ponttuset J, Perazzi F, Caelles S,et al. The 2017 Davis challenge on video object segmentation. Computer Vision and Pattern Recognition, 2017,arXiv:1704.00675.

基于网格流的视频修补网络

MOVIE: Mesh oriented video inpainting network

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 10

编辑推荐

Metrics

本文评价

[1]	Dabwan Yousef N, 裴刚, 关学新, 赵斌. 一种基于线性菲涅耳反射镜的混合太阳能预热中冷燃气轮机[J]. 中国科学技术大学学报, 2021, 51(7): 505-520.
[2]	童谣. 非期望产出存在时考虑多值指标的目标导向DEA方法[J]. 中国科学技术大学学报, 2021, 51(7): 521-541.
[3]	孔冰娴, 时笑, 杨锋. 社交媒体中乡村旅游信息质量对其旅游地意愿的影响机理研究[J]. 中国科学技术大学学报, 2021, 51(7): 542-561.
[4]	凌华泽, 薛开平, Wei David S.L., 李睿栋. 多用户场景支持多关键词模糊搜索的可搜索加密方案[J]. 中国科学技术大学学报, 2021, 51(7): 562-576.
[5]	李廷芳, 王潇. 企业的“抗疫”能力:来自中国的证据[J]. 中国科学技术大学学报, 2021, 51(4): 288-298.
[6]	刘兴华, 白丹丹, 孙宝仁, 文家燕, 吕文君, 李鲲. 欺骗攻击下多区域互联电力系统的事件触发滑模负荷频率控制[J]. 中国科学技术大学学报, 2021, 51(1): 33-42.
[7]	何浩楠, 任玮, 王作航, 赵成勇, 王善勇, 马飞. 基于随机Petri网的个人碳交易机制下电动汽车采纳行为演化研究[J]. 中国科学技术大学学报, 2021, 51(1): 53-64.
[8]	刘佩琳, 陆宇阳, 郑东昌, 何陵辉, 倪勇∗ . 应用修正Stoney公式原位测量硅薄膜电极中的锂化应力 [J]. 中国科学技术大学学报, 2020, 50(12): 1516-1524.
[9]	胡心颖, 何钰, 孙广中. 基于概率图模型的计算机课程教学认知诊断框架[J]. 中国科学技术大学学报, 2021, 51(1): 12-21.
[10]	郭传凤, 杜昕泽, 吴钦宇, 毛甜甜. 基于RDEU模型的风险度量[J]. 中国科学技术大学学报, 2021, 51(1): 65-74.