基于级联卷积神经网络的轻量级视频插帧算法

张培健; 滕奇志; 何小海

Chinese

您当前的位置：

首页 >

文章列表页 >

基于级联卷积神经网络的轻量级视频插帧算法

更新时间：2026-09-04

- 基于级联卷积神经网络的轻量级视频插帧算法
- Lightweight cascade network for video frame interpolation
- 微电子学与计算机 2021年38卷第3期页码：39-45
- 作者机构：
  
  四川大学电子信息学院,四川,成都,610065
- 作者简介：
- 基金信息：
  
  国家自然科学基金 61871279成都市产业集群协同创新项目 2016-XT00-00015-GX
- DOI：
  中图分类号：
- 纸质出版：2021
- 稿件说明：
移动端阅览
张培健, 滕奇志, 何小海. 基于级联卷积神经网络的轻量级视频插帧算法[J]. 微电子学与计算机, 2021, 38(3): 39-45.

ZHANG Pei-jian, TENG Qi-zhi, HE Xiao-hai. Lightweight cascade network for video frame interpolation[J]. 2021, 38(3): 39-45.
张培健, 滕奇志, 何小海. 基于级联卷积神经网络的轻量级视频插帧算法[J]. 微电子学与计算机, 2021, 38(3): 39-45. DOI：

ZHANG Pei-jian, TENG Qi-zhi, HE Xiao-hai. Lightweight cascade network for video frame interpolation[J]. 2021, 38(3): 39-45. DOI：

摘要

针对基于卷积神经网络的视频插帧算法模型参数过大、实时性差、内存占用高、难以广泛应用的问题，提出了一种基于双向光流和多尺度特征融合的轻量级级联推理网络模型.将视频插帧任务分解为帧间运动合成和纹理重建两个步骤，设计了一种轻量级的级联双向光流预测网络，并提出了一种多尺度空间与纹理特征融合网络模型，实现了对视频帧多尺度纹理特征和复杂运动特征的充分提取和利用.该模型使用相邻的两个视频帧和所需中间帧的位置作为网络输入，首先计算两个输入帧的空间金字塔特征与纹理金字塔特征；然后使用空间金字塔特征计算帧间的多尺度双向光流；随后结合双向光流，计算出中间帧的空间特征和纹理特征；最后融合中间帧的多尺度空间、纹理特征得到最终所需的视频帧.在Vimeo90K和UCF101数据集上的实验表明，在保证精度的前提下，本文算法在计算速度和模型参数量上具有更好的表现.

Abstract

Aiming at the problems that the model parameters of video interpolation algorithm based on convolutional neural network are too large

poor real-time

high memory occupation and difficult to be widely used

a lightweight cascading inference based on bidirectional optical flow and multi-scale feature fusion is proposed. The network model decomposes the task of video frame insertion into two steps of inter-frame motion synthesis and texture reconstruction

designs a lightweight two-way optical flow prediction network

and proposes a multi-scale spatial and texture feature fusion network model. The multi-scale texture features and complex motion features of video frames are fully extracted and utilized. The model uses the position of two adjacent video frames and the required intermediate frames as network inputs. First

the spatial pyramid features and texture pyramid features of the two input frames are calculated. Then the spatial pyramid features are used to calculate the multi-scale bidirectional optical flow between frames

meanwhile

calculating the spatial and texture features of the intermediate frame. Finally

a fusing network is introduced to generate the multi-scale space and texture features of the intermediate frame to generate the final required video frame. Experiments on the Vimeo90K and UCF101 datasets show that

under the premise of guaranteeing accuracy

the algorithm in this paper has better performance in terms of calculation speed and model parameters.

关键词

Keywords

references

HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778. DOI: 10.1109/CVPR.2016.90.

SOOMRO K, ZAMIR A R, SHAH M. UCF101: a dataset of 101 human actions classes from videos in the wild[J]. arXiv preprint arXiv: 1212.0402, 2012.

NIKLAUS S, LIU F. Context-aware synthesis for video frame interpolation[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 1701-1710. DOI: 10.1109/CVPR.2018.00183.

MEYER S, DJELOUAH A, MCWILLIAMS B, et al. PhaseNet for video frame interpolation[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 498-507. DOI: 10.1109/CVPR.2018.00059.

SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv preprint arXiv: 1409.1556, 2014.

ILG E, MAYER N, SAIKIA T, et al. FlowNet 2.0: evolution of optical flow estimation with deep networks[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 2462-2470. DOI: 10.1109/CVPR.2017.179.

BAO W B, LAI W S, MA C, et al. Depth-aware video frame interpolation[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA: IEEE, 2019: 3703-3712.

JADERBERG M, SIMONYAN K, ZISSERMAN A. Spatial transformer networks[C]//Advances in Neural Information Processing Systems. 2015: 2017-2025.

LIU Z W, YEH R A, TANG X O, et al. Video frame synthesis using deep voxel flow[C]//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 4463-4471. DOI: 10.1109/ICCV.2017.478.

RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]//Proceedings of the 18th International Conference International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich, Germany: Springer, 2015: 234-241. DOI: 10.1007/978-3-319-24574-4_28.

NIKLAUS S, MAI L, LIU F. Video frame interpolation via adaptive separable convolution[C]//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 261-270. DOI: 10.1109/ICCV.2017.37.

XUE T F, CHEN B A, WU J J, et al. Video enhancement with task-oriented flow[J]. International Journal of Computer Vision, 2019, 127(8): 1106-1125. DOI: 10.1007/s11263-018-01144-2.

CHOI B D, HAN J W, KIM C S, et al. Motion-compensated frame interpolation using bilateral motion estimation and adaptive overlapped block motion compensation[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2007, 17(4): 407-416. DOI: 10.1109/TCSVT.2007.893835.

SU S C, DELBRACIO M, WANG J, et al. Deep video deblurring for hand-held cameras[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 1279-1288. DOI: 10.1109/CVPR.2017.33.

NIKLAUS S, MAI L, LIU F. Video frame interpolation via adaptive convolution[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 670-679. DOI: 10.1109/CVPR.2017.244.

DENG J, DONG W, SOCHER R, et al. ImageNet: a large-scale hierarchical image database[C]//Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, FL: IEEE, 2009: 248-255. DOI: 10.1109/CVPR.2009.5206848.

DOSOVITSKIY A, FISCHER P, ILG E, et al. FlowNet: learning optical flow with convolutional networks[C]//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile: IEEE, 2015: 2758-2766. DOI: 10.1109/ICCV.2015.316.

LEE W H, CHOI K, RA J B. Frame rate up conversion based on variational image fusion[J]. IEEE Transactions on Image Processing, 2014, 23(1): 399-412. DOI: 10.1109/TIP.2013.2288139.

JIANG H Z, SUN D Q, JAMPANI V, et al. Super SloMo: high quality estimation of multiple intermediate frames for video interpolation[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 9000-9008.

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

大模型发展综述：从BERT到Deepseek的技术演进

基于区块链的AI生成内容溯源框架设计研究

面向处理器分支预测单元的Spectre攻击轻量化防御方法

一种用于极低压boost系统的高效率电荷泵

超低功耗实时时钟电路的设计