Lightweight cascade network for video frame interpolation

ZHANG Pei-jian; TENG Qi-zhi; HE Xiao-hai

Chinese

您当前的位置：

首页 >

文章列表页 >

Lightweight cascade network for video frame interpolation

更新时间：2026-09-04

- Lightweight cascade network for video frame interpolation
- Vol. 38, Issue 3, Pages: 39-45(2021)
- 作者机构：
  
  四川大学电子信息学院,四川,成都,610065
- 作者简介：
- 基金信息：
- DOI：
  CLC：
- Published：2021
- 稿件说明：
移动端阅览
张培健, 滕奇志, 何小海. 基于级联卷积神经网络的轻量级视频插帧算法[J]. 微电子学与计算机, 2021, 38(3): 39-45.

ZHANG Pei-jian, TENG Qi-zhi, HE Xiao-hai. Lightweight cascade network for video frame interpolation[J]. 2021, 38(3): 39-45.
张培健, 滕奇志, 何小海. 基于级联卷积神经网络的轻量级视频插帧算法[J]. 微电子学与计算机, 2021, 38(3): 39-45. DOI：

ZHANG Pei-jian, TENG Qi-zhi, HE Xiao-hai. Lightweight cascade network for video frame interpolation[J]. 2021, 38(3): 39-45. DOI：

摘要

针对基于卷积神经网络的视频插帧算法模型参数过大、实时性差、内存占用高、难以广泛应用的问题，提出了一种基于双向光流和多尺度特征融合的轻量级级联推理网络模型.将视频插帧任务分解为帧间运动合成和纹理重建两个步骤，设计了一种轻量级的级联双向光流预测网络，并提出了一种多尺度空间与纹理特征融合网络模型，实现了对视频帧多尺度纹理特征和复杂运动特征的充分提取和利用.该模型使用相邻的两个视频帧和所需中间帧的位置作为网络输入，首先计算两个输入帧的空间金字塔特征与纹理金字塔特征；然后使用空间金字塔特征计算帧间的多尺度双向光流；随后结合双向光流，计算出中间帧的空间特征和纹理特征；最后融合中间帧的多尺度空间、纹理特征得到最终所需的视频帧.在Vimeo90K和UCF101数据集上的实验表明，在保证精度的前提下，本文算法在计算速度和模型参数量上具有更好的表现.

Abstract

Aiming at the problems that the model parameters of video interpolation algorithm based on convolutional neural network are too large

poor real-time

high memory occupation and difficult to be widely used

a lightweight cascading inference based on bidirectional optical flow and multi-scale feature fusion is proposed. The network model decomposes the task of video frame insertion into two steps of inter-frame motion synthesis and texture reconstruction

designs a lightweight two-way optical flow prediction network

and proposes a multi-scale spatial and texture feature fusion network model. The multi-scale texture features and complex motion features of video frames are fully extracted and utilized. The model uses the position of two adjacent video frames and the required intermediate frames as network inputs. First

the spatial pyramid features and texture pyramid features of the two input frames are calculated. Then the spatial pyramid features are used to calculate the multi-scale bidirectional optical flow between frames

meanwhile

calculating the spatial and texture features of the intermediate frame. Finally

a fusing network is introduced to generate the multi-scale space and texture features of the intermediate frame to generate the final required video frame. Experiments on the Vimeo90K and UCF101 datasets show that

under the premise of guaranteeing accuracy

the algorithm in this paper has better performance in terms of calculation speed and model parameters.

关键词

Keywords

references

HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas: IEEE, 2016: 770-778. DOI: 10.1109/CVPR.2016.90.

SOOMRO K, ZAMIR A R, SHAH M. UCF101: a dataset of 101 human actions classes from videos in the wild[J]. arXiv preprint arXiv: 1212.0402, 2012.

NIKLAUS S, LIU F. Context-aware synthesis for video frame interpolation[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 1701-1710. DOI: 10.1109/CVPR.2018.00183.

MEYER S, DJELOUAH A, MCWILLIAMS B, et al. PhaseNet for video frame interpolation[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 498-507. DOI: 10.1109/CVPR.2018.00059.

SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[J]. arXiv preprint arXiv: 1409.1556, 2014.

ILG E, MAYER N, SAIKIA T, et al. FlowNet 2.0: evolution of optical flow estimation with deep networks[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 2462-2470. DOI: 10.1109/CVPR.2017.179.

BAO W B, LAI W S, MA C, et al. Depth-aware video frame interpolation[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, CA: IEEE, 2019: 3703-3712.

JADERBERG M, SIMONYAN K, ZISSERMAN A. Spatial transformer networks[C]//Advances in Neural Information Processing Systems. 2015: 2017-2025.

LIU Z W, YEH R A, TANG X O, et al. Video frame synthesis using deep voxel flow[C]//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 4463-4471. DOI: 10.1109/ICCV.2017.478.

RONNEBERGER O, FISCHER P, BROX T. U-Net: convolutional networks for biomedical image segmentation[C]//Proceedings of the 18th International Conference International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich, Germany: Springer, 2015: 234-241. DOI: 10.1007/978-3-319-24574-4_28.

NIKLAUS S, MAI L, LIU F. Video frame interpolation via adaptive separable convolution[C]//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 261-270. DOI: 10.1109/ICCV.2017.37.

XUE T F, CHEN B A, WU J J, et al. Video enhancement with task-oriented flow[J]. International Journal of Computer Vision, 2019, 127(8): 1106-1125. DOI: 10.1007/s11263-018-01144-2.

CHOI B D, HAN J W, KIM C S, et al. Motion-compensated frame interpolation using bilateral motion estimation and adaptive overlapped block motion compensation[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2007, 17(4): 407-416. DOI: 10.1109/TCSVT.2007.893835.

SU S C, DELBRACIO M, WANG J, et al. Deep video deblurring for hand-held cameras[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 1279-1288. DOI: 10.1109/CVPR.2017.33.

NIKLAUS S, MAI L, LIU F. Video frame interpolation via adaptive convolution[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu: IEEE, 2017: 670-679. DOI: 10.1109/CVPR.2017.244.

DENG J, DONG W, SOCHER R, et al. ImageNet: a large-scale hierarchical image database[C]//Proceedings of 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, FL: IEEE, 2009: 248-255. DOI: 10.1109/CVPR.2009.5206848.

DOSOVITSKIY A, FISCHER P, ILG E, et al. FlowNet: learning optical flow with convolutional networks[C]//Proceedings of 2015 IEEE International Conference on Computer Vision. Santiago, Chile: IEEE, 2015: 2758-2766. DOI: 10.1109/ICCV.2015.316.

LEE W H, CHOI K, RA J B. Frame rate up conversion based on variational image fusion[J]. IEEE Transactions on Image Processing, 2014, 23(1): 399-412. DOI: 10.1109/TIP.2013.2288139.

JIANG H Z, SUN D Q, JAMPANI V, et al. Super SloMo: high quality estimation of multiple intermediate frames for video interpolation[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City: IEEE, 2018: 9000-9008.

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Steganography model guided by attention mechanism and improved dense convolution

Monte carlo analysis of V-layout trench SiC neutron detector

Aerial vehicle detection algorithm based on multi-scale fusion and small-object enhancement

Research on assembly action recognition based on MSS-YOLO-MPHands fusion architecture

Lightweight countermeasures against Spectre attacks on the branch prediction unit

Related Author

No data

Related Institution

No data

AI问答

Postal code：100079
Tel：（010）53879206 Email：tmw@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09082226号-64 京公网安备11010602201714号
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰