A Parallel Acceleration Strategy for Distributed DBN in Spark

HUANG Zhen; QIAN Yu-rong; YU Jiong; Ying Chang-tian; ZHAO Jing-xia

Chinese

您当前的位置：

首页 >

文章列表页 >

A Parallel Acceleration Strategy for Distributed DBN in Spark

更新时间：2026-09-04

- A Parallel Acceleration Strategy for Distributed DBN in Spark
- Vol. 35, Issue 11, Pages: 100-105(2018)
- 作者机构：
  
  1. 新疆大学软件学院,新疆,乌鲁木齐,830091
  2. 新疆大学信息科学与工程学院,新疆,乌鲁木齐,830046
  3. 新疆大学电气工程学科博士后科研流动站,新疆,乌鲁木齐,830000
- 作者简介：
- 基金信息：
- DOI：
  CLC：
- Published：2018
- 稿件说明：
移动端阅览
黄震, 钱育蓉, 于炯, 英昌甜, 赵京霞. 一种Spark下分布式DBN并行加速策略[J]. 微电子学与计算机, 2018, 35(11): 100-105.

HUANG Zhen, QIAN Yu-rong, YU Jiong, et al. A Parallel Acceleration Strategy for Distributed DBN in Spark[J]. 2018, 35(11): 100-105.
黄震, 钱育蓉, 于炯, 英昌甜, 赵京霞. 一种Spark下分布式DBN并行加速策略[J]. 微电子学与计算机, 2018, 35(11): 100-105. DOI：

HUANG Zhen, QIAN Yu-rong, YU Jiong, et al. A Parallel Acceleration Strategy for Distributed DBN in Spark[J]. 2018, 35(11): 100-105. DOI：

摘要

Spark下分布式深度信念网络(Distributed Deep Belief Network

DDBN)存在数据倾斜、缺乏细粒度数据置换、无法自动缓存重用度高的数据等问题

导致了DDBN计算复杂高、运行时效性低的缺陷.为了提高DDBN的时效性

提出一种Spark下DDBN数据并行加速策略

其中包含基于标签集的范围分区(Label Set based on Range Partition

LSRP)算法和基于权重的缓存替换(Cache Replacement based on Weight

CRW)算法.通过LSRP算法解决数据倾斜问题

采用CRW算法解决RDD(Resilient Distributed Datasets)重复利用以及缓存数据过多造成内存空间不足问题.结果表明:与传统DBN相比

DDBN训练速度提高约2.3倍

通过LSRP和CRW大幅提高了DDBN分布式并行度.

Abstract

DDBN(Distributed Deep Belief Network

DDBN) has many problems in Spark

such as data skew

lack of fine-grained data replacement

and unable to cache data with high re-usability automatically

resulting in high complexity and low timeliness of DDBN computing. In order to improve the timeliness of DDBN

a parallel acceleration strategy is proposed for DDBN in Spark

which includes LSRP(Label Set based on Range Partition

LSRP) algorithm and CRWS(Cache Replacement based on Weight Statistics

CRWS) algorithm. The problem of data skew is solved by LSRP algorithm

and CRW algorithm is used to solve the problem of RDD reuse and cached data caused by insufficient memory space. The results show that compared with the traditional DBN

the training speed of DDBN is increased by about 2.3 times

and the distributed parallelism of DDBN is greatly improved through LSRP and CRW.

关键词

Keywords

references

Hodge V J, O'Keefe S, Austin J. Hadoop neural network for parallel and distributed feature selection[J]. Neural Networks the Official Journal of the International Neural Network Society, 2016 :78 :24-35. DOI: 24

Maillo J, Ramírez S, Triguero I, et al.KNN-IS:An Iterative Spark-based design of the k-Nearest Neighbors Classifier for BigData[J]. Knowledge-Based Systems, 2016.

Arias J, Gamez J A, Puerta J M.Learning istributed discrete Bayesian Network Classifiers under Map Reduce with Apache Spark[J].Knowledge-Based Systems, 2017.16-26 DOI: 16

王诏远, 王宏杰, 邢焕来, 等.基于Spark的蚁群优化算法[J].计算机应用, 2015, 35(10): 2777-2780. DOI: 2777

饶萍, 王建力, 王勇, 等.基于多特征决策树的建设用地信息提取[J].农业工程学报, 2014, 30(12): 233-240. DOI: 233

谭熊, 余旭初, 秦进春, 等.高光谱影像的多核SVM提取[J].信息安全与技术, 2014, 35(2): 405-411. DOI: 405

毛毅, 陈稳霖, 郭宝龙, 等.基于密度估计的逻辑回归模型[J].自动化学报, 2014, 40(1): 62-72. DOI: 62

Seide F, Fu H, Droppo J, et al. 1-bit stochastic gradient descent and its applicationto data-parallel distributed training of speech DNNs[J]. Interspeech, 2014:1058-1062. DOI: 1058

Alham N K, Li M, Liu Y, et al. A Map Reduce-based distributed SVM algorithm for automatic image annotation[J]. Computers % Mathematics with Applications, 2011, 62(7):2801-2811. DOI: 2801

张伐伐, 李卫忠, 卢柳叶, 等. SVM多窗口纹理土地利用信息提取技术[J].遥感学报, 2012, 16(1): 67-78. DOI: 67

Veetil S, Gao Q. Chapter 18-Real-time Network Intrusion Detection Using Hadoop-Based Bayesian Classifier[J]. Emerging Trends in Ict Security, 2014:281-299. DOI: 281

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Research on assembly action recognition based on MSS-YOLO-MPHands fusion architecture

Simulation on the DC characteristics of double heterojunction p-GaN HEMT with AlN/GaN hybrid buffer layer

A feature-oriented context understanding and slice analysis network for tuberculosis detection

Aerial small object detection algorithm based on improved RT-DETR

Monte carlo analysis of V-layout trench SiC neutron detector

Related Author

No data

Related Institution

No data

AI问答

Postal code：100079
Tel：（010）53879206 Email：tmw@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09082226号-64 京公网安备11010602201714号
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰