南京工业大学 机构 计算机科学与技术学院,江苏,南京,211816
纸质出版:2018
移动端阅览
许明杰, 蔚承建, 沈航. 基于Spark的并行K-means算法研究[J]. 微电子学与计算机, 2018, 35(5): 95-99.
XU Ming-jie, YU Cheng-jian, SHEN Hang. Research on K-means Algorithm of Spark Parallelization[J]. 2018, 35(5): 95-99.
针对K-means算法在海量数据的处理过程中
由迭代计算次数加大导致的内存不足的问题
提出Spark并行化的K-means算法.将粒子群优化(PSO) 与K-means结合
利用PSO来提高K-means的全局搜索能力
得到初始聚类中心.利用Spark的迭代计算能力
将K-means算法与Spark并行框架结合
提升该算法模型对数据的处理速度
缩短算法的整体运行时间.通过疾病检测数据进行实验
结果表明Spark并行化的PSOK-means算法在保证准确率的同时大大提高了算法的效率
本算法对于海量数据的聚类研究有着很好的应用场景.
In view of the problem of insufficient memory caused by the increase of iterative computation in the process of mass data processing in K-means algorithm
this paper proposes K-means algorithm of Spark parallization.the algorithm uses particle swarm optimization (PSO) to improve the global search ability of K-means to get the initial clustering center.Through the iterative computing power of Spark
the K-means algorithm is combined with the Spark parallel framework to improve the processing speed of the model and reduce the overall running time of the algorithm.The experiment was carried out by disease detection data
the experimental results show that the Spark parallelized PSOK-means algorithm greatly improves the efficiency and accuracy of the algorithm
It will be good application scenarios for the clustering of massive data.
Liu J M, HAN Lichuan, HOU Liwen. Cluster analysis based on particle swarm optimizaiton algorithm[J]. System Engineering Theory and Practice, 2005, 6: 118-123. DOI: 118
Shboul B A, Myaeng S H. Initializing K-Means using Genetic Algorithms[C]//World Academy of Science, Engineering and Technology. IEEE, 2009.
Pal S K, Wang P P. Genetic algorithm for pattern recognition[M]. CRC Press, 1996.
Ahmadyfard A, Modares H. Combining PSO and kmeans to enhance data clustering[C]//International Symposium on Telicommunications. IEEE, 2008.
Alsabti K, Ranka S, Singh V. An efficient K-Means clustering algorithm[J]. Electrical Engineering and Computer Science, 1997, 23 (11): 1067-1080. DOI: 1067
Li J J, Yang X, Lu Y M, et al. Survey of particle swarm clustering algorithms[J]. Application Research of Computers, 2009, 26 (12): 4423-4427. DOI: 4423
0
浏览量
13
下载量
0
CSCD
关联资源
相关文章
相关作者
相关机构
京公网安备11010602201714号