Research on K-means Algorithm of Spark Parallelization

XU Ming-jie; YU Cheng-jian; SHEN Hang

Chinese

您当前的位置：

首页 >

文章列表页 >

Research on K-means Algorithm of Spark Parallelization

更新时间：2026-09-04

- Research on K-means Algorithm of Spark Parallelization
- Vol. 35, Issue 5, Pages: 95-99(2018)
- 作者机构：
  
  南京工业大学机构计算机科学与技术学院,江苏,南京,211816
- 作者简介：
- 基金信息：
- DOI：
  CLC：
- Published：2018
- 稿件说明：
移动端阅览
许明杰, 蔚承建, 沈航. 基于Spark的并行K-means算法研究[J]. 微电子学与计算机, 2018, 35(5): 95-99.

XU Ming-jie, YU Cheng-jian, SHEN Hang. Research on K-means Algorithm of Spark Parallelization[J]. 2018, 35(5): 95-99.
许明杰, 蔚承建, 沈航. 基于Spark的并行K-means算法研究[J]. 微电子学与计算机, 2018, 35(5): 95-99. DOI：

XU Ming-jie, YU Cheng-jian, SHEN Hang. Research on K-means Algorithm of Spark Parallelization[J]. 2018, 35(5): 95-99. DOI：

摘要

针对K-means算法在海量数据的处理过程中

由迭代计算次数加大导致的内存不足的问题

提出Spark并行化的K-means算法.将粒子群优化(PSO) 与K-means结合

利用PSO来提高K-means的全局搜索能力

得到初始聚类中心.利用Spark的迭代计算能力

将K-means算法与Spark并行框架结合

提升该算法模型对数据的处理速度

缩短算法的整体运行时间.通过疾病检测数据进行实验

结果表明Spark并行化的PSOK-means算法在保证准确率的同时大大提高了算法的效率

本算法对于海量数据的聚类研究有着很好的应用场景.

Abstract

In view of the problem of insufficient memory caused by the increase of iterative computation in the process of mass data processing in K-means algorithm

this paper proposes K-means algorithm of Spark parallization.the algorithm uses particle swarm optimization (PSO) to improve the global search ability of K-means to get the initial clustering center.Through the iterative computing power of Spark

the K-means algorithm is combined with the Spark parallel framework to improve the processing speed of the model and reduce the overall running time of the algorithm.The experiment was carried out by disease detection data

the experimental results show that the Spark parallelized PSOK-means algorithm greatly improves the efficiency and accuracy of the algorithm

It will be good application scenarios for the clustering of massive data.

关键词

Keywords

references

Liu J M, HAN Lichuan, HOU Liwen. Cluster analysis based on particle swarm optimizaiton algorithm[J]. System Engineering Theory and Practice, 2005, 6: 118-123. DOI: 118

Shboul B A, Myaeng S H. Initializing K-Means using Genetic Algorithms[C]//World Academy of Science, Engineering and Technology. IEEE, 2009.

Pal S K, Wang P P. Genetic algorithm for pattern recognition[M]. CRC Press, 1996.

Ahmadyfard A, Modares H. Combining PSO and kmeans to enhance data clustering[C]//International Symposium on Telicommunications. IEEE, 2008.

Alsabti K, Ranka S, Singh V. An efficient K-Means clustering algorithm[J]. Electrical Engineering and Computer Science, 1997, 23 (11): 1067-1080. DOI: 1067

Li J J, Yang X, Lu Y M, et al. Survey of particle swarm clustering algorithms[J]. Application Research of Computers, 2009, 26 (12): 4423-4427. DOI: 4423

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Research on the design of blockchain-based AI-generated content traceability framework

Monte carlo analysis of V-layout trench SiC neutron detector

Microservices anomaly detection based on the correlation of monitoring metrics

Steganography model guided by attention mechanism and improved dense convolution

Aerial small object detection algorithm based on improved RT-DETR

Related Author

No data

Related Institution

No data

AI问答

Postal code：100079
Tel：（010）53879206 Email：tmw@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09082226号-64 京公网安备11010602201714号
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰