基于Spark的主动重叠K-means聚类算法

柴变芳; 李有熠

Chinese

您当前的位置：

首页 >

文章列表页 >

基于Spark的主动重叠K-means聚类算法

更新时间：2026-09-04

- 基于Spark的主动重叠K-means聚类算法
- Active overlapping K-means clustering algorithm based on spark
- 微电子学与计算机 2021年38卷第1期页码：70-76
- 作者机构：
  
  河北地质大学机构信息工程学院,河北,石家庄,050031
- 作者简介：
- 基金信息：
  
  河北省自然科学基金 F2019403070
- DOI：
  中图分类号：
- 纸质出版：2021
- 稿件说明：
移动端阅览
柴变芳, 李有熠. 基于Spark的主动重叠K-means聚类算法[J]. 微电子学与计算机, 2021, 38(1): 70-76.

CHAI Bian-fang, LI You-yi. Active overlapping K-means clustering algorithm based on spark[J]. 2021, 38(1): 70-76.
柴变芳, 李有熠. 基于Spark的主动重叠K-means聚类算法[J]. 微电子学与计算机, 2021, 38(1): 70-76. DOI：

CHAI Bian-fang, LI You-yi. Active overlapping K-means clustering algorithm based on spark[J]. 2021, 38(1): 70-76. DOI：

摘要

别大规模数据的潜在模式.但其存在两个问题：多次迭代Master和Worker节点间数据交换，导致算法运行效率低；对初始聚类中心敏感，导致聚类结果不稳定且收敛速度慢.为提高算法运行效率和结果稳定性，提出了一种主动重叠K-means聚类算法.其在各个分区上执行重叠K-means算法获得局部聚类中心，将结果汇总回收到Master节点，在Master节点运行重叠K-means算法聚合所有聚类中心，作为最终聚类中心；同时采用并行化主动选择策略获得更优的初始簇中心，提高算法准确性、收敛速度.实验结果表明，改进后的主动重叠聚类算法提高了算法准确性，降低了算法运行时间.

Abstract

Parallel Overlapping K-means clustering algorithm (POKM) based on Spark framework can effectively identify potential pattern of large-scale data. But multiple iterations of data exchange between the Master and the Worker nodes lead to low efficiency of the algorithm

and it is sensitive to the initial clustering center

resulting in unstable clustering results and slow convergence rate. In order to improve the performance and stability of the algorithm

an active overlapping K-means clustering algorithm is proposed. It performs the overlapping K-means algorithm on each worker and obtains the local cluster center

and then collects the centers and runs the overlapping K-means algorithm on the Master node. At the same time

the parallel active selection strategy is adopted to obtain a better initial cluster center to improve the accuracy and convergence speed. Experiment results show that the improved active overlapping clustering algorithm improves the accuracy and reduces the running time.

关键词

Keywords

references

何玉林, 黄哲学.大规模数据集聚类算法的研究进展[J].深圳大学学报(理工版), 2019, 36(1): 4-17. DOI: 10.372 4/SP.J.12 49.2019.0100 4. HE Y L, HUANG Z X. A review on clustering algorithms for large-scale data sets[J]. Journal of Shenzhen University (Science % Engineering), 2019, 36(1): 4-17. DOI: 10.372 4/SP.J.12 49.2019.0100 4.

JAIN A K, MURTY M N, FLYNN P J. Data clustering: a review[J]. ACM Computing Surveys, 1993, 31(3): 264-323. DOI: 10.1145/331499.331504.

ZAYANI A, N'CIR C E B, ESSOUSSI N. Parallel clustering method for non-disjoint partitioning of large-scale data based on spark framework[C]//Proceedings of 2016 IEEE International Conference on Big Data (Big Data). Washington: IEEE, 2016. DOI: 10.1109/BigData.2016.7840708.

王铭坤, 袁少光, 朱永利, 等.基于Storm的海量数据实时聚类[J].计算机应用, 2014, 34(11): 3078-3081. DOI: 10.11772/j.issn.1001-9081.2014.11. 3078. WANG M K, YUAN S G, ZHU Y L, et al. Real-time clustering for massive data using Storm[J]. Journal of Computer Applications, 2014, 34(11): 3078-3081. DOI: 10.11772/j.issn.1001-9081.2014.11. 3078.

ZHAO L, CHEN Z K, YANG L. Incremental CFS clustering on large data[C]//Proceedings of 2017 IEEE Global Conference on Signal and Information Processing. Montreal, Canada: IEEE, 2017: 687-690. DOI: 10.1109/GlobalSIP.2017.8309047.

CHEN X J, HONG W J, NIE F P, et al. Directly minimizing normalized cut for large scale data[C]//Proceedings of ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, USA: ACM, 2018: 1206-1215.

CLEUZIOU G. An extended version of the k-means method for overlapping clustering[C]//Proceedings of the 19th International Conference on Pattern Recognition. Tampa: IEEE, 2009. DOI: 10.1109/ICPR.2008.4761079.

杜世强.基于维数约简的无监督聚类算法研究[D].兰州: 兰州大学, 2017. DU S Q. Unsupervised clustering algorithm based on dimension reduction[D]. Lanzhou: Lanzhou University, 2017.

马国峻, 王水波, 裴庆祺, 等.基于主成分分析和k-means聚类的平行坐标可视化技术研究[J].网络与信息安全学报, 2017, 3(8): 18-27. DOI: 10.11959/j.issn.2096-109x.2017.00 189. MA G J, WANG S B, PEI Q Q, et al. Research on parallel coordinate visualization technology based on principal component analysis and K-means clustering[J]. Chinese Journal of Network and Information Security, 2017, 3(8): 18-27. DOI: 10.11959/j.issn.2096-109x.2017.00 189.

魏丞昊, 黄哲学, 何玉林.基于统计感知的大数据系统计算框架[J].深圳大学学报(理工版), 2018, 35(5): 441-443. DOI: 10.3724/SP.J.1249.2018.05 441. WEI C H, HUANG Z X, HE Y L. A statistical aware based big data system computing framework[J]. Journal of Shenzhen University (Science % Engineering), 2018, 35(5): 441-443. DOI: 10.3724/SP.J.1249.2018.05 441.

虞倩倩, 戴月明.基于MapReduce的并行模糊C均值算法[J].计算机工程与应用, 2013, 49(14): 133-137. DOI: 10.3778/j.issn.1002-8331.1111-0261. YU Q Q, DAI Y M. Parallel fuzzy C-means algorithm based on MapReduce[J]. Computer Engineering and Applications, 2013, 49(14): 133-137. DOI: 10.3778/j.issn.1002-8331.1111-0261.

BASU S, BANERJEE A, MOONEY R J. Active semi-supervision for pairwise constrained clustering[C]//Proceedings of the 4th SIAM International Conference on Data Mining. Lake Buena Vista: Society for Industrial and Applied Mathematics, 2004: 333-344.

MENG X R, BRADLEY J, YAVUZ B, et al. MLlib: machine learning in apache Spark[J]. The Journal of Machine Learning Research, 2016, 17(1): 1235-1241. DOI: 1235

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

V型沟槽SiC中子探测器的蒙特卡洛分析

基于反向体偏置的高增益稳定性高温运放设计

注意力机制与改进密集卷积引导的信息隐藏模型

大模型发展综述：从BERT到Deepseek的技术演进

超低功耗实时时钟电路的设计