Active overlapping K-means clustering algorithm based on spark

CHAI Bian-fang; LI You-yi

Chinese

您当前的位置：

首页 >

文章列表页 >

Active overlapping K-means clustering algorithm based on spark

更新时间：2026-09-04

- Active overlapping K-means clustering algorithm based on spark
- Vol. 38, Issue 1, Pages: 70-76(2021)
- 作者机构：
  
  河北地质大学机构信息工程学院,河北,石家庄,050031
- 作者简介：
- 基金信息：
- DOI：
  CLC：
- Published：2021
- 稿件说明：
移动端阅览
柴变芳, 李有熠. 基于Spark的主动重叠K-means聚类算法[J]. 微电子学与计算机, 2021, 38(1): 70-76.

CHAI Bian-fang, LI You-yi. Active overlapping K-means clustering algorithm based on spark[J]. 2021, 38(1): 70-76.
柴变芳, 李有熠. 基于Spark的主动重叠K-means聚类算法[J]. 微电子学与计算机, 2021, 38(1): 70-76. DOI：

CHAI Bian-fang, LI You-yi. Active overlapping K-means clustering algorithm based on spark[J]. 2021, 38(1): 70-76. DOI：

摘要

别大规模数据的潜在模式.但其存在两个问题：多次迭代Master和Worker节点间数据交换，导致算法运行效率低；对初始聚类中心敏感，导致聚类结果不稳定且收敛速度慢.为提高算法运行效率和结果稳定性，提出了一种主动重叠K-means聚类算法.其在各个分区上执行重叠K-means算法获得局部聚类中心，将结果汇总回收到Master节点，在Master节点运行重叠K-means算法聚合所有聚类中心，作为最终聚类中心；同时采用并行化主动选择策略获得更优的初始簇中心，提高算法准确性、收敛速度.实验结果表明，改进后的主动重叠聚类算法提高了算法准确性，降低了算法运行时间.

Abstract

Parallel Overlapping K-means clustering algorithm (POKM) based on Spark framework can effectively identify potential pattern of large-scale data. But multiple iterations of data exchange between the Master and the Worker nodes lead to low efficiency of the algorithm

and it is sensitive to the initial clustering center

resulting in unstable clustering results and slow convergence rate. In order to improve the performance and stability of the algorithm

an active overlapping K-means clustering algorithm is proposed. It performs the overlapping K-means algorithm on each worker and obtains the local cluster center

and then collects the centers and runs the overlapping K-means algorithm on the Master node. At the same time

the parallel active selection strategy is adopted to obtain a better initial cluster center to improve the accuracy and convergence speed. Experiment results show that the improved active overlapping clustering algorithm improves the accuracy and reduces the running time.

关键词

Keywords

references

何玉林, 黄哲学.大规模数据集聚类算法的研究进展[J].深圳大学学报(理工版), 2019, 36(1): 4-17. DOI: 10.372 4/SP.J.12 49.2019.0100 4. HE Y L, HUANG Z X. A review on clustering algorithms for large-scale data sets[J]. Journal of Shenzhen University (Science % Engineering), 2019, 36(1): 4-17. DOI: 10.372 4/SP.J.12 49.2019.0100 4.

JAIN A K, MURTY M N, FLYNN P J. Data clustering: a review[J]. ACM Computing Surveys, 1993, 31(3): 264-323. DOI: 10.1145/331499.331504.

ZAYANI A, N'CIR C E B, ESSOUSSI N. Parallel clustering method for non-disjoint partitioning of large-scale data based on spark framework[C]//Proceedings of 2016 IEEE International Conference on Big Data (Big Data). Washington: IEEE, 2016. DOI: 10.1109/BigData.2016.7840708.

王铭坤, 袁少光, 朱永利, 等.基于Storm的海量数据实时聚类[J].计算机应用, 2014, 34(11): 3078-3081. DOI: 10.11772/j.issn.1001-9081.2014.11. 3078. WANG M K, YUAN S G, ZHU Y L, et al. Real-time clustering for massive data using Storm[J]. Journal of Computer Applications, 2014, 34(11): 3078-3081. DOI: 10.11772/j.issn.1001-9081.2014.11. 3078.

ZHAO L, CHEN Z K, YANG L. Incremental CFS clustering on large data[C]//Proceedings of 2017 IEEE Global Conference on Signal and Information Processing. Montreal, Canada: IEEE, 2017: 687-690. DOI: 10.1109/GlobalSIP.2017.8309047.

CHEN X J, HONG W J, NIE F P, et al. Directly minimizing normalized cut for large scale data[C]//Proceedings of ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, USA: ACM, 2018: 1206-1215.

CLEUZIOU G. An extended version of the k-means method for overlapping clustering[C]//Proceedings of the 19th International Conference on Pattern Recognition. Tampa: IEEE, 2009. DOI: 10.1109/ICPR.2008.4761079.

杜世强.基于维数约简的无监督聚类算法研究[D].兰州: 兰州大学, 2017. DU S Q. Unsupervised clustering algorithm based on dimension reduction[D]. Lanzhou: Lanzhou University, 2017.

马国峻, 王水波, 裴庆祺, 等.基于主成分分析和k-means聚类的平行坐标可视化技术研究[J].网络与信息安全学报, 2017, 3(8): 18-27. DOI: 10.11959/j.issn.2096-109x.2017.00 189. MA G J, WANG S B, PEI Q Q, et al. Research on parallel coordinate visualization technology based on principal component analysis and K-means clustering[J]. Chinese Journal of Network and Information Security, 2017, 3(8): 18-27. DOI: 10.11959/j.issn.2096-109x.2017.00 189.

魏丞昊, 黄哲学, 何玉林.基于统计感知的大数据系统计算框架[J].深圳大学学报(理工版), 2018, 35(5): 441-443. DOI: 10.3724/SP.J.1249.2018.05 441. WEI C H, HUANG Z X, HE Y L. A statistical aware based big data system computing framework[J]. Journal of Shenzhen University (Science % Engineering), 2018, 35(5): 441-443. DOI: 10.3724/SP.J.1249.2018.05 441.

虞倩倩, 戴月明.基于MapReduce的并行模糊C均值算法[J].计算机工程与应用, 2013, 49(14): 133-137. DOI: 10.3778/j.issn.1002-8331.1111-0261. YU Q Q, DAI Y M. Parallel fuzzy C-means algorithm based on MapReduce[J]. Computer Engineering and Applications, 2013, 49(14): 133-137. DOI: 10.3778/j.issn.1002-8331.1111-0261.

BASU S, BANERJEE A, MOONEY R J. Active semi-supervision for pairwise constrained clustering[C]//Proceedings of the 4th SIAM International Conference on Data Mining. Lake Buena Vista: Society for Industrial and Applied Mathematics, 2004: 333-344.

MENG X R, BRADLEY J, YAVUZ B, et al. MLlib: machine learning in apache Spark[J]. The Journal of Machine Learning Research, 2016, 17(1): 1235-1241. DOI: 1235

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Aerial small object detection algorithm based on improved RT-DETR

A high-efficiency charge pump for ultra-low voltage boost systems

A review of large model development: technological evolution from BERT to Deepseek

Steganography model guided by attention mechanism and improved dense convolution

Research on assembly action recognition based on MSS-YOLO-MPHands fusion architecture

Related Author

No data

Related Institution

No data

AI问答

Postal code：100079
Tel：（010）53879206 Email：tmw@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09082226号-64 京公网安备11010602201714号
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰