基于Hadoop的大数据频繁模式挖掘算法

李校林; 杜托; 谢勇

Chinese

您当前的位置：

首页 >

文章列表页 >

基于Hadoop的大数据频繁模式挖掘算法

更新时间：2026-09-04

- 基于Hadoop的大数据频繁模式挖掘算法
- Algorithm for Mining Frequent Patterns in Big Data Based on Hadoop
- 微电子学与计算机 2018年35卷第9期页码：14-19
- 作者机构：
  
  1. 重庆邮电大学机构通信新技术应用研究中心,重庆,400065
  2. 重庆信科设计有限公司,重庆,40065
- 作者简介：
- 基金信息：
  
  重庆市研究生科研创新基金 CYS15166
- DOI：
  中图分类号：
- 纸质出版：2018
- 稿件说明：
移动端阅览
李校林, 杜托, 谢勇. 基于Hadoop的大数据频繁模式挖掘算法[J]. 微电子学与计算机, 2018, 35(9): 14-19.

LI Xiao-lin, DU Tuo, XIE Yong. Algorithm for Mining Frequent Patterns in Big Data Based on Hadoop[J]. 2018, 35(9): 14-19.
李校林, 杜托, 谢勇. 基于Hadoop的大数据频繁模式挖掘算法[J]. 微电子学与计算机, 2018, 35(9): 14-19. DOI：

LI Xiao-lin, DU Tuo, XIE Yong. Algorithm for Mining Frequent Patterns in Big Data Based on Hadoop[J]. 2018, 35(9): 14-19. DOI：

摘要

针对传统的频繁模式挖掘算法不能满足大数据环境下的挖掘需要

提出一种高效挖掘大型数据库中频繁模式的并行算法H_PrePost.首先从压缩数据库、简化数据表示以及采用高效的连接和剪枝策略等方面对PrePost算法进行改进

以提高单机模式下的挖掘效率.然后将改进算法迁移到Hadoop平台上

利用MapReduce模型进行并行计算

同时提出一种负载均衡策略保证集群高效运行.最后使用kulczynski度量和不平衡比对所挖掘的频繁模式进行评估

以确保所挖掘模式具有实际应用价值.实验结果表明

H_PrePost算法可以有效挖掘大数据集中的频繁模式.

Abstract

Aiming at the traditional frequent pattern mining algorithm can not meet the needs of mining in big data environment

a parallel algorithm for efficiently mining frequent patterns in large databases is proposed. Firstly

PrePost algorithm is improved from compressing database

simplifying data representation and using efficient connection and pruning strategy

which improve the efficiency of mining in stand-alone mode. Then

the improved algorithm is migrated to the Hadoop platform and the MapReduce model is used for parallel computing. A load balancing strategy is proposed to ensure the efficient operation of the cluster. Finally

the frequent pattern mining is evaluated using kulczynski metric and unbalance ratio to ensure that the mining pattern has practical value. Experimental results show that this algorithm can effectively mine the frequent patterns in big data sets.

关键词

Keywords

references

Sahoo J, Das A K, Goswami A. An efficient approach for mining association rules from high utility itemsets[J]. Expert Systems with Applications, 2015, 42(13): 5754-5778. DOI: 5754

Agrawal R, Srikant R. Fast algorithms for mining association rules[C]//Proc. 20th int. conf. very large data bases, VLDB. Zurich, Switzerland, 1994(1215): 487-499.

Song W, Yang B, Xu Z. Index-BitTableFI: An improved algorithm for mining frequent itemsets[J]. Knowledge-Based Systems, 2008, 21(6): 507-513. DOI: 507

Deng Z H, Wang Z H, Jiang J J. A new algorithm for fast mining frequent itemsets using N-lists[J]. Science China Information Sciences, 2012, 55(9): 2008-2030. DOI: 2008

Aji A, Wang F, Vo H, et al. Hadoop gis: a high performance spatial data warehousing system over mapreduce[J]. the VLDB Endowment, 2013, 6(11): 1009-1020. DOI: 1009

Qibing Q, Long T. An effective hybrid algorithm for fast mining frequent itemsets[C]//Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC), 2016 International Conference on. Zhengzhou, China, IEEE, 2016: 91-95.

Han J, Pei J, Yin Y. Mining frequent patterns without candidate generation[C]// Acm Sigmod International Conference on Managemenet of data. Dallas, Texas: ACM Press, 2000, 29 (2) : 1-12.

Dam T L, Li K, Fournier-Viger P, et al. An efficient algorithm for mining top-rank-k frequent patterns[J]. Applied Intelligence, 2016, 45(1): 96-111. DOI: 96

Pyun G, Yun U, Ryu K H. Efficient frequent pattern mining based on linear prefix tree[J]. Knowledge-Based Systems, 2014(55): 125-139. DOI: 125

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

云服务环境下的大数据多标签属性分类技术

基于MapReduce的并行加权FIUT算法

基于FPGA的高效数据过滤技术研究

面向钻井大数据的数据集成及分析系统的设计与实现

面向大数据的并行分类混合算法研究