Algorithm for Mining Frequent Patterns in Big Data Based on Hadoop

LI Xiao-lin; DU Tuo; XIE Yong

Chinese

您当前的位置：

首页 >

文章列表页 >

Algorithm for Mining Frequent Patterns in Big Data Based on Hadoop

更新时间：2026-09-04

- Algorithm for Mining Frequent Patterns in Big Data Based on Hadoop
- Vol. 35, Issue 9, Pages: 14-19(2018)
- 作者机构：
  
  1. 重庆邮电大学机构通信新技术应用研究中心,重庆,400065
  2. 重庆信科设计有限公司,重庆,40065
- 作者简介：
- 基金信息：
- DOI：
  CLC：
- Published：2018
- 稿件说明：
移动端阅览
李校林, 杜托, 谢勇. 基于Hadoop的大数据频繁模式挖掘算法[J]. 微电子学与计算机, 2018, 35(9): 14-19.

LI Xiao-lin, DU Tuo, XIE Yong. Algorithm for Mining Frequent Patterns in Big Data Based on Hadoop[J]. 2018, 35(9): 14-19.
李校林, 杜托, 谢勇. 基于Hadoop的大数据频繁模式挖掘算法[J]. 微电子学与计算机, 2018, 35(9): 14-19. DOI：

LI Xiao-lin, DU Tuo, XIE Yong. Algorithm for Mining Frequent Patterns in Big Data Based on Hadoop[J]. 2018, 35(9): 14-19. DOI：

摘要

针对传统的频繁模式挖掘算法不能满足大数据环境下的挖掘需要

提出一种高效挖掘大型数据库中频繁模式的并行算法H_PrePost.首先从压缩数据库、简化数据表示以及采用高效的连接和剪枝策略等方面对PrePost算法进行改进

以提高单机模式下的挖掘效率.然后将改进算法迁移到Hadoop平台上

利用MapReduce模型进行并行计算

同时提出一种负载均衡策略保证集群高效运行.最后使用kulczynski度量和不平衡比对所挖掘的频繁模式进行评估

以确保所挖掘模式具有实际应用价值.实验结果表明

H_PrePost算法可以有效挖掘大数据集中的频繁模式.

Abstract

Aiming at the traditional frequent pattern mining algorithm can not meet the needs of mining in big data environment

a parallel algorithm for efficiently mining frequent patterns in large databases is proposed. Firstly

PrePost algorithm is improved from compressing database

simplifying data representation and using efficient connection and pruning strategy

which improve the efficiency of mining in stand-alone mode. Then

the improved algorithm is migrated to the Hadoop platform and the MapReduce model is used for parallel computing. A load balancing strategy is proposed to ensure the efficient operation of the cluster. Finally

the frequent pattern mining is evaluated using kulczynski metric and unbalance ratio to ensure that the mining pattern has practical value. Experimental results show that this algorithm can effectively mine the frequent patterns in big data sets.

关键词

Keywords

references

Sahoo J, Das A K, Goswami A. An efficient approach for mining association rules from high utility itemsets[J]. Expert Systems with Applications, 2015, 42(13): 5754-5778. DOI: 5754

Agrawal R, Srikant R. Fast algorithms for mining association rules[C]//Proc. 20th int. conf. very large data bases, VLDB. Zurich, Switzerland, 1994(1215): 487-499.

Song W, Yang B, Xu Z. Index-BitTableFI: An improved algorithm for mining frequent itemsets[J]. Knowledge-Based Systems, 2008, 21(6): 507-513. DOI: 507

Deng Z H, Wang Z H, Jiang J J. A new algorithm for fast mining frequent itemsets using N-lists[J]. Science China Information Sciences, 2012, 55(9): 2008-2030. DOI: 2008

Aji A, Wang F, Vo H, et al. Hadoop gis: a high performance spatial data warehousing system over mapreduce[J]. the VLDB Endowment, 2013, 6(11): 1009-1020. DOI: 1009

Qibing Q, Long T. An effective hybrid algorithm for fast mining frequent itemsets[C]//Cyber-Enabled Distributed Computing and Knowledge Discovery (CyberC), 2016 International Conference on. Zhengzhou, China, IEEE, 2016: 91-95.

Han J, Pei J, Yin Y. Mining frequent patterns without candidate generation[C]// Acm Sigmod International Conference on Managemenet of data. Dallas, Texas: ACM Press, 2000, 29 (2) : 1-12.

Dam T L, Li K, Fournier-Viger P, et al. An efficient algorithm for mining top-rank-k frequent patterns[J]. Applied Intelligence, 2016, 45(1): 96-111. DOI: 96

Pyun G, Yun U, Ryu K H. Efficient frequent pattern mining based on linear prefix tree[J]. Knowledge-Based Systems, 2014(55): 125-139. DOI: 125

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Big Aata Multi-label Attribute Classification Technology in Cloud Service Environment

Parallel Weighted FIUT Algorithm Based on MapReduce

Research on Efficient Data Filtering Based on FPGA

The Implementation of Data Integration and Analysis System for Big Data of Drilling

Research on Parallel Classification Hybrid Algorithm for Big Data

Related Author

LIN Qian-yu

Li Qi

ZHANG Ping-kang

ZHANG Xin

ZHANG Hang

LIANG Zhuang

ZHANG Xiu-zhi

CHEN Bao-yuan

Related Institution

Chengyi University College, Jimei University

College of Big Data and Information Engineering, Guizhou University

The Higher Education Key Laboratory for Measuring & Control Technology and Instrumentations of Heilongjiang Province, Harbin University of Science & Technology

CNPC Changqing Drilling Engineering Company

Northwesern Polytechnical University

AI问答

Postal code：100079
Tel：（010）53879206 Email：tmw@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09082226号-64 京公网安备11010602201714号
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰