LI Cheng-yan, XIN Xue, ZHAO Shuai, et al. Sp-IEclat: A Big Data Parallel Association Rule Mining Algorithm[J]. 2021, 26(4): 109-118.
DOI:
LI Cheng-yan, XIN Xue, ZHAO Shuai, et al. Sp-IEclat: A Big Data Parallel Association Rule Mining Algorithm[J]. 2021, 26(4): 109-118.DOI: 10.15938/j.jhust.2021.04.015.
Sp-IEclat: A Big Data Parallel Association Rule Mining Algorithm
摘 要: 针对大数据环境下关联规则数据挖掘效率不高的问题,采用 Eclat 算法使用垂直数据库将事务的合并转换成集合操作的方法。研究了一种大数据并行关联规则挖掘算法 - Sp-IEclat ( Improved Eclat algorithm on Spark Framework) ,该算法基于内存计算的 Spark 框架,减少磁盘输入输出降低 I /O 负载,使用位图运算降低交集的时间代价并减少 CPU 占用,采用前缀划分的剪枝技术减少求交集运算的数据量,降低运算时间。使用 mushroom 数据集和 webdocs 数据集在两种大数据平台下实验,结果表明,Sp-IEclat 算法的时间效率优于 MapReduce 框架下的 Eclat 算法及 Spark框架下的 FP-Growth 算法和 Eclat 算法。从对集群的性能监控得到的数值表明,同 Spark 框架下的FP-Growth 算法和 Eclat 算法相比,Sp-IEclat 算法的 CPU 占用和 I /O 集群负载都较小。
Abstract
Aiming at the problem of inefficient data mining of association rules in a big data environment,the Eclat algorithm is used to use a vertical database to convert the merging of transactions into collective operations. We researched a big data parallel association rule mining algorithm-Sp-IEclat ( Improved Eclat algorithm on Spark Framework) . The algorithm is based on the Spark framework of memory computing,reduces disk input and output, reduces I /O load,and uses bitmap operations to reduce the time of intersection and CPU usage. The pruning technique of prefix division is used to reduce the amount of data in the intersection operation to reduce the operation time. The mushroom dataset and the webdocs dataset are used to test under two big data platforms. The experimental results show that the time efficiency of the Sp-IEclat algorithm is better than the Eclat algorithm under the MapReduce framework and the FP-Growth algorithm and the Eclat algorithm under the Spark framework. The value obtained from the performance monitoring of the cluster shows that,compared with the FP-Growth algorithm and the Eclat algorithm under the Spark framework,the CPU usage and I /O cluster load of Sp-IEclat are smaller.