Parallel High Utility Itemset Mining Algorithm on the Spark

Abstract

In the field of efficient utility itemset mining, considering both internal and external utility values provides a more comprehensive approach compared to traditional frequency-based methods. However, the increased complexity of computations and the generation of numerous candidate itemsets pose challenges for efficient mining on large-scale datasets. To address these challenges, this paper proposes a parallel mining algorithm based on the Spark framework. The algorithm leverages a vertical dataset structure to efficiently store and process the data. A utility table is utilized to store the data items along with their corresponding transaction utility values. By utilizing the utility table, the algorithm can directly access transaction utility values, simplifying the computation process and reducing overhead. To further enhance efficiency, the algorithm combines a prefix partitioning strategy with a minimum utility threshold. By employing this strategy, the generation of candidate itemsets is effectively reduced, resulting in a smaller search space and enhancing the efficiency of the mining process. The algorithm is implemented on the Spark framework, leveraging its capabilities in parallel processing and scalability. By leveraging the distributed computing capabilities of Spark, the algorithm can efficiently mine efficient utility frequent item-sets from large-scale datasets. Experimental results demonstrate the effectiveness and efficiency of the proposed algorithm in performing efficient utility itemset mining. It surpasses traditional approaches and showcases its ability to handle large-scale datasets while maintaining high performance.