Research on Parallel Text Classification System Based on Non-Balanced LSH

ZHAO Tong; LIU Bin; LI Tao

Chinese

您当前的位置：

首页 >

文章列表页 >

Research on Parallel Text Classification System Based on Non-Balanced LSH

更新时间：2026-09-04

- Research on Parallel Text Classification System Based on Non-Balanced LSH
- Vol. 34, Issue 12, Pages: 67-73(2017)
- 作者机构：
  
  南京工业大学机构计算机科学与技术学院,江苏,南京,211816
- 作者简介：
- 基金信息：
- DOI：
  CLC：
- Published：2017
- 稿件说明：
移动端阅览
赵彤, 刘斌, 李涛. 基于非均衡局部敏感哈希的并行文本分类研究[J]. 微电子学与计算机, 2017, 34(12): 67-73.

ZHAO Tong, LIU Bin, LI Tao. Research on Parallel Text Classification System Based on Non-Balanced LSH[J]. 2017, 34(12): 67-73.
赵彤, 刘斌, 李涛. 基于非均衡局部敏感哈希的并行文本分类研究[J]. 微电子学与计算机, 2017, 34(12): 67-73. DOI：

ZHAO Tong, LIU Bin, LI Tao. Research on Parallel Text Classification System Based on Non-Balanced LSH[J]. 2017, 34(12): 67-73. DOI：

摘要

针对KNN分类算法在面对海量文本处理时效率低下的问题

提出了一种基于超平面的非均衡局部敏感哈希分类算法

该分类算法相比于传统的局部敏感哈希算法在提高分类的准确性和实时性上有显著的效果.同时

为了进一步降低分类算法的执行时间

提高分类效率

将该分类算法与Spark并行计算模型结合

在大数据处理平台Hadoop上实现了一种高效的并行文本分类系统.实验结果表明

所设计的文本分类系统在具有较高分类速度的同时保持了较高的分类准确性.

Abstract

In order to solve the problem of low efficiency of the K-Nearset Neighbors(KNN) classification algorithm in face of massive text

a non-balanced local sensitive hash classification algorithm based on hyper-plane is proposed

which has a more significant effect than the traditional local sensitive hash algorithm on improving the accuracy and real-time performance. At the same time

in order to further reduce the execution time of the classification algorithm and improve the classification efficiency

an efficient parallel text classification system baseed on Hadoop is designed which combines the classification algorithm and the Spark parallel computing model. The experimental results show that such text classification system has a high classification speed and a high classification accuracy.

关键词

Keywords

references

Zhu P, Zhan X, Qiu W. Efficient k-Nearest Neighbors Search in High Dimensions Using MapReduce[C]//Big Data and Cloud Computing (BDCloud), 2015 IEEE Fifth International Conference on. IEEE, 2015: 23-30.

Datar M, Immorlica N, Indyk P, et al. Locality-sensitive hashing scheme based on p-stable distributions[C]//Proceedings of the twentieth annual symposium on Computational geometry. ACM, 2004: 253-262.

彭凯, 汪伟, 杨煜普. 基于余弦距离度量学习的伪K近邻文本分类算法[J]. 计算机工程与设计, 2013, 34(6): 2200-2203. DOI: 2200

Yao B, Li F, Kumar P. K nearest neighbor queries and knn-joins in large relational databases (almost) for free[C]//Data engineering (ICDE), 2010 IEEE 26th international conference on. IEEE, 2010: 4-15.

Park Y, Hwang H, Lee S. A Fast k-Nearest Neighbor Search Using Query-Specific Signature Selection[C]//Proceedings of the 24th ACM International on Conference on Information and Knowledge Management. ACM, 2015: 1883-1886.

Beijing Gao Yanjie. Data Processing with Spark[M]. Beijing: China Machine Press, 2015.

苏毅娟, 邓振云, 程德波, 等. 大数据下的快速KNN分类算法[J]. 计算机应用研究, 2016, 33(4): 1003-1006. DOI: 1003

Indyk P, Motwani R. Approximate nearest neighbors: towards removing the curse of dimensionality[C]//Proceedings of the thirtieth annual ACM symposium on Theory of computing. ACM, 1998: 604-613.

Jagadish H V, Ooi B C, Tan K L, et al. iDistance: An adaptive B+-tree based indexing method for nearest neighbor search[J]. ACM Transactions on Database Systems (TODS), 2005, 30(2): 364-397. DOI: 364

戴上平, 冯鹏, 刘盛英杰, 等. 基于余弦距离的敏感哈希的KNN算法在中文文本上的快速分类[J]. 计算机工程与科学, 2015, 37(10): 1971-1976. DOI: 1971

余鹰, 苗夺谦, 刘财辉, 等. 基于变精度粗糙集的KNN分类改进算法[J]. 模式识别与人工智能, 2012, 25(4): 617-623. DOI: 617

刘怀亮, 张治国, 马志辉, 等. 基于SVM与KNN的中文文本分类比较实证研究[J]. 情报理论与实践, 2008, 31(6): 143-146. DOI: 143

马建斌, 李滢, 滕桂法, 等. KNN和SVM算法在中文文本自动分类技术上的比较研究[J]. 河北农业大学学报, 2008, 31(3): 120-123. DOI: 120

陈佑雄, 向阳, 张骐, 等. 基于LSH和MapReduce的近邻模型推荐算法[J]. 微电子学与计算机, 2013, 30(12): 47-49. DOI: 47

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Implementation and verification of a dual-chain power gating strategy for low power design

Steganography model guided by attention mechanism and improved dense convolution

Aerial small object detection algorithm based on improved RT-DETR

A feature-oriented context understanding and slice analysis network for tuberculosis detection

Microservices anomaly detection based on the correlation of monitoring metrics

Related Author

No data

Related Institution

No data

AI问答

Postal code：100079
Tel：（010）53879206 Email：tmw@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09082226号-64 京公网安备11010602201714号
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰