A Parallel Webpage Duplicate Removal Algorithm Based on Character String

XIE Yao-bing

Chinese

您当前的位置：

首页 >

文章列表页 >

A Parallel Webpage Duplicate Removal Algorithm Based on Character String

更新时间：2026-09-04

- A Parallel Webpage Duplicate Removal Algorithm Based on Character String
- Vol. 32, Issue 2, Pages: 69-72(2015)
- 作者机构：
  
  同济大学电子与信息工程学院,上海,201804
- 作者简介：
- 基金信息：
- DOI：
  CLC：
- Published：2015
- 稿件说明：
移动端阅览
谢瑶兵. 基于特征串的网页文本并行去重算法[J]. 微电子学与计算机, 2015, 32(2): 69-72.

XIE Yao-bing. A Parallel Webpage Duplicate Removal Algorithm Based on Character String[J]. 2015, 32(2): 69-72.
谢瑶兵. 基于特征串的网页文本并行去重算法[J]. 微电子学与计算机, 2015, 32(2): 69-72. DOI：

XIE Yao-bing. A Parallel Webpage Duplicate Removal Algorithm Based on Character String[J]. 2015, 32(2): 69-72. DOI：

摘要

针对海量网页文本去重效率不高问题

提出了一种高效的并行网页去重算法.该算法利用Hadoop框架的Map/Reduce机制

通过对网页文本提取特征串

使用Google的Simhash算法对提取的特征串进行哈希映射得到相应的哈希码

然后对产生的哈希码进行海明距离比较

从而得到重复的网页数据.实验表明

与相关去重算法相比

所提算法有效地提高了文本去重计算效率.

Abstract

Against the inefficiency of the huge amount of webpage duplicate removal method

proposes a parallel webpage duplicate removal algorithm based on character string. Using the MapReduce model in Hadoop to extract character string from webpage content

and compute the character string into hash code by Simhash algorithm. Then compare the haming distance between all the hash code to find duplicate webpages. The Algorithm is proved to be more efficient than related algorithms based on experimental results.

关键词

Keywords

references

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Sequential Pattern Mining Based on Pattern Space Division in Map/Reduce Cluster

Design of ultra-low power real-time clock circuits

Implementation and verification of a dual-chain power gating strategy for low power design

High gain stability high temperature operational amplifier based on reverse body bias

Research on assembly action recognition based on MSS-YOLO-MPHands fusion architecture

Related Author

LIU Qian

CHEN Ming

Related Institution

Department of Computer Science and Technology,China University of Petroleum

AI问答

Postal code：100079
Tel：（010）53879206 Email：tmw@bjxintong.com.cn
Technical support is provided by Beijing Founder electronics co., LTD 京ICP备09082226号-64 京公网安备11010602201714号
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰