Biblioteca Digital do Instituto Politécnico de Bragança   Instituto Politécnico de Bragança

Biblioteca Digital do IPB >
Escola Superior de Tecnologia e Gestão >
Informática e Comunicações >
IC - Publicações em Proceedings Indexadas ao ISI >

Please use this identifier to cite or link to this item: http://hdl.handle.net/10198/4406

Título: Efficient partitioning strategies for distributed web crawling
Autor: Exposto, José
Macedo, Joaquim
Pina, António
Alves, Albano
Rufino, José
Issue Date: 2008
Citação: Exposto, José; Macedo, Joaquim; Pina, António; Alves, Albano; Rufino, José (2008) - Efficient partitioning strategies for distributed web crawling. Lecture Notes in Computer Science. 5200, p.544-553.
Resumo: This paper presents a multi-objective approach toWeb space partitioning, aimed to improve distributed crawling efficiency. The in- vestigation is supported by the construction of two different weighted graphs. The first is used to model the topological communication infras- tructure between crawlers and Web servers and the second is used to represent the amount of link connections between servers’ pages. The values of the graph edges represent, respectively, computed RTTs and pages links between nodes. The two graphs are further combined, using a multi-objective partition- ing algorithm, to support Web space partitioning and load allocation for an adaptable number of geographical distributed crawlers. Partitioning strategies were evaluated by varying the number of parti- tions (crawlers) to obtain merit figures for: i) download time, ii) exchange time and iii) relocation time. Evaluation has showed that our partition- ing schemes outperform traditional hostname hash based counterparts in all evaluated metric, achieving on average 18% reduction for download time, 78% reduction for exchange time and 46% reduction for relocation time.
Arbitragem científica: yes
URI: http://hdl.handle.net/10198/4406
Appears in Collections:IC - Publicações em Proceedings Indexadas ao ISI

Files in This Item:

File Description SizeFormat
LNCS2008.pdf250,89 kBAdobe PDFView/Open

Statistics
FacebookTwitterDeliciousLinkedInDiggGoogle BookmarksMySpaceOrkut
Formato BibTex mendeley Endnote Logotipo do DeGóis 

Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.

 


  © Instituto Politécnico de Bragança - Biblioteca Digital - Feedback - Statistics
  Estamos no RCAAP Governo Português separator Ministério da Educação e Ciência   Fundação para a Ciência e a Tecnologia

Financiado por:

POS_C UE