Performance Optimization of Web crawler

dc.contributor.guidePrasad J.S., Hilal S.
dc.coverage.spatial
dc.creator.researcherGoel K.
dc.date.accessioned2019-04-11T11:44:34Z
dc.date.available2019-04-11T11:44:34Z
dc.date.awarded11/01/2019
dc.date.completed2018
dc.date.registered17/09/2012
dc.description.abstractnewlineOptimizing performance of crawler is the requirement in the area of crawling and searching. newlineAs web contains a huge volume of information and it is a challenging task to extract information, there is a need of the user interface for a user to extract information from the web. A Search engine is the user interface to extract information from web and crawler is a tool used by search engine to create a database. Search results generated by the search engine are derived from the results given by crawler. If crawler provides better results, it will give relevant results in searching also. Finding useful information from the web has inherent issues of page freshness, crawling multimedia contents, and duplicate contents. Crawling and indexing similar contents and URLs implies wastage of resources. Crawler gives duplicate results because of the bad crawling algorithm, poor quality ranking algorithm. newlineThis thesis contributes to the area of optimizing crawler s performance by removing duplicate URLs. Removing duplicate URLs at crawling level improves crawler s efficiency in terms of time and space also. newlineSix popular search engines are at first analyzed for identifying the presence of redundancy in the content over 44 categories of user search interest. Further, the new algorithm based on the URL normalization in query parameter and categorization is developed. To test the effectiveness of the proposed algorithm, a proposed crawler has been developed. To compare and analyze the results another crawler i.e. base crawler based on breadth-first search has been developed. The results of proposed crawler are compared with results of the base crawler, and encouraging performance improvement in terms of crawling time, space, search engine execution time and reduction in the number of duplicates has been observed. The percentage improvement of crawling time between base crawler and proposed crawler varies from 0.086% to 17.44%. The proposed algorithm crawls a URL in a particular category which yields more relevant results
dc.description.note
dc.format.accompanyingmaterialDVD
dc.format.dimensions
dc.format.extent
dc.identifier.urihttp://hdl.handle.net/10603/238366
dc.languageEnglish
dc.publisher.institutionComputer Science Engineering
dc.publisher.placeHaryana
dc.publisher.universityMVN University,Palwal
dc.relation
dc.rightsuniversity
dc.source.universityUniversity
dc.subject.keywordEngineering and Technology,Computer Science,Computer Science Information Systems
dc.titlePerformance Optimization of Web crawler
dc.title.alternative
dc.type.degreePh.D.

Files

Original bundle

Now showing 1 - 5 of 18
Loading...
Thumbnail Image
Name:
01_title page.pdf
Size:
17.85 KB
Format:
Adobe Portable Document Format
Description:
Attached File
Loading...
Thumbnail Image
Name:
02_certificate.pdf
Size:
76.72 KB
Format:
Adobe Portable Document Format
Loading...
Thumbnail Image
Name:
03_abstract.pdf
Size:
81.47 KB
Format:
Adobe Portable Document Format
Loading...
Thumbnail Image
Name:
04_acknowledgement.pdf
Size:
81.79 KB
Format:
Adobe Portable Document Format
Loading...
Thumbnail Image
Name:
05_tableofcontents.pdf
Size:
113.62 KB
Format:
Adobe Portable Document Format

License bundle

Now showing 1 - 1 of 1
Loading...
Thumbnail Image
Name:
license.txt
Size:
1.79 KB
Format:
Plain Text
Description: