nutch爬蟲系統分析-資料下載頁

　　

【正文】 ges, 0 errors, pages/s, 256 kb/s, 20090511 09:50:01,578 INFO TaskRunner Task 39。attempt_local_0005_m_000000_039。 done.20090511 09:50:01,593 INFO LocalJobRunner 20090511 09:50:01,593 INFO Merger Merging 1 sorted segments20090511 09:50:01,593 INFO Merger Down to the last mergepass, with 1 segments left of total size: 72558 bytes20090511 09:50:01,593 INFO LocalJobRunner 20090511 09:50:01,671 INFO CodecPool Got brandnew pressor20090511 09:50:01,734 INFO CodecPool Got brandnew pressor20090511 09:50:01,765 INFO CodecPool Got brandnew pressor20090511 09:50:01,765 INFO PluginRepository Plugins: looking in: D:\work\workspace\nutch_crawl\bin\plugins省略插件加載日志…20090511 09:50:01,921 INFO Configuration found resource at file:/D:/work/workspace/nutch_crawl/bin/20090511 09:50:01,984 INFO CodecPool Got brandnew pressor20090511 09:50:02,015 INFO CodecPool Got brandnew pressor20090511 09:50:02,062 INFO CodecPool Got brandnew pressor20090511 09:50:02,093 INFO CodecPool Got brandnew pressor20090511 09:50:02,125 INFO CodecPool Got brandnew pressor20090511 09:50:02,140 WARN RegexURLNormalizer can39。t find rules for scope 39。outlink39。, using default20090511 09:50:02,171 INFO TaskRunner Task:attempt_local_0005_r_000000_0 is done. And is in the process of miting20090511 09:50:02,171 INFO LocalJobRunner reduce reduce20090511 09:50:02,187 INFO TaskRunner Task 39。attempt_local_0005_r_000000_039。 done.20090511 09:50:44,062 INFO JobClient Running job: job_local_000520090511 09:51:31,328 INFO JobClient Job plete: job_local_000520090511 09:51:32,984 INFO JobClient Counters: 1120090511 09:51:33,000 INFO JobClient File Systems20090511 09:51:33,000 INFO JobClient Local bytes read=33642420090511 09:51:33,000 INFO JobClient Local bytes written=70039420090511 09:51:33,000 INFO JobClient MapReduce Framework20090511 09:51:33,000 INFO JobClient Reduce input groups=120090511 09:51:33,000 INFO JobClient Combine output records=020090511 09:51:33,000 INFO JobClient Map input records=120090511 09:51:33,000 INFO JobClient Reduce output records=320090511 09:51:33,000 INFO JobClient Map output bytes=7254520090511 09:51:33,000 INFO JobClient Map input bytes=7820090511 09:51:33,000 INFO JobClient Combine input records=020090511 09:51:33,000 INFO JobClient Map output records=320090511 09:51:33,000 INFO JobClient Reduce input records=320090511 09:51:47,750 INFO Fetcher Fetcher: done parse方法描述：解析下載頁面內容 update方法描述：添加子鏈接到爬取數據庫20090511 10:04:20,890 INFO CrawlDb CrawlDb update: starting20090511 10:04:22,500 INFO CrawlDb CrawlDb update: db: 20090508/crawldb20090511 10:05:53,593 INFO CrawlDb CrawlDb update: segments: [20090508/segments/20090511094102]20090511 10:06:06,031 INFO CrawlDb CrawlDb update: additions allowed: true20090511 10:06:07,296 INFO CrawlDb CrawlDb update: URL normalizing: true20090511 10:06:09,031 INFO CrawlDb CrawlDb update: URL filtering: true20090511 10:07:05,125 INFO CrawlDb CrawlDb update: Merging segment data into db.20090511 10:08:11,031 INFO JvmMetrics Cannot initialize JVM Metrics with processName=JobTracker, sessionId= already initialized20090511 10:09:00,187 WARN JobClient Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.20090511 10:09:00,375 WARN JobClient No job jar file set. User classes may not be found. See JobConf(Class) or JobConfsetJar(String).20090511 10:10:03,531 INFO FileInputFormat Total input paths to process : 320090511 10:16:25,125 INFO FileInputFormat Total input paths to process : 320090511 10:16:25,203 INFO MapTask numReduceTasks: 120090511 10:16:25,203 INFO MapTask = 10020090511 10:16:25,343 INFO MapTask data buffer = 79691776/9961472020090511 10:16:25,343 INFO MapTask record buffer = 262144/32768020090511 10:16:25,343 INFO PluginRepository Plugins: looking in: D:\work\workspace\nutch_crawl\bin\plugins省略插件加載日志…20090511 10:16:25,750 INFO Configuration found resource at file:/D:/work/workspace/nutch_crawl/bin/20090511 10:16:25,796 WARN RegexURLNormalizer can39。t find rules for scope 39。crawldb39。, using default20090511 10:16:25,796 INFO MapTask Starting flush of map output20090511 10:16:25,984 INFO MapTask Finished spill 020090511 10:16:26,000 INFO TaskRunner Task:attempt_local_0006_m_000000_0 is done. And is in the process of miting20090511 10:16:26,000 INFO LocalJobRunner file:/D:/work/workspace/nutch_crawl/20090508/crawldb/current/part00000/data:0+14320090511 10:16:26,000 INFO TaskRunner Task 39。attempt_local_0006_m_000000_039。 done.20090511 10:16:26,031 INFO MapTask numReduceTasks: 120090511 10:16:26,031 INFO MapTask = 10020090511 10:16:26,140 INFO MapTask data buffer = 79691776/9961472020090511 10:16:26,140 INFO MapTask record buffer = 262144/32768020090511 10:16:26,156 INFO CodecPool Got brandnew depressor20090511 10:16:26,171 INFO PluginRepository Plugins: looking in: D:\work\workspace\nutch_crawl\bin\plugins省略插件加載日志…20090511 10:16:26,687 INFO Configuration found resource at file:/D:/work/workspace/nutch_crawl/bin/20090511 10:16:26,718 WARN RegexURLNormalizer can39。t find rules for scope 39。crawldb39。, using default20090511 10:16:26,734 INFO MapTask Starting flush of map output20090511 10:16:26,750 INFO MapTask Finished spill 020090511 10:16:26,750 INFO TaskRunner Task:attempt_local_0006_m_000002_0 is done. And is in the process of miting20090511 10:16:26,750 INFO LocalJobRunner file:/D:/work/workspace/nutch_crawl/20090508/segments/20090511094102/crawl_parse/part00000:0+402620