web crawlers

基本解釋

Once the Sitemap XML is completed, you can make it available to Web crawlers by submitting a URL to the sitemap to those sites that support the protocol.
完成 Sitemap XML 文件后，便可以向支持這一協(xié)議的站點(diǎn)提交站點(diǎn)地圖的 URL，這樣 web 爬蟲程序便可以使用該文件了。
Sitemaps 0.90 is a simple and intuitive way for webmasters to provide the right level of information to Web crawlers so that they can efficiently crawl a Web site.
Sitemaps 0.90為 Web 站點(diǎn)管理員提供了簡單而且直觀的方法，使他們可以向 web 爬蟲程序提供正確的信息，從而使爬蟲程序能夠高效地爬行某個 Web 站點(diǎn)。
Also known as "invisible web", the term "deep web" refers to a vast repository of underlying content, such as documents in online databases that general-purpose web crawlers cannot reach.
“深網(wǎng)”也被稱做“隱形網(wǎng)”，指的是網(wǎng)絡(luò)中存儲數(shù)量浩繁的基本信息的空間，例如存儲于在線數(shù)據(jù)庫中的文件，一般用途的網(wǎng)絡(luò)搜索器是找不到這些文件的。

"Think algorithms and models and different software and Web crawlers, " says Christine Elliott, an American Express spokesperson.
FORBES: Magazine Article
Industry specific web crawlers or vulnerability discovery bots, network scans, and resource exhaustion attacks against web servers are rampant.
FORBES: Clean That Network Traffic
Through the use of proprietary Web crawlers, this news organization had breached the firewall to provide its readers with an upper hand.
FORBES: Data Security: A Shareholder Protection Imperative

網(wǎng)絡(luò)爬蟲
Web page lifetime analysis from users perspective is important to enhance the performance of web crawlers and search engines,and to improve the efficiency of web advertising.
從用戶的角度對網(wǎng)頁的生命周期進(jìn)行分析可以提高網(wǎng)絡(luò)爬蟲和搜索引擎的性能,改善網(wǎng)絡(luò)廣告的效果。