• Create BookmarkCreate Bookmark
  • Create Note or TagCreate Note or Tag
  • PrintPrint
Share this Page URL
Help

Chapter 3. Making the Most of Your Web S... > Peering into the Invisible Web

Peering into the Invisible Web

Spiders are the intelligent agents that are used by search engines to index pages on the Web. When you see a list of Web pages on your favorite search engine, it’s because that engine’s spider has indexed that site.

What many people don’t know, however, is that spiders cannot index databases. For instance, if you look up an author’s name in a search engine, you might well find listings from Amazon.com. However, if you go to the Library of Congress catalog (http://www.loc.gov), and look for that author’s name, you find additional pages that will never be found by a search engine. Often, the only way to find these pages is by querying the database on that site, because the pages are dynamically generated when you create the database query. They don’t exist as static pages on the Web that the spider can find as it searches through the site. When a spider gets to a site containing a database, all it can see is the “front door”—the database form. It can’t get to the data inside. That data is invisible to the search engines—hence the term, “Invisible Web.”


PREVIEW

                                                                          

Not a subscriber?

Start A Free Trial


  
  • Creative Edge
  • Create BookmarkCreate Bookmark
  • Create Note or TagCreate Note or Tag
  • PrintPrint