deep web - briefing
Deep Web? Invisible Web? We use both terms in this section as both terms refer to the fact that a significant portion of the Web's new content is not visible to spiders, rendering the data inaccessible through current search engines. Much of this is caused by content providers migrating from static HTML pages to dynamic databases to maintain and display their data on the Web, but other types of data, such as PDF files and files with a ? in the URL are not accessible either. swatrecruiting's deep Web section will keep you current on what is out there in the deep Web and how to find it. This briefing is broken into two components: Understanding the deep Web and searching the deep Web.
understanding the deep web
| Site | Description |
| Learn More About the Invisible Web | Article by Chris Sherman, co-author of The Invisible Web, provides a good high-level understanding of the issues forming the invisible web. |
| The Invisible Web: What it is, Why it exists, How to find it, and its inherent ambiguity | A tutorial from UC Berkeley - an in-depth look at the technical issues behind the invisible web and how to search it. |
| Complete Planet's Deep Web White Paper | A detailed study of the deep Web. |
| The Invisible Web | Book by Chris Sherman and Gary Price, Order or purchased online at Amazon or Barnes&Noble. |
| USA Today - Search Technologies Explore the Invisible Web | A really good overview article on the Invisible or Deep Web, that includes some of the frustrations encountered in searching it. |
| The Invisible Web | Ken Wiseman's excellent high-level presentation. A great first read when learning about the Deep or Invisible Web. |
searching the invisible web
|
Site |
Description |
| Bright Planet Deep Query Manager | An enterprise tool to search all the web, including over 90,000 databases, message boards, chat rooms and hard to find content. Features include full indexing of all Web content, scheduling and monitoring, results filtering, ranking, annotation and saving; as well as the ability to share and archive sites. Offers a free 21 day trial. |
| Deep Web Directory | A turnkey content solution for Web portals. Intended to help manage and deliver content to users, not to find content on the Web. |
| LexiBot | A desktop invisible Web search tool, LexiBot supports simple text, natural language or Boolean queries; searches 2,200 deep Web databases and search engines, and offers 200 pre-configured information channels. Features include results filtering, and ranking; ability to refine results by terms, sources or documents; and the ability to publish results as Web pages. |
| Complete Planet | Free site containing 90,000 searchable databases and specialty search engines. |
| xRefer | Contains dictionaries, encyclopedias, thesauri, books of quotations, and a growing list of subject-specific titles. A great jumping off point for research. |
| Aviation Information System | The operating status of the nation's largest airports is now brought directly to your wireless device, pager, phone, PDA, or e-mail client in real-time as it happens. A great way to stay abreast of travel conditions at airports that affect you, real time, during travel. |
| Direct Search | Direct Search is a growing compilation of links to the search interfaces of resources that contain data not easily or entirely searchable/accessible from general search tools like Alta Vista, Google, or Hotbot. Although these "general" tools are essential for the retrieval of Internet based data, searchers often fail to realize that a massive amount of information is not easily or entirely searchable/accessible via these search tools. Material "hidden" from the general search tools is said to reside on the Invisible Web. |
| InfoMine | Searches 20,000 academically valuable resources. Databases include: biological, agricultural and medical sciences; Government information; Instructional resources: K-12; Instructional resources: university; Internet enabling tools; Maps & GIS; Physical sciences, engineering, CS, math; Social sciences and humanities; Visual and performing arts; and Electronic journals. |
| Beaucoup | Lists over 2,500 specialized databases and directories. |
| Internet Public Library Reference Center | A virtual library that provides a good starting point for finding reference works, subject guides, and specialized databases. |
| Fossick | This meta-search tool contains links to hundreds of specialized databases. |
| The Invisible Web | A well-organized, comprehensive directory to thousands of specialized databases |
| LibrarySpot | Contains links to over 2,500 libraries. |
| Librarian's Index to the Internet | A virtual library that is both searchable and browsable, this is an excellent source for specialized databases. |
| The Scout Report | The Scout Report is a good way to keep up with new search tools, especially specialized databases. You can view its weekly report and its archive of previous Scout Reports on the Web. You can also have the report delivered to you via email by subscribing through a listserv. Send an email message to scout-report-request@cs.wisc.edu. Type subscribe to scout-report in the body of the message. |
| WebData | Claims to be a comprehensive guide to searching thousands of searchable databases. |
| IncyWincy | An Invisible Web search engine. |
| Quigo | A Deep Web search tool |
| Gary Price's List of Lists | Gary Price's extensive directory of Web pages that present information in the form of rankings of different people, organizations, companies, etc. |
| News Center | A huge directory of links to up to the minute news stories on any subject imaginable. |
| The Big Hub | Contains an index of over 3,000 subject specific searchable databases in over 300 categories. |
| YBLost.com | A good search tool for public records, government information, and people searches. |
| Profusion | Excellent all purpose search engine includes Invisible Web interface. |