There are an ever-increasing and effective number of Internet directories and search tools. Which of these you choose should depend mostly on how much you know about the information you're trying to find. General questions like "What's out there
about college basketball?" demand a different search strategy than more specific queries such as "Where can I find a shareware NCAA Tournament program for Windows?"
Browsing through directories "by hand" will give you much better answers to a general inquiry like the first question, but you can quickly answer the second sort of question by doing an automated search. This chapter shows you how to use both
approaches to search the World Wide Web, as well as how to search all FTP sites or Gopher menus in the world at one time. And, in case you're interested, the chapter answers the questions in the previous paragraph along the way.
The Directory of Virtual Libraries (see Figure 8.1) at
http://www.w3.org/hypertext/DataSources/bySubject/Virtual_libraries/Overview.html
is essentially a "directory of directories." It lists all the major subject-oriented guides to the World Wide Web and the Internet in one place. (The word directory here means more of a "collection" or "catalog," as
opposed to the technical meaning of "directory folder on a disk," which is more often used in relation to computers.)
All of the directories on the list in Figure 8.1 include links to a wide variety of carefully selected sites organized by subject. Figures 8.2 and 8.3 show two typical examples: the Whole Internet Catalog (http://gnn.com/gnn/wic/index.html) and the Excite directory (http://www.excite.com/Subject/). The choice between various directories is largely a matter of personal
choice, though by using the Search function of Internet Explorer, you'll always be pointed to the most effective directories on the Net. Most directories focus primarily on Web sites, though you can find major FTP and Gopher sites among the mix if you
look
carefully.
Figure 8.3. Excite is one of several very effective searchable Internet directories around.
Browsing through a topical directory is so simple that it requires no explanation; just click on the links that look interesting.
TIP
Put a bunch of Web surfers in a room, and they'll probably start an argument about which directory or search engine is really the best. However, if you poll those same people six months later, half of their opinions won't be the same. Things change fast on the Net, and the clear leader today might be way behind the times in a month. One thing is certain, however: searching will only become easier and more powerful. What should you do? Try several of the best and use the one you find the best for your needs.
First, it might be helpful to remind you where to always start using Internet Explorer. With a single click on the Search button (see Chapter 6, "Browsing the Web," for a review of Internet Explorer toolbar buttons), you will always go
directly to a choice of the best directories available on the Internet (see Figure 8.4). Of course, if you find a particular directory you like, you can always make it a favorite (see Chapter 7 for more details) and access it that way.
Figure 8.4. The Search button always takes you directly to the best search tools on the Web (http://home.microsoft.com/access/allinone.asp).
NOTE
Clicking on a search engine from the MSN Search page doesn't necessarily take you to the URL or the search engine itself. Instead, you will more likely go to an MSN "copy" or "mirror" of the search engine. For example, clicking on the Yahoo! and browse links after clicking the Explorer button takes you to http://msn.yahoo.com/msn/ and not http://www.yahoo.com/. Many of these "mirrors" might not be identical and might not have the same features.
Most of these search engines enable you to either do a direct search using their engine or browse through categorized subject links. For example, a click on the Subject tab at the Excite directory (http://www.excite.com/) brings up a list of more than a dozen categories for you to choose from. Yahoo! (http://www.yahoo.com/) places its categories directly below the search field at
the top of the first page.
Almost all directories offer the capability to search their listings for specific words by using an interactive search form. In addition, almost every directory offers ways of conducting a more advanced search using special tips, options, and switches.
It would be impossible to cover all of these options here, but Yahoo!'s options are presented here because they cover most of the different types of choices you are likely to run into.
TIP
Almost all the Internet search engines discussed in this chapter offer some or all of the options available on Yahoo! (sometimes given slightly different names). If you take the time to play with a few searches and learn how to use these, you'll be able to apply the same searching skills every time you need to find anything on the Internet.
First, go directly to the Yahoo! page at http://www.yahoo.com/. Next, to conduct more refined searches, click on the Options link next to the Search button. When you do, you will see the form in Figure 8.5.
If you're not a database expert or computer programmer, the meaning of the choices listed might not be obvious. The first thing you'll need to understand is that the words or letters you enter in the search box are called keys. Here's how the various
options work:
Figure 8.6 shows the results of searching for the word swords. Notice that Yahoo! also provides links to broader categories in which your search term was found so that you can expand your search to related areas.
You can find one of the nicest features of Yahoo! by scrolling to the bottom of a search results page (see Figure 8.7). At the end of every Yahoo! search results page, you'll find links to all the major search engines on the World Wide Web. When you
click on one of these links, the same search word or words you gave Yahoo! are sent to the search site you have chosen, and the appropriate result page comes back to you.
Everything happens automatically and invisibly, and you just get the links you were looking for. You don't need to access different search forms or learn new terms. For example, if you clicked on Lycos after you scroll down to the links pictured in
Figure 8.7, you would get a whole new listing, such as the one shown in Figure 8.8. Some hits would probably be the same, but you would also get quite a few additional hits that Yahoo! didn't return.
The Lycos database, as well as others, contains many entries because of the way it is built. Instead of relying on slow humans to find and describe good sites, computer programs—pretending to be humans—visit thousands of pages a day and then
automatically generate an outline of each site's content based on the major headings and more frequently used words.
Several other similar programs, including America Online's WebCrawler, are out on the Web building their own index databases. You can search WebCrawler, or Alta Vista or Magellan, right from a Yahoo! search results page, too.
InfoSeek is another "robot-generated" index that is more heavily assisted by humans than are Lycos and WebCrawler. The full InfoSeek search service costs money, but a free "quick search" of their database is available from the
Yahoo!
page. InfoSeek is also available from the MSN search page.
There are, of course, many more search engines that search different areas, topics, and databases. In fact, there are already in excess of 250 different search engines designed to find everything from e-mail addresses to area codes to the quickest way
of getting to Albuquerque from New York (see Figure 8.9). It's obvious that they can't all be covered here. However, Table 8.1 provides a listing of some of the more efficient, accessible, and popular search engines. Table 8.2 lists some of the better
specialized search engines.
Figure 8.9. This A-to-Z listing of search engines at InfoSeek can help you find almost anything you might want to look for. (http://www.search.com/alpha.html)
| Search Engine | URL |
| Yahoo! | http://www.yahoo.com/ |
| Excite | http://www.excite.com/ |
| Infoseek | http://www.infoseek.com/ |
| Alta Vista | http://www.altavista.digital.com/ |
| Magellan | http://www.mckinley.com/ |
| Lycos | http://www.lycos.com/ |
| Open Text | http://www.opentext.com/ |
| WebCrawler | http://www.webcrawler.com/ |
| Inktomi | http://inktomi.cs.berkeley.edu/ |
| Search Engine | Purpose | URL |
| shareware.com | Searches for Shareware | http://www.shareware.com/ |
| DejaNews | Searches Usenet Newsgroups | http://www.dejanews.com/ |
| Four11 | E-mail address searching | http://www.four11.com/ |
| MetaCrawler | Can search by geographic region | http://metacrawler.cs.washington.edu:8080/ |
TIP
Even though all search engines try to be comprehensive, you will almost always find sites with one index that you wouldn't find with another. So if you search one index and don't find quite what you're looking for, try another—and another.
Although some people view the multiple indexes of the Internet as a waste of resources, experienced searchers are grateful for the multiple coverage. As the Web continues to grow exponentially, it seems ever more unlikely that any one procedure or program would ever be able to successfully index it all. But by the time you search three or four indexes on the same topic and follow a few links to related sites, you'll have a very thorough picture of what's available.
You might notice that search engines for FTP and Gopher were conspicuously left out of Table 8.2. That's because they are covered in more detail in the following sections. In addition to these powerful search engines, the next sections cover the
Configurable Unified Search Interface (CUSI). All provide essential search services not available from the directories and Web index sites mentioned so far in this chapter.
Want to find a specific file? Good news! There is a master database of every filename in every public FTP archive in the world. Even better, copies of it are maintained on many different servers around the globe. This searchable database is called
Archie (a pun on "archive").
Like many things on the Internet, there is more than one way to access Archie. Most people still use special dedicated programs like WS_Archie for Windows or Fetch for the Mac (both of which are on the CD-ROM that comes with this book). However,
installing a separate program just to do an occasional Archie search is unnecessary; you can access Archie through ArchiePlex gateways on the World Wide Web just as easily.
A list of every ArchiePlex server (see Figure 8.10) can be found at
http://pubweb.nexor.co.uk/public/archie/servers.html
Figure 8.10. You can conduct Archie searches through these World Wide Web gateways.
TIP
The document at
http://pubweb.nexor.co.uk/public/archie/servers.html
is very popular, and the computer that delivers it is often overburdened; therefore, you might have a long delay when you load this page. To avoid having to download it every time you want to do an Archie search, you can load it once and then save it on your own computer. (Select File | Save As.) Then you can just open the file on your computer instead of accessing the one in the United Kingdom. Another option is to find a couple of ArchiePlex servers that you like and add them to your list of Favorites.
The NASA ArchiePlex form page at the following address is shown in Figure 8.11:
http://www.lerc.nasa.gov/archieplex/doc/form.html
To run a search, simply enter a search term in the Search for: box. There are a number of options that you will want to pay attention to in order to make your search more effective.
For the type of search, you'll almost always want Case Insensitive Substring Match. You should use Exact Match and Case Sensitive Substring Match when you know exactly what you're looking for. You can use a Regular Expression Match when you know only a
partial filename or category of files. For example, if you are looking for a WinSock program, but aren't sure exactly what to look for, you might Search for winsock* as a Regular Expression Match. Such a search will bring up hits with WinSock in them, as
shown in Figure 8.12.
The settings for Restricting and Sorting results (which can be found if you scroll down the page shown in Figure 8.11) are really a matter of personal taste. However, keep in mind that the more results you want displayed, the longer a search will take.
As for "impact on other users," which controls how much of the server's attention your query will get, please choose Nice; doing so will spread out the traffic so that others can also use the service effectively.
Selection of which Archie servers can be used usually depends on the time of day because most servers are overburdened during daylight hours. If you're burning the midnight oil, use the server closest to you. But if it's ten in the morning and you're
in
North America, choose a server in New Zealand or Taiwan, where it's the middle of the night and the computers have nothing better to do than chat over the Internet with foreigners. You'll get your answer much faster that way, and the people trying to
access the busy local servers will get their answers faster, too.
Click on the Submit button, and in a few seconds (sometimes longer at busy times), you will have a list of every publicly archived file in the world containing the letters you specified in its name. As you can see from Figure 8.12, the result pops
right
up as a nicely formatted Web page that you can then save on your computer if you wish.
Archie doesn't just look at filenames; it indexes directory names, too. For example, the directory pub/pc/win/winsock at rpub.msu.edu, as well as files, was turned up by the Archie search in Figure 8.12. If you use long words like astronomy for
Archie searches, you won't find any DOS files (because of the eight-character limit on DOS filenames), but you might find FTP directories containing DOS files. Note that Archie also finds files that are in Gopher systems because Gopher is essentially just
an interface laid over regular FTP.
TIP
In theory, all Archie servers and gateways are created equal. But Archie and ArchiePlex, like many programs on the Net, aren't perfect. For example, a search of one Archie server might bring up nothing, while a search on a different server could bring up 100. The moral of the story? If you don't get any matches, try a different form/server combination. It will probably work.
Chapter 9, "Accessing Other Services with Internet Explorer 3.0," discusses some of the searching capabilities built into most Gopher sites. All of which is pretty snazzy, as file systems go. After all, cruising Gopherspace sure beats plowing
through FTP directories. But Veronica is what makes Gopherspace really show its stuff. Veronica is a master searchable index of public Gopherspace in its entirety, much like Archie is for FTP. All that Veronica needs from you is a few letters or words to
search for, and away she goes to fetch every file in the global Gopherspace that matches your query. Almost all of the files Archie has access to are also in Gopherspace, so Veronica can find them, too.
Often, Veronica has access to longer descriptions of the files than Archie does. More important, Veronica knows what type of file she's looking at, and she tells you by displaying the appropriate icon on the search result Gopher menu. By the way, in
case you're interested, Veronica stands for Very Easy Rodent-Oriented Netwide Index to Computerized Archives.
The next paragraph steps you through a Veronica search so you can see how it works.
First, pull up the list of Veronica servers at the following (see Figure 8.13):
http://www.scs.unr.edu/veronica.html
Notice the nifty simplified search feature on this page that offers to automatically try servers in random order until it finds one that's not busy. It sounds good. Often, though, you'll actually get a quicker response just by choosing a server from
the
list. Be sure to save this page on your hard disk (so you won't have to download it again every time you want to do a Veronica search) before selecting either a server or the simplified search link. For this example, click on the veronica server at PSINet
link.
The Veronica search form is pretty basic (see Figure 8.14). You just click in the box, type the search term(s), and press Enter. Along with the words to search for (which can be entered in any order), you can also give special commands to Veronica. For
more information on the command language, choose the How to Compose veronica Queries link from the Veronica server list page. For our example, conduct a search for pigs.
TIP
Veronica searches often turn up thousands upon thousands of matching files, only 200 or so of which are shown unless you request to see more. Even with handy icons to guide you, weeding through 200 files or more for what you want is no fun.
To limit a search to files of only certain types, you can include a special command after your search terms, as described in the following list. Type -t (that's a hyphen followed by a lowercase t), immediately followed by as many of the following codes as you like:
0 Plain Text File 1 Directory 7 Gopher menu 5 PC Binary (program) File 9 Other Binary File s Sound g GIF image I Image (other than GIF) h Web page (hypertext)
For example, taking this example, if I wanted to only see directories and graphics about pigs, I would search for pigs-t1g.
NOTE
Internet Explorer will display most image files automatically when you click on their icons on a Gopher menu or Veronica search result list. You can then choose File | Save As to save the image to your hard disk.
Some image types, however, aren't yet supported by Internet Explorer. If you try to load one of those images, you'll just see a generic icon representing a graphics file instead of the actual image. You can still save the file as if it had been loaded, and it will be transferred to your hard disk.
Searching the Internet successfully might take a few tries with different word combinations and different search engines, but the basic idea is still just about as simple as anything could be: type in a word, push a button, and away you go—that
is,
in theory. In reality, one of the biggest headaches you'll experience while searching is the wait for overburdened servers to respond to your queries. These free services are increasingly essential for finding anything on the Net. Even major universities
can't afford enough hardware and bandwidth to keep up with the millions upon millions of searches that flood their servers.
If one search service is busy, you can always try another. However, this can often take a lot of time and effort. Fortunately, you can find the ultimate search interface, called CUSI (Configurable Unified Search Index), at the following address (see
Figure 8.14):
http://pubweb.nexor.co.uk/public/cusi/cusi.html
The idea here is simple: Put on one Web page access to every major Internet search tool.
Figure 8.14. CUSI gives you quick access to many search services from a single Web page.
CUSI becomes useful only if you have some basic understanding of the search engines that are hiding behind it. Otherwise, it would just look like a confusing bunch of pull-down lists of gibberish. But now that you know what Excite, Yahoo!, InfoSeek,
Veronica, and ArchiePlex all mean, you can use CUSI to access them all, and many other search tools besides.
TIP
Like some other services discussed in this chapter, CUSI has mirror sites throughout the world. You can choose any one of them from the list at
http://pubweb.nexor.co.uk/public/cusi/doc/list.html
But each of the CUSI forms is slightly different, and the one at Northwestern University is well-organized and accessible. So just load the NWU version of CUSI at
http://www.eecs.nwu.edu/susi/cusi.html
save the Web page on your own computer, and bookmark it. That way, you never have to fend off the traffic to access CUSI. It will load right off your hard drive and will use only the Internet to submit the search itself. (See Chapter 7 for details on how to save files as favorites.)
Figure 8.15 shows the Robot generated WWW Indices section of the CUSI form page at NWU. To submit a search to Lycos for the keyword, just type it in the Search term box and click Submit.
Figure 8.15. Searching the index of your choice from CUSI is as simple as type, click.
TIP
You might have noticed that Archie and Veronica are on the CUSI super-search form, too. Although you can access them through CUSI, you might find that the ArchiePlex and Veronica search servers on your favorite CUSI page are often too busy to answer your queries. You might also find that Veronica sometimes doesn't understand commands (like the -t command) issued through CUSI. If you encounter these problems, just hop over to the lists of ArchiePlex or Veronica servers mentioned earlier in this chapter to send your request to a different server.
Be sure to take a peek at the bottom part of the CUSI form, too. The search tools located there enable you to search software libraries, dictionaries, Internet standards documents, and several other reference works.
The two preceding chapters show you how to browse, bookmark, and leave a trail to your favorite places on the Net. This chapter introduces you to the tools available for searching it. Chapter 10, "Using E-Mail and Newsgroups," shows you how
to
access other services—such as FTP, Gopher, and Telnet—using Internet Explorer.
