Pier 39 in San Francisco

Humans aren’t the only ones who submit queries through search engines.

There are individuals and organizations that use search agents to find and collect information through search engines, focusing upon a number of different informational needs including search augmentation services that help searchers as they surf the web, and agents used by recommendation sites, metasearch engines, and shopping comparison sites.

A study from November, 2006, provides some details about search agents, and how their interactions may differ from human searchers, and what that might mean for search engines. The paper is on the fairly techical side, so I decided to look up a number of the search agents referred to in the document, which are kind of interesting in their diversity and multiple purposes.

I listed and linked to a number of the agents, or to papers that describe them, and I’d recommend looking at a number of the agents to get a sense of their scope and uses before tackling the paper, which I’ll point to first.

Automated Gathering of Web Information: An In-depth Examination of Agents Interacting with Search Engines (pdf)

The paper is written by Bernard J. Jansen and Tracy Mullen of the Pennsylvania State University, Amanda Spink of the University of Pittsburgh, and Jan Pedersen of Yahoo.

Thier research looked at three sets of queries and page views from search agents which used Excite and AltaVista from 1997 to 2002, encompassing around 900,000 queries from over 3,000 agents.

Findings include:

(1) agent sessions are extremely interactive, with sometimes hundreds of interactions per second

(2) agent queries are comparable to human searchers, with little use of query operators,

(3) Web agents are searching for a relatively limited variety of information, with only 18% of the terms used being unique, and;

(4) the duration of agent – Web search engine interaction typically spans several hours.

The paper doesn’t provide links to search agents that they discuss in the paper, so I looked up a number of them, so that we could see firsthand what kinds of uses these things might be making of search engines. Some of these point to pages for the agents themselves, while others are papers about how the agents work:

The study is limited to data from queries submitted to Excite and Altavista between 1997 and 2003, and many of the agents studied were academic search agents, a number of which don’t look like they are active any more. It would be interesting to see some more modern data and results, though you have to wonder if Google or Yahoo or Microsoft or Ask would share this type of information with researchers.

Interesting, though. When we talk about user behavior in connection to queries performed on search engines, we aren’t always talking about users that are people.