Friday, January 30, 2009

Microsoft Searches for Group Advantage

A new search algorithm looks for connections between people.

By Robert Lemos --- technologyreview.com

As part of its efforts to better compete with Google, Microsoft is plumbing the connections between searchers and their contacts to produce better results.

Microsoft researchers are exploring whether using data from several members of a social group--a technique that the company calls "groupization"--can improve search results. Their initial findings, based on experiments involving around 100 participating Microsoft employees, suggest that tapping into different types of groups could produce significantly better search results.

Credit: Technology Review

The team has developed an algorithm that, on average, pinpoints at least one search result for all members of a group that they judge to be better than the results returned using conventional algorithms. The results will be presented at the Web Search and Data Mining Conference in Barcelona in mid-February.

The Microsoft team believes that the approach could help the company overcome an industry-wide plateau in the quality of search results. "Today, search engines are really challenged and are sort of at the cusp of having to know individuals better," says Jaime Teevan, a computer scientist at Microsoft Research and lead author of the paper. "This [research] has the opportunity to enrich that."

The new research is part of Microsoft's efforts to erode Google's massive lead in search. Google currently attracts 63 percent of all searches, according to a 2008 survey by consumer-analysis firm Nielsen, far outpacing both Yahoo's 17 percent share and Microsoft's 10 percent share. Last year, Microsoft attempted to increase its share by acquiring Yahoo, but its initial advances were rejected. Yahoo later wanted to return to the bargaining table, but for the time being, Microsoft is focusing on increasing its audience by enhancing its own search offering.

With an eye on refining search results, Teevan and her colleagues--Meredith Morris and Steve Bush--looked at the way that people with similar interests or attributes search for information. The researchers grouped people using explicit factors, such as their age, gender, participation in certain mailing lists, and job function. In some cases, implicit groups--such as people who appeared to be conducting the same task or appeared to have the same interest--were inferred. The researchers acknowledged that gathering such data in the real world could be tricky. But it could perhaps be collected through registration, by caching previous searches or by tapping into social-networking software.

The Microsoft team found that groups defined by demographics such as age and location have little in common for most searches. However, groups of people with similar interests tend to rank similar search terms highly. The researchers also found that, while people believe that they phrase their queries in similar ways, the idiosyncrasies of search terms vary tremendously.

When asked to identify the pros and cons of telecommuting, for example, one searcher searched for "telecommuting," while others queried "working at home cost benefit" and "economic comparison telecommuting versus office." Knowing that these people have a shared interest could mean better results, Teevan says. "I don't talk about things the same way that you talk about things," she says. "And by using those different ways, [Microsoft is] more likely to find a page where someone talks about something in their own way."

Even if tapping into social groups improves search results, Microsoft will have to significantly improve its search service or introduce major new features to win over Google's loyal followers, says Andrew Frank, research vice president with business-intelligence firm Gartner's media group. "I think that the search category has been so successful for Google, and their dominance is so extreme, that it is hard to imagine a specific tactic that could be a silver bullet to change the trajectory of things," he says. "It will take a lot of effort and a lot of different things to change the overall picture of search."

Efforts to compare the quality of search results from Google, Microsoft, and Yahoo have found that about half of people still prefer Google--a smaller number than Google's actual market share. The difference is the attraction of Google's brand, Frank says. "You have to kind of change the game with search," he says. "It is almost impossible to get people to switch on a large scale just on a feature-function comparison."

In 2008, Google kicked off an experiment in which it allowed users to change the look of their search results, mapping them or placing them on a timeline. Early this year, the company added ability function that lets users reorder their search results through a service called SearchWiki. Yahoo has expanded its research-and-development efforts to try to match its rival's efforts.

Microsoft's Teevan believes, however, that there's still plenty of room for improvement. "Search is a really huge activity on the Web, but right now, we only have a single search tool--the search box--and a list of results," she says. "Groups can teach us a lot."