July 9, 2010
Internet, July 9, 2010. Yandex has developed and implemented a new search technology that allows users to find the latest information minutes after it appears online.
A significant portion of search queries (from 2% on a quiet day to 8% on a day when something happens) refer to the latest events. Web users making these queries expect to see, among other things, the documents that have just been created. To deliver helpful results to such queries, Yandex, first, has learned to recognize this type of searches among all other, and second, it launched a new searchbot, Orange. The new Yandex spider crawls the web in real time. It finds and indexes the new documents as soon as they appear online and sends them to the index servers in a matter of seconds.
The frequency at which websites update their content can vary; some sites are quite slow, while others, like news sites, for instance, upload new material all the time. The Orange crawler visits these sites and adds the newly uploaded documents to the search index as they appear, without any delay.
To understand if responding to a specific search query requires the most recently updated information, Yandex has developed a number of detectors. Using these programs allows to analyze a range of information, such as the topics currently covered in mass media or an increase in specifically-themed search queries.
“The web is offering more and more real-time information, something that people want to know right there and then. We’d like to give our users an opportunity to find what they need right when they need it,” says Fedor Romanenko, the manager of Search Quality at Yandex. “It’s only the first step, of course, we will continue to improve search quality for the news searches. With the new crawler, we can do this relatively easily.”
The Orange indexer and the real-time search technology were developed by the company’s California-based Yandex.Labs team with the help from their colleagues in Moscow. To read more about Yandex’s new technologies, please, visit the company blog (in Russian).
