SearchImagesVideoTranslateMail
Login

Products & Technologies

Duplicate Images and Yandex Image Search

Sometimes a picture is really worth a thousand words. This is exactly when an image search engine turns out to be quite handy. Obviously, to someone who just wants to know what a [fennec fox] is, to look at a picture makes more sense than to read all about the shape of the animal’s ears or the length of its tail. A graphic image is complementary to almost any search result and is essential to queries like [Rothko], which retrieve on Yandex not only the information about the artist, but also images of his masterpieces.

Yandex retrieves images using the text that somehow refers to this image — it could be a tag, text description, html title, page title, a link to this image from another website. Of the billions of images available through Yandex, only half are unique. The remaining half is the so-called duplicate images, those that do not differ at all or differ insignificantly.

Yandex.Images, a specialized image search service, categorizes duplicate copies it finds into four classes:

Exact duplicates — absolutely identical copies of an image that do not differ a bit.

Thumbnail duplicates — image copies that differ only in size, i.e. a full image of a picture on an art gallery website and a thumbnail image of this picture in the navigation menu on this web site.

Semi-duplicates — watermarked, slightly colour-enhanced, framed or cropped copies of the same image.

Enhanced semi-duplicates — considerably colour-enhanced, altered or fragmented copies of the same image.

Every image available on the web has, on average, three duplicate copies. To filter out identical images in its search results, Yandex.Images sorts them into groups before processing.

Grouping duplicate images

For a computer to recognize images and find duplicates, visual content has to be translated into digital language, a language a computer can understand. Yandex.Images has a special computer system to do this. This system, whose programs find images on the web and process them, extracts the necessary information about each image such as size, colour, format (JPG, PNG, etc.), and creates a signature, a digital description of this image.

To create an image signature, the system chooses a meaningful fragment of the image, shrinks this fragment to 16х16 pixels and assigns to each of the 256 pixels a number matching its brightness. The resulting number sequence is a signature of the image.

The images with identical fragments (and, consequently, identical signatures) are grouped together. Then, the images within these groups are sorted into smaller groups of even more identical images — those sharing no less than two fragments. The most closely matching images are classed as potential duplicates. The program marks image areas that include all of the matching fragments, shrinks these areas down to about 60x60 pixels, digitalizes them and compares the signatures. The images with matching signatures are classed as duplicates.

This method of grouping allows Yandex.Images to processes over 2bn images very quickly.

Using duplicate images

Most images on the web have text descriptions. This exactly is what the Yandex.Images search technology uses to perform the searches. Copies of the same image on different websites are likely to have different text descriptions. When grouping images, the service reads and compares descriptions of all duplicates to find common or similar text fragments and filter out odd images that happen to have a matching text description. Matching text description with visual content allows the service boost the relevance of search results.

Let’s say, a photo of a long blue car has forty duplicate copies. Fifteen of these duplicates are tagged “car”, ten of them have a “blue car” tag, five duplicates are tagged “green car” and the remaining ten are tagged “long”. The ratio of the number of times each tag is used to the total number of duplicates shows how well this tag matches the image it describes:

[car] — 0.75 (30 images out of 40);

[blue] — 0.25 (10 images out of 40);

[long] — 0.25 (10 images out of 40);

[green] — 0.125 (5 images out of 40).

If 30 out of 40 images of a car have the tag ‘car’, most likely, this text description matches what it describes. This image will be quite relevant to the user looking for [car] on Yandex.Images. A duplicate copy with text description other than ‘car’ might also be relevant to the search query [car] and even to queries like [blue car] or [long car], since the degree of match between the content of the car-image and its description is still quite high, even if not complete. If text descriptions of some duplicates in the search engine’s database are mutually exclusive like ‘green car’ and ‘blue car’ for essentially identical images, the service returns that image whose description has a higher rate of occurrence.

Along with being instrumental to providing relevant search results within milliseconds, grouping images also expands search experience for users. After clicking on an image in the Yandex.Images search results, the user is directed to the page, which shows a list of copies of this image and a link to the page with every copy of this image the search engine could find. The list of copies allows the user to choose the image of the right size or find out which websites host this image or what photo bank it can be purchased from.

The duplicate detecting technology also helps identify adult content on websites and use this information for refining search results in family search or in moderated search. Yandex’s signature database contains numerical characteristics for images hosted on adult content websites. Using a special algorithm, the search engine checks a newly indexed website for adult content if signatures of the images it hosts match the signatures of the adult content images in its database. This mechanism allows users enable Family search to filter out from their search results controversial websites or images.

Most copied images

Duplicates of the same image, a photo of a popular model of a cell phone, for instance, may have tens of thousands copies. Photos of celebrities and consumer products are the most likely subjects for copying.

Yandex N.V.
Registered Office in Amsterdam
Address: Schiphol Boulevard 165
1118 BG Schiphol
The Netherlands
tel.: +31 (0) 20 206 6970
Yandex LLC
Headquarters in Russia
Office in 
Moscow
  • Moscow
  • Saint Petersburg
  • Ekaterinburg
  • Novosibirsk
  • Kazan
  • Rostov-on-Don
  • Nizhny Novgorod
  • Kiev
  • Odessa
  • Simferopol
  • Minsk
  • San Jose
  • Istanbul
  • Lucerne
  • Zürich
  • Berlin
Pic / Map
Address
16, Leo Tolstoy St., Moscow 119021, Russia
Reception
tel. +7 495 739-70-00, fax +7 495 739-70-70
Marketing & Media Relations
pr@yandex-team.ru
Investor Relations
tel. +7 495 974 3538
Office in 
Saint Petersburg
  • Moscow
  • Saint Petersburg
  • Ekaterinburg
  • Novosibirsk
  • Kazan
  • Rostov-on-Don
  • Nizhny Novgorod
  • Kiev
  • Odessa
  • Simferopol
  • Minsk
  • San Jose
  • Istanbul
  • Lucerne
  • Zürich
  • Berlin
Pic / Map
Address
Piskarevskiy prospekt, building 2, block 2, 4th floor, Benois Business Centre, 195027, Saint Petersburg, Russia
Reception
tel. +7 812 633-36-00, fax +7 812 633-36-99
Toll-free number for regional users 8 800 333-9639 (or 8 800 333-YNDX)
Office in 
Ekaterinburg
  • Moscow
  • Saint Petersburg
  • Ekaterinburg
  • Novosibirsk
  • Kazan
  • Rostov-on-Don
  • Nizhny Novgorod
  • Kiev
  • Odessa
  • Simferopol
  • Minsk
  • San Jose
  • Istanbul
  • Lucerne
  • Zürich
  • Berlin
Pic / Map
Address
10 Hohryakova St., Ekaterinburg 620014, Russia
Reception
fax +7 343 385-01-99
Sales Office
tel. +7 343 385-01-00 uralsales@yandex-team.ru
Toll-free number for regional users 8 800 333-9639 (or 8 800 333-YNDX)
Office in 
Novosibirsk
  • Moscow
  • Saint Petersburg
  • Ekaterinburg
  • Novosibirsk
  • Kazan
  • Rostov-on-Don
  • Nizhny Novgorod
  • Kiev
  • Odessa
  • Simferopol
  • Minsk
  • San Jose
  • Istanbul
  • Lucerne
  • Zürich
  • Berlin
Pic / Map
Address
35 Krasnoyarskaya St., Novosibirsk 630004, Russia
Reception
fax +7 343 385-01-99
Sales Office
tel. +7 383 230-43-06 sales-nsk@yandex-team.ru
Office in 
Kazan
  • Moscow
  • Saint Petersburg
  • Ekaterinburg
  • Novosibirsk
  • Kazan
  • Rostov-on-Don
  • Nizhny Novgorod
  • Kiev
  • Odessa
  • Simferopol
  • Minsk
  • San Jose
  • Istanbul
  • Lucerne
  • Zürich
  • Berlin
Pic / Map
Address
6 Spartakovskaya Street, 11th floor, right wing, Kazan 420107, Russia
Reception
tel. +7 843 524-71-71
Sales Office
kzn@yandex-team.ru
Office in 
Rostov-on-Don
  • Moscow
  • Saint Petersburg
  • Ekaterinburg
  • Novosibirsk
  • Kazan
  • Rostov-on-Don
  • Nizhny Novgorod
  • Kiev
  • Odessa
  • Simferopol
  • Minsk
  • San Jose
  • Istanbul
  • Lucerne
  • Zürich
  • Berlin
Pic / Map
Address
70D, Gvardeysky Business Centre, Dolomanovsky Lane, Rostov-on-Don, 344011, Russia
Tel. +7 (863) 2-688-300
Office in 
Nizhny Novgorod
  • Moscow
  • Saint Petersburg
  • Ekaterinburg
  • Novosibirsk
  • Kazan
  • Rostov-on-Don
  • Nizhny Novgorod
  • Kiev
  • Odessa
  • Simferopol
  • Minsk
  • San Jose
  • Istanbul
  • Lucerne
  • Zürich
  • Berlin
Pic / Map
Address:
Lobachevsky Plaza, ulitsa Alekseevskaya, 10/16, Nizhny Novgorod, 603000
Tel. +7 (831) 233 06 06
Office in 
Kiev
  • Moscow
  • Saint Petersburg
  • Ekaterinburg
  • Novosibirsk
  • Kazan
  • Rostov-on-Don
  • Nizhny Novgorod
  • Kiev
  • Odessa
  • Simferopol
  • Minsk
  • San Jose
  • Istanbul
  • Lucerne
  • Zürich
  • Berlin
Pic / Map
Address
Suite 30, 19–21 Bohdana Khmelnytskoho Street, Kyiv, 01030 Ukraine
Reception
+38 044 586 41 48, fax. +38 044 586 41 48 ext. 6665
Marketing & Media Relations
pr@yandex-team.com.ua
For general inquiries
info@yandex-team.com.ua
Client Service Department
+38 044 586 41 48 ext. 2601
Skype
yandex.ukraine (reception)
Office in 
Odessa
  • Moscow
  • Saint Petersburg
  • Ekaterinburg
  • Novosibirsk
  • Kazan
  • Rostov-on-Don
  • Nizhny Novgorod
  • Kiev
  • Odessa
  • Simferopol
  • Minsk
  • San Jose
  • Istanbul
  • Lucerne
  • Zürich
  • Berlin
Pic / Map
Address
Polsky Uzviz 11, Morsky-2 Business Center (8th floor), Odessa 65026, Ukraine
Reception
tel./fax: +38 0487 37 44 10, +38 0445 86 41 48
Marketing & Media Relations
pr@yandex-team.com.ua
For general inquiries
info@yandex-team.com.ua
Client Service Department
+38 044 586 41 48 ext. 2601
Skype
yandex.ukraine (reception)
Office in 
Simferopol
  • Moscow
  • Saint Petersburg
  • Ekaterinburg
  • Novosibirsk
  • Kazan
  • Rostov-on-Don
  • Nizhny Novgorod
  • Kiev
  • Odessa
  • Simferopol
  • Minsk
  • San Jose
  • Istanbul
  • Lucerne
  • Zürich
  • Berlin
Pic / Map
Address
1a (4th Floor), Kazanskaya St., Simferopol
Reception
tel. +7 495 739 70 00, fax +7 495 739 70 70
Office in 
Minsk
  • Moscow
  • Saint Petersburg
  • Ekaterinburg
  • Novosibirsk
  • Kazan
  • Rostov-on-Don
  • Nizhny Novgorod
  • Kiev
  • Odessa
  • Simferopol
  • Minsk
  • San Jose
  • Istanbul
  • Lucerne
  • Zürich
  • Berlin
Pic / Map
Address
Office 308, Rubin Plaza Business Center, 5 Dzerzhinskogo prospekt, Minsk, 220036, Belarus
Reception
+375 17 328-19-61, fax. +375 17 328-15-14
Office in 
San Jose
  • Moscow
  • Saint Petersburg
  • Ekaterinburg
  • Novosibirsk
  • Kazan
  • Rostov-on-Don
  • Nizhny Novgorod
  • Kiev
  • Odessa
  • Simferopol
  • Minsk
  • San Jose
  • Istanbul
  • Lucerne
  • Zürich
  • Berlin
Pic / Map
Address
2001 Gateway place, Suite 400W, San Jose, CA, 95110
Tel. +1-617-398-7870
Office in 
Istanbul
  • Moscow
  • Saint Petersburg
  • Ekaterinburg
  • Novosibirsk
  • Kazan
  • Rostov-on-Don
  • Nizhny Novgorod
  • Kiev
  • Odessa
  • Simferopol
  • Minsk
  • San Jose
  • Istanbul
  • Lucerne
  • Zürich
  • Berlin
Pic / Map
Address
Mecidiyeköy Yolu Caddesi No: 12,
Trump Towers 2. Kule, 15. Kat, Ofis No: 1501-1502
34387 Mecidiyeköy, Şişli, İstanbul - Türkiye
Tel. +90 212 386 87 60 (pbx), fax. +90 212 284 46 48
Marketing & Media Relations
pr@yandex.com.tr
Contact Us
Office in 
Lucerne
  • Moscow
  • Saint Petersburg
  • Ekaterinburg
  • Novosibirsk
  • Kazan
  • Rostov-on-Don
  • Nizhny Novgorod
  • Kiev
  • Odessa
  • Simferopol
  • Minsk
  • San Jose
  • Istanbul
  • Lucerne
  • Zürich
  • Berlin
Pic / Map
Address:
Citybay Business Center, Werftestrasse 4, CH 6005 Lucerne, Switzerland
Tel. +41-41-248-08-60, fax. +41-41-248-08-63
Office in 
Zürich
  • Moscow
  • Saint Petersburg
  • Ekaterinburg
  • Novosibirsk
  • Kazan
  • Rostov-on-Don
  • Nizhny Novgorod
  • Kiev
  • Odessa
  • Simferopol
  • Minsk
  • San Jose
  • Istanbul
  • Lucerne
  • Zürich
  • Berlin
Pic / Map
Address:
Odeonhaus Limmatquai 2, 8001 Zürich, Switzerland
Tel. + 41 44 252 50 00
Office in 
Berlin
  • Moscow
  • Saint Petersburg
  • Ekaterinburg
  • Novosibirsk
  • Kazan
  • Rostov-on-Don
  • Nizhny Novgorod
  • Kiev
  • Odessa
  • Simferopol
  • Minsk
  • San Jose
  • Istanbul
  • Lucerne
  • Zürich
  • Berlin
Pic / Map
Address:
Karl-Liebknecht-Straße, 1, 10178 Berlin, Germany
enEn