Image-enabled query / search

Search engines have come a long way since the beginning of the Web. However, all search related problems have not yet been solved. A good example would be natural language search. Natural language search or query is the kind of query expression we use in our everyday communication. A good one is for example a child asking her mom whether she's seen the cat: "Mom, have you seen my cat?". Of course, search engines not being involved in individuals' home affairs, they would not be able to provide an answer to such a question. So a more relevant one would be: "Where can I buy an original edition of Romeo and Juliet?". This for example is a query that a [good] search engine ought to be able to give results to. However, this kind of queries are not yet understandable by the current search engines on the market. It is my understanding however that there is tremendous research being done for making search engines more supportive of this kind of query.


That said, Image-Enabled Search or IES for short is the kind of search capability I have always dreamed would one day be available in search engine software.

Research has shown that images are much better understood, stored and interpreted by the human brain than words. That is the main reason why for example in engineering and in the IT industry, people rely a lot more on diagrams, pictures, schemas than written text.

Also, it seems that when we recall an object or a person or an animal from our memory where it had been stored in both picture and word formats, we seem to remember the picture of it much better than the word or expression associated with it. Actually, sometimes we don't even remember the words at all.

Now, imagine that you had pictures of an object or photos of a person or celebrity or anything else on your computer. Imagine that the picture file had a random name and that there was no way you could associate that name with the picture or the photo. A good example would be a photo of let's say "Emily Mortimer" with a file name of 1234frijld980_lop.jpg. Imagine that your in brain, you sort of remember having seen her somewhere but you really don't remember that she is Emily Mortimer.

Now, think about the fact that you have an Internet connection with access to great search engines (Yahoo, Google, Ask.com, whatever your poison is) that could have helped you find out more about the photo, except that as we discussed earlier you do not know or remember the name "Emily Mortimer" and cannot provide that as a search query. The question is, what's the use?

Now, just as for Natural Language Queries, wouldn't it be great if you could submit the photo or picture file you have as query and get the same results you would normally get after a normal word or expression query? Wouldn't it be great if you could submit file 1234frijld980_lop.jpg to Google and get results back? Well that is an idea I've been thinking about for almost a year and that in my opinion is what Google Image is missing.

I am not really sure what it would involve in terms of software design, algorithms, search logic and so on. I have not even done any research on the subject. For example are pixels from a particular digital picture / photo unique? If they were, would that help? How about the lighting, the contrast, the saturation etc... Is there really a way of building such a software? Let's agree that there was; the submitted picture query would be compared to milliions if not billions of other pictures in a database built through crawling (an infrastructure already exists for that kind of database: Google Image Search) and once a corresponding or a "likely to be" picture is found, words and expressions related to that picture would be returned as a response to the query. The question would be how long would that take? Knowing that an average Web user cannot wait for more than 15 seconds for a page to load, the underlying algorithm for such a search engine would have to be extremely fast and efficient.

I have never worked with images, I don't even know how it would be possible to algorithmically define a concept such as "likely to be" when it comes to images. This might be all Technological Utopia but things have been done that were deemed impossible. Whenever I have time or get a long break from school work, I would do some serious research on this but otherwise, I am just throwing this out in the wild for anyone to think about it and even come up with a piece of code that solves the problem.

In the end, if one takes his/her imagination a bit further, once the problem has been solved with images, the same principle could be applied to other things such as audio and video files. For example submitting a small piece of recorded voice to a search engine and get details of the artist or person who was recorded. The possibilities are endless....

I believe that computer technology (hardware and software) has come to a great level of advancement although more remains to be discovered, implemented and advanced. I also believe that the future of technology in general would be based on Users and not Machines. If we, as IT professionals, researchers, inventors and students cannot make the machines and what runs them more user-friendly and user-oriented, eventually, our discoveries and inventions would amount to nothing (well almost to nothing).