Cognitive Search: Balancing the Exploration-Exploitation Equation

Our 5-minute guide to cognitive search notes that psychologists and technologists agree that cognition:

• Transforms sensory/data inputs into understandable formats
• Reduces massive amounts of data to a concise, usable summary
• Elaborates, extends, and enrichs the information processed
• Stores and recovers information efficiently
• Uses information to solve a meaningful problem

Of course, most of us don’t think about this when we use a cognitive search engine, but, “behind the veil,” natural language processing, machine learning, and text analytics function to execute those five tasks as quickly as possible.

Reducing exploration and increasing exploitation

Researchers in various fields have published a lot of very nerdy papers about search and cognition. One of the concepts that surfaces in this literature is the idea that search has two phases: an exploration phase and an exploitation phase.

In the exploration phase, we try to find a resource—physical or informational. In the exploitation phase we use that resource. For the knowledge worker, cognitive search tilts the balance in the favor of exploitation.

This is one of the important aspects of search that influences the way Google tunes its algorithms and makes adjustments to its user interface. Inherent in this exploration-exploitation balancing act is the idea that after someone conducts a successful search, there’s still a lot of work to be done. Search is just the beginning of trying to “solve a meaningful problem.”

Search Engine Wars: Google v. Inktomi

In a recent blog on the website Search Engine Watch—yes, that’s a real website—the author revisits a faceoff between Inktomi and Google. He also supplies a link to a post from former Inktomi engineer Diego Basch who was there when it happened. Interesting stuff.

At one time—1999 into the early 2000s—Inktomi was the the number one search engine in the world. But, as the number of pages search engines needed to index grew staying, there became an ultimately solvable problem for Inktomi. Scale is the downfall of us all. Or it can be.

Basch recalls, “… there was one huge red flag: engineers at Inktomi were starting to use Google as our search engine.” He goes on to write, “Google had realized that a search engine wasn’t about finding ten links for you to click on. It was about satisfying a need for information.”

Again, there’s that balance between exploration and exploitation. Even before it had a name, the need for cognitive search was emerging.

To learn more about what makes cognitive search cognitive, check out our 5-minute Guide: On the Path to Cognitive Search.