Why Machine Learning Is So Important In Search

We’ve been hearing about Artificial Intelligence for years now – enough so that even the most casual AI watcher has a high-level understanding of what AI is: the art and science of getting computers to act and respond just like humans do. Casual AI watchers are also likely to recognize that AI is getting smarter - less artificial and more intelligent - by the day. In part, that’s thanks to machine learning, an AI technique that helps AI become more intelligent by learning over time, on its own, without having to be programmed. It does so by observing real-world interactions between humans and computers, and analyzing the data and information associated with these interactions.

Be wary of claims about machine learning

Not all machine learning is created equal, and sometimes when search providers claims that they’re using machine learning, they’re really not.

Take the case of search platforms that are supposed to be providing better answers to user queries.

Many search platforms purporting to use machine learning analyze only structured or previously categorized data, relying on key words and tags to identify relevant information. When these search platforms produce answers to queries their ability to find answers is only as good as an information source’s key words and tags. They also use a voting approach to decide what the best answers are by counting clicks associated with different bits of content and boosting those sources based on the results.

Attivio has a smarter approach

There is a smarter approach to machine learning, and that’s the one that Attivio uses.

Like other search platforms, Attivio does use clicks as a signal because it is a useful piece of information to know what end users are looking at. But Attivio machine learning goes beyond, analyzing user behaviors and building relevancy models that learn and improve as content, data, and user activity grows and evolves. Attivio recognizes that it’s critical to understand exactly what end users are after.

Here’s where another AI concept comes into play, and that’s natural language processing. This technique lets humans communicate with a computer in a way that’s actually natural to them, as if they’re having a conversation with another person or thinking out loud. Attivio also relies on text analytics to categorize, classify, and mine unstructured information – the sort of unstructured information that often contains the answers people are looking for.

Natural language processing – figuring out what end users and asking - and text analytics – figuring out where the answers are - lay the foundation for Attivio’s machine learning. Machine learning then deploys statistical techniques that enable computers to learn from data. For Attivio, the focus of machine learning is using algorithms to identify information that’s relevant in general to the topic at hand and relevant in particular: personal to the end user searching for information.

Click counting vs. the Attivio approach: what makes Attivio smarter and better

The “vote with your clicks” approach deployed by so many other search platforms is little more than a popularity contest. With the click-counting approach, older answers are more popular. They’re pushed to the top of search results even if they’re completely outdated. Newer content is punished: it hasn’t been around long enough to become popular. When newer content is ignored, there’s a good possibility that better answers remain hidden. With this so-called machine learning approach, improving search performance requires human intervention – better categorization of information sources. And that’s not very smart at all.

Voting systems tell you which documents people have liked up until now, but they don’t tell you anything about why they liked that document or what they might like in the future. Attivio’s machine learning does just that. It learns why users like one document more than another and uses that information to find answers, not just documents. And knowing why people might like something—not just what they clicked on in the past—is applicable when they ask new questions.Voting systems always fail when someone asks something new.

The technology at play here is very complex and sophisticated, so I’ll use a simple example of what we mean by Attivio’s machine language knowing why an answer would be most relevant to a specific user.

In one instance, an HR professional searching for Java is likely looking for resumes of potential hires who have this skill. Conversely, when a developer searches for the word Java, they’re likely looking for information on how to program in Java.

My point is that there’s more to machine learning than counting clicks. Machine learning that’s actually learning something as it goes along does so by making sure that it’s continually reevaluating old information, sifting through new information, and acquiring additional insights about the humans asking questions and looking for answers. And that’s what makes Attivio’s machine language approach smarter and better.

Curious how machine learning can improve your search experience? Read our 5-minute guide on machine learning, or connect with our team to show you a demo.