Adding Dynamic Popularity Weighting to Relevance Rankings

Relevancy isn't really about the mathematics of TF, IDF, and other ranking algorithms. Relevancy is about presenting results to the end user in an order that is useful to them. If you're selling paint or hardware, then popularity of your products doesn't change very quickly. But if you are selling CDs or tracking hot news and trends that have very high popularity swings, it's more difficult to judge which items are the most popular today. What are you doing to update your search capabilities so items that are popular today are ranked higher than those that were popular yesterday?

If you are an online retailer, for example, you have a gold mine in your sales transaction database. You know what products are selling better today than yesterday, and which solid sellers are slowly tailing off. Why not use this transactional information to dynamically adjust relevance?

Several features of the Attivio Intelligence Engine (AIE) combine to make this almost trivial to implement. First is our Database Connector technology, which allows you to easily import the result of any SQL query into AIE. Suppose you have raw product purchases in a database table like this:

CREATE TABLE Purchases (
   User_ID int,
   Product_ID int,
   PurchaseDate date;
)

It would be easy to write a SQL query to count the transactions in the last week to determine how popular each product ID is:

SELECT Product_ID, COUNT(Product_ID)
FROM Purchases AS Popularity_LastWeek
WHERE PurchaseDate BETWEEN DATEADD( DAY, -7, getdate())AND getdate()
ORDER BY Product_ID  GROUP BY Product_ID

But popularity is really a weighted average of sales over (say) the last week, month, and year, with a strong bias towards the last week. With AIE, it's easy to import these transaction counts from your database, and then compute their weighted average using a transformer component. The configuration might look something like this:

<component "weightedSum" class="ComputeWeightedSum">
   <map name="input">
     <property name="Popularity_LastWeek"      value="0.7" />
     <property name="Popularity_LastMonth"     value="0.2" />
     <property name="Popularity_LastYear"      value="0.1" />
   </map>
   <property name="output" value="popularity" />
</component>

Next, we would want to update the popularity information in the AIE index. Most search engines require that you re-index the product information (product name, artist, price, etc.) with the new popularity values. But with AIE, all you need to do is implement popularity as a Real-Time Field, and no other data needs to be modified.

Finally, AIE allows any numeric field to be used to dynamically adjust the score during the relevance computation. So, just add the popularity field to the relevancy declaration, and voila! Results ranking includes the popularity component.

When you put this all together, you have a system that dynamically ranks search results based upon the popularity of the results, where the popularity is a weighted average of sales over several recent time intervals. Updating the popularity field for every item based on historic sales numbers takes just a few minutes, and can be initiated at any time without overloading either the transaction database or AIE.

TrackBack URI for this entry

Comments (0)