Article

TECHNOLOGY NEWS

A Technology that Comes Highly Recommended

Neal Leavitt

Today's recommendation systems use new approaches that let companies more effectively suggest products and services to potential customers.

Retail and other online businesses have been using recommendation technology for years to gain an edge in selling products and services to potential customers.

Recommendation engines typically collect information about users' online purchases, preferences, and behaviors to help retailers and other organizations determine what customers might want to buy and then to present them with suggestions.

The technology is used by shopping websites such as Amazon, which receives about 35 percent of its revenue via product recommendations. It is also used by coupon sites like Groupon; by travel sites to suggest flights, hotels, and rental cars; by social-networking sites such as LinkedIn; by video sites like Netflix to recommend movies and TV shows; and by music, news, and food sites to suggest songs, news stories, and restaurants, respectively.

Until recently, though, noted Omer Trajman, vice president of field operations for big-data applications vendor WibiData, recommendation engines have relied on limited contextual information and generated only narrow insights into user interests.

Meanwhile, developers hadn't optimized the technology's ability to leverage data from sources such as mobile devices.

Now, recommendation vendors are improving their technology's capacity for handling large amounts of data and its ability to provide personalized services to users.

Ultimately, this will let businesses better target goods and services via specially crafted homepage banners or email messages, noted Neil Hamilton, director of recommendation-technology vendor Emailvision.

"We've seen large jumps in recommender accuracy, more comprehensive evaluation metrics, and a general proliferation of recommenders across domains," said Eric Bieschke, director of playlist technology for Pandora Internet radio.

Even financial-services firms recently began using recommender systems to provide alerts for investors about key market events in which they might be interested.

MAKING RECOMMENDATIONS

Recommendation technology is one of many tools that companies use to sell more products and services, keep visitors from leaving websites, and create customer loyalty, said WibiData's Trajman.

A brief history

Recommender systems emerged from the research community in the early 1990s.

In 1992, Bell Communications Research (now Telcordia Technologies) introduced LyricTime, a personalized digital jukebox that automatically learned music preferences and recommended songs from user activity and feedback.

Shortly thereafter, teams from the Xerox PARC, the University of Minnesota's GroupLens Research lab, and the MIT Media Lab released recommendation-related applications.

Academic teams and others soon began creating companies to release recommendation products, noted University of Minnesota professor Joseph A. Konstan.

Amazon was one of the first companies to use recommendation technology on its website.

Technical backgrounder

Early versions of recommender systems used two approaches.

The user-centric technique was based almost completely on past consumer purchases. This is not always the best way to predict future activity, particularly in product areas not related to the original sale.

The item-centric approach determines that many customers who bought one product also bought another and then recommended that all buyers of the first item also look at the second. This has proven to be fairly effective, said WibiData's Trajman.

The first systems were simpler because they didn't have to process large volumes of data from multiple types of sources in real time.

Today, on the other hand, many organizations interact with customers online, via fixed and mobile devices, and in physical stores. Each of these channels, noted Trajman, produces a stream of contextual information that recommendation engines cab use.

Also, early systems were batch oriented and computed recommendations in advance for each customer, even before they visited the e-commerce website. They thus could not always react to a customer's most recent behavior.

TODAY'S RECOMMENDATION TECHNOLOGY

Today's recommendation approaches are much more sophisticated and in some cases can even, for example, try to "upsell" a customer, said Peter Sheldon, a principal analyst for Forrester Research.

Recommendation engines work by trying to establish a statistical relationship between prospective customers and products or services they might be interested in buying.

The systems establish these relationships with information about shoppers—from e-commerce websites, call centers, or physical stores—and about products.

In some cases, systems that have detailed product information can make recommendations even without extensive customer data, noted Martin Rugfelt, chief marketing officer of recommendation-technology vendor Expertmaker.

Gathering data

Recommender systems collect data via APIs; transaction databases; or cookies, which can help with Web-log sessionization (identifying browsing sessions from recorded clicks).

New sources are becoming available through social networks, ad hoc and marketing networks, and other external sources.

For example, data can be obtained from users' general browsing history accessed via tracking cookies, as well as nonpurchasing activity on e-commerce sites and search engines.

All this enables recommendation engines to take a more holistic view of the customer, said Jack Aaronson, CEO of the Aaronson Group, a market research firm.

Using greater amounts of data lets recommendation engines find connections that might otherwise go unnoticed, yielding better suggestions, said Rugfelt. However, this also sometimes requires recommendation systems to use big-data analysis techniques.

Online public profiles and preference listings on social-networking sites such as Facebook add useful data, noted Rugfelt.

However, the development of ways to connect social posts with product recommendations is just starting, said Rhett Ryder, CEO of recommendation-technology vendor The Filter.

Algorithms and techniques

Most recommendation engines, observed Forrester's Sheldon, use complex algorithms to translate user activities into suggested purchases.

"More data drives the need for better algorithms, analytics, and reporting," said The Filter's Ryder.

Some systems use algorithms that employ personalized collaborative filtering, which uses multiple agents or data sources to identify patterns and draw conclusions. This approach helps determine that numerous users who have liked one type of product in the past may also like a second type in the future.

Many systems use expert-adaptive approaches, explained Emailvision's Hamilton. These techniques create new sets of suggestions, analyze their performance, and adjust the recommendation pattern for similar users. This lets systems adapt quickly to new trends and behaviors.

Rules-based systems enable businesses to establish rules that optimize recommendation performance, Hamilton added. For example, if a customer is looking for parts for a specific truck, rules would keep the system from offering parts for another vehicle, noted Ken Levy, CEO of recommendation-technology vendor 4-Tell.

Some rules-based systems include filters such as those that ensure that all suggestions are for products with at least a 75 percent price markup, added the University of Minnesota's Konstan.

Recommendation engines use data mining, as well as AI and other techniques, to recognize customer-behavior patterns in datasets. Some also use Bayesian technology to estimate the likelihood of certain customer-related actions occurring given the presence of other information and events.

Companies like Amazon utilize the path that users travel through their websites to suggest products for purchase.

New information that a company collects about potential customers could cause a recommendation system to change its evaluation of their preferences or even the types and combinations of algorithms it uses.

In recent years, noted Aaronson, recommendation systems that use multiple algorithms—such as those from MyBuys and RichRelevance—have done a better job of automatically choosing the right algorithm based on the data available.

Some advanced recommender algorithms require training. To properly train them, companies must acquire past data for customers and their purchases so that the system can detect patterns, Rugfelt said.

Rather than analyzing customers on a daily or otherwise regular basis, WibiData's system re-evaluates them as soon as they begin shopping, taking up-to-date information into account.

New touches

Upgrades to the infrastructure that collects, stores, indexes, and serves data are a major reason why recommendation quality has improved, according to WibiData's Trajman. Also, systems' ability to quickly update their recommendation models has improved.

In the past year, vendors have introduced several new recommendation approaches.

For example, some systems develop a purchase-related model for each shopper and a customer-oriented model for each product or service, said University of California, San Diego, professor Charles Elkan.

These systems recommend items based on a function that combines the two models. They frequently use algorithms that employ matrices in which each row corresponds to a customer and each column represents an item, explained Elkan.

According to Emailvision's Hamilton, hybrid rule-based algorithm systems let businesses configure their recommender systems around their business goals, their products or services, and visitor behavior.

Many systems, he added, also use algorithms to conduct persuasion profiling of specific customers to automatically tailor sales messages and the customer experience.

New recommendation engines can better handle dynamic changes to circumstances that could affect how users make buying decisions. For example, if a combination of the time of day, day of week, and weather forecast makes a pair of gloves more attractive to a customer in a certain area, a system could make that suggestion when it might not otherwise do so.

Amazon is starting to let customers edit their recommendation profile by, for example, listing their preferences or deleting information about purchases—such as gifts—that don't reflect their taste. This is important, say recommendation-system vendors, because it allows accurate personalization of suggestions.

"When a visitor interacts with your online business, they're giving you their data in exchange for a relevant brand experience, so it's imperative that a customer doesn't become just a number," said Hamilton.

RECOMMENDED IMPROVEMENTS

Recommendation technology inherently relies upon data mining to be effective. This creates privacy-related concerns, particularly involving companies' collection, sharing, and use of individuals' personal information, noted Paul Stephens, director of policy and advocacy for the Privacy Rights Clearinghouse.

In addition, retailers could use recommendation technology to determine that someone might pay more for the same product or that they should offer a customer a higher-priced item, noted WibiData's Trajman.

"The use of such data to engage in price discrimination represents a concrete example of the consumer harm resulting from this technology," Stephens said.

Recommendation engines are currently underusing social-networking information, added Emailvision's Hamilton. Accessing and utilizing such data could prove valuable in making more meaningful suggestions, he explained.

Today's recommendation technology will have to work for all retailers, not just the biggest ones, to become more useful and to offer an engaging shopping experience in all channels, including PCs, mobile devices, the Web, email, or in stores, stated 4-Tell's Levy.

To do this, the systems would have to work out of the box and instantly integrate with existing infrastructures so that complex consulting isn't necessary.

Understanding and analyzing user behavior from large amounts of data is challenging, noted Trajman. The algorithms for accomplishing this are still relatively new, and network latency can cause problems.

As mature retailers move from simple-recommendation to personalized-recommendation systems, getting the additional customer data this approach requires can be a challenge, added Hamilton.

ONWARD

Companies making recommendation engines include 4-Tell, Barilliance, BayNote, ChoiceStream, Emailvision, Expertmaker, The Filter, LiftSuggest, Strands Recommender, and ThinkAnalytics.

Their business models are typically based on retailers paying recommendation-system vendors for the number of suggestions their engine is asked to make. Some vendors also take a percentage of additional sales attributable to recommendations.

In some cases, universities are even using the approach to recommend classes to students.
Big data will continue to have an impact on recommendation technology and the algorithms it uses, added Bill Guild, ChoiceStream's vice president for product and marketing.

Over the next five years, said Emailvision's Hamilton, the technology will get smarter via accessing more data, improved predictive algorithms, and better AI.

Eventually, he added, recommendation systems might use speech to communicate with customers.

However, said the Privacy Rights Clearinghouse's Stephens, vendors will have to be aware of customers' privacy concerns.

And as people increasingly use smartphones and tablets, noted Aaronson, recommendation systems will have to provide information in a format that works on mobile devices' smaller screens.

Neal Leavitt is president of Leavitt Communications (www.leavcom.com), a Fallbrook, California-based international marketing communications company with affiliate offices in Brazil, France, Germany, Hong Kong, India, and the UK. He writes frequently on technology topics and can be reached at neal@leavcom.com.