Will Recommendation Technology Boost E-Commerce?

Most consumers who have used Amazon.com or other major e-commerce Web sites are accustomed to receiving recommendations for a book, music CD, DVD, or article of clothing they might be interested in buying.

Initially, recommendation technology was relatively crude. It basically just recommended products like other items the buyer had purchased. However, the technology has become considerably more sophisticated and is now an essential part of many online retailers' economic models.

The approach uses complex algorithms to analyze large volumes of data and determine what products that potential consumers might want to buy based on their stated preferences, online shopping choices, and the purchases of people with similar tastes or demographics. It is creating new revenue opportunities and increasing both customer retention and the number of shoppers who become buyers.

Major recommendation-technology vendors include AgentArts, ChoiceStream, ExpertMaker, Google, and Mavice. Big users include Amazon.com, Apple Computer, and the Netflix DVD-rental Web site.

The technology is helping drive online sales, particularly in the booming music industry. "With such a huge market, it's easy to see that personalizing makes sense," said ChoiceStream CEO Michael Strickman "One of our clients has seen users download almost four times more songs than before implementation."

While recommendation technology continues to improve, it has generated concerns. For example, some statistical approaches rely on large volumes of data that aren't available to many smaller retailers. Also, the technology's gathering and use of data from online retail activity and the creation of consumer profiles have sparked privacy-related concerns.

WHAT DO YOU RECOMMEND?

Recommendation technology began to take shape in the early 1990s. The University of Minnesota's GroupLens Research Project (www.cs.umn.edu/research/GroupLens/index.html) was an early pioneer. Some of the first recommendation-technology users included Amazon.com and Net Perceptions, a vendor of Web site personalization software.

The driving forces for companies to integrate recommendation engines into their e-commerce sites include the desire to get consumers to not only buy more products but also to return to their sites in the future. Customers would return to sites because recommendation engines make it faster and easier to find products they want, and they offer personalization that yields helpful purchasing suggestions, said Brian Stack, Mavice's chief technology officer and cofounder.

Vendors want to present customers with products they are interested in but didn't plan to buy. Currently, fewer than 25 percent of online shoppers make unplanned purchases, a far smaller percentage than customers at traditional stores, said analyst Patti Freeman Evans with JupiterResearch, a market research firm.

Online vendors also use recommendation tools as a way to promote and generate revenue from older or low-demand items, such as niche CDs and DVDs, said Gartner Research Director Mike McGuire. In addition, businesses use the technology to collect customer data to improve marketing and sales, particularly of available stock and promotional products.

The business models that recommendation-technology vendors generally employ involve either hosting the service for a company or licensing their engines to e-commerce businesses to run themselves.

TYPES OF SYSTEMS

Implicit engines provide recommendations based on analysis of multiple customers' activities while browsing a company's Web site. These engines basically tell users that customers who bought Product A subsequently purchased Product B.

Explicit engines make recommendations based on customers entering words or phrases to describe the type of products they are searching for.

Content-based systems recommend items similar to ones the customer has preferred in the past. These systems gather information about a customer's preferences via questionnaires or past purchasing histories stored in databases.

Collaborative-based engines provide recommendations derived from the purchasing preferences of customers with similar interests or demographics, based on questionnaire responses or profiles generated from consumers' online activities.

"Hybrid models can use various parts of content-based and collaborative models," noted Mavice's Stack.

HOW THEY WORK

Recommendation engines come up with suggestions in different ways, including expert, rules-based approaches and Bayesian techniques, which make decisions via statistically based probability inferences.

For example, said ExpertMaker CEO Lars Hard, "To perform well with small amounts of information, some vendors add encoded knowledge into their system to reflect the expertise of a skilled salesperson."

By knowing the attributes of the product, whether it is clothing or movies, vendors can better analyze user activity to provide relevant suggestions, added ChoiceStream's Strickman.

Some recommendation-technology vendors, such as ExpertMaker, are starting to use one or more types of artificial intelligence, such as neural networks, to learn to make more accurate suggestions with greater consistency.

Most major vendors provide simple APIs so that their engines can easily integrate with Web sites, instant messaging programs, TV-based or wireless applications, and other systems. A recommendation engine can act as a Web service for one or more online stores. In these cases, the engine and the e-commerce application act together over an IP backbone using XML; the Simple Object Access Protocol; the Web Services Description Language; and Universal Description, Discovery, and Integration registries.

Vendors such as ChoiceStream offer remote management, reporting, and testing capabilities as part of a packaged recommendation service for e-commerce vendors. In recommendation engines, relational databases store user profiles, purchase history, and product cross-linking information. "Thus, a scalable and robust database schema is critical to success," said Mavice CEO Greg Russell.

Data mining

Recommendation engines generally use data mining technology, which looks for hidden patterns and relationships in large amounts of data to predict future behavior.

Data mining algorithms analyze customers' shopping and purchasing history, answers to user-preference questionnaires and surveys, and profiles resulting from this information, and translate consumer activity into product and preference relationships that could indicate what consumers might be interested in buying in the future.

Recommendation engines' data mining systems typically update content relationships nightly by looking at changes in user profiles that have occurred during the day, noted AgentArts CEO Andrew Coates.

When deciding which products to recommend, some engines give the newest purchases more consideration than older ones to better reflect a user's current interests.

Collaborative filtering

Many recommendation engines, such as those that Amazon.com and Netflix use, utilize collaborative filtering, which computes personalized recommendations by comparing multiple customers' information—collected via profiles, questionnaires, and purchasing histories—and finding those with similar tastes.

Collaborative filtering uses pattern-matching techniques to determine correlations between products that customers have purchased and potential future items of interest, and between the buying choices of similar customers.

User-based filtering examines and then leverages the history, preferences, and similarities among a current online customer and previous consumers, Russell explained. However, user-based filtering often doesn't scale well because the analysis and comparison processes become more complex for Web sites with many customers and products.

Item-based filtering identifies similarities among items, rather than users, explained Michael Strickman, ChoiceStream's chief technology officer. In other words, the technique, which companies such as ChoiceStream and Mavice utilize, determines that users who liked Item 1 might also like the similar Item 2. At its simplest, this approach could recommend a sports-related product, such as a football book or golf clubs, to someone who purchased a sports-related computer game.

Item-based filtering is generally more scalable than the user-based technique because it's easier to draw correlations among a limited number of products, which are easy to categorize, than among millions of users, whose activities must be examined and analyzed, said Strickman.

QUALIFIED RECOMMENDATION

Companies have several concerns about recommendation technology, including system scalability. According to Netflix chief product officer Neil Hunt, his company's customer base grew rapidly from 2.6 million in 2004 to 4.2 million in 2005 and is expected to increase to an estimated 5.9 million this year and 20 million by 2012. "When you look at that growth track, everything has to be scalable," he said.

Many users are also concerned about privacy and don't like e-commerce sites collecting personal information based on their purchases.

Accuracy

A key issue for recommendation engines is making suggestions that accurately reflect a buyer's interests. According to AgentArts' Coates, this could be a particular problem for low-traffic e-commerce sites, which have relatively little customer data from which to derive useful content relationships. This could also be trouble for sites that offer many content items, which makes selecting recommendations more complex.

However, the problem can also afflict large e-commerce operations. WalMart.com recently took down its Web site's recommendation system when customers who looked at a boxed set of DVDs that included Martin Luther King: I Have a Dream and Unforgivable Blackness: The Rise and Fall of Jack Johnson were told they might also enjoy a number of other titles, including some that were potentially insulting, particularly to African-Americans.

The company has apologized for the incident, saying it occurred when its e-commerce site offered simple, direct links—rather than those determined by customer or product analysis—among a large number of unrelated DVDs. WalMart said this was done to promote video sales, without checking to determine whether the links between some movie pairs could be offensive.

Cost

Most recommendation systems entail significant vendor start-up costs, including those for hardware and software. Another key reason why the systems can be expensive is user-interface development, said Coates. "The way the storefront presents recommendations is crucial in generating benefits," he explained. "The UI layer is the critical difference as to how visible and accessible recommendations really are and, therefore, how easy it is for customers to discover new content without having to do any work."

The Liveplasma.com beta music and movie file-sharing site uses an unusual interface for its recommendation technology. The site graphically maps users' potential interests, showing clusters of circles of various sizes. Each cluster indicates the songs or movies in which customers with similar preferences in one area might have interest. The bigger circles indicate greater potential interest.

Recommendation engines rely heavily on user profiling, which requires storing, backing up, and encrypting enormous amounts of data, which can increase operating costs. In addition, many of the techniques can require considerable processing power.

Meanwhile, return on investment can be difficult to measure. Thus, small and medium-size companies might not feel they can justify implementing recommendation technology.

However, Coates added, large e-commerce providers might want to do so because even a small increase in their sales and customer-retention rates could mean millions of dollars in additional revenue.

Recommendation technology could take several new directions in the near future. For example, noted Gartner's McGuire, the technology could add a location-based component. "I might have a wireless communications device with GPS capability," he said, "and as I get near a store, I could get pinged with a message saying there's a sale there on a CD I might like."

One burgeoning development is matching consumer tastes across Web businesses, such as using knowledge of customers' tastes in one area, like music, to sell them products in another area, like books. This will depend on future business alliances and partnerships, rather than advances in the technology, said Mavice's Russell.

Marketplace adoption of recommendation technology is still in its early stages. Recommendation systems will have to increase personalization by collecting and processing additional data without being intrusive and time consuming. In the future, consumers will not want to utilize the technology unless it is highly functional and easy to use, said Rob Enderle, principal analyst of the Enderle Group, a market research firm.

"Recommendation technology must also be able to reach out to small and medium-size businesses and be robust and cost-effective," said Stack. "Once this occurs, we'll see recommendation technology truly become a ubiquitous piece of an online consumer's daily Internet experience."

Neal Leavitt is president of Leavitt Communications (www.leavcom.com), a Fallbrook, California-based international marketing communications company with affiliate offices in Brazil, France, Germany, Hong Kong, India, and the UK. He writes frequently on technology topics and can be reached at neal@leavcom.com.

Return to: Articles Index