Recommendation systems and machine learning: approaches and case studies
September 26, 2023
Machine learning-based recommendation systems are powerful engines using machine learning (ML) algorithms to segment customers based on user data and behavioral patterns (such as purchase and browsing history, likes, or reviews) and target them with personalized product or content suggestions.
Let's explore these tools’ nature and different design approaches and find out how machine learning experts can help various businesses adopt recommendation engines to boost customer engagement and improve user experience.
Recommendation system market trends
67%
of consumers expect relevant product or service recommendations from brands
- McKinsey
76%
of consumers are more likely to purchase from brands that personalize services
- McKinsey
+26%
of revenue by leveraging product recommendations based on artificial intelligence and machine learning
- Salesforce
Market Snapshot
Fastest Growing Market:
Asia-Pacific
Largest Market:
Asia Pacific
Major players
Scheme title: Global recommendation engine market forecast
Data source: mordorintelligence.com — The recommendation engine market
Approaches to recommendation system design
Most recommendation engines fall into three major sub-categories depending on the approach used to select and recommend products or services.
Collaborative filtering: a focus on user similarity
How it works
Recommendation engines in this category rely on machine learning algorithms, such as clustering models, regression, user-based k-nearest neighbors, matrix factorization, and Bayesian networks, to survey customers’ perceptions of products. After identifying customer preferences for certain products, described via a user-item matrix, the engine will offer items already bought by other users with similar tastes. The system can define users via explicit data collection (asking them to compile a list of favorite items and rate previously purchased products) or implicit data collection (scanning user interactions on social media powered by AI or monitoring user behavior like purchases and browsing on ecommerce websites).
Collaborative filtering
Pros
Collaborative filtering systems can be extremely accurate and provide effective suggestions, especially when relying on context-aware filtering.
These systems can predict customers' interest in a product they didn't know existed by observing what caught the attention of similar users.
Recommendation engines based on this approach perform well even without understanding the nature of each item, which eliminates the need for detailed product descriptions.
Cons
Cold start problem: providing valuable suggestions to new users with no purchase history can be challenging considering the only available parameters (gender, age, etc.).
Scalability: using this algorithm to search for purchase patterns among a growing number of customers and products requires significant computational power.
Rich-get-richer effect: algorithms generally recommend products with many excellent reviews, increasing their popularity at the expense of new items.
Data sparsity: in cases with a large product catalog, each item may not have a sufficient number of user reviews to analyze, reducing the recommendations' accuracy.
Shilling attacks: new products are vulnerable to user rating manipulations (such as negative reviews from competitors).
Content-based filtering: a focus on product similarity
How it works
This approach mainly considers the item's characteristics, such as price, category, and other features defined by assigned keywords and tags, along with user preferences interpreted from their purchases and related feedback. Based on these metrics, a machine learning algorithm (be it Bayesian classifiers, decision trees, clustering, etc.) will investigate customers' purchase patterns and recommend other products sharing similar features with those previously bought and positively reviewed.
Content-based filtering
Pros
This approach mitigates the problem with new products on sale due to the assigned keywords.
The most advanced content-based recommendation systems can enrich the set of examined variables by probing text reviews through AI-based natural language processing (NLP).
Cons
The tagging procedure implies a massive workload, especially on large platforms or marketplaces.
The cold start issue of collaborative filtering is still there, although less critical than in collaborative filtering, as the historical data associated with new customers is very limited.
Algorithms can be rather conservative, recommending categories of products and content already purchased by a user and avoiding new, potentially interesting categories.
Hybrid recommender systems: the best of both worlds
How it works
Many recommendation systems embrace a hybrid approach combining collaborative and content-based filtering. There are several ways to hybridize them. The mixed hybridization technique involves providing users with both collaborative and content-based suggestions at the same time. The weighted technique, on the other hand, merges the score calculated via two different approaches. Another combination trick, namely meta-level, implies using the output of the first approach (basically the machine learning model built by algorithms) as an input source for the second one.
Pros
According to recent studies, hybrid approaches significantly enhance recommendation systems' performance.
A hybrid engine can use the best approach for each specific task. Netflix’s and Spotify’s systems spot similarities among users through collaborative filtering but identify movies and songs with the same attributes via content-based filtering.
Cons
Merging both mechanisms into a single system requires more complex architectures and superior computing power.
Consult our experts to implement machine learning recommendation systems
How ML-powered recommendation systems work
Whether based on collaborative or content-based filtering, ML-powered recommendation systems will follow a multi-stage pipeline to turn product and customer data into personalized suggestions.
Data collection
A machine learning system needs large data sets to segment customers, namely categorize them into a certain archetype or buyer persona according to their attributes, and target them with suitable suggestions. Relevant metrics can include browsing behavior, purchase history, content usage, personal information from user profiles, product reviews, access devices, and many more. The system can gather this information via explicit or implicit data collection, while product features can be obtained from the related tags.
Data storage
Data sets should be consolidated into a suitable repository depending on the type of data a recommender system needs to analyze. Along with traditional SQL databases designed to efficiently store structured data, you can rely on NoSQL databases that handle complex formats, such as unstructured data. Data warehouses, on the other hand, can easily integrate information from multiple sources and prepare it for analysis, while data lakes act as flexible repositories to ingest any data format.
Scheme title: AWS-based recommender system architecture with an Amazon S3 data lake
Data source: AWS — Architecting near real-time personalized recommendations with Amazon Personalize
Data analysis and decision making
The recommendation system leverages machine learning algorithms to process data sets, identify patterns and correlations among multiple variables, and build ML models portraying them. For example, algorithms can identify a recurring connection between the age of customers and their preference for one brand over another. Trained models can make predictions on user preferences and recommend the most suitable products or content which companies then base their decisions upon.
Scheme title: Machine learning modeling
Data source: medium.com — docs.microsoft.com—What is a machine learning model?
Top recommendation systems
Nowadays, all major digital service providers and ecommerce enterprises rely on large-scale recommendation systems in a variety of use cases to deliver a customized user experience and enhance sales performance or advertising revenues.
Amazon leverages a recommendation algorithm for products or search results, combining in-site suggestions based on several strategies (recommended for you, bought together, recently viewed, etc.) with off-site recommendations via email. The ecommerce leader deployed its collaborative filtering-based recommender engine between 2011 and 2012, recording an outstanding 29% sales increase in the second fiscal quarter of 2012. However, Amazon is now shifting from item-based collaborative filtering (users who bought A also bought B) to a system based on autoencoder neural networks.
YouTube implemented a recommendation system to prioritize certain videos, suggest channel subscriptions, and provide relevant news. The engine takes into account a variety of parameters, defined as "signals" to better frame user interests, including clicks, likes and dislikes, survey responses, watch time, and shares. In addition to personalizing user experience, this system aims to promote high-quality information while demoting problematic content, such as sensationalistic tabloid news or racy and violent videos.
Facebook uses a recommendation engine based on deep learning and neural networks (known as DLRM or deep learning recommendation model) for friend suggestions and news feed sorting, but also to recommend groups, pages you could be interested in, or products on its marketplace. The DLRM processes both continuous features (such as age) and categorical features (including product category) detailing items and platform users. Categorical features are described through embeddings, which are representations of words via real-valued vectors.
Netflix relies on a deep learning-powered recommendation system to provide movie recommendations. Its algorithms consider variables such as user features (including browsing history and ratings issued), movie type and popularity, and item-item similarity with previous content to sort the groups of movies displayed in horizontal rows on its home page. The engine also monitors strictly contextual features, such as the day of the week, seasonality, national festivities, device, and more.
LinkedIn deployed a recommendation system to suggest job ads, connections, and courses. One of its core applications is LinkedIn Recruiter, a comprehensive HR tool that compiles lists of suitable candidates for an open position and ranks them depending on their skills, experience, and location. The talent search engine considers both the relevance of certain candidates to a given query and the mutual interest between such professionals and recruiters (essentially, if the candidate positively replies to an InMail message from the recruiter).
ML-based recommender engines implementation tips
ML-based recommender engines are complex tools with intricate architectures, so you can’t reap their benefits without proper implementation. Here are some factors to keep in mind:
Approaches
Collaborative, content-based, and hybrid approaches determine recommendation systems' underlying mechanics and the variables to prioritize. The design choice depends on your business scenario, including your target audience and product range, as well as the data available. For example, major retail platforms with a broad service or product offering can find the hybrid approach to achieve high performance and address the wide range of tasks they have.
User-driven strategies
The most effective recommender systems can switch between different recommendation strategies depending on the customer journey stage. For example, first-time users could be directed towards the "most popular" products, as we still can't frame their interests, while visitors with a long purchase history can be targeted with tailored recommendations based on user affinity with certain categories of merchandise.
Page context-driven strategies
Different page contexts can also call for specific recommendation strategies. For instance, a standard choice for home pages is the “most popular” strategy, while recommendations based on “similar items” can work better for product pages. Regarding cart pages, it's worth embracing a “bought together” strategy to encourage upselling.
Platform-based solutions
Nothing prevents you from building recommender systems fully tailored to your needs if you're ready to face higher upfront costs for its development. However, you can also rely on cloud-based services and tools providing built-in algorithms, pre-trained ML models, and APIs to receive model output. These include Salesforce Interaction Studio, Adobe Target, Amazon Personalize, Optimizely, IBM Watson Real-Time Personalization, and more.
ML-based recommendation system benefits
Retail companies and digital service providers are relying on machine learning-powered recommendation systems and other big data ecommerce solutions to achieve six essential goals:
Better user experience
Recommendation systems help replicate in-store customer care and personalized guidance offered by a real salesperson in a virtual environment, boosting user engagement and satisfaction.
Focus on the right product
Recommender systems mitigate the so-called information overload as they direct customers towards the product they want, hidden amid an overwhelming range of merchandise and content.
Sales and revenue drive
Personalizing shopping experience and highlighting relevant products result in a higher number of items per order, superior average order value, enhanced customer retention and lifetime value, and, therefore, revenue growth.
Data-driven decision-making
Recommendation systems gather customer and sales data which can be used to compile detailed reports, providing managers with valuable insights into customers’ preferences and enhancing decision-making for marketing, logistics, and pricing strategies.
Optimized marketing costs
Along with targeted advertising and triggered communications, recommendation systems can help mitigate marketing costs. In this regard, McKinsey highlighted that product recommendations can improve marketing-spend efficiency by 10-30%.
Driving personalization with machine learning
Recommendation systems offer a synthesis between customers' wishes for a customized user experience and personalized digital services and the need for retail companies and digital service providers to improve their sales performance. Machine learning in ecommerce and retail, as well as HR and eLearning has further fostered this synergy, allowing enterprises to fully personalize customer journeys.
On the flip side, these data-driven technologies are computationally demanding and can create tension between sheer performance and the growing demand for privacy and data protection from both the public and legislators. To streamline their implementation and ensure compliance, rely on an expert partner like Itransition.
Rely on Itransition’s ML solutions to personalize your digital services
FAQ
What is the most common type of recommendation system?
According to Grand View Research, collaborative filtering-based engines are currently the most popular type on the market, while the hybrid system segment seems set to expand at the highest CAGR.
Which algorithm is best for a recommendation system?
Recommendation systems relying on deep neural networks outperform those based on "traditional" machine learning algorithms in a variety of studies, according to Amazon’s research. Thanks to their complex, multi-layered structures with interconnected nodes, neural networks can better identify non-linear relationships among data points. However, they require superior computational resources, wider training data sets, and extensive hyperparameter optimization to tune the learning process.
Do recommendation systems use supervised or unsupervised learning?
Recommender engines can leverage both supervised and unsupervised learning. For instance, they rely on supervised learning to handle labeled data, such as product ratings. However, they can also use k-means clustering, an unsupervised learning algorithm, to make inferences from data sets without labels and aggregate data points with certain similarities.
Service
Machine learning consulting services & solutions we deliver
Explore our range of machine learning consulting services, along with related technologies, use cases, implementation roadmap, and payoffs.
Service
ML services and solutions from certified machine learning experts
Get ML consulting and development services from machine learning experts. Proven expertise in artificial intelligence, machine learning, and data science.
Case study
An ML solution for brand analytics and reporting
Find out how Itransition’s team designed and developed an ML tool for brand tracking and analytics that processes images 50% faster than the legacy solution.
Case study
A shoppable video platform for AiBUY
Find out how Itransition’s dedicated team helped AiBUY release their innovative machine learning-driven shoppable video platform.
Service
Ecommerce consulting services: key options, platform expertise, and benefits
Discover how retailers can improve their digital commerce strategy and significantly enhance their online store performance with ecommerce consulting services.
Case study
Custom ML algorithms for an insurance platform
We developed and trained an AI model that predicts insurance application conversion, helping the customer select targeted user price policies and discounts.
Insights
AI in social media: use cases, top tools & adoption challenges
Explore key use cases and real-life examples of AI in social media, along with tools and guidelines to implement effective social media strategies.
Insights
Accelerating ecommerce growth with predictive analytics
In this article, we discuss how ecommerce companies can implement predictive analytics to boost customer loyalty, optimize operations, and increase profits.