An Agent-based Movie Recommender System
Combining the Results Computed based on
Heterogeneous Semantic Datasets
Andreas Lommatzsch, Benjamin Kille, and Sahin Albayrak
DAI-Lab, TU Berlin
Ernst-Reuter-Platz 7, D-10587 Berlin, Germany
{andreas.lommatzsch,benjamin.kille,sahin.albayrak}@dai-labor.de
Abstract. In a growing number of domains, recommender systems support users’ decision making processes in finding entities (such as book,
movies, or newspaper articles) matching their individual preferences. Individual preferences vary extensively thus hampering the creation of good
recommender systems. Additionally, the relevance of items to users depends on a plethora of criteria. Such relevance criteria may consider
users’ current contexts, their ratings (“likes”/“dislikes”) and content describing the entities (e. g., a movie’s leading actors). In this paper, we
introduce an agent-based recommender approach that integrates heterogeneous recommender agents and combines their results based on a
personalized weighting scheme. The framework is open for new agents,
allowing us to integrate new algorithms and data sources. The use of
a personalized weighting scheme for combining the results from heterogeneous agents as well as the integration of personalized recommender
agents allows the system to adapt the results to individual users’ preferences. For improving the trust in unknown items, the system provides
detailed explanations based on the semantic relations. We present a recommender system for the movie domain (based on the framework) and
discuss the evaluation results.
1
Introduction
Recommender systems support users in finding relevant entities by reducing the
number of alternatives to those that they will likely prefer. Traditional recommender systems focus on Collaborative Filtering (CF). User-based CF is built
on the assumption that entities (items or products) are potentially relevant for
users if like-minded users liked them (or rated them positively). As discussed by
Schein et al. [1], CF cannot provide adequate recommendations unless a comprehensive set of ratings is available for all users and all items. This phenomenon
is called the “cold start problem” [2]. New users as well as items entering the
system are particularly affected thereof. The problem incurs as similarities between existing entities and the newly added ones are undefined. In addition,
CF-based algorithms cannot provide good human understandable explanations.
The lack of explanations describing in what extent a recommended entity is
relevant, often leads to missing trust in the system [3]. CF-based recommendations are often biased towards popular items, probably already known to the
users. Recommending already known entities to the user is neither useful nor
serendipitous.
In contrast to Collaborative recommender algorithms, content-based recommender algorithms do not handle entities as black boxes. Instead, an entity’s
relevance is determined by its properties. Thus, content-based recommenders are
able to compute similarities for new (still unrated) entities. Moreover, contentbased recommender systems may provide explanations describing which properties of the recommended entities match the user’s preferences. The recommendation process requires detailed descriptions to ensure high-quality recommendations. This has been a big problem for years, due to the fact that adding these descriptions is costly and time consuming. With the rapidly growing availability of
big semantic data collections, content-based recommender systems receive growing attention. The fine-grained machine readable, “semantically represented”
reusable, often “open” descriptions for huge entity sets provide a good basis for
content-based recommendations. Comprehensive semantic data collections for
the entertainment domain are available, for instance from Freebase1 , DBpedia2 , and IMDb3 . Semantic data sets represent knowledge as graphs consisting
of nodes (representing entities) and edges (representing relationships between
nodes). Nodes as well as edges are equipped with labels providing detailed information about them. Entities in semantic dataset are identified by unique
URIs allowing interlinking entities with other datasets and additional features.
This universal representation of information is well-suited for content-based and
user-specific data. Since every dataset and relationship set might have specific
semantics, optimized edge algebras and models must be learned for every dataset.
Since every user has specific preferences and criteria on what properties determine an entity’s relevance, recommender systems should provide personalized
recommendations. Typically, preferences can be seen as a complex, continuously
changing mixture of many different criteria (e.g., influenced by friends or new
experiences). Moreover, new data sources, providing relevant information, may
become available. Thus, the recommender system’s design should support the
integration of new recommender strategies and be able to constantly adapt to
changing user preferences.
In this paper, we present an agent-based approach that encapsulates various
semantic recommender algorithms and combines their suggestions based on individual user preferences. We evaluate the recommender models and show that
the personalized aggregation of results from heterogeneous recommender agents
provides highly precise recommendation. The remaining paper is organized as
follows: Section 2 reviews related work and discusses current approaches. Section 3 describes the analyzed scenarios and formulates the research questions.
1
2
3
http://www.freebase.com
http://dbpedia.org
http://www.imdb.com
Section 4 introduces the implemented system and discussed the developed algorithms in detail. A discussion of our evaluation results can be found in Section 5.
Finally, we conclude and give an outlook to future work in Section 6.
2
Related Work
In this section we discuss the properties of different recommender approaches.
Collaborative and Content-based Recommender Collaborative recommender
systems suggest items based on preference ratings [4]. If sufficient ratings for
each user and item exist, CF-based systems provide high-quality results. In case
there is only a small number of ratings available, CF-based recommenders cannot provide good recommendations. This phenomenon occurs particularly for
new users and items entering the system. In addition, CF does not provide good
explanations.
Content-based filtering algorithms compute the relevance of entities based on
their features [5]. Textual as well as semantic representations of such features
have been investigated. Textual investigations focus on the most important
terms. Having extracted those from textual descriptions, the recommendation
algorithm computes similarities between entities [6, 7]. The system recommends
the most similar entities for a given entity (e.g., based on a weighted term overlap). The need to handle multi-lingual content, homonyms, synonyms as well
as anaphors constitutes an additional challenge for approaches utilizing textual
data on items. The increasing availability of so-called linked open data enables
item comparison based on well-structured, “semantic” features. Content-based
approaches are generally less personalized compared to collaborative methods,
due to focusing on item features rather than user features.
Semantic Recommenders Semantic recommenders compute recommendations using semantically represented data. Blanco-Fernández et al. [8] introduced
the AVATAR TV program recommender system. Their approach uses an ontology of TV-related entities. The authors determine an entity’s relevance based
on semantic intensity, connectedness and user preferences. A relevance threshold
allows disregarding unappealing TV programs reducing computational complexity. Cantador et al. [9] present the News @ hand system for recommending news
articles based on semantic relations. Their approach encapsulates recommendation algorithms using semantic similarities for either individual users, groups of
users or even user clusters. Challenges building semantic recommenders consist
in defining adequate metrics and efficient prediction models considering the semantics of the edges and reducing the noise in large datasets [10, 11]. Semantic
recommenders can be seen as hybrid recommenders due to the fact that collaborative and content-based knowledge (semantically represented) are combined for
improving the recommendation quality [12].
2.1
Problems with existing Approaches
Current recommender approaches are usually tailored for one specific scenario
even though approaches considering several criteria at a time are reported to
provide superior results [13]. Choosing an adequate parameter setting requires
expert knowledge with respect to the specific scenario [4]. Furthermore, most
recommender systems cannot handle large semantic networks with differently
labeled edges but rely on bipartite datasets (such as preference relations). Existing hybrid recommenders often statically combine a content-based and a collaborative approach based on expert defined rules. Such adaptions require manual
adjustments once a new recommender system is started. Most existing frameworks for processing semantic data do not fulfill the requirements of openness
and flexibility. Many systems are designed for a single scenario but lack extensibility and the support for several semantic relationship types. In the following
section, we introduce our recommender framework that provides a comprehensive collection of components for processing large semantic data collections and
for building adaptive recommender applications.
3
Problem description
In our research, we investigate the problem of combining different recommender
approaches in one system and analyze approaches for the personalized aggregation of suggestions from different agents taking into account individual user
preferences.
3.1
Computing Recommendations based on Semantic Data Sets
In recent years, the amount of information published as Linked Open Data grows
rapidly. For the entertainment domain several large data sets are available as “semantically represented data” providing comprehensive knowledge about movies,
actors, singers, authors, songs, etc. These datasets (e.g., DBpedia) contain data
for a huge number of entities and the relations between them. Unfortunately, the
unified knowledge representation based on a huge graph does not imply a unified
similarity measure for entity types. Due to the fact that relations have different
semantic meanings, additional effort is required, to define similarity measures
optimized for computing recommendations based on the relations. In general,
encyclopedic knowledge (e.g., the relationship movie-director) needs to be
handled differently compared to user generated knowledge (e.g., the relationship
movie-tags). Thus, a recommender system tailored to heterogeneous semantic
data sets should provide several different algorithms optimized to the specific
properties of the respective relationship sets.
3.2
Aggregating Recommendations from Different Agents
An advanced recommender system should be able to consider many different
criteria (semantic relationships) when computing personalized recommendations.
Since new semantic datasets might become available in the future, recommender
systems should support the integration of new recommender strategies as well
as the integration of additional data sources, thus keeping the recommender
system open and adaptive. Due to the fact, that every user has her individual
preferences, recommender systems should learn optimal, personalized weights for
each criterion. The weights can be learned based on user feedback (e.g., ratings)
and should be continuously adapted to match the changes within her preferences.
User preferences may relate to the properties of recommended entities (similarity
of a suggested entity with the user profile) or to properties of the result set (e.g.,
diversity, coverage).
In the next section, we present an approach that allows us to build an adaptive recommender system for heterogeneous semantic dataset taking into account
the discussed challenges.
4
Approach and System Architecture
We define an agent-based architecture for a recommender system optimized for
semantic datasets. Each semantic relationship is wrapped by at least one agent.
The wrapping allows us to learn optimized recommender models for each relationship set. A manager agent aggregates the agents’ recommendations into one
result list considering individual user preferences. This approach ensures high
flexibility, since we can easily add new data or algorithms (wrapped in an agent)
to the system. We achieve the personalization by combining recommendations
from different agents based on user specific weights as well as by combining
results computed based on (“general”, content-based) encyclopedic data and
results based on user ratings (“collaborative data”). This pattern allows us to
provide personalized recommendations using encyclopedic data. Thus, users can
share recommender agents based on encyclopedic data, significantly reducing the
amount of required resources.
Figure 1 visualizes the system’s architecture. The recommender system’s architecture follows the idea that users demand items relevant according to sev-
Σ
actor RDF
director RDF
Σ
Σ
writer RDF composer RDF costume RDF
designer
Fig. 1. The movie recommender system’s architecture: Each semantic relationship is
represented by one recommender agent optimized with respect to its specific characteristics. The resulting recommendations are aggregated taking individual user preferences
into account. The application server provides users with personalized recommendations
as a web application. Since each user has individual preferences, not every semantic
relationship set must be considered by every personal result aggregation agent.
eral criteria such as preferred genres, favorite actors, and directors. Additionally,
users may have different expectations to the list of recommended items: Some
users may prefer homogeneous result list (strongly focused on “globally” optimal recommendations); some may prefer diverse result lists (created based on
“locally” optimal results from each agents), animating them to browse through
the results and to discover relevant, but less popular items. In the movie recommendation domain the system must decide whether the user in interested in the
most relevant movie according to actor, director, etc. or whether the user prefers
movies having the highest score over all criteria. For this reason, our movie recommender provides different algorithms for combining the recommender agents’
results.
4.1
Computing Recommendations
Semantic datasets represent knowledge as a graph consisting of nodes and edges.
Nodes represent entities while edges describe the relationship between entities.
Both nodes and edges are annotated with features describing them in detail.
We must define appropriate similarity measures to compute recommendations
based on semantic data. In general, entities connected by several parallel paths
are more similar than entities connected by a single path. Additionally, we consider the path length: Entities connected by short paths (consisting of two edges)
are more similar than entities connected by longer paths (e. g., consisting of four
edges). We need to assign appropriate edge weights to algebraically compute
recommendations on the semantic graph. The weights must reflect the entities’ similarity induced by the semantic meaning (“label”) as well as the edge’s
relevance. Usually, edge weights are scaled to values in the interval [0; 1]. We
calculate the similarity between two entities by analyzing all parallel paths between them with a defined length. The weights for each path are computed by
multiplying the edge weights along the path. Weights computed for the parallel
paths are summed up. This ensures that two entities are the more similar the
more parallel paths exist and the higher the weights are for the connecting path.
4.2
Combining Results from different Recommender Agents
A movie recommender should consider several criteria when computing recommendations. Our movie recommender wraps each criterion into one agent. The
system delegates user requests to a set of appropriate agents. In a second step,
the manager agent combines the recommender agents’ results. The manager
agent has different aggregation approaches available:
(1) The final list contains each recommender agent’s best ranked entities. This
creates a heterogeneous recommendation list and ensures that the suggested entities are highly relevant according to at least one criterion. E.g., in a movie
recommender scenario, the aggregated recommendation list contains the movie
most similar to the given user profile based on the relationship movies-actors
along with the entity most similar according to the relationship movie-genre.
Since this strategy disregards the sum over the relevance predictions, it might
fail to recommend movies relevant to several criteria but not listed at the top of
any recommender agent’s list.
(2) We aggregate the recommender agents’ suggestions by computing a weighted
sum of their relevance predictions. This approach follows the idea, that movies
are the more relevant the more recommender agents assigned them a positive
relevance rating and the higher these relevance scores. This is based on the
observation that good recommendations should be relevant according to several
criteria. The weights for each recommender agent control the importance of each
criterion to the user.
Both approaches have specific advantages. The aggregation of results by summing up the relevance predictions (CombSUM [14]) shows a higher recommendation precision (in the evaluation using a rating datasets); but the results contain
a higher fraction of popular, probably already known movies. Aggregating the
best ranked movies from each agent increases the result list’s diversity and motivates the user to discover new, less popular movies.
5
A Hybrid Movie Recommender
Based on the presented architecture, we have implemented a movie recommender
system (in Java). We use Freebase and IMDb as semantic data sources. The
data is represented as graph consisting of approximately 6 million entities and 29
million edges (Table 1). The user interface is implemented as a web application
(using Grails).
Table 1. The Table lists the number of nodes and edges extracted from IMDb. This
data provide the basis for the implemented movie recommender prototype. The entity
type misc annotates various persons supporting the production such as location scouts
or stuntmen.
entity type # entities
actors
2,269,883
composers
113,759
directors
244,852
genres
30
keywords
115,359
misc
601,563
writers
334,269
movies
2,043,004
relationship set
movies – actors
movies – composers
movies – directors
movies – genres
movies – keywords
movies – misc
movies – writers
# edges
16,381,711
635,377
1,381,188
1,267,667
3,845,036
3,230,682
2,253,637
density
1.76E-06
2.73E-07
5.28E-07
6.07E-07
1.65E-06
9.24E-07
7.98E-07
Queries and User Profiles Queries for the semantic recommender system consist of a list of entities describing user preferences, e. g. movies a user liked.
Keeping the entrance barrier as low as possible, users may use the recommender
system anonymously. In this case, the system stores user data in the session and
users have to enter their favorite entities explicitly (Fig. 2). The auto-completion
Fig. 2. The screenshot shows how users define an entity based query in the web application. The system compares the character input provided by the user with movie
titles (“entity labels”) and lists the most similar movies. This allows the system to use
unique entity URIs avoiding linguistic problems, e. g. ambiguity of words.
ensures that a valid entity input is found. Due to the fact that the IMDb and
Freebase datasets are user-generated, the number of edges (data connected
with the movies) varies between entities. In general, the more popular a movie
the more details are available for its node representation. Additional to the
semantic dataset, IMDb provides text based movie descriptions and movie reviews. These data are handled using standard text processing techniques such
as language detection, stemming, and stop word removal.
Recommender Agents As discussed in Section 4.1, we model each semantic relationship with one agent. We restrict each agent to a bipartite dataset. Paths
connecting three or more entity set (e. g., movie-actor-location-movie) are
disregarded, since the semantic meaning of “complex” paths is controversial
and may confuse users. Experiments on benchmark data sets have shown that
complex paths barely affect predicted movies’ relevance. Moreover, we limit the
path length to 4, ensuring that users understand why an entity has been recommended. Considering longer paths increases the computational complexity, but
(in general) does not improve the recommendation quality.
The movie recommender system integrates agents computing recommendations
based on either semantic relationship sets (e. g., movie-actor) or on text-based
similarities (e. g., movie-plot). Recommender agents based on semantic relationship sets determine the entities best-connected according to the respective
entities’ relationship set (e. g., movies) to the entities the user liked in the past
(see Fig. 3). The considered paths are visualized to provide explanations to the
user. The text-based recommender agents compute the similarity between two
movies based on the assigned texts’ similarity. We compute the term overlap as
similarity measure for texts [15]. We calculate each term’s weight based on the
tf-idf-value (term frequency – inverse document frequency) of terms appearing
in the considered texts (e.g. plot); stop words are excluded.
Combination Strategies The recommender strategies presented in the previous
paragraph compute recommendations based on one semantic relationship set.
The results provided by such “simple” recommendation agents are aggregated
into a unified result set. We implement the aggregation by summing up the relevance ratings. The weights assigned to each recommender agent are predefined
(computed based on a training data set). Users may adapt the weights according
to their individual preferences. This allows the system to provide personalized
recommendations based on the “simple” recommender agents. In addition to
the weights, users may influence the result set’s diversity by assigning higher
weights to movies relevant according to many different criteria. Assigning higher
weights to movies recommended by many recommender agents reduces the risk
of suggesting bad movies.
Fig. 3. The Figure shows how recommendations are visualized. The left area displays a
list of aggregated recommended movies (ordered by relevance). For each recommended
movie the system provides an explanation as well as additional information such as
IMDb’s description and Freebase’s data. Additionally, users may use the links to
YouTube to watch the movie’s trailer. Movie descriptions and explanations (e. g., the
graph deployed for computing the results) are presented in the page’s right section.
Users may explore each single agent’s recommendations by browsing the tabs Actor,
Director, . . . (located below the query input field). The button Refine allows users
to personalize the weights for each recommender agent. Moreover, users may activate
additional filters for movie types (TV serial, movie, video game), for age classification
as well as for controlling the aggregated result list’s diversity.
5.1
User-centric Evaluation
We conducted a user study to evaluate our recommender system. In discussions
with participants, we realized that movies unknown to the user cause problems in
the evaluation: On the one hand, users expect the recommender to provide them
with unknown entities (following the idea that a recommender should provide
new suggestions); on the other hand, users cannot assess the system’s recommendation quality, if all recommended items are unknown to them. The metarecommender (aggregating the results from the “simple” recommender agents)
should consider the popularity of entities and ensure that the result sets presented to users contain popular as well as less popular entities to cope with this
problem: Relevant popular entities (probably known to users) improve the trust
in the system. Less popular entities (and thus probably unknown to users) cover
the requirement for providing serendipitous, new movies. Especially for “unknown” movies, the system should provide human understandable explanations
describing why the suggested movie matches the individual user preferences and
motivating users to spend time or money on watching it. Explanations generated
based on the semantic data are very helpful, since they describe the aspects in
which a recommendation is potentially relevant to users. Additional information,
such as trailers, images, reviews or plot summaries help users to get a comprehensive impression of the movie without much effort.
When discussing with participants about the movie recommender, we found
that the users’ preferences differ from one another. E.g., some participants are
only interested in the final result list and the movie trailers, while others prefer browsing the recommendations from the different agents in detail. Moreover,
each user has specific expectation what criteria a good movie should fulfill. The
combination of different recommender agents has been seen as a good approach
to consider different preference schemes. The explanations help users to improve
the requests and to adapt the recommender agent’s weighting scheme.
5.2
Data-driven Evaluation
We evaluate our recommender based on a dataset. For ensuring the results’ reproducibility, we use a movie dataset originally created for the second international
Workshop on Information Heterogeneity and Fusion in Recommender Systems4 .
The dataset has been created by combining data from the MovieLens dataset
and IMDb. In contrast to the data used in the movie recommender web application, the HetRec dataset provides only a fraction of movie features; ratings
(extracted from MovieLens) describe at what time a movie has been rated by
a user. We try to predict what movies a user will rate in the future (based on
data about the rating behavior in the past). We split the dataset based on the
rating timestamps into a training set and a test set. Users having less than ten
ratings in at least one of the sets are filtered out to ensure that an adequate
number of ratings exists while learning and evaluating the recommenders (in the
evaluation 760 user profiles have been considered). We compute the recommendation quality for each recommender agent, such as the actor, director, and
genre agents. In the next step, we create a recommender ensemble aggregating
the rating predictions using weighted CombSUM. We compute the optimal weight
for each agent over all users (“global optimization”). In addition, we learn for
4
HetRec ’2011, http://ir.ii.uam.es/hetrec2011/
0.20
0.18
MAP
0.16
Memory-based
Recommender
(Path length = 2)
Memory-based
Recommender
(Path length = 4)
Ensemble
Recommender
(globally optimized weights)
Personalized Ensemble
Recommender
(personalized weights)
0.14
0.12
0.10
0.08
0.06
0.04
0.02
0.00
Fig. 4. The Figure depicts an excerpt of our evaluation. The agents’ weights in the ensemble are displayed in brackets. The recommender ensembles achieve precision superior to any single semantic relation based recommender. The personalized recommender
ensembles achieve an improvement compared to the ensembles based on globally optimized weights.
each user an optimized recommender ensemble resulting in personalized weighting schemes for the recommender agents. The evaluation (Fig. 4) shows that the
combination of different content-based recommenders highly improves the precision. The recommender ensembles’ (based on globally optimized weights) mean
average precision (MAP) outperforms the best one-feature based recommenders.
The personalized recommender ensemble achieves even superior MAP. Thus, allowing users to adjust the weights for the respective movie features as well as
learning the optimal weights based on user feedback is a promising approach for
personalized recommendation.
6
Conclusion and Outlook to Future Work
In this paper, we present a framework for efficiently processing large semantic
datasets (by means of a variety of supported recommender models). We show
how to create an adaptive movie recommender system based on this framework.
Wrapping the recommender algorithms and semantic relationships in agents allows us to integrate new recommender algorithms and relevance criteria at runtime. For combining the recommender agents’ results different algorithms have
been implemented controlling the results’ diversity and reliability. We achieve
personalization of the results by learning weighting schemes, assigning personalized weights to every agent. This weighting is subject to be continuously adapted
by algorithms based on user feedback. The evaluation shows that high-quality
personalized recommender systems can be implemented based on comprehensive,
semantic knowledge. Providing explanations for recommendations improves the
trust in the system (especially when new, unknown movies are suggested). En-
abling the user to explicitly control the weights for criteria used to determine the
relevance of recommended entities allows users to adapt the system to the individual preferences quickly. In future work, we will focus on improving the support
of unstructured data. Therefore, we will integrate an additional component for
Named Entity Recognition and Name Entity Disambiguation in our framework.
Additionally, we plan to test the context-aware adaption of personalized weights
for the agents.
References
1. Schein, A., Popescul, A., Ungar, L., Pennock, D.: Methods and metrics for coldstart recommendations. In: Proc. of the 25th ACM SIGIR Conf. on Research and
Development in IR. SIGIR ’02, ACM 253–260
2. Koren, Y., Bell, R., Volinsky, C.: Matrix factorization techniques for recommender
systems. IEEE Computer (2009)
3. Adomavicius, G., Zhang, J.: On the stability of recommendation algorithms. In:
Proceedings of the fourth ACM conference on Recommender systems. (2010) 47–54
4. Adomavicius, G., Tuzhilin, A.: Towards the next generation of recommender systems: A survey of the state-of-the-art and possible extensions. Transactions on
Knowledge and Data Engineering 17 (2005)
5. Lops, P., de Gemmis, M., Semeraro, G.: 3. In: Content-based Recommender
Systems: State of the Art and Trends. Springer (2011)
6. Lang, K.: Newsweeder: Learning to filter netnews. In: Proc. of the 12th ICML,
Morgan Kaufmann (1995) 331–339
7. Melville, P., Mooney, R., Nagarajan, R.: Content-boosted collaborative filtering
for improved recommendations. In: 18th national conf. on AI, Menlo Park, CA,
USA, AAAI (2002) 187–192
8. Blanco-Fernández, Y., Pazos-Arias, J., Gil-Solla, A., Ramos-Cabrer, M., LópezNores, M.: Providing entertainment by content-based filtering and semantic reasoning in intelligent recommender systems. IEEE Transactions on Consumer Electronics 54(2) (2008)
9. Cantador, I., Bellogin, A., Castells, P.: News@hand: A semantic web approach
to recommending news. Adaptive Hypermedia and Adaptive Web-based Systems
5149 (2008) 279–283
10. Kunegis, J., Schmidt, S., Lommatzsch, A., Lerner, J., Luca, E.W.D., Albayrak, S.:
Spectral analysis of signed graphs for clustering, prediction and visualization. In:
Proc. SIAM Int. Conf. on Data Mining. (2010) 559–570
11. Fouss, F., Yen, L., Pirotte, A., Saerens, M.: An experimental investigation of five
graph kernels on a collaborative recommendation task. In: Proc. of the 6th IEEE
Intl. Conf. on Data Mining (ICDM 2006). (2006)
12. Katz, G., Ofek, N., Shapira, B., Rockach, L., Shani, G.: Using wikipedia to boost
collaborative filtering techniques. In: Proceedings of the fifth ACM conference on
Recommender systems. RecSys ’11, New York, NY, USA, ACM (2011) 285–288
13. Bell, R., Koren, Y.: Lessons from the netflix prize challenge. ACM SIGKDD
Explorations 9(2) (2007)
14. Hull, D., Pedersen, J., Schütze, H.: Method combination for document filtering.
In: Proc. of the 19th ACM SIGIR conf. SIGIR ’96, NY, NY, USA (1996) 279–287
15. Baeza-Yates, R., Ribeiro-Neto, B.: Modern Information Retrieval. Addison Wesley
(1999)