Factorization MachinesAuthor: Steffen Rendle
Conference: ICDM 2010
Abstract In this paper, we introduce Factorization Machines (FM) which are a new model class that combines the advantages of Support Vector Machines (SVM) with factorization models. Like SVMs, FMs are a general predictor working with any real valued feature vector. In contrast to SVMs, FMs model all interactions between variables using factorized parameters. Thus they are able to estimate interactions even in problems with huge sparsity (like recommender systems) where SVMs fail. We show that the model equation of FMs can be calculated in linear time and thus FMs can be optimized directly. So unlike nonlinear SVMs, a transformation in the dual form is not necessary and the model parameters can be estimated directly without the need of any support vector in the solution. We show the relationship to SVMs and the advantages of FMs for parameter estimation in sparse settings.
On the other hand there are many different factorization models like matrix factorization, parallel factor analysis or specialized models like SVD++, PITF or FPMC. The drawback of these models is that they are not applicable for general prediction tasks but work only with special input data. Furthermore their model equations and optimization algorithms are derived individually for each task. We show that FMs can mimic these models just by specifying the input data (i.e. the feature vectors). This makes FMs easily applicable even for users without expert knowledge in factorization models.
SoftwareIn the experiments of this paper, libFM is applied to the Netflix dataset. Additionally for the task of tag recommendation, a factorization machine (FM) based model is compared to the pairwise interaction tensor factorization model (PITF) [WSDM 2010]. For experimental reproducibility, we provide implementations for these methods.
- libFM is a library for Factorization Machines [ICDM 2010]. In this paper, stochastic gradient descent (SGD) is used for optimization.
- To use the SGD optimization in libFM, use the command line option --method sgd when calling libFM.
- This software can be used to reproduce the first experiment.
- Download libFM software
- The PITF tag recommender [WSDM 2010] is based on a pairwise interaction tensor factorization model.
- To run the tag recommender based on the PITF model, call the tag recommender with the option --method pitf.
- The FM model is integrated into the tag recommender tool. It can be chosen by running the tag recommender with the option --method fm.
- The tool contains two FM variants. One is more integrated and thus faster, the other uses the generic gradient and prediction methods provided by libFM. The (slower) one using the generic methods of libFM can be called by --method fmgeneric; the faster one is called by --method fm. Besides empirical runtime differences, both the generic and the more integrated implementation result in exactly the same predictions and gradients and also their runtime complexity class is the same.
- Download Tag Recommender software
ReferencesTo cite this paper, please use the reference [ICDM 2010].
|Steffen Rendle (2010): Factorization Machines, in Proceedings of the 10th IEEE International Conference on Data Mining (ICDM 2010), Sydney, Australia.||Supplementary Material||BibTeX|
|Steffen Rendle, Lars Schmidt-Thieme (2010): Pairwise Interaction Tensor Factorization for Personalized Tag Recommendation, in Proceedings of the Third ACM International Conference on Web Search and Data Mining (WSDM 2010), ACM.||BibTeX|