Recommendation systems create personalized list of items that might interest the user by analyzing the user’s history of past purchases and/or consumption. For rating based systems, most of the traditional methods for recommendation focus on the absolute ratings provided by the users to the items. In this paper, we extend the traditional Matrix Factorization approach for recommendation and propose pairwise relation based factor modeling. While modeling the items in the system, the use of pairwise preferences allow information flow between the items through the preference relations as an additional information. Item feedbacks are available in the form of reviews apart from the rating information. The reviews have textual information that can be really helpful to represent the item’s latent feature vector appropriately. We perform topic modeling of the item reviews and use the topic vectors to guide the joint factor modeling of the users and items and learn their final representations. The proposed method shows promising results in comparison to the state-of-the-art methods in our experiments. Copyright © 2018 by the paper’s authors.