Incorporating fairness into the design of machine learning (ML) systems is a central component of responsible technology design. However, in machine learning, and particularly within the subdomain of recommendation systems, applying concepts of fairness requires attention to various stakeholders’ complex and often-conflicting needs. Since fairness is a socially constructed, theoretical, and context-dependent construct, there are numerous definitions that span multiple disciplines. Still, it is rare for machine learning researchers to develop their metrics in close consideration of their social context, or with the input of those who are most impacted by these systems. More often, standard definitions are adopted and assumed to be applicable across contexts and stakeholders. This complexity of fairness definitions and contexts coupled with the proliferation of fairness metrics in research literature have led to a challenging decision-making space for practitioners to navigate. In addition, the practical incentives, motivations, and constraints of doing fairness work in industry settings limit what is possible for practitioners.

In this dissertation, I confront these challenges by uncovering potential best practices and methods for operationalizing fairness in multistakeholder recommender systems. Drawing on five qualitative research studies involving over 100 end-users, item providers, practitioners, and ML fairness experts, I examine the intricate process of translating subjective perspectives and theoretical fairness frameworks from moral philosophy into empirical investigations suitable for observable settings. I define “pragmatic fairness” as the path from business constraints to imperfect (though satisfactory) fairness evaluations, and I explore how fairness evaluation can still be successful within this practical setting. Finally, I discuss the tradeoffs between best practices and realistic practices when operationalizing fairness in industry, and I point towards promising future work to help bridge this gap.