October 21, 2021
3 min read

Where do biases in ML come from? #1 👉 Introduction

Searching for bias in ML
Jean-Marie John-Mathews, Ph.D.
Searching for bias in ML

Research papers list plenty of bias sources of /. One review that particularly interests us is the one from Mehrabi (2019). It formalizes the AI development process as a feedback loop with three components:

  • Data
  • Algorithm
  • End-user

Along this circular process, multiple biases can arise:

A survey on Bias and Fairness in Machine Learning

With the widespread use of AI systems and applications in our everyday lives, it is important to take fairness issues…

1. Between data and algorithm

Data scientists train models in a social world with many imperfections: discriminations, social inequalities amongst other power relations. These imperfections can induce hidden biases inside the model despite data scientists’ efforts to develop good models (Suresh & Guttag, 2019).

2. Between algorithm and end-user

AI models are used by humans to make decisions, often through user interfaces. For instance, a credit scoring algorithm computes the default probability to help bank advisors accept or deny loan requests. Depending on the advisor and the design of the interface, the same probability can lead to very different decisions (Friedman & Nissenbaum, 1996).

3. Between end-user and data

In a production system, end-users can produce behavioral data that are fed back to the model. These behaviors can vary depending on enunciation contexts. For example, one single user can have very different behaviors across social networks. Consequently, models are only valid in a specific context and can fail to generalize (Baeza-Yates, 2018).

At Giskard, we help AI professionals to identify these hidden biases. In an upcoming series of posts, we are going to deep dive into the multiple categories of biases and how to address them.

Want to know more? ✋ Contact us at