When the Algorithm Gets You Wrong: The Right to Be an Exception
In the modern era we are surrounded by algorithms. They are being used to make life-changing decisions like who gets parole, hired for a job, or admitted to college. Algorithms rely on patterns and often averages, but not everyone fits into an average mold. That is what the authors of The Right to Be an Exception to a Data-Driven Rule call a data-driven exception. What are your rights when you are a data-driven exception? How do you even know if you are one?
Data-driven Rules
Algorithms are making life changing decisions in our everyday lives that we are often unaware about. Data-driven rules are basically the decision making framework behind an algorithm. For any algorithm to make a prediction, it has to follow patterns or averages in a set of data. Take linear regression for a simple example:
There will always be outliers in real data, but a model is generally fit so that outliers do not skew the prediction for the majority of the data. What happens when this model is used on the outlier?
Imagine we are using test scores on a standardized test to predict whether a student should be admitted into an advanced class. In the majority of cases, a higher test score is associated with higher performance in the class. In this case, though, there is one student who scored only 10 on the exam, yet when admitted to the program received an A average (99%).
Data-driven Exceptions
If admissions officers used this algorithm to decide on who to admit, they would not have admitted this student in the first place. This person is called a data-driven exception. Data-driven exceptions are people whose circumstances are not adequately captured by the model (the generalization made does not fit them).
In some cases, the actual outcome can’t be determined. For students that are outliers, they would not get admitted in the first place, would not have the ability to receive a grade, and therefore would never know they were a data-driven exception. This also applies to other algorithms, like those that determine whether an incarcerated person is granted parole.
The student is an example of partial observability, which is when the algorithm does not receive all of the data necessary to make an accurate prediction. Perhaps our student had the death of a parent on the morning of the exam. That would certainly be relevant information for a human making a decision, but it is not information that is easy to standardize for an algorithm. Other causes of data-driven exceptions can include:
- Sampling Bias: the exception is not represented in the training data
- Model capacity: creating too generalized predictions, like fitting a linear model on non-linear data
- Distribution Shift: An algorithm trained on weather patterns in Death Valley is deployed in Antarctica
Data-driven exceptions can be caused at multiple points in the modeling process, from data collection, to model training and deployment. Some data-driven exceptions are difficult to eliminate though, because of the nature of human decisions. People are able to use discretion and take in much more nuanced information than a model is. For example, Applicant Tracking Systems (ATS) are widely used in hiring decisions today. If you don’t use the correct buzzwords on your resume, it could be automatically rejected even if you are otherwise qualified for the job. I know many people who struggle getting an interview because their resume is rejected automatically.
If you take five different hiring managers and give them the same resumes, they will all have different personal opinions and biases that impact who they favor. If all of these hiring managers use ATS instead, it causes systemic harm because the same person will be an exception for all five jobs. Depending on the complexity, algorithms can sometimes be a ‘black box’ that not even the developers can explain fully. If your resume is being rejected by every single job, it could be difficult to find out why, even if you have access to the algorithm. A regression or decision tree are generally easy to interpret, but more complex models like ensemble models and neural networks are not.
Eliminating Exceptions
Individualizing the algorithms by adding more data seems like a great idea on the surface. If we can just add more data for the model, why wouldn’t it provide better predictions? The first issue with individualization is privacy. Most people would prefer to give less information about themselves to an algorithm. If you’re applying for a job, should they ask you about your mental health history or childhood? It would be a clear violation of privacy to collect the amount of information necessary to make predictions closer to human-made decisions. Additionally, it is often not feasible to collect and process this data. There are some circumstances that don’t easily translate to a categorical or numerical variable in a model.
You can’t eliminate all exceptions. Unfortunately, there will always be error. There is a certain amount of randomness to data that is unavoidable. This error (aleatoric uncertainty) can’t be reduced with adding additional data or changing the model.
If we can’t eliminate all error, what is the next step? It is to acknowledge that with every model there is uncertainty. When applying data-driven rules to high-stakes situations like parole, hiring, or college admissions, we must be even more cognizant of the uncertainty of predictions. We must also stop expecting data-driven exceptions to carry the burden of being an exception. But if a person isn’t responsible for being an exception, who is responsible for treating data-driven exceptions ethically, and what steps should they be taking to prevent harm?
Final Thoughts
This article was dense and had a lot to unpack, but I tried to step back academically a little bit and think about specific situations I’ve experienced (school admissions, hiring) to make this concept relatable. It made me wonder if I’ve ever been overlooked for a school or job because of a data-driven rule. There is little transparency when these algorithms are being used. Was my college admission dependent on an algorithm like this? Almost certainly. How would I even know if I hadn’t been admitted somewhere based on an arbitrary reason? I guess I wouldn’t. There is no recourse if you never know what could have happened, and I think that is the most concerning part about data-driven exceptions. The authors suggested the duty to protect exceptions be on data-driven decision makers, but I don’t see that happening without legislation that can be enforced.
