In games with precisely defined rules and 'win conditions,' like chess and go, machine learning algorithms have also done well. However, medical professionals don't always agree on how to treat patients, and even the winning criteria of being 'healthy' is not universally accepted. Doctors know what it means to be ill, and we have the most data on patients when they're at their worst, as according to Ghassemi. However, since people are less likely to visit doctors when they are healthy, we don't get as much information from them.

Even mechanical devices have the potential to produce inaccurate data and treatment inequalities. For instance, pulse oximeters that have been calibrated primarily on people with light skin cannot reliably assess the blood oxygen levels of people with darker complexion. And when oxygen levels are low, which is precisely when accurate readings are most crucial, these deficits are at their most severe. In a similar vein, Ghassemi and Nsoesie argue that women are more likely to have risks after 'metal-on-metal' hip replacements 'partially due to anatomic differences that aren't taken into account in implant design.' These kinds of details might be concealed in the data provided to computer models, undermining their conclusions.

According to Ghassemi, the output of machine-learning algorithms has 'the shine of objectivity' when it comes from computers. But it can be misleading and hazardous because it's more difficult to identify the inaccurate data sent to a computer in bulk than it is to ignore the advice of a single potentially incompetent (and possibly racist) physician. She believes that the issue is not with machine learning itself. It's the people. Because human caretakers are fallible, they may produce inaccurate data.

She nonetheless maintains that machine learning can contribute to more effective and equitable suggestions and practices in the field of healthcare. Improving the quality of data, which is a difficult challenge, is one way to fulfill the promise of machine learning in the healthcare industry.

'Imagine if we could share data from physicians who perform well with other physicians with less training and experience,' adds Ghassemi. We must audit and get this data immediately.

She points out that the problem in this situation is that there are no incentives or rewards for data collection. 'Getting a grant for something or asking students to work on it takes work. Why should I give my data away for free when I can sell it to a firm for millions, data providers may ask?

But access to data should not be constrained by issues like 'What paper will I get my name on in exchange for providing you access to data that sits at my institution?'

The only way to improve health care, according to Ghassemi, is to obtain better data, and the only way to obtain better data is to encourage the release of that data.

It involves more than just gathering data. There is also the issue of who will gather and examine it. Ghassemi advises putting together varied research teams made up of doctors, statisticians, medical ethicists, and computer scientists.

She claims that the Patterns paper's goal is not to deter technologists from using their machine learning skills to the medical industry. Before endorsing a specific computer model, they just need to be aware of any treatment gaps and other complications that need to be taken into account.