Introduction
Artificial intelligence is becoming a standard feature of modern business. From screening job applications and assessing creditworthiness to detecting fraud and allocating public support, automated systems increasingly influence decisions that affect people’s lives.
Although these systems can process information quickly, they may also reproduce discrimination contained in historical data. When an automated system makes an unfair decision, responsibility cannot simply be attributed to “the machine”. The organisation choosing to use the technology will generally remain accountable for its consequences.
The Company Using the Tool
Under the Equality Act 2010, discrimination caused by an automated process may be unlawful in the same way as discrimination caused by human conduct. This includes:
- Direct discrimination, where someone is treated less favourably because of a protected characteristic.
- Indirect discrimination, where an apparently neutral practice places people with a protected characteristic at a particular disadvantage.
For example, a recruitment system may reject applicants because it was trained on historical data that undervalued women, disabled people or ethnic minorities. The employer may face legal consequences even if it did not intend to discriminate. The Equality Act therefore applies to discrimination in employment and employment services, including arrangements for selecting applicants.
Businesses using artificial intelligence should assess whether a system is suitable, monitor its results and address discriminatory outcomes. They cannot avoid responsibility by pointing to an external software provider.
The Software Developer
Developers may also face responsibility where they design an unreliable system, use unsuitable training data or fail to communicate known limitations. However, determining liability can be complex. The UK currently regulates artificial intelligence through existing legal frameworks and sector-specific regulators, rather than a single, comprehensive AI statute.
Responsibility may depend on factors such as:
- who designed the system
- who supplied or selected the training data
- who configured the tool
- who made the final decision
Contracts may allocate risks between developers and users, but they cannot remove individuals’ rights under equality or data protection law.
Data Protection and Human Oversight
The UK General Data Protection Regulation (GDPR) provides additional safeguards where decisions are made solely through automated processing and have legal or similarly significant effects. Subject to specific exceptions, individuals may have the right to:
- receive information about the processing
- express their views
- request human intervention
- challenge the decision
Human involvement must therefore be meaningful. A manager who simply accepts an AI‑generated score without examining the evidence is not providing effective oversight. Proper review requires the decision‑maker to understand the system’s role, consider the individual circumstances and have authority to change the outcome. Therefore, human oversight should prevent decision‑makers from accepting an automated result without scrutiny.
Shared Accountability
Responsibility is shared, but not necessarily equal. Developers should create systems that are tested, transparent and appropriately designed. Organisations using them must conduct due diligence, monitor outcomes and provide accessible complaints procedures. Legal teams and data protection specialists may assist with equality advice, data protection impact assessments, procurement, contractual risk allocation and regulatory investigations.
Ultimately, artificial intelligence should remain a decision‑support tool, not an independent judge. When a business allows software to influence someone’s employment, finances or access to services, it must remain accountable for the result.
Fairness depends not only on reliable technology, but also on responsible governance, meaningful human judgement and a clear route for correcting mistakes.