White Coat Oversight of Black-Box Algorithms: Ethical Challenges in the Application of Artificial Intelligence in Healthcare

Introduction

Artificial intelligence (AI) has transitioned from a conceptual tool to a practical component of modern healthcare. Its earliest applications, such as MY- CIN in the 1970s, aimed to emulate clinical reason- ing but were limited by inflexible logic and narrow datasets [1,2]. These early models exposed a foun- dational problem in AI development: the tension between technical innovation and ethical inclusiv- ity. With the emergence of machine learning (ML) Julian Lloyd Bruce (Julian.Bruce.MD@gmail.com) is with the EUCLID (Euclid University) and Affiliated Institutes US Executive Office 1101 30th St NW, Ste 500 Washington DC 20007 (USA) DOI: 10.52609/jmlph.v5i3.216

in the late 1990s, AI gained the ability to identify patterns from large datasets. Technologies like con- volutional neural networks (CNNs) have since transformed medical imaging, often outperforming human readings using traditional diagnostic tools [3]. However, many systems have advanced with- out adequate oversight, as demonstrated by the Therac-25 incident, where software faults led to pa- tient deaths—underscoring the potential dangers of insufficient transparency and regulation [4]. The COVID-19 pandemic accelerated AI’s integra- tion into clinical practice, with tools used for re- source planning and risk prediction demonstrating real-time utility during crisis scenarios [5]. Innova- tions like AlphaFold, which predicts protein struc- tures, also showcased AI's growing role in biomed- ical research [6]. However, enthusiasm for these advancements must be tempered by persistent bias and inequity in deployment. A well-known exam- ple involved a triage algorithm that reduced care access for black patients due to flawed assumptions in training data [7]. In response, regulatory bodies have begun establishing ethical standards and boundaries. The European Union's AI Act and the U.S. FDA's guidance on Software as a Medical De- vice (SaMD) represent attempts to classify high- risk AI systems and impose necessary compliance standards [8]. These efforts emphasize transpar- ency, explainability, and safety, though they re- main in the early stages of harmonization. Looking ahead, the future of AI in healthcare de- pends on learning from past shortcomings. Through inclusive design and interdisciplinary col- laboration, it is possible to create AI systems that support clinical decision-making while respecting patient rights and reducing systemic disparities [9]. Ethical and regulatory safeguards should not be viewed as barriers to innovation but as essential in- frastructure that protects patients and strengthens long-term public trust.

Methods

A comprehensive literature review was conducted using PubMed, Google Scholar, and IEEE Xplore to examine ethical challenges and regulatory con- siderations in healthcare AI. The review prioritised

English-language, peer-reviewed articles, confer- ence papers, and official guidelines published within the past decade, with particular emphasis on work from the last five years to ensure relevance and currency. The analysis focused on core ethical concerns, in- cluding AI ethics, data privacy, informed consent, patient autonomy, and algorithmic bias. Addition- ally, studies exploring XAI tools were incorporated to provide insights into transparency and accounta- bility in AI-driven healthcare systems. Selected studies offered both historical perspectives and contemporary discussions, ensuring a broad under- standing of evolving ethical and regulatory trends.

Results

Ethical and Regulatory Frameworks The ethical deployment of artificial intelligence in healthcare requires more than technical sophistica- tion. It depends on strong legal and moral frame- works that guide how these tools interact with hu- man lives. While principles such as autonomy, be- neficence, justice, and confidentiality remain rele- vant, AI introduces unique challenges related to opaque algorithms, shifting accountability, and dis- parities in data representation [10]. One of the cen- tral concerns is the lack of interpretability in many AI systems, with clinicians and patients often asked to trust recommendations without clear explana- tions. Tools such as SHAP and LIME help make predictions more understandable by revealing how input features influence outputs [11,12]; however, these tools are not always accessible to users with- out technical training, and their effectiveness varies by model complexity. Traceability and accountability must be built into every phase of AI development. Decision pathways should be logged, data sources clearly documented, and roles defined across the development and clin- ical use spectrum. Developers, hospitals, and regu- lators must share responsibility for monitoring per- formance and addressing failures. Ethical design also includes the principle of design justice, which encourages the inclusion of patients, especially those from underserved communities, in shaping technologies that affect them [13]. Additionally, the location of data centres is critical for security and regulatory compliance. Jurisdictional differ- ences shape the governance of health data, necessi- tating careful selection of storage sites that align with privacy laws and accessibility needs [14]. Fur- thermore, mechanisms must be implemented to

track and monitor access to sensitive medical data, in order to safeguard patient confidentiality and prevent unauthorized use. Transparent access logs and audit trails can reinforce trust in AI-driven healthcare systems [15]. Privacy is another foundational issue, as AI sys- tems are typically trained on large health datasets containing sensitive personal information. The World Health Organization has emphasized that patient trust depends on responsible data govern- ance, with its 2021 guidance highlighting the need to balance benefits and risks while prioritizing fea- sibility, equity, and transparency in digital health systems [16]. These regulations aim to reinforce accountability, but implementation and enforce- ment remain imbalanced across different regions. Despite these efforts, regulatory enforcement var- ies due to differences in national policies, resource availability, and cultural values, complicating the establishment of consistent global standards. The development of these frameworks signals an evolving understanding that AI is not value-neu- tral—it must be governed with careful attention to its social impact. Ensuring AI systems do not exac- erbate existing disparities or introduce new ethical concerns requires coordinated efforts among gov- ernments, healthcare institutions, and developers. The industry must adopt a proactive stance to en- sure that AI serves healthcare equitably and ethi- cally. Ethical and regulatory safeguards should not be treated as barriers to innovation, but as essential infrastructure that protects patients and strengthens long-term public trust [17]. By prioritizing trans- parency, inclusivity, and accountability, AI devel- opers and healthcare stakeholders can work toward responsible integration that enhances clinical deci- sion-making without compromising fundamental ethical principles. Autonomy, Consent, and Privacy As artificial intelligence becomes more embedded in healthcare, core ethical principles such as auton- omy, informed consent, and data privacy face new challenges. These concepts, once applied in straightforward clinical contexts, now require rein- terpretation to remain meaningful in an era shaped by complex algorithms. Patient autonomy hinges on the ability to make informed decisions, yet many AI systems operate in ways that are difficult to ex- plain—even to trained clinicians. Tools like Corti, which detects cardiac arrest in real time, illustrate how AI can support decision-making while raising questions about transparency [18]. When patients

are unaware of how an algorithm contributes to their care, their ability to provide meaningful con- sent is compromised. Traditional informed consent requires the explanation of risks, benefits, and al- ternatives, but in the context of AI, it must also in- clude disclosure of how predictions are generated, the data on which models are trained, and any lim- itations that may affect clinical judgment. The Eu- ropean Union's General Data Protection Regulation (GDPR) addresses this issue by granting individu- als the right to an explanation of automated deci- sions [19]. However, making this right operational requires clear, accessible tools that do not under- mine data security or burden patients with technical complexity. Privacy concerns remain central as AI systems rely on extensive health data for training and optimiza- tion. High-profile incidents, such as the Royal Free NHS Foundation Trust’s collaboration with Deep- Mind, highlight how data can be used without suf- ficient patient awareness or consent [20]. To main- tain trust, institutions must integrate strong safe- guards into the design of AI tools, including en- cryption, data de-identification, and limits on data sharing with third parties. Legal frameworks alone are not enough—public trust depends on visible, enforceable governance structures that uphold pri- vacy and consent beyond mere compliance. Court cases like Dinerstein v. Google and the controversy surrounding Project Nightingale illustrate the con- sequences of failing to involve patients in decisions about their personal data [20]. Protecting autonomy and privacy in AI-enabled healthcare requires more than technical adjustments; it calls for continuous attention to communication, trust, and ethical de- sign. Ensuring that innovation supports, rather than overrides, patients' rights is essential in maintain- ing ethical AI implementation [17]. Clinicians play a crucial role in bridging the gap between technical systems and patient understand- ing. To foster their informed participation, they must be trained to explain clearly the functions, benefits, and limitations of AI tools. This empow- ers patients to remain actively involved in their own care, even as decision-making becomes in- creasingly data-driven. Additionally, ensuring pa- tients’ access to understandable explanations and meaningful engagement in AI-driven healthcare decisions reinforces trust and autonomy. Strong pa- tient education and clinician training will be critical in mitigating ethical risks associated with opaque algorithms.

By integrating transparency, accountability, and patient-centred education into AI development, healthcare systems can strike a balance between technological advancement and ethical responsibil- ity. AI-driven healthcare must not only prioritize efficiency, but also safeguard fundamental ethical principles that protect patient autonomy, privacy, and informed decision-making. A proactive ap- proach will ensure that AI enhances rather than di- minishes trust in digital healthcare solutions. Ensuring Safety, Accountability, and Transpar- ency AI systems used in healthcare must meet high standards for safety, especially given the serious consequences of diagnostic or treatment errors. These technologies are often introduced into com- plex clinical environments where even minor inac- curacies can lead to patient harm, making reliabil- ity not just a technical challenge but an ethical and institutional responsibility. Failures such as IBM Watson for Oncology highlight the importance of robust evaluation. Despite early promise, Watson underperformed in clinical settings due to its reli- ance on synthetic datasets and unrepresentative scenarios [21]. This case reinforces the need to train and test models on diverse, real-world data. Regulatory bodies such as the U.S. Food and Drug Administration have increasingly emphasized real- world evidence (RWE) as a key component of ap- proval and oversight for AI-enabled medical de- vices [22]. It is essential to establish clear lines of accountability—developers must ensure that algo- rithms are robust and transparent, capable of de- tecting and correcting bias, while clinicians must evaluate AI recommendations within the context of professional judgment rather than as unquestiona- ble outputs. Institutions play a critical role by main- taining oversight, conducting audits, and respond- ing to performance failures. Legal frameworks are evolving to address these needs. The European Union's Artificial Intelligence Act places medical AI in the high-risk category and requires documentation, testing, and human oversight [23]. Similarly, the FDA's SaMD guidelines call for rigorous validation and ongoing performance monitoring in the United States, shifting the focus from one-time approval to long- term accountability and patient safety. Transparency is also crucial in AI-driven healthcare — explainable AI ( XAI ) tools help users understand how predictions are made, allowing

clinicians to make informed formed decisions. Standards from the National Institute of Standards and Technology (NIST) encourage developers to prioritize interpretability, even when it comes at the cost of marginal accuracy [24,25]. However, the liability gap remains a key issue: clinicians are typically held accountable for medical decisions, yet developers may not bear responsibility when AI tools cause harm. The Therac-25 incident serves as an historical warning about the dangers of poorly defined accountability [4]. Legislation such as the U.S. Algorithmic Account- ability Act seeks to address this issue by requiring stronger oversight for high-impact AI systems [26,27]. Real-world deployments reinforce the need for continuous evaluation. For example, a sep- sis detection tool used in U.S. hospitals failed to align with clinical outcomes, highlighting the risks of insufficient validation [28]. Ensuring long-term safety necessitates institutional commitment to sys- tems that support feedback, revision, and transpar- ency. By embedding safety checks and fostering shared accountability, healthcare systems can inte- grate AI in ways that prioritize patient welfare and reinforce trust [29]. Proactive monitoring and adaptability in AI governance will be essential to mitigate risks and improve healthcare outcomes. Mitigating Bias and Promoting Inclusivity Bias in artificial intelligence systems remains one of the most persistent ethical concerns in the field of healthcare AI. Algorithms trained on non-repre- sentative datasets often underperform for patients from marginalized groups, resulting in inaccurate diagnoses, delayed treatment, or exclusion from services. In dermatology, for example, AI models trained primarily on images of lighter skin tones have shown reduced accuracy in identifying condi- tions in darker-skinned patients [30]. Such dispari- ties illustrate how AI systems can reinforce existing inequalities if inclusivity is not prioritized from the outset. Addressing these issues requires the cura- tion of diverse training datasets that reflect the full range of human variation. Developers should apply stratified sampling and continuous bias monitoring to ensure fairness in algorithm performance across demographic groups [31]. Inclusivity also extends to the design process itself. Involving patients, clinicians, and representatives from underserved communities at the development stage allows AI tools to be shaped by those they are intended to serve. This participatory approach mir- rors the push for more inclusive clinical trials that

followed the NIH Revitalization Act of 1993, which emphasized the importance of representation in improving health outcomes [32]. Bridging the digital divide remains a significant challenge, as limited infrastructure in many regions restricts ac- cess to AI-enabled tools. The African Union’s Dig- ital Transformation Strategy has emphasized the need for increased investment in technology and connectivity to support equitable access to digital health services [33]. Public-private partnerships play a vital role in distributing resources and adapt- ing AI systems to local needs, ensuring that ad- vancements reach underserved populations. Accountability structures are also essential in ad- dressing discriminatory outcomes. The European Union’s AI Act proposes reporting mechanisms for algorithmic harms and mandates transparency in high-risk systems [34]. Developers can further pro- mote fairness by conducting algorithmic impact as- sessments (AIAs) and publishing performance au- dits that examine how models behave across gen- der, racial, and socioeconomic lines [35]. Real- world examples continue to highlight the dangers of biased algorithms, such as a healthcare risk-pre- diction model used in the United States that sys- tematically underestimated the needs of black pa- tients by using healthcare costs as a proxy for health status—unintentionally reinforcing histori- cal disparities in access to care [34]. Building inclusive AI systems is an ongoing re- sponsibility, not a one-time correction. Bias can emerge at any point in the system’s life cycle, from data collection to deployment. Ensuring that AI serves all patients equitably requires sustained at- tention, stakeholder engagement, and ethical over- sight. Strengthening trust in digital healthcare tools depends on continuous refinement and proactive measures to prevent algorithmic bias. By embed- ding fairness measures and prioritizing transpar- ency in AI development, healthcare providers and developers can mitigate risks and foster equitable AI integration. Validation and Ethical Compliance of AI Systems Ensuring that artificial intelligence systems in healthcare are both practical and ethically sound re- quires rigorous validation at every stage of devel- opment and deployment. Without this process, tools risk causing harm, reinforcing bias, or under- mining patient trust. Validation should assess tech- nical performance alongside ethical alignment, en- suring that systems support fairness, transparency,

and accountability. The European Union's Artifi- cial Intelligence Act follows a similar logic, desig- nating healthcare AI as high-risk and requiring doc- umentation of system behaviour, data quality, and human oversight [36]. These requirements help to ensure that AI tools meet consistent standards be- fore and after their introduction into clinical work- flows, and reinforce the need for long-term moni- toring. Ethical compliance is equally critical. AI systems must reflect core values such as autonomy, benefi- cence, and justice—principles first formalized in the Belmont Report that continue to guide healthcare innovation today [33]. Developers must assess whether their tools treat all patient groups equitably, particularly those historically un- derrepresented in medical research. This includes auditing datasets for demographic imbalances and tracking performance across population subgroups [36]. Post-deployment monitoring plays a vital role in maintaining system reliability. Institutions should implement safety reporting mechanisms and conduct routine performance audits, similar to post-market surveillance practices in drug develop- ment, to detect and correct problems that may not be evident during pre-launch testing. Training is also essential. Clinicians need to under- stand the capabilities and limitations of AI tools to use them appropriately and know when to inter- vene. Agencies such as the National Institute of Standards and Technology (NIST) have developed guidelines to help institutions manage AI oversight and clinician training [37]. Beyond technical edu- cation, fostering an ethical understanding of AI de- cision-making ensures that healthcare providers re- main engaged and prepared to challenge AI-gener- ated outcomes when necessary. A proactive ap- proach to training and accountability reduces the risk of over-reliance on automated recommenda- tions while reinforcing clinician expertise in pa- tient-cantered care. Global collaboration will be key to creating unified standards. The World Health Organization’s digital health strategy promotes coordinated efforts to val- idate and monitor AI systems, especially in low-re- source settings [38]. Shared ethical frameworks and consistent validation protocols can help ensure that AI technologies are safe, inclusive, and adapt- able across diverse healthcare systems. Strengthen- ing international cooperation will be instrumental in setting globally recognized best practices that balance innovation with ethical responsibility in

AI-driven healthcare. Limitations and Risks in Decision-Making While AI offers powerful tools for clinical deci- sion-making, it also introduces significant risks that must be carefully managed. AI systems often strug- gle to adapt to the complexity of real-world clinical environments. Algorithms trained on static or nar- row datasets may perform well in controlled condi- tions but fail when applied to new settings or di- verse patient populations. The case of MYCIN il- lustrates this challenge: although it demonstrated high diagnostic accuracy in early tests, it was never widely adopted because it could not respond effec- tively to the unpredictability of clinical practice [2]. Today’s AI models, while more advanced, still face similar obstacles. Assuming that high predictive accuracy guarantees clinical utility can lead to overconfidence in tools that may not generalize well across contexts, increasing the likelihood of misjudgements in patient care. This over-reliance can reduce clinician vigilance. When AI systems routinely provide recommenda- tions that appear accurate, clinicians may defer to these tools without critically evaluating the output. This is known as automation bias and has been ob- served in fields such as criminal justice, where risk assessment algorithms have influenced parole de- cisions without adequate transparency [39]. Healthcare is not immune to these risks, particu- larly when clinicians are pressured to adopt AI tools without sufficient training or oversight. An- other major concern is opacity—many AI systems function as black boxes, offering no insight into how decisions are made. This lack of explainability makes it difficult for users to assess whether rec- ommendations are appropriate or ethically sound. The European Commission’s 2019 guidelines em- phasize that trustworthy AI must be transparent, subject to review, and explainable to non-technical users [40]. From an ethical standpoint, black-box systems raise concerns about nonmaleficence and benefi- cence. If clinicians cannot interpret or challenge AI outputs, they may unknowingly act on flawed rec- ommendations, compromising patient safety and weakening their ability to fulfil professional re- sponsibilities. To address these risks, healthcare systems must invest in clinician training that high- lights the limitations of AI tools. It is essential that clinicians understand when to rely on AI and when to question or override its suggestions. Developers should also prioritize usability, creating interfaces

with clear explanations and confidence scores to guide decision-making, thereby ensuring that AI complements, rather than replaces, human judg- ment. Institutions must have regular auditing and feed- back protocols, including mechanisms for identify- ing and reporting AI-related errors. Such safe- guards help to ensure that AI systems evolve along- side clinical practice, rather than becoming static tools that fall out of sync with patient needs. AI sys- tems should support human judgment, not replace it. Their limitations—in terms of context sensitiv- ity, opacity, and the risk of over-reliance—require ongoing attention. Responsible deployment de- pends on aligning technology with ethical practice and maintaining transparency in every decision that affects patient care [41].

Conclusion and Emerging Chal-

LENGES This paper examines the historical development of AI in healthcare, its ethical challenges, and the reg- ulatory frameworks needed to mitigate its risks. Core themes include transparency, accountability, privacy, and inclusivity, as well as discussing ex- plainability of AI tools, algorithmic bias, and data protection measures. These considerations empha- size the need for AI technologies to enhance patient care while maintaining ethical integrity. Collaboration among technologists, clinicians, and policymakers is essential to achieving responsible AI implementation. Nonetheless, AI’s increasing role in healthcare raises concerns about its potential impact on clinical autonomy, job displacement, and evolving professional responsibilities. Future re- search should explore how AI-driven decision- making might shift the dynamic between healthcare professionals and automated systems, and how to ensure that technology complements, rather than replaces, human expertise.

Acronyms AI: Artificial Intelligence; AIA: Algorithmic Im- pact Assessments; CNN: Convolutional Neural Networks; FDA: Food and Drug Administration; GDPR: General Data Protection Regulation; ML: Machine Learning; NIST: National Institute of Standards and Technology; RWE: Real-World Ev- idence; SaMD: Software as a Medical Device; WHO: World Health Organization; XAI: Explain- able Artificial Intelligence.

V. CONFLICT OF INTEREST The sole author of this paper, Julian Lloyd Bruce, serves as the president of Deeply Human Inc., an artificial intelligence and semiconductor startup based in Austin, Texas. As the company's primary focus is not within the fields of healthcare or clini- cal research, there are no potential conflicts of in- terest concerning this paper's content, analysis, or conclusions.

Acknowledgements

The author wishes to thank Euclid University for its support throughout the research and writing pro- cesses. Special thanks to Professor Laurent Cleenewerck de Kiev for his guidance, insights, and advice on the writing and submission of this paper; his expertise and encouragement were in- strumental in its successful completion.