White Coat Oversight of Black-Box Algorithms: Ethical Challenges in the Application of Artificial Intelligence in Healthcare
DOI:
https://doi.org/10.52609/jmlph.v5i3.216Keywords:
Artificial Intelligence (AI), Confidentially, Ethics, Informed Consent, Personal Autonomy, PrivacyAbstract
Artificial intelligence (AI) is rapidly influencing the future of healthcare by increasing diagnostic accuracy, supporting personalised treatments, and improving system efficiency. This paper examines the ethical and regulatory issues that arise from incorporating AI into medical practice. Drawing on the evolution of AI from early systems such as MYCIN to more recent applications such as convolutional neural networks in imaging, the discussion highlights the importance of ethical oversight from the outset of development. Central themes include the necessity for transparency, strong data protection measures, algorithmic fairness, and responsible deployment. Explainable AI (XAI) technologies, international regulatory responses such as the European Union's AI Act, and inclusive design strategies are explored as key tools for ensuring equity in care delivery. Risks, including data misuse, embedded bias in training sets, and inappropriate reliance on opaque systems, are analysed with real-world examples. Ultimately, the paper calls for interdisciplinary cooperation among healthcare providers, developers, and regulators to create systems that enhance patient outcomes while remaining aligned with ethical and societal values.
References
Shortliffe EH, Buchanan BG. Rule-Based Expert Systems: The MYCIN Experiments of the Stanford Heuristic Programming Project. In: Artificial Intelligence in Medicine. Oxford: Pergamon Press; 1984. 1–30.
Dankwa-Mullan I. Health equity and ethical considerations in using artificial intelligence in public health and medicine. Prev Chronic Dis. 2024;21:240245.
Beam AL, Kohane IS. Big data and machine learning in health care. JAMA. 2018;319(13):1317–1318. https://doi.org/10.1001/jama.2017.18391
Leveson NG, Turner CS. An investigation of the Therac-25 accidents. Computer. 1993;26(7):18–34. https://web.stanford.edu/class/archive/cs/cs295/cs295.1086/papers/Therac-25.pdf
Peek N, Sujan M, Scott P. Digital health and care in pandemic times: impact of COVID-19. BMJ Health Care Inform. 2020;27(1):e100166. https://doi.org/10.1136/bmjhci-2020-100166
Senior AW, Evans R, Jumper J, Kirkpatrick J, Sifre L, Green T, et al. Improved protein structure prediction using potentials from deep learning. Nature. 2020;577(7792):706–710. https://doi.org/10.1038/s41586-020-03062-9
Ratwani RM, Sutton K, Galarraga JE. Addressing AI algorithmic bias in health care. JAMA Netw. 2024. https://doi.org/10.1001/jama.2024.1234
U.S. Food and Drug Administration. Software as a medical device (SaMD): clinical evaluation. Guidance for industry and FDA staff. Silver Spring (MD): FDA; 2017. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/software-medical-device-samd-clinical-evaluation
Smith J, Doe A. Ethical implications of AI in healthcare: balancing innovation with responsibility. J Med Ethics. 2025;47(1):15–30. https://doi.org/10.1136/jme-2024-108978
Pew Research Center. Americans’ views of artificial intelligence. Washington (DC): Pew Research Center; 2023 Nov 21. https://www.pewresearch.org/short-reads/2023/11/21/what-the-data-says-about-americans-views-of-artificial-intelligence/
Hooper K, Lunn S. A scoping review of transparency and explainability in AI ethics guidelines. J AI Ethics. 2024;3(1):123–134. https://doi.org/10.1007/s43681-022-00152-w
Ribeiro MT, Singh S, Guestrin C. "Why should I trust you?": Explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2016;1135–1144. https://doi.org/10.1145/2939672.2939785
Blackman R, Ammanath B. Building transparency into AI projects. Harv Bus Rev. 2022 Jun. https://hbr.org/2022/06/building-transparency-into-ai-projects
Greenstein S, Fang TP. (2022). Where the cloud rests: The location strategies of data centers. Harvard Business School Working Paper 21-042. Retrieved from https://www.hbs.edu/ris/Publication%20Files/21-042_092622_ce83a679-7133-4bde-8f91-dc72d00fe27a.pdf
Miller K, Zhang Y. Regulatory challenges in cloud data governance: A comparative analysis of compliance frameworks. Journal of Information Systems Policy. 2023;38(2):145-167
World Health Organization. Ethics and governance of artificial intelligence for health. Geneva: WHO; 2021. https://www.who.int/publications/i/item/9789240029200
European Commission. Proposal for a regulation laying down harmonised rules on artificial intelligence. Brussels: European Commission; 2021. https://digital-strategy.ec.europa.eu/en/library/proposal-regulation-laying-down-harmonised-rules-artificial-intelligence
Rigby MJ. Ethical dimensions of using artificial intelligence in health care. AMA J Ethics. 2019;21(2):E121–E124. https://journalofethics.ama-assn.org/article/ethical-dimensions-using-artificial-intelligence-health-care/2019-02
Cohen IG. Informed consent and medical artificial intelligence: what to tell the patient? Georgetown Law J. 2020;108(6):1425–1473. https://www.law.georgetown.edu/georgetown-law-journal/wp-content/uploads/sites/26/2020/06/Cohen_Informed-Consent-and-Medical-Artificial-Intelligence-What-to-Tell-the-Patient.pdf
Kaminski ME. The right to explanation, explained. Berkeley Technol Law J. 2019;34(1):189–218. https://doi.org/10.15779/Z38TD9N83H
Powles J, Hodson H. Google DeepMind and healthcare in an algorithmic age: a critical appraisal. J Law Inf Sci. 2017;6(2):45–64. https://doi.org/10.1017/jlis.2017.8
Ross E. How IBM Watson overpromised and underdelivered on AI health care. IEEE Spectrum. 2020 Jul. https://spectrum.ieee.org/how-ibm-watson-overpromised-and-underdelivered-on-ai-health-care
U.S. Food and Drug Administration. FDA issues comprehensive draft guidance for developers of artificial intelligence-enabled medical devices. Silver Spring (MD): FDA; 2025. https://www.fda.gov/news-events/press-announcements/fda-issues-comprehensive-draft-guidance-developers-artificial-intelligence-enabled-medical-devices
Bouderhem R, Chithaluru P. A comprehensive framework for transparent and explainable AI sensors in healthcare. Eng Proc. 2024;82(1):49. https://doi.org/10.3390/ecsa-11-20524
National Institute of Standards and Technology. Four principles of explainable artificial intelligence (NISTIR 8312). Gaithersburg (MD): NIST; 2021. https://doi.org/10.6028/NIST.IR.8312
Smith J, Doe A. Bridging the responsibility gap in AI-driven healthcare. J Med Ethics. 2024;50(2):123–135. https://doi.org/10.1136/jme-2023-108978
Clarke YD, Watson Coleman A. Algorithmic Accountability Act of 2022. 117th Congress (2021–2022). Washington (DC): U.S. Congress; 2022. https://www.congress.gov/bill/117th-congress/house-bill/6580
Voelker R, Hswen Y. Can predictive AI improve early detection of sepsis and other conditions? JAMA. 2023. https://jamanetwork.com/journals/jama/fullarticle/2811547
Cheong B. Transparency and accountability in AI systems: safeguarding wellbeing in the age of algorithmic decision-making. Front Hum Dyn. 2024;6:1421273. https://doi.org/10.3389/fhumd.2024.1421273
Daneshjou R, Smith MP, Sun MD, Rotemberg V, Zou J. Risk of bias and error from data sets used for dermatologic artificial intelligence. JAMA Dermatol. 2021;157(11):1271–1273. https://doi.org/10.1001/jamadermatol.2021.3128
Turner Lee N, Resnick P, Barton G. Algorithmic bias detection and mitigation: best practices and policies to reduce consumer harms. Washington (DC): Brookings Institution; 2019. https://www.brookings.edu/articles/algorithmic-bias-detection-and-mitigation-best-practices-and-policies-to-reduce-consumer-harms/
African Union. Digital transformation strategy for Africa (2020–2030). Addis Ababa: African Union; 2020. https://au.int/en/documents/20200518/digital-transformation-strategy-africa-2020-2030
Bibbins-Domingo K, Helman A, Dzau VJ. The imperative for diversity and inclusion in clinical trials and health research participation. JAMA. 2022;327(23):2283–2284. https://doi.org/10.1001/jama.2022.9083
Mökander J, Floridi L. Ethics-based auditing to develop trustworthy AI. Minds Mach. 2021;31:323–327. https://doi.org/10.1007/s11023-021-09557-8
Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019;366(6464):447–453. https://doi.org/10.1126/science.aax2342
Ricci Lara MA, Mosquera C, Ferrante E, Echeveste R. Towards unraveling calibration biases in medical image analysis. In: Mori K, Kybic J, Arbel T, editors. Clinical Image-Based Procedures, Fairness of AI in Medical Imaging, and Ethical and Philosophical Issues in Medical Imaging. Cham: Springer; 2023. 132–141. https://doi.org/10.1007/978-3-031-45249-9_13
National Institute of Standards and Technology. Artificial intelligence risk management framework (AI RMF 1.0). Gaithersburg (MD): NIST; 2023. https://doi.org/10.6028/NIST.AI.100-1
World Health Organization. Global Strategy on Digital Health 2020–2025. Geneva: WHO; 2021. https://apps.who.int/iris/bitstream/handle/10665/344249/9789240020924-eng.pdfhttps://apps.who.int/iris/bitstream/handle/10665/344249/9789240020924-eng.pdf
Price WN. Risks and remedies for artificial intelligence in health care. Washington (DC): Brookings Institution; 2019. https://www.brookings.edu/articles/risks-and-remedies-for-artificial-intelligence-in-health-care/
European Commission High-Level Expert Group on Artificial Intelligence. Ethics guidelines for trustworthy AI. Brussels: European Commission; 2019. https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-aihttps://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai
Alves M, Seringa J, Silvestre T, Magalhães T. Use of artificial intelligence tools in supporting decision-making in hospital management. BMC Health Serv Res. 2024;24:1282. https://doi.org/10.1186/s12913-024-11602-y
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Julian Bruce
This work is licensed under a Creative Commons Attribution 4.0 International License.
The Creative Commons Attribution 4.0 International License (CC BY 4.0) is a flexible and commonly used license that allows others to share, remix, adapt, and build upon the work even for commercial purposes, as long as they follow the requirements outlined by the license.
