AI in Healthcare — Governance for Australian Health Organisations
For hospitals, clinics, allied health practices, PHNs, and health technology teams. How to govern AI adoption across clinical, administrative, and patient-facing contexts while meeting Australian health sector obligations.
AI is entering every part of Australian healthcare — from diagnostic imaging and clinical decision support to patient triage chatbots, administrative automation, and medical record summarisation. The potential is significant, but so are the risks. Health data is the most sensitive category of personal information under Australian law, and the consequences of AI errors in healthcare are measured in patient outcomes, not just business impact.
This guide is for health sector leaders, clinical governance teams, IT directors, practice managers, and health technology specialists. For individual consumer guidance on sharing health information with AI, see our Health & AI Safety guide.
The Regulatory Landscape
Healthcare AI in Australia operates under a layered regulatory framework that is more restrictive than most other sectors:
Privacy Act 1988 — Health Information
Health information receives special protection under the Australian Privacy Principles. APP 3 restricts collection to what is reasonably necessary. APP 6 limits use and disclosure. APP 8 restricts cross-border data flows. APP 11 requires security proportionate to the sensitivity of the information. Health information is explicitly recognised as sensitive information under the Act, triggering higher compliance thresholds for every APP.
The practical implication: any AI tool that processes patient data — including clinical notes, diagnostic information, pathology results, or patient communications — must meet these higher thresholds. Consumer AI tools (ChatGPT, Gemini, Copilot) almost certainly do not, because they store data offshore, may use it for training, and provide insufficient contractual protections for health information.
My Health Records Act 2012
If your organisation accesses or contributes to the My Health Record system, the My Health Records Act imposes specific restrictions on how that data is used. AI processing of My Health Record data raises questions about purpose limitation, secondary use, and data minimisation that require careful legal analysis before deployment.
TGA — Software as a Medical Device (SaMD)
The Therapeutic Goods Administration regulates AI software that makes or supports clinical decisions as a medical device. If an AI tool is intended to diagnose, prevent, monitor, treat, or alleviate disease — or to provide clinical decision support that directly influences treatment — it may be classified as Software as a Medical Device (SaMD) and require TGA registration.
This applies regardless of whether the AI is a standalone product or a feature embedded in an existing clinical system. The TGA's classification depends on the intended purpose, not the underlying technology. Key considerations include whether the output is intended to inform or influence clinical decisions, the seriousness of the health condition involved, and the criticality of the AI's contribution to the clinical decision.
State and Territory Health Privacy
NSW (Health Records and Information Privacy Act 2002), Victoria (Health Records Act 2001), and the ACT (Health Records Privacy and Access Act 1997) impose additional requirements that overlay the federal framework. If your organisation operates across jurisdictions, compliance with the most restrictive applicable framework is the safest approach.
Professional Registration and Obligations
AHPRA-registered health practitioners have professional obligations that are not suspended because they used AI. A GP who relies on an AI diagnostic tool is still professionally accountable for the clinical decision. A pharmacist who uses AI to check interactions is still responsible for dispensing safely. Practitioners must be able to explain and justify their clinical reasoning regardless of whether AI informed it.
Categories of Healthcare AI Use
Different AI use cases carry different risk profiles and governance requirements:
Healthcare-Specific AI Governance Framework
Beyond standard AI governance, health organisations should implement these additional controls:
1. Clinical Validation Requirements
Any AI tool that informs clinical decisions must demonstrate clinical validity — meaning it has been tested on relevant patient populations and shown to produce accurate, reliable results. Ask your vendor: What datasets was the model trained on? Were Australian or comparable patient populations included? What is the known error rate? How does performance vary across different demographics, conditions, and edge cases? A model validated on US patient populations may not perform identically on Australian patients.
2. Bias and Equity Assessment
AI models in healthcare have documented biases. Dermatology AI performs worse on darker skin tones. Chest X-ray AI trained predominantly on male patients may miss findings in women. Predictive models using socioeconomic proxies can disadvantage First Nations patients and patients from lower socioeconomic backgrounds. Before deploying any clinical AI, assess whether it has been tested for performance equity across the populations you serve.
3. Data Sovereignty for Health Information
Health data leaving Australian jurisdiction creates significant compliance and security risks. At minimum, health organisations should require Australian data residency for any AI system processing identifiable patient information. Ideally, inference (the actual AI processing) should also occur onshore. This rules out most consumer AI tools and requires enterprise-grade solutions with contractual data residency guarantees. See our AI Tools Comparison for data residency details by provider.
4. Training Data Exclusion
Ensure contractual guarantees that patient data will not be used to train AI models — even in de-identified form. Health data has a particularly high re-identification risk, and de-identification standards vary. Your contract with any AI vendor should explicitly prohibit the use of your organisation's data for model training, improvement, or any secondary purpose.
5. Incident Response for AI Errors
Establish a clear process for handling AI errors that have clinical impact. This includes how to identify that an AI contributed to an adverse event, how to report it through existing clinical incident systems, how to assess whether the AI should be suspended pending investigation, how to communicate with affected patients, and how to notify regulators (TGA for SaMD, state health departments for clinical incidents).
6. Informed Consent
Patients should be informed when AI is used in their care in a way that is meaningful and proportionate. For diagnostic AI that materially influences treatment decisions, explicit consent may be appropriate. For administrative AI that summarises notes, transparency through privacy notices may suffice. The principle: patients should not be surprised to learn that AI was involved in their care.
Common Mistakes in Healthcare AI Adoption
- Using consumer AI for clinical documentation. Staff using personal ChatGPT accounts to summarise patient notes is a data breach. Provide approved alternatives or prohibit the practice with clear policy and enforcement
- Assuming the vendor handles compliance. AI vendors sell technology, not compliance. Your organisation remains responsible for meeting health privacy obligations regardless of what the vendor promises in marketing materials. Read the DPA, not the brochure
- Skipping clinical validation. An AI tool that performs well in the vendor's demonstration may not perform well on your patient population. Demand evidence of clinical validation in comparable settings
- Deploying without clinician engagement. AI tools implemented by IT without meaningful clinical input often fail in practice. Clinicians need to understand the tool's capabilities, limitations, and intended role in their workflow
- Overlooking the secondary AI layer. Your electronic health record vendor may embed AI features that process patient data through third-party models. Use our Secondary AI Tools framework to identify the full data flow chain
Procurement Checklist for Healthcare AI
Before procuring any AI tool that will process health information:
Related Resources
Draft AI Safety Guidelines for Health Contexts
Copy this prompt into any AI tool. Fill in the [BRACKETS] with your details.
You are creating AI safety guidelines for [INSERT HEALTH ORGANISATION/PRACTICE]. Draft practical guidance covering: (1) what health information should never be entered into commercial AI tools, (2) de-identification requirements before using AI for clinical support, (3) TGA and APRA requirements for AI in healthcare settings, (4) patient communication about AI use in their care, (5) specific privacy obligations for health data under the Privacy Act. Include scenario-based examples for clinicians. Format as a clinical governance policy.