AI in Healthcare — Governance for Australian Health Organisations

For hospitals, clinics, allied health practices, PHNs, and health technology teams. How to govern AI adoption across clinical, administrative, and patient-facing contexts while meeting Australian health sector obligations.

AI is entering every part of Australian healthcare — from diagnostic imaging and clinical decision support to patient triage chatbots, administrative automation, and medical record summarisation. The potential is significant, but so are the risks. Health data is the most sensitive category of personal information under Australian law, and the consequences of AI errors in healthcare are measured in patient outcomes, not just business impact.

This guide is for health sector leaders, clinical governance teams, IT directors, practice managers, and health technology specialists. For individual consumer guidance on sharing health information with AI, see our Health & AI Safety guide.

Critical principle: AI in healthcare must always support — never replace — clinical judgement. No AI system should make autonomous clinical decisions without qualified human oversight. This isn't just good practice; it's a legal and ethical requirement under Australian health regulation.

The Regulatory Landscape

Healthcare AI in Australia operates under a layered regulatory framework that is more restrictive than most other sectors:

Privacy Act 1988 — Health Information

Health information receives special protection under the Australian Privacy Principles. APP 3 restricts collection to what is reasonably necessary. APP 6 limits use and disclosure. APP 8 restricts cross-border data flows. APP 11 requires security proportionate to the sensitivity of the information. Health information is explicitly recognised as sensitive information under the Act, triggering higher compliance thresholds for every APP.

The practical implication: any AI tool that processes patient data — including clinical notes, diagnostic information, pathology results, or patient communications — must meet these higher thresholds. Consumer AI tools (ChatGPT, Gemini, Copilot) almost certainly do not, because they store data offshore, may use it for training, and provide insufficient contractual protections for health information.

My Health Records Act 2012

If your organisation accesses or contributes to the My Health Record system, the My Health Records Act imposes specific restrictions on how that data is used. AI processing of My Health Record data raises questions about purpose limitation, secondary use, and data minimisation that require careful legal analysis before deployment.

TGA — Software as a Medical Device (SaMD)

The Therapeutic Goods Administration regulates AI software that makes or supports clinical decisions as a medical device. If an AI tool is intended to diagnose, prevent, monitor, treat, or alleviate disease — or to provide clinical decision support that directly influences treatment — it may be classified as Software as a Medical Device (SaMD) and require TGA registration.

This applies regardless of whether the AI is a standalone product or a feature embedded in an existing clinical system. The TGA's classification depends on the intended purpose, not the underlying technology. Key considerations include whether the output is intended to inform or influence clinical decisions, the seriousness of the health condition involved, and the criticality of the AI's contribution to the clinical decision.

State and Territory Health Privacy

NSW (Health Records and Information Privacy Act 2002), Victoria (Health Records Act 2001), and the ACT (Health Records Privacy and Access Act 1997) impose additional requirements that overlay the federal framework. If your organisation operates across jurisdictions, compliance with the most restrictive applicable framework is the safest approach.

Professional Registration and Obligations

AHPRA-registered health practitioners have professional obligations that are not suspended because they used AI. A GP who relies on an AI diagnostic tool is still professionally accountable for the clinical decision. A pharmacist who uses AI to check interactions is still responsible for dispensing safely. Practitioners must be able to explain and justify their clinical reasoning regardless of whether AI informed it.

Categories of Healthcare AI Use

Different AI use cases carry different risk profiles and governance requirements:

Category Examples Risk Level Key Requirements
Clinical Decision Support Diagnostic imaging AI, pathology screening, drug interaction checking, clinical risk scoring CRITICAL TGA assessment, clinical validation, human oversight mandatory, professional liability coverage, incident reporting
Patient-Facing AI Triage chatbots, symptom checkers, patient communication bots, appointment scheduling AI HIGH Health information privacy, informed consent, clear limitations disclosure, clinical escalation pathways, accessibility
Clinical Documentation Medical record summarisation, discharge summary drafting, referral letter generation, clinical note transcription HIGH Data residency, training data exclusion, accuracy verification, clinician sign-off, audit trail
Operational & Administrative Rostering optimisation, billing automation, supply chain, facility management, HR MODERATE Standard AI governance, staff data protection, workforce impact assessment
Research & Population Health Cohort analysis, epidemiological modelling, clinical trial matching, health service planning HIGH Ethics approval (HREC), de-identification standards, consent requirements, research governance

Healthcare-Specific AI Governance Framework

Beyond standard AI governance, health organisations should implement these additional controls:

1. Clinical Validation Requirements

Any AI tool that informs clinical decisions must demonstrate clinical validity — meaning it has been tested on relevant patient populations and shown to produce accurate, reliable results. Ask your vendor: What datasets was the model trained on? Were Australian or comparable patient populations included? What is the known error rate? How does performance vary across different demographics, conditions, and edge cases? A model validated on US patient populations may not perform identically on Australian patients.

2. Bias and Equity Assessment

AI models in healthcare have documented biases. Dermatology AI performs worse on darker skin tones. Chest X-ray AI trained predominantly on male patients may miss findings in women. Predictive models using socioeconomic proxies can disadvantage First Nations patients and patients from lower socioeconomic backgrounds. Before deploying any clinical AI, assess whether it has been tested for performance equity across the populations you serve.

3. Data Sovereignty for Health Information

Health data leaving Australian jurisdiction creates significant compliance and security risks. At minimum, health organisations should require Australian data residency for any AI system processing identifiable patient information. Ideally, inference (the actual AI processing) should also occur onshore. This rules out most consumer AI tools and requires enterprise-grade solutions with contractual data residency guarantees. See our AI Tools Comparison for data residency details by provider.

4. Training Data Exclusion

Ensure contractual guarantees that patient data will not be used to train AI models — even in de-identified form. Health data has a particularly high re-identification risk, and de-identification standards vary. Your contract with any AI vendor should explicitly prohibit the use of your organisation's data for model training, improvement, or any secondary purpose.

5. Incident Response for AI Errors

Establish a clear process for handling AI errors that have clinical impact. This includes how to identify that an AI contributed to an adverse event, how to report it through existing clinical incident systems, how to assess whether the AI should be suspended pending investigation, how to communicate with affected patients, and how to notify regulators (TGA for SaMD, state health departments for clinical incidents).

6. Informed Consent

Patients should be informed when AI is used in their care in a way that is meaningful and proportionate. For diagnostic AI that materially influences treatment decisions, explicit consent may be appropriate. For administrative AI that summarises notes, transparency through privacy notices may suffice. The principle: patients should not be surprised to learn that AI was involved in their care.

Common Mistakes in Healthcare AI Adoption

  • Using consumer AI for clinical documentation. Staff using personal ChatGPT accounts to summarise patient notes is a data breach. Provide approved alternatives or prohibit the practice with clear policy and enforcement
  • Assuming the vendor handles compliance. AI vendors sell technology, not compliance. Your organisation remains responsible for meeting health privacy obligations regardless of what the vendor promises in marketing materials. Read the DPA, not the brochure
  • Skipping clinical validation. An AI tool that performs well in the vendor's demonstration may not perform well on your patient population. Demand evidence of clinical validation in comparable settings
  • Deploying without clinician engagement. AI tools implemented by IT without meaningful clinical input often fail in practice. Clinicians need to understand the tool's capabilities, limitations, and intended role in their workflow
  • Overlooking the secondary AI layer. Your electronic health record vendor may embed AI features that process patient data through third-party models. Use our Secondary AI Tools framework to identify the full data flow chain

Procurement Checklist for Healthcare AI

Before procuring any AI tool that will process health information:

☐ Data residency — confirm all patient data is processed and stored in Australia
☐ Training data exclusion — contractual guarantee patient data won't train models
☐ TGA classification — determine if the tool is SaMD and whether registration is required
☐ Clinical validation — evidence of testing on relevant patient populations
☐ Bias assessment — tested for equity across demographics your organisation serves
☐ Human oversight model — clear definition of AI's role relative to clinical judgement
☐ Secondary AI identification — map all third-party AI providers in the data flow chain
☐ Incident response — plan for AI errors with clinical impact
☐ Patient consent — determine informed consent requirements for the specific use case
☐ HREC approval — if the use involves research or novel application of patient data
Governance tools: Our Governance Starter Kits include AI use policy templates, vendor assessment frameworks, and due diligence checklists adaptable for healthcare settings. The Due Diligence Framework provides a structured approach to evaluating any AI tool before procurement.

Related Resources

Draft AI Safety Guidelines for Health Contexts

Copy this prompt into any AI tool. Fill in the [BRACKETS] with your details.

You are creating AI safety guidelines for [INSERT HEALTH ORGANISATION/PRACTICE]. Draft practical guidance covering: (1) what health information should never be entered into commercial AI tools, (2) de-identification requirements before using AI for clinical support, (3) TGA and APRA requirements for AI in healthcare settings, (4) patient communication about AI use in their care, (5) specific privacy obligations for health data under the Privacy Act. Include scenario-based examples for clinicians. Format as a clinical governance policy.
Want all 28 governance prompts and templates? AI Safety Governance Pack — ready to use, no assembly required.
Browse All Resources

Share this page

AI safety knowledge shouldn't be locked behind a paywall, a postgrad degree, or an expensive AI consultant.