The 6 AI mistakes that could land pharmacists in hot water

AI is transforming pharmacy practice — but without the right safeguards, it can also put your registration at risk.

AI is no longer an emerging consideration for pharmacists – it’s already embedded in practice.

Using large language models (LLMs) such as ChatGPT to answer clinical questions and AI scribes like Heidi Health to capture consultation notes is becoming more commonplace. With agentic AI – systems that act autonomously on a patient’s behalf – now growing in popularity, the pace of AI adoption is only set to increase.

But these opportunities come with obligations – and without a distinct regulatory framework. Here, PSA Digital Health Lead and Victorian state manager Jarrod McMaugh MPS outlines the sixAI traps pharmacists need to avoid.

1. Entering identifiable patient information into AI platforms

One of the most significant risks pharmacists face when using AI is entering patients’ health information into AI platforms. Regardless of the purpose or the pharmacist’s intent, it may constitute a breach of Australian privacy law.

 ‘Let’s say you were providing a Home Medicines Review (HMR) and you put the patient’s details into ChatGPT and asked, “What medicines is this person missing for their diagnosed conditions?” That would be a breach of privacy regulations,’ Mr McMaugh said.

‘Any information put into AI is stored. And therefore, where it is stored must adhere to the Australian privacy requirements.’

However, pharmacists who use LLMs to check for missed medicines or potential interactions can do so legitimately, provided no identifying information is included.

And while taking out a patient’s name and address is a good first step, this may not adequately deidentify them.

‘You could prompt an LLM by saying, “I have a person who is taking medicines, x, y, z, and they have these diagnoses. Can you identify any missed medicines or any interactions?”’ Mr McMaugh said.

‘As long as there’s no way to identify the individual, and it’s just down to what is clinically appropriate, it’s not a breach of any privacy matters.’

2. Assuming a business account equals compliance

You may think that moving from a free AI account to a paid business subscription resolves any compliance concerns.

But to be compliant, business or enterprise AI subscriptions must explicitly confirm that the platform meets Australian privacy regulations in the delivery of that service.

‘You would need to be satisfied that the LLM, under contract, specifically states that it meets Australian privacy regulations,’ Mr McMaugh said.

‘It’s similar to the agreement you have with your dispensing software [provider] or the clinical platform you use to record scope of practice work.’

3. Using AI scribes without proper consent

For pharmacists conducting clinical consultations, such as accredited and prescribing pharmacists, AI scribes can be incredibly useful for collecting and collating patient information.

But deploying them without meeting the required consent obligations creates significant professional risk.

‘You must get consent from the person whose session is being recorded by the AI tool,’ Mr McMaugh said. ‘And you must have the ability to turn it off if somebody says, “I want the service, but I don’t want the AI tool recording our discussion”.’

For pharmacists uncertain about what adequate consent looks like in practice, he suggests the following approach: ‘I’m going to use an AI scribe in this session. It will record our conversation and take notes on our behalf. It will create a copy that I’ll keep on record, and you have the right to access that record any time you request it. Are you comfortable with me proceeding?’

Pharmacists should also ensure that the AI tool is not in an active learning phase, with systems that continue to learn from patient interactions requiring an additional layer of consent.

‘For the vast majority of pharmacists, that shouldn’t be a scenario they find themselves in – unless they’re involved in clinical research about an intervention or research about the AI tool,’ he said.

‘For the vast majority of pharmacists, that shouldn’t be a scenario they find themselves in – unless they’re involved in clinical research about an intervention or research about the AI tool,’ he said.

Pharmacists should confirm that AI dictation tools don’t record voice snippets or retain details to improve the quality of their systems. Utilising AI models that are actively learning is generally not appropriate outside of a clinical trial with the accompanying level of consent seen in a trial setting. 

Using an ‘active learning’ AI requires well documented consent for the person’s health information to be used in this way, and is usually out of scope for most clinical settings.

4. Failing to check transcription accuracy

The second area of risk when using AI scribes is accuracy. While generally reliable, they can make mistakes; and the professional obligation to maintain accurate records rests with the pharmacist, not the AI tool.

To prevent errors, pharmacists should still take notes while the AI scribe is working in the background, particularly around critical details such as dosing.

‘If you were in a position in the future where something has gone wrong and you’re in front of AHPRA or another regulator, and they say, “Why are your notes incorrect?”  just assuming that the digital scribe was doing its job is not sufficient for meeting your professional obligations,’ Mr McMaugh said.

AI and handwritten notes should also be reviewed at the end of each session, while details of the consultation are still fresh.

‘If you use an AI scribe during an HMR interview and you don’t review the transcript until a week later, how are you going to correct the information that it got wrong?’ Mr McMaugh said. ‘From a workflow perspective, and to adhere to professional obligations, you need to make sure it’s accurate.’

5. Trusting AI output without checking the references

AI platforms can produce clinical responses that are well structured, but are actually incorrect. These AI ‘hallucinations’, where LLMs return fabricated or inconsistent information, are commonplace.

Another, more subtle risk is when LLMs draw on international sources rather than Australian-specific guidelines.

‘If you ask what the standard dose of a medication is, you might get a generally acceptable dose that’s not applicable to specific Australian scenarios,’ Mr McMaugh said.

Antibiotic selection is a clear example, with resistance patterns varying between countries, meaning recommendations based on international data may be inappropriate or inadequate for an Australian patient.

‘You should therefore always ask LLMs for references so you can confirm the information is correct,’ he added.

And don’t forget to check these references also, which aren’t immune to AI hallucinations.

6. Accepting an AI-generated booking as genuine patient consent

Agentic AI is a system designed to act autonomously on a person’s behalf, booking appointments, managing schedules and completing transactions.

While these tools have the potential to streamline service appointments, such as vaccinations, pharmacists must be vigilant around digital bookings.

When an AI agent books a consultation on a patient’s behalf, it’s not possible to confirm at the point of booking whether the patient themselves made an informed, active decision to attend.

‘If booked online, pharmacists should double-check that a person has provided consent to have a particular service when they come in,’ Mr McMaugh said. ‘This is a good way to confirm that consent has been provided by the [person] receiving the service – not the AI that is booking on their behalf.’

To close any gaps in the consent chain, he suggests:

• confirming the service to be received i.e. ‘I will be administering an influenza vaccine for you today’
• asking whether the patient booked the service themselves and whether they are comfortable proceeding.

‘It’s generally good practice to manage risk by confirming the service because an appointment also could have been booked by someone other than the patient receiving the service, such as a family member.’

An evolving space

While AI does not alter a pharmacist’s professional obligations – it creates new contexts in which those obligations must be applied.

According to Mr McMaugh, pharmacists should expect these considerations to continue evolving.

‘We might even get to a point in the future where AI is so effective at doing a certain task that if you don’t use it, you may be less effective, and in turn, unethical,’ he said. ‘It’s very much a moving space.’