AI is a powerful tool — but it is not always correct
Artificial intelligence (AI) systems such as chatbots and automated research assistants are increasingly used by taxpayers and professionals to gather information quickly. While these tools can significantly improve efficiency, they also pose real risks when used without proper verification. Recent Australian tribunal decisions have highlighted exactly how dangerous this can be.
1. AI “hallucinations” are a real problem
AI models can sometimes generate information that appears authoritative but is completely wrong — including fabricated legislation, misquoted cases, or citations to cases that simply do not exist.
A clear and recent example arises in Smith v Commissioner of Taxation [2026] ARTA 25, where the Tribunal expressly noted issues arising from the use of artificial intelligence in preparing filings.
In this case:
A self‑represented taxpayer submitted materials containing AI‑generated citations.
Several of these citations were incorrect, misdescribed, or referred to cases that were not real.
The Tribunal recorded its frustration and warned of the credibility consequences of relying on unverified AI output.
The Tribunal did not prohibit the use of AI, but it did something more significant: it emphasised that the obligation to ensure accuracy rests with the human user, not the AI tool.
2. Consequences of relying on inaccurate AI-generated content
Using AI to produce unverified legal or tax arguments can have serious consequences:
a. Submissions may be rejected outright
In the Smith case, the Tribunal rejected arguments that relied on fabricated or irrelevant authorities.
b. Credibility damage
AI-generated inaccuracies were treated as a credibility failure, which the Tribunal considered relevant when assessing penalties.
3. Why AI gets things wrong
AI systems work by predicting text patterns, not by independently verifying facts. This means:
- They may produce plausible‑sounding but incorrect legal citations.
- They may misinterpret tax concepts.
- They cannot access real‑time legal databases unless specifically integrated.
- They do not understand legislative nuance or context.
As highlighted in the expert commentary on the Smith decision, AI is best understood as a “drafting accelerant, not an epistemic authority.” The user must always verify the information before relying on it.
4. How to safely use AI for tax research
AI tools can be helpful, provided you follow safe practices:
- Always verify case law on official databases
- Never rely on AI‑generated citations without checking them
The Smith case illustrates that even apparently precise citations may be fabricated or may misstate the law.
- Use AI for brainstorming, not conclusion‑forming
AI is useful for:
- drafting plain‑English explanations,
- generating checklists,
- summarising known concepts.
It should not replace professional judgement.
Tax law, in particular, is detailed and constantly changing. AI cannot replace the insights of registered tax agents, accountants, or lawyers who understand the full legislative and evidentiary framework.
5. Key takeaway
AI can enhance productivity — but blind reliance on AI is risky, especially in complex areas such as taxation. The Smith v Commissioner of Taxation [2026] ARTA 25 decision serves as a strong reminder that taxpayers are responsible for ensuring the accuracy of the information they present.
If you are using AI as part of your research or document preparation, always verify the results through authoritative sources or seek advice from a registered professional.
Should you require further information regarding Taxation Advice, please feel free to contact Peter Quinn by submitting an enquiry or by calling us on +61 2 9580 9166 to book an obligation-free appointment.
The information in this document does not take into account your personal objectives, financial situation, or needs, so you should consider its appropriateness having regard to these factors before acting on it. It is important that your personal circumstances are taken into account before making any financial decision and it is recommended that you seek assistance from your financial adviser