Results Across 7 ChatGPT Generated Information Sheets Among 3 Expert Assessors
| GPT-3.5 | GPT-4 | |||||||
|---|---|---|---|---|---|---|---|---|
| Parameter | Poor | Below average | Average | Above average | Poor | Below average | Average | Above average |
| Overall accuracy | 16.3% | 72.8% | 10.9% | 0% | 4.8% | 20.4% | 60.5% | 14.3% |
| Procedure is accurately explained | 11 | 9 | 1 | 0 | 3 | 1 | 15 | 2 |
| Preparation is accurate | 9 | 10 | 2 | 0 | 3 | 4 | 11 | 3 |
| Postprocedure requirements are correct | 1 | 16 | 4 | 0 | 0 | 6 | 12 | 3 |
| Potential side effects or risks are correctly outlined | 0 | 18 | 3 | 0 | 0 | 5 | 12 | 4 |
| Content is relevant to procedure | 1 | 17 | 3 | 0 | 0 | 5 | 11 | 5 |
| Information is evidence-based | 1 | 17 | 3 | 0 | 0 | 5 | 14 | 2 |
| All information is accurate | 1 | 20 | 0 | 0 | 1 | 4 | 14 | 2 |
| Overall appropriateness | 0% | 11.4% | 88.6% | 0% | 5.7% | 4.8% | 52.4% | 37.1% |
| Medical terminology is appropriate and explained in layperson’s terms | 0 | 0 | 21 | 0 | 0 | 1 | 12 | 8 |
| Language and tone are appropriate for target patients and their families | 0 | 0 | 21 | 0 | 0 | 1 | 12 | 8 |
| Information is presented in clear, organized manner | 0 | 6 | 15 | 0 | 0 | 3 | 7 | 11 |
| Any cultural or linguistic considerations have been considered | 0 | 6 | 15 | 0 | 6 | 0 | 14 | 1 |
| Professional tone is used in patient-appropriate way | 0 | 0 | 21 | 0 | 0 | 0 | 10 | 11 |
| Overall currency | 21.4% | 57.1% | 21.4% | 0% | 8.3% | 26.2% | 53.6% | 13.1% |
| Content is up to date | 1 | 19 | 1 | 0 | 0 | 4 | 13 | 4 |
| Information reflects current best practice | 4 | 17 | 0 | 0 | 2 | 6 | 10 | 3 |
| Information is free from bias | 0 | 4 | 17 | 0 | 0 | 1 | 17 | 3 |
| There is no key information omitted | 13 | 8 | 0 | 0 | 5 | 11 | 5 | 1 |
| Is this adequate for purposes of informed consent? | 12 | 7 | 1 | 0 | 3 | 11 | 4 | 3 |
Mode is highlighted in bold. There were no excellent responses.