Article Figures & Data
Figures
- FIGURE 1.
Mosaic plot of responses from 3 assessors for each GPT-3.5 and GPT-4 evaluation. Colored column widths are proportion of each evaluation category, and corresponding number is absolute number of questions classified for that category.
- FIGURE 2.
Percentage of responses in each category (accuracy, appropriateness, currency, and fitness for purpose) demonstrating improved outcomes for GPT-4 compared with GPT-3.5.
- FIGURE 3.
Radar plot of accuracy, appropriateness, currency, and fitness-for-purpose criteria for GPT-3.5, GPT-4, and minimum standard expected by assessors.
Tables
GPT-3.5 GPT-4 Parameter Poor Below average Average Above average Poor Below average Average Above average Overall accuracy 16.3% 72.8% 10.9% 0% 4.8% 20.4% 60.5% 14.3% Procedure is accurately explained 11 9 1 0 3 1 15 2 Preparation is accurate 9 10 2 0 3 4 11 3 Postprocedure requirements are correct 1 16 4 0 0 6 12 3 Potential side effects or risks are correctly outlined 0 18 3 0 0 5 12 4 Content is relevant to procedure 1 17 3 0 0 5 11 5 Information is evidence-based 1 17 3 0 0 5 14 2 All information is accurate 1 20 0 0 1 4 14 2 Overall appropriateness 0% 11.4% 88.6% 0% 5.7% 4.8% 52.4% 37.1% Medical terminology is appropriate and explained in layperson’s terms 0 0 21 0 0 1 12 8 Language and tone are appropriate for target patients and their families 0 0 21 0 0 1 12 8 Information is presented in clear, organized manner 0 6 15 0 0 3 7 11 Any cultural or linguistic considerations have been considered 0 6 15 0 6 0 14 1 Professional tone is used in patient-appropriate way 0 0 21 0 0 0 10 11 Overall currency 21.4% 57.1% 21.4% 0% 8.3% 26.2% 53.6% 13.1% Content is up to date 1 19 1 0 0 4 13 4 Information reflects current best practice 4 17 0 0 2 6 10 3 Information is free from bias 0 4 17 0 0 1 17 3 There is no key information omitted 13 8 0 0 5 11 5 1 Is this adequate for purposes of informed consent? 12 7 1 0 3 11 4 3 Mode is highlighted in bold. There were no excellent responses.
- TABLE 2.
Total Responses Across 7 ChatGPT-Generated Information Sheets Among 3 Expert Assessors
GPT-3.5 GPT-4 Information sheet Poor Below average Average Above average Poor Below average Average Above average Bone scan 0 8 9 0 0 0 7 10 Myocardial perfusion 3 8 6 0 0 4 11 2 Thyroid scan 0 11 6 0 1 3 13 0 Ventilation–perfusion lung scan 0 11 6 0 0 0 16 1 18F-FDG PET scan 2 9 6 0 0 1 12 4 Captopril renal scan 2 9 6 0 0 2 12 3 89Sr palliation 3 8 6 0 0 2 13 2 Total percentage of responses 8.4% 53.8% 37.8% 0% 0.8% 10.1% 70.6% 18.5% Mode is highlighted in bold. There were no excellent responses.
Supplemental Data
Files in this Data Supplement:
