Evaluating the Alignment of Artificial Intelligence‐Generated Recommendations With Clinical Guidelines Focused on Soft Tissue Tumors

ABSTRACT Background The integration of artificial intelligence (AI), particularly, in oncology, has significantly shifted the paradigms of medical diagnostics and treatment planning. However, the utility of AI, specifically OpenAI's ChatGPT, in soft tissue sarcoma treatment, remains unclear. Methods We evaluated ChatGPT's alignment with the Japanese Orthopaedic Association (JOA) clinical practice guidelines on the management of soft tissue tumors 2020. Twenty‐two clinical questions (CQs) were formulated to encompass various aspects of sarcoma diagnosis, treatment, and management. ChatGPT's responses were classified into “Complete Alignment,” “Partial Alignment,” or “Nonalignment” based on the recommendation and strength of evidence. Results ChatGPT demonstrated an 86% alignment rate with the JOA guidelines. The AI provided two instances of complete alignment and 17 instances of partial alignment, indicating a strong capability to match guideline criteria for most questions. However, three discrepancies were identified in areas concerning the treatment of atypical lipomatous tumors, perioperative chemotherapy for synovial sarcoma, and treatment strategies for elderly patients with malignant soft tissue tumors. Reassessment with guideline input led to some adjustments, revealing both the potential and limitations of AI in complex sarcoma care. Conclusion Our study demonstrates that AI, specifically ChatGPT, can align with clinical guidelines for soft tissue sarcoma treatment. It also underscores the need for continuous refinement and cautious integration of AI in medical decision‐making, particularly in the context of treatment for soft tissue sarcoma.