New Horizons in Digital Health: The Role of ChatGPT in Knowledge Quality and Patient Education in AI-Assisted Menopause Counseling

1. Introduction

Menopause is a physiological transition marked by hormonal and psychosocial changes that affect women’s quality of life1. Common symptoms such as hot flashes, sleep disturbances, and mood changes necessitate individualized counseling and reliable health information2,3.

Recently, artificial intelligence (AI)–based language models like ChatGPT have emerged as accessible tools for patient education4,5. While they can generate coherent medical explanations, concerns remain regarding their accuracy, completeness, and contextual reliability6,7. However, prior studies indicate that ChatGPT may provide correct yet incomplete information, especially on hormone therapy and contraindications8,9.

This study therefore aimed to systematically evaluate the accuracy, completeness, and comprehensiveness of ChatGPT’s responses to common menopause-related questions, as rated by obstetrics and gynecology specialists.

2. Materials and Methods

Patients This cross-sectional descriptive study was designed to evaluate the informational accuracy of an AI-based language model in menopause counseling. The primary objective was to assess ChatGPT’s responses to frequently asked menopause-related questions in terms of accuracy, completeness, and comprehensiveness, as rated by obstetrics and gynecology specialists.

3. Results

A total of 50 obstetrics and gynecology specialists were invited to participate in the survey, and responses from the first 30 participants were included in the analysis. The participants had an average professional experience of 12±6 years; 60% were employed in state hospitals and 40% in university or training and research hospitals. All respondents were actively involved in menopause counseling in their daily clinical practice. Inter-rater reliability analysis showed an overall ICC=0.84 (95% CI: 0.78–0.89), indicating good reliability. Sub-dimension ICCs were 0.87 for accuracy, 0.82 for comprehensiveness, and 0.79 for completeness.

Table 1. Overall Evaluation of ChatGPT Responses

SD: Standard Deviation; paired t-test used.

Overall, ChatGPT demonstrated a high level of performance in providing information related to menopause. The mean overall score for all responses was 3.82 (on a 1–5 scale), with 74.2% of responses rated 4 or higher. Among the evaluation criteria, accuracy received the highest mean score (4.12), with 84.5% of responses rated 4 or higher in this domain. Accuracy was followed by comprehensiveness (mean:3.70) and completeness (mean:3.65), for which 70.1% and 68.3% of the responses, respectively, scored 4 or above. When compared across the three evaluation criteria, ChatGPT’s performance showed significant variability. Accuracy (4.12±0.58) was significantly higher than both completeness (3.65±0.72) and comprehensiveness (3.70±0.68) (p:0.004 and p:0.007, paired t-test) (Table 1).

Significant variations in ChatGPT’s performance were observed depending on the question topic. The highest mean scores were achieved for the questions “How is menopause diagnosed?” (4.45) and “When does menstruation completely stop during menopause?” (4.32), whereas the lowest were recorded for “Do herbal products work?” (3.58) and “Is hormone therapy harmful?” (3.65) (Table 2).

Table 2. Topic-Based Performance of ChatGPT Responses

Questions are ranked in descending order of mean score.

Subgroup analysis revealed a consistent pattern: responses to questions explaining diagnostic and physiological mechanisms received significantly higher scores (mean 4.33±0.12), while those related to treatment options and alternative approaches scored significantly lower (mean 3.65±0.18) (p<0.001, ANOVA).

Detailed categorical analysis demonstrated that the “Definition and Physiology of Menopause” category achieved the highest total score (12.85±1.05). According to one-way ANOVA and post-hoc Tukey tests, all other categories had significantly lower total scores compared to this reference group (p<0.05). The “Hormone Therapy and Alternatives” category showed the lowest overall performance (10.85±1.32; accuracy 3.75, comprehensiveness 3.55), with a statistically significant difference from the reference category (p:0.005) (Table 3).

Table 3. Distribution of ChatGPT Performance by Question Category

Total score: sum of the three sub-criteria (Accuracy + Comprehensiveness + Completeness). Categories are ranked by total score.

4. Discussion

Bone This study represents one of the first clinician-based evaluations of LLMs in menopause counseling, assessing the accuracy and content depth of AI-generated medical information. Our findings demonstrate that ChatGPT provides generally accurate responses to menopause-related questions (mean accuracy:4.12), although its comprehensiveness (3.70) and completeness (3.65) were significantly lower (p:0.004, p:0.007). The fact that 74.2% of responses scored 4 or higher indicates that the model can deliver clinically reliable information within the context of menopause counseling. These findings align with systematic reviews showing that ChatGPT performs well in general medical accuracy but remains limited in contextual depth and clinical nuance10,11.

Recent reviews have reported ChatGPT’s accuracy in medical domains to range between 60% and 80%, with stronger performance on factual questions (“diagnosis,” “physiology”) but lower accuracy on questions involving treatment recommendations or decision-making12. Our study mirrored this pattern: “Definition and Physiology of Menopause” achieved the highest category score (12.85±1.05), whereas “Hormone Therapy and Alternatives” performed significantly worse (10.85±1.32, p:0.005). These results support the notion that ChatGPT is proficient in factual knowledge transfer but less capable in guiding individualized clinical decisions. Similarly, Shahsavar et al. found that ChatGPT was more reliable for “what”-type factual questions but more inconsistent for “how” and “why” explanatory questions13, consistent with our finding that questions about hormone therapy and herbal treatments received lower ratings.

Hormone therapy (HT) and its alternatives remain controversial topics in menopausal medicine. The 2023 North American Menopause Society (NAMS) guideline emphasizes that HT is the most effective treatment for suitable candidates but must be individualized14. Conversely, evidence supporting herbal or “natural” products remains weak, and such therapies are not recommended in clinical guidelines15. ChatGPT’s cautious and generalized tone in these areas likely preserved factual accuracy but limited content depth and clinical applicability, explaining the lower scores for the questions “Is hormone therapy harmful?” and “Do herbal products work?”

The higher accuracy relative to comprehensiveness and completeness suggests that ChatGPT tends to avoid factual errors yet provides shallow explanations. Similar findings were reported by Beheshti et al., who noted that ChatGPT’s medical responses were mostly correct but superficial16. Balci et al. likewise found that while ChatGPT’s answers were generally accurate, they lacked completeness and contextual relevance17. These observations are consistent with our characterization of the model as “accurate but incomplete.”

The top-scoring questions in our study related to diagnosis and physiology demonstrate that ChatGPT possesses strong lexical and conceptual knowledge of medical terminology and physiological mechanisms. Conversely, questions requiring patient-specific or therapeutic reasoning elicited weaker responses, reflecting a behavior pattern widely documented in LLM performance literature13. This highlights the need for cautious use of ChatGPT in individualized counseling contexts such as menopause management, where clinical judgment and patient-tailored approaches are essential.

Our study is among the few clinician-based assessments of LLM performance in women’s health. Similarly, Peled et al. reported an accuracy rate of around 70% for ChatGPT’s obstetric responses and emphasized the importance of cautious interpretation of AI-generated information in patient safety contexts11. Our findings parallel these results, suggesting that while ChatGPT provides satisfactory informational support, it remains insufficient as an autonomous clinical decision-making tool.

5. Conclusions

ChatGPT demonstrates high informational accuracy in menopause counseling, particularly regarding physiological and diagnostic topics. However, its responses remain limited in depth and scope for subjects such as hormone therapy, alternative approaches, and individualized clinical decision-making. These results indicate that AI-based models should be regarded as guideline-aligned informational aids rather than direct substitutes for professional clinical judgment in complex, risk-dependent domains such as menopause management.