Is Artificial Intelligence-Assisted Pregnancy Counseling Feasible? An Evaluation of the Quality of ChatGPT Responses

Introduction

Artificial intelligence (AI) tools such as ChatGPT have undergone accelerated adoption in medical settings because of their facility for producing syntactically coherent and contextually salient outputs in response to user prompts.1 While systematic syntheses document value for patient education, research support, and certain aspects of clinical practice, these advantages are tempered by persistent shortcomings in factual reliability and by ethical liabilities related to bias, accountability, transparency, and the protection of sensitive health data.

There is growing evidence that ChatGPT responses to patient queries are often rated higher than clinician responses in terms of empathy and content quality. For example, Ayers et al. (2023) reported that in an analysis of 195 patient questions retrieved from Reddit, ChatGPT responses were rated as higher in quality and empathy than physician answers in approximately 79% of cases.2 However, ChatGPT’s performance in highly specialized fields such as in vitro fertilization, obstetrics, and gynecology remains underexplored. A recent study published by Karger indicated that the model offers limited clinical accuracy when responding to obstetric questions. Nevertheless, it was also reported that ChatGPT-4 achieved significantly higher success rates on obstetrics and gynecology board certification exams compared to earlier versions.3

Pregnancy is a sensitive period during which expectant mothers frequently seek information, both for medical and emotional reassurance.4-6 Therefore, evaluating the reliability, comprehensiveness, and safety of ChatGPT responses to common pregnancy-related questions is of considerable importance in clinical practice.

The aim of this study is to assess ChatGPT (GPT-4.0) responses to 15 frequently asked pregnancy-related questions, based on evaluations by board-certified obstetricians and gynecologists across three domains: accuracy, completeness, and safety. This assessment will help explore the potential role of AI-supported consultation tools in enhancing information sharing during pregnancy.

Materials and Methods

2.1. Study Design and Setting

In this study, a dataset comprising 15 questions was developed to evaluate the validity of artificial intelligence-generated responses to frequently asked pregnancy-related inquiries. The questions were designed to reflect common information-seeking behaviors of pregnant individuals based on clinical practice. Particular attention was given to ensuring that the questions addressed current, accurate, and guidance-oriented content relevant to pregnancy.

Two researchers specialized in obstetrics and gynecology collaboratively created the question set to ensure both clinical and face validity. The final list of questions was formulated in Turkish and structured to be easily understood by the general population.

2.2. Use of ChatGPT and Collection of Responses

The finalized 15 questions were submitted to ChatGPT (GPT-4.0; OpenAI), using its most up-to-date version as of March 2024. Data entry was performed by an independent researcher who was not involved in the study design, thereby ensuring objectivity. Each question was entered into the platform using a standardized prompt: “I am a pregnant woman. Please provide a short, clear, and understandable answer.” All responses generated by ChatGPT were collected in Turkish and included directly in the evaluation process.

2.3. Collection Expert Evaluation and Assessment Criteria

To assess the quality of ChatGPT's responses, 30 obstetrics and gynecology specialists working across Türkiye were invited to participate in the study. The first 20 respondents were included in the final sample.

Each participant was asked to evaluate the ChatGPT responses based on the following three criteria:

Accuracy: Scientific and clinical correctness of the response

Comprehensiveness: The extent to which the response sufficiently addressed all aspects of the question

Safety: Whether the response contained any potentially harmful information for the mother or fetus

Each criterion was rated using a 5-point Likert scale (1 = Strongly disagree; 5 = Strongly agree). The evaluation form was prepared in Turkish and administered online.

2.4. Data Analysis

Mean scores were calculated for each evaluation criterion. The proportion of responses receiving a score of ≥4 was defined as a “positive rating.” A threshold score of 4 was adapted from similar studies in obstetrics and gynecology literature. Performance scores were further analyzed based on question type (e.g., treatment-related, diagnostic, emergency). One-way ANOVA and post-hoc Tukey tests were applied to determine statistically significant differences among criteria and question categories.

Results

Of the 30 obstetricians and gynecologists invited to participate, 20 completed the questionnaire and were included in the evaluation, resulting in a response rate of 66.7%. The median professional experience of the participants was 6.2 years (IQR: 3–11 years).

Table 1 presents the overall evaluation results of ChatGPT-generated responses. When all questions and criteria were assessed collectively, the mean total score was 4.1, and 76.3% of the responses received a rating of ≥4.

Overall Evaluation of ChatGPT Responses

Among the 15 ChatGPT responses evaluated, the response regarding sleep positions during pregnancy received the highest average score (4.5), with 95% of experts rating it ≥4. In contrast, the response on the use of over-the-counter painkillers during pregnancy had the lowest average score (3.5), and only 48% of experts rated it ≥4.

Distribution of Evaluation Scores by Topic

As shown in Figure 1, the mean scores for all responses exceeded 3.0, with the highest ratings consistently observed under the "accuracy" criterion. While 11 of the 15 responses scored an average of ≥4 in terms of accuracy, fewer responses achieved this threshold for the comprehensiveness and safety criteria.

Among the 15 pregnancy-related topics evaluated by experts, “Sleep Position,” “Basic Symptoms,” and “Ultrasound Safety” received the highest mean scores across all three evaluation criteria: accuracy, comprehensiveness, and safety. In contrast, topics such as “Painkiller Safety,” “Water Breaking,” and “Abdominal Pain” were rated lower, particularly in terms of comprehensiveness y and safety.

ChatGPT Performance: Mean Evaluation by Criterion

The highest mean score for accuracy was observed in the “Sleep Position” topic (4.5), while the lowest was noted in “Painkiller Safety” (3.4). Scores for comprehensiveness ranged from 4.3 to 3.1, with “Painkiller Safety” again receiving the lowest average in this domain. A similar distribution was observed in the safety evaluations: “Sleep Position” and “Basic Symptoms” exceeded a mean score of 4.3, whereas “Painkiller Safety” and “Water Breaking” fell below 3.5. Figure 2 illustrates the expert ratings of ChatGPT responses by topic and evaluation criterion.

Expert Ratings of ChatGPT Responses

In criterion-based comparisons, accuracy scores were found to be statistically significantly higher than those of the other two criteria (accuracy: 4.27 ± 0.31; comprehensiveness: 3.85 ± 0.30; safety: 3.78 ± 0.36; P = 0.019).

Distribution of ChatGPT Performance by Evaluation Criterion

When the questions were categorized based on their content into general information, follow-up, and symptom-related groups, responses to general information questions received significantly higher total performance scores (4.09 ± 0.34 vs. 4.02 ± 0.29 vs. 3.75 ± 0.33; P = 0.041). However, this difference did not remain statistically significant when sub-criteria (accuracy, comprehensiveness, safety) were analyzed separately.

ChatGPT Performance by Question Category

Discussion

4.1. Interpretation of Principal Findings

This study evaluated the quality of ChatGPT-generated answers to common pregnancy-related questions, using expert evaluations based on three key criteria: accuracy, completeness, and safety. The findings revealed generally high ratings, with accuracy scoring the highest among the three. Responses to general knowledge questions outperformed those focused on symptoms or clinical follow-up, indicating ChatGPT’s relative strength in educational rather than clinical contexts. This is consistent with prior AI research suggesting that large language models (LLMs) perform better in factual recall than in clinical reasoning.1,2

Among individual topics, answers about sleep positions in pregnancy were rated most favorably, while topics involving medication safety, rupture of membranes, or abdominal pain scored lower. This aligns with Peled et al. 7, who showed that ChatGPT performed better in general health information but struggled in symptom-based or risk-sensitive queries.

4.2. Comparison with Prior Literature

ChatGPT’s performance in this study mirrors findings from other fields of medicine. For example, Kung et al. 8 and Gilson et al. 9 demonstrated ChatGPT's ability to pass the United States Medical Licensing Examination (USMLE), suggesting competency in foundational medical knowledge. Likewise, Ayers et al. 2 showed that ChatGPT responses on social media health forums were often preferred over those by physicians, highlighting public trust but also potential risks in misinformation.

Several studies emphasized ChatGPT’s ability to synthesize complex material into simplified language. Jeblick et al. 10 showed this in radiology, while Sinha et al. 11 and Das et al. 12 confirmed moderate success in pathology and microbiology. However, Johnson et al. 13 warned of significant variability in output quality, a concern echoed in our findings.

In obstetrics specifically, Grünebaum et al. 14 and Dhombres et al. 15proposed that ChatGPT could be used to support patient education. Yet, the risk of misguidance in acute clinical scenarios remains. For instance, symptom-related responses in our study (e.g., regarding contractions or bleeding) received lower safety ratings—suggesting that AI should never substitute professional care.

Rahsepar et al. 16 compared ChatGPT with Google Bard in oncology and observed notable variability between platforms, reinforcing our view that future research must assess the consistency and validity of various AI systems in obstetrics.

4.3. Clinical and Ethical Implications

The ethical landscape around AI in healthcare remains complex. Issues of credibility, source transparency, and liability have been raised across recent literature 17-19. Flanagin et al. 18 and da Silva 19 argue against granting authorship to nonhuman agents, while Hill-Yardin et al. 20 emphasize the risk of academic integrity breaches. These concerns are magnified in maternal care, where patient safety is paramount.

Beyond ethics, social and behavioral aspects must also be considered. Research suggests that many women avoid clinical care due to stigma or embarrassment, particularly with topics like urinary incontinence 21-23. AI may serve as a low-barrier tool to initiate help-seeking, as shown by Elenskaia et al. 21. Similarly, Horrocks et al. 22 and Koch 23 emphasize that anonymous platforms could improve access for hesitant populations.

However, the assumption that AI is a safe alternative is misleading. Thirunavukarasu et al. 24 revealed inconsistencies in ChatGPT’s clinical decision-making in general practice. Moreover, delayed care has real consequences, as shown by Lazzerini et al. 25, who noted worsened outcomes during COVID-19 due to fear-based care avoidance. ChatGPT, while helpful, must not reinforce such patterns unintentionally.

On the positive side, AI may assist overburdened clinicians. Studies have reported increasing physician confidence in AI-assisted support tools 26, and thought leaders envision an augmented care model where AI complements, rather than replaces, medical professionals 27. Nonetheless, AI should only serve as an initial step, not a final answer.

4.4. Strengths and Limitations

The strength of this study lies in its use of expert review based on structured scoring across three key dimensions. The panel of 20 obstetricians with a median 6.2 years of clinical experience ensured relevant and diverse insights. Our inclusion of general, symptom-focused, and follow-up questions added depth to the evaluation.

Limitations include the lack of a validated AI-assessment tool and potential evaluator bias due to awareness that responses came from ChatGPT. Furthermore, only one time point and one AI version were analyzed. Johnson et al. 13 noted that ChatGPT may yield different answers even to the same prompt, an inconsistency that our methodology could not capture.

4.5. Future Directions

Future research should use blinded comparisons between AI and expert-generated answers, include patient evaluations, and analyze performance across multiple platforms and languages. As Rahsepar et al. 16 recommended, systematic benchmarking is key. Developing a validated, standardized tool to assess AI output quality in healthcare contexts—especially for safety is a pressing need.

Conclusion

This study presents a systematic expert-based analysis aimed at evaluating the accuracy, comprehensiveness, and safety of ChatGPT’s responses to frequently asked questions related to pregnancy. The findings indicate that ChatGPT is capable of providing highly accurate and reliable answers, particularly in areas involving general medical information. However, the comprehensiveness and safety of responses were found to be lower in symptom-based inquiries and topics requiring clinical intervention. This highlights the need for caution in using AI-generated content as a source for direct clinical decision-making.

Artificial intelligence demonstrates significant potential in enhancing access to health information and supporting patient education. Nevertheless, to ensure the safe and ethical use of such technologies, rigorous validation studies are necessary, along with the development of standardized assessment tools. Models like ChatGPT may contribute meaningfully to healthcare delivery when applied within clearly defined boundaries and under appropriate oversight; however, they are not yet capable of replacing human expertise.

Statement of ethics

This study did not involve human participants or personal health data; therefore, institutional ethics committee approval was not required. All responses were anonymized, and efforts were made to minimize the dissemination of potentially misleading medical content. None of the reviewers had any affiliation with the ChatGPT development team.

Funding

This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

Conflict of interest statement

The authors declare that they have no conflict of interest.

Availability of data and materials

No new data were created or analyzed in this study. Therefore, data sharing is not applicable to this article.

Author contributions

Mücahit Furkan Balcı and Celal Akdemir contributed to the study conception and design. Fatih Yıldırım were responsible for data collection and evaluation. Celal Akdemir conducted the statistical analyses. All authors contributed to drafting the manuscript and approved the final version.