Testing LLM-enhanced healthcare surveys with pregnant momsTesting LLM-enhanced healthcare surveys with pregnant moms

Testing LLM-enhanced healthcare surveys with pregnant momsTesting LLM-enhanced healthcare surveys with pregnant moms

Carien Moolman

To embed a Youtube video, add the URL to the properties panel.

We designed 4 surveys typically sent to pregnant moms in MomConnect via WhatsApp and enhanced these with an LLM. In July of this year we did one-on-one in-person qualitative usability testing of these surveys as well as their statically designed counterparts with 17 pregnant moms in Khayelitsha, Cape Town. Each mom sat with a single facilitator and was asked to complete a survey twice: first the static version, and then the LLM-enhanced version, while facilitators observed and made notes. Right after completion, the moms were asked which one they preferred and why they preferred that one. We accounted for ordering effects by asking half of our moms to begin with the static version, and the other half to begin with the LLM-enhanced version. The surveys we tested were: - a demographic information survey - a 5-question decision-making ability assessment - a knowledge, attitudes, and behaviour assessment regarding pregnancy - a short survey typically sent after an antenatal clinic appointment We also tested a feature where moms could ask any pregnancy health-related questions, as freeform text, and get an answer from the LLM, where the LLM was limited to approved healthcare information provided by our content team. Our findings were fascinating and point to: (1) the risks of using LLM-enhancement and how this might affect the credibility of data gathered (2) the need to keep humans in the loop on both the design side and the data moderation side of these kinds of studies (3) the immense potential of LLM-enhancement in the provision of digital healthcare information and services

About

Reunion.ai logo

Delivering independent journalism, thought-provoking insights, and trustworthy reporting to keep you informed, inspired, and engaged with the world every day.

About

Reunion.ai logo

Delivering independent journalism, thought-provoking insights, and trustworthy reporting to keep you informed, inspired, and engaged with the world every day.

About

Reunion.ai logo

Delivering independent journalism, thought-provoking insights, and trustworthy reporting to keep you informed, inspired, and engaged with the world every day.

Talks from Day 1

Talks from Day 2