Students prefer AI chatbots, until they know it is one

It’s a question Joshua Lambert, PhD, an associate professor and biostatistician in the University of Cincinnati College of Nursing, is pondering. He’s turned to a group of his students to find out their thoughts about the helpfulness and satisfaction of a custom AI education chatbot.

Lambert piloted his custom chatbot by examining how a small group of Doctor of Nursing Practice (DNP) students evaluated answers to a set of questions from three different sources: a professor, a graduate assistant and a chatbot. 

The results of the study have been published in the peer-reviewed Journal of Nursing Education. This pilot project used a randomized, blinded, within-subjects comparison study with a survey-based evaluation.

Seven doctoral students in the study submitted statistical questions related to their capstone projects and received blind responses from the professor, graduate assistant and chatbot. The students rated each response on helpfulness, satisfaction and likelihood of use on a scale of one to five with the best being five. They then guessed which response came from the chatbot.

Other researchers who are co-authors of the study from the UC College of Nursing include Robyn Stamm, DNP, associate professor of clinical nursing; Shannon White, DNP, assistant professor in the doctor of nursing practice program; and Melanie Kroger-Jarvis, DNP, associate dean for graduate clinical and leadership programs. Bailey Martin, PhD, a postdoctoral research fellow at the University of Colorado Anschutz Medical Campus, is also a co-author of  the study. Martin is also a graduate of the nursing doctoral program at UC.

Researchers in the study acknowledge that while the small sample size is appropriate for a pilot study, it is insufficient for determining adequate effectiveness. They suggest that larger studies, replicated in multiple sites with additional qualitative and quantitative data are needed to thoroughly evaluate AI chatbot tools in nursing education and advising.

“For this reason, the descriptive results should be considered an initial ‘first step’ toward understanding how such a tool may assist in student learning and consultation,” the researchers wrote in their study.

Lambert says he considered using the chatbot because students, like others, are sometimes hesitant to ask questions to another individual, particularly a professor, that might seem silly or make them appear as not so smart.

However, the chatbot won’t judge you based on your questions, he adds.

“Sometimes the topics we cover are challenging or intimidating. Educators want something that will lower the barrier so students can ask any questions they like," says Lambert — whether that’s with a person or an AI chatbot.

The study is available on Journal of Nursing Education website.

Funding for the study came from an internal grant from the University of Cincinnati College of Nursing to support conference attendance, participant reimbursement and software fees. The authors of the study report no potential conflicts of interest.