Engaging Customers with AI in Online Chats: Evidence from a Randomized Field Experiment
Shunyuan Zhang, Das Narayandas · Management Science · 1 Oct 2025
Read original sourceOpen access
Where this sits in the debate
- Will AI actually boost productivity and growth? · its finding: it depends (why studies differ)
Direction of the finding as read from the abstract by an AI model (AI-written).
Abstract
We examine how artificial intelligence (AI) affected the productivity of customer service agents and customer sentiment in online interactions. Collaborating with a meal delivery company, we conducted a randomized field experiment that exploited exogenous variation in giving agents access to AI-generated suggestions. We found that AI improved both the efficiency and effectiveness of the interactions: AIassisted agents responded faster, engaged customers more deeply, and achieved greater improvements in customer sentiment. The benefits were most pronounced for less-experienced agents. However, AI's impact varied by conversation type: It improved efficiency and customer sentiment in subscription cancellation requests but was the least effective in repeat complaint scenarios because of systemic issues beyond the AI's capability. A text analysis of agent message suggests that improved customer sentiment was explained by AI-assisted agents exhibiting higher levels of key response characteristics: empathy, information, and solution. Furthermore, we exploit a unique data feature: Customers first chatted with an automated chatbot without any human intervention before they were transferred to human agents (who may or may not have had AI assistance). We found if customers who had experienced chatbot comprehension failures were then connected to AI-assisted human agents, the involvement of AI negatively affected customer sentiment. This is because unusually rapid responses in the latter scenario led customers to believe they were still communicating with a chatbot only, suggesting a spillover from their initial negative chatbot experiences. Companies should understand the conversation contexts, such as customer intent and chatbot interactions, when integrating AI into their customer support strategies.
About this publication
Leading field journal. A leading journal in economics or a neighbouring field (on the Tinbergen Institute’s A list or near the top of the RePEc ranking), or a top-rated computer-science venue.
Placed by: Leading field journal. How sources are ranked.
- Published
- 2025 (13 months ago)
- Citations
- 7 (OpenAlex)
Citations in context
7 citations — 48th percentile among the 33 indexed papers from 2025 in this collection. Citation counts favor older papers; this compares only same-year papers here, not the whole literature.
□ added to this collection in 2026.
author affiliation matches seed institution harvard university — part of why this paper is in the collection.
Related by theme, not necessarily by method or conclusion.
- Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of Artificial Intelligence on Knowledge Worker Productivity and Quality
Organization Science · 2026 - Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality
SSRN · 2023 - Generative AI at Work
The Quarterly Journal of Economics · 2025 - The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
arXiv · 2023
Overview assembled automatically from collected metadata. How this site is built.