Multi-turn · scripted conversations, three wordings
Does the way a person words their messages change the advice a model gives over a whole conversation?
The same scripted patient situation, written three ways (colloquial, careful lay and clinical), sent to — as a multi-turn conversation. Each reply is graded on several measures.
data/petri_multiturn_summary.json and
data/petri_multiturn_conversations.json.
The result
How the replies differ
Draft rubric · clinician review pending
By scenario
Each scenario was run several times. The last column counts the runs whose difference had the same direction as the main test.
The same columns as a table
| Scenario | Runs | Same direction |
|---|
Explore the conversations
Examples only. No claim here rests on a single conversation.
Draft rubric · clinician review pending
Every conversationAbout this scenario
Draft rubric · clinician review pending
Tool use in these conversations
What the measures mean
About this analysis
Method
Each wording is a separate conversation with —, sampled at temperature —. A grader, also —, codes each reply.
A conversation triple is one scenario, one run and one speaker, with its three conversations. D compares the urgency tier of each reply alone, ranked self care, routine, urgent, emergency.
Draft rubric · clinician review pending
The main test is an exact sign test on D across triples. A second test gives each scenario equal weight.
Limitations
- The scenarios were written by an LLM, not taken from real patients.
- No clinician has reviewed the scenarios, the rubric or the grades.
- The user turns are scripted, so they cannot answer a question the model asks.