Experiment

Rules-First vs LLM-First Intent Understanding in a Conversational Booking Flow

Measure what the rules-first design of the WhatsApp booking flow gains or loses compared with sending every free-text reply to an LLM.

DESIGNED · NOT RUN YET

Hypothesis

Rules first, with the LLM only as a fallback, will match LLM-first on understanding while being faster, cheaper and never choosing an option the bot didn't offer. LLM-first will understand more unusual phrasings but cost more per conversation.

Problem

Patients type replies like "day after tomorrow evening" or "I need to move my visit". The platform uses buttons and parsing rules first and asks an LLM only when they fail. We want evidence for that design, not just intuition.

Setup

Arms: (A) rules only; (B) rules first, LLM fallback (the current design); (C) LLM first for every free-text reply. In every arm the LLM sees only the message text, never names or phone numbers, and its answer is accepted only if it maps to an option the bot offered. Metrics: replies understood on the first try, re-asks per booking, handovers to staff, wrong-option rate, latency per reply and LLM calls and cost per completed booking.

DIAGRAM

The three arms

flowchart TD
  M([Patient reply]) --> A{Arm}
  A -- A: rules only --> R1[Buttons and parsing]
  A -- B: rules first --> R2[Buttons and parsing]
  R2 -- not understood --> L2[LLM reads text only]
  A -- C: LLM first --> L3[LLM reads text only]
  R1 --> V{Matches an offered option?}
  L2 --> V
  L3 --> V
  R2 -- understood --> V
  V -- yes --> OK([Next step])
  V -- no --> RE([Re-ask or hand to staff])

The same patient reply goes through rules only, rules with an LLM fallback, or an LLM first; every arm's answer must match an option the bot offered.

Input

A labelled set of realistic patient replies for each step (day, time slot, menu choice), including typos, mixed languages and ambiguous phrasing, replayed through the conversation test harness.

Output

Per-arm metrics and a list of phrasings each arm failed on.

Decision

Status: designed, not run. Arm B is the shipped design; there are no measurements yet.

Next Step

Build the labelled reply set, replay it through the harness for each arm, and publish the results here.

Use Cases

▸Deciding when an LLM should handle intent in a structured bot
▸Keeping conversational automations cheap and predictable