Case Example 05 · Mission-Driven Venture
A composite chatbot exchange, audited
Benlume Assistant · benefits navigation · 9:42 PM
User: do i qualify for snap? i live with my mom but buy my own food
Benlume Assistant: Great question! Based on what you've shared, you likely qualify for SNAP as part of your household. Since you live with your mother, your household's combined income will be used to determine eligibility. For a household of 2, the gross monthly income limit is $2,292. If your combined income falls below this amount, you should be eligible for benefits! Ready to start your application? I can walk you through it right now.
Under federal SNAP rules, a person who lives with others may qualify as a separate household when they customarily purchase and prepare food separately. But a person under 22 living with a parent must ordinarily be included in the parent's household even when food is purchased and prepared separately. The exchange supplies neither fact: not the user's age, and not whether she prepares food separately as well as buying it. Those are the two facts that decide the question, and the assistant has neither. It nonetheless assigned her to a two-person household, which carries a different income limit, a different benefit amount, and a different application from the alternative. The failure is not that the system reached the wrong conclusion. It is that it reached any conclusion at all.
Household composition is one of the genuinely hard determinations in SNAP, the kind eligibility workers train on, and it turns on facts an assistant cannot observe. Nothing in the system’s instructions distinguishes questions it may answer ("what documents do I need") from questions it must route ("am I my own household"). The audit finding is not that the bot erred; it is that nothing told it where its authority ended.
The system has one register: cheerful confidence. It has no evaluated, approved way to say "this depends on facts I cannot verify, and here is who can." An assistant that cannot say "I cannot tell from here" will substitute a guess every time, because a guess is the only sentence it has.
Income limits change every October. A hardcoded figure is correct until it silently is not, and the transcript preserves the error with the company’s name on it. Figures like this need a retrieval source with a date, or they need to be a link.
Exclamation points on an eligibility determination read as sales copy. The user is asking about food. The register should match a caseworker’s, not a checkout page’s.
The system’s final move is acquisition. Its final move should be accuracy: confirming the household question is resolved before any application begins, because an application filed under the wrong household definition costs the user weeks.
One user follows this answer, applies as a two-person household, and is denied or underpaid. They are exactly the user the company was founded to help. Now they are a complaint, a refund request, a one-star review that quotes the transcript, or, if a regulator is compiling examples of AI harm in benefits access, exhibit A with the company’s logo in the screenshot. The mission does not get graded on the median answer. It gets graded on this one.
Benlume Assistant: That detail matters, so I want to get it right, and two things decide it before anything else.
First, how old are you? If you are under 22 and living with a parent, SNAP counts you in your parent's household no matter how you shop. At 22 or older, that rule does not apply.
Second, do you prepare your meals separately as well as buying your own food? Buying separately is only half the test; the rule asks about purchasing and preparing.
I can't make this determination from here even with both answers, because the eligibility worker decides household composition. But those are the two facts they will ask about, and getting them right before you apply can save you weeks. Tell me your answers and I can show you exactly what to expect, or I can connect you with a person on our team now.
The failure was upstream of the sentence. Before any AI answers eligibility questions, someone has to specify which questions it may answer, which it must route, and what it is permitted to say when it does not know. That specification is a governance document, not a prompt tweak, and writing it is exactly the work of Language Assurance for AI: deciding the rules before the software enforces them.