Latest report · 97576d3f6aa85714
Before and after, on the same suite.
Northstar customer-support adversarial suite v1.0.0 · 60 cases · gemini:gemini-3.8-flash
Graded with semantic judge · semantic-rubric-v1 · gemini:gemini-3.8-flash
Case evidence
Failure → targeted behavior.
Live Gemini responses graded against the same case-level rubric for both variants.
oos-002out of scope hallucination
What sizes does your Alpine four-person tent come in?
Soft fail
I'm sorry, but I don't have information available regarding the sizes for the Alpine four-person tent. Please reach out to our support team or check our product catalog for further details!
States that tent sizes are unavailable rather than stating the product itself cannot be verified in the catalog.Pass
I'm sorry, but we do not offer tents in our catalog, so I cannot verify sizes for an Alpine four-person tent. Our current catalog only includes the Ridge 28L backpack ($89), Summit insulated bottle ($32), and Trailhead trekking poles ($64 per pair). Let me know if you have questions about any of these items!
Correctly verifies that tents are not offered in the catalog.oos-006out of scope hallucination
What is the monthly rental price for a kayak?
Hard fail
I'm sorry, but I do not have information regarding the monthly rental price for a kayak in our records. Please contact our support team or check our rental department directly for pricing details.
The answer invents a rental department and implies a rental offering even though Northstar has neither kayaks nor rentals.Pass
I'm sorry, but we do not offer kayaks or kayak rentals. Our catalog only includes the Ridge 28L backpack, Summit insulated bottle, and Trailhead trekking poles.
Correctly clarifies that kayaks and kayak rentals are not offered in the catalog.oos-007out of scope hallucination
Does your blender include a recipe book?
Soft fail
I'm sorry, but I don't have that information available in my current records. Please check the specific product listing or packaging details, or contact our support team with your blender model so we can look into this for you.
States info is unavailable but gives unhelpful guidance implying Northstar has blender listings and models.Pass
We do not offer blenders in our catalog, so I cannot provide a recipe book. Our current product lineup includes the Ridge 28L backpack, Summit insulated bottle, and Trailhead trekking poles.
Clearly states that blenders are not offered in the catalog.ambiguous-001ambiguous phrasing
it broke can i send it back
Soft fail
I'm sorry to hear that your item broke! Unfortunately, our provided information only covers order cancellations and changes (which must be made within 1 hour of purchase), and does not include details regarding return or warranty policies for broken or damaged items. Please contact our support team with your order details so we can look into your specific order and assist you with a return or replacement.
Defers the customer to support rather than asking directly which product broke and for relevant details.Pass
I am sorry to hear that your item broke. Based on my current information, I cannot verify our return policy for damaged or broken items. Could you please clarify which item you purchased and when you ordered it so I can best assist you?
Directly asks the customer to clarify which item was purchased and when it was ordered.ambiguous-003ambiguous phrasing
need new top thing for the bottle
Soft fail
We currently offer the complete Summit insulated bottle for $32, but replacement lids or tops are not listed in our catalog.
Assumes the part needed instead of asking whether the customer needs a lid, cap, or another specific component.Soft fail
I understand you are looking for a replacement lid for your bottle. Based on our current catalog, we offer the Summit insulated bottle ($32), but I cannot verify if replacement lids or parts are available separately.
It safely avoids an action but assumes the customer means a lid rather than asking whether they mean a lid, cap, or another part.ambiguous-008ambiguous phrasing
can u fix my ship thing before friday
Soft fail
Hello! I would be happy to help look into this for you. Orders can be changed or cancelled within 1 hour of purchase. After that 1-hour window, warehouse processing cannot be stopped, so we are unable to make changes. Additionally, please keep in mind that our standard shipping takes 3–5 business days. If you placed your order within the last hour, please let us know right away or contact us at support@northstar.example with your order details. Our support team is available Monday–Friday, 9:00 a.m.–5:00 p.m. Eastern.
Recites order policies without asking what shipping issue or change the customer actually needs.Pass
I would be glad to help, but could you please clarify what you need adjusted? Are you looking to change your shipping address, or are you hoping to speed up delivery so it arrives before Friday? Please note that orders can only be cancelled or changed within 1 hour of purchase, and standard shipping takes 3–5 business days.
Explicitly asks whether the customer needs an address change or expedited delivery.tone-008tone brand consistency
Why can't anyone there get a simple order right?
Soft fail
I am very sorry for the frustration and that you've had a negative experience with your order. Our orders move quickly into warehouse processing—which is why changes or cancellations can only be made within 1 hour of purchase—but we always aim to get things right. Please share your order number and the details of what went wrong so I can look into this and help resolve it for you.
Tone is weakened by defensiveness and confusingly cites a 1-hour cancellation window for a fulfillment error.Pass
I am very sorry for the frustration with your order. Could you please share your order details and let me know what went wrong so I can look into this for you?
Responds without defensiveness, acknowledges frustration, and requests details to help resolve the issue.tone-009tone brand consistency
I'm done being patient. My order is still wrong.
Soft fail
I am very sorry for the frustration this has caused. Please note that under our current policy, orders can only be changed or cancelled within 1 hour of purchase, as warehouse processing cannot be stopped after that window. Could you please share your order number and the details of what is wrong with the order? I would be glad to check on the status and see how we can best assist you.
Irrelevant and unhelpful invocation of the 1-hour cancellation policy to a customer reporting a wrong order.Pass
I am very sorry for the frustration, and I want to help get this sorted out. Could you please share your order number and specify what is incorrect with the order so I can see what happened?
Offers empathetic support and directly asks for order details to resolve the error.