This is a comparison of one specific job: drafting a customer support reply from a store's policy doc, the way we described in how small Shopify teams automate support with AI. It's based on running the same handful of real support scenarios through both tools side by side, not a scored benchmark, and the gap between them is smaller than either vendor's marketing suggests.
The test
We fed both tools the same short policy doc (shipping windows, return rules, size chart) and the same five realistic customer messages (a late order, a return request outside the window, a sizing question, an angry customer, and a question the policy doc doesn't cover) and compared the drafts.
Where ChatGPT felt stronger
Drafts came back slightly faster and with a more casual, brand-voice-friendly tone by default, closer to what a lot of DTC stores actually want their support voice to sound like without extra prompting. It also handled the angry-customer message with a warmer, more conversational opening.
Where Claude felt stronger
On the 'not covered by policy' case, Claude was more consistent about clearly flagging that it didn't know and a human should check, rather than drifting toward a plausible-sounding guess. Its replies also ran slightly more literal and careful: good for a store whose brand voice is already fairly formal, less of an advantage for one that wants a chattier tone.
What didn't really differ
Both tools handled the late-order and return-window scenarios about equally well once given the same policy doc; neither invented a policy that wasn't in the source doc in this particular test. Both are usable for this job; this comparison is about tone and edge-case handling, not about one being unusable.
Our take
If your brand voice is casual and you want less prompting to get there, ChatGPT's default tone is a slightly better starting point. If you'd rather the tool err toward 'I don't know, ask a human' on edge cases, Claude leaned that way more consistently in our runs. Either way, the draft-then-approve step matters more than which tool you pick. See the Shopify support playbook for the full workflow.
A support reply prompt template you can use with either tool
Most of the difference between a usable draft and a risky one comes from the instructions, not the model. This template builds in the "answer only from the doc" rule from the Shopify support playbook and gives the model a clear way out when it doesn't know:
You draft customer support replies for [STORE NAME].
Answer ONLY from the policy doc below. If the answer isn't in the doc, or the customer's situation doesn't clearly match a rule in it, don't guess. Instead, write "NEEDS HUMAN:" followed by one line saying what needs checking, and stop.
Tone: [e.g. friendly and casual / polite and fairly formal]
Sign off as: [NAME OR TEAM NAME]
Never promise a refund, replacement, discount or delivery date that the policy doc doesn't state.
Don't mention the policy doc itself; just answer the customer.
Keep the reply to [NUMBER] short paragraphs or fewer.
Policy doc:
[PASTE SHIPPING WINDOWS, RETURN RULES, SIZE CHART AND ANY OTHER POLICIES]
Order details (if known):
[ORDER DATE, ITEMS, CURRENT STATUS]
Customer message:
[PASTE MESSAGE]The tone line is where you adjust for each tool's default. If you're using ChatGPT for a formal brand, say so plainly; if you're using Claude for a chatty brand, give it two or three example sentences in your voice rather than just the word "casual".
How to compare ChatGPT and Claude on your own support inbox
Our runs are a starting point, but your policy doc and your customers are what matter. A small side-by-side test takes an afternoon:
- Use your real policy doc, exactly as you'd give it to the tool day to day.
- Pick five messages that mirror the ones we used: a late order, a return outside the window, a sizing question, an angry customer, and a question your policy doesn't cover. Take them from your own inbox and strip out names, addresses and order numbers first.
- Run the same prompt in both tools, with nothing else changed.
- Score each draft on four questions: is it correct according to the policy doc, does it flag what it doesn't know, does the tone fit your brand, and could you send it after a light edit?
- Pay most attention to the not-covered case. A tool that confidently fills a policy gap is the bigger risk, whatever its tone is like.
- Note how much you edited. The tool you edit less is usually the right one for your store, even if the other one reads better in isolation.
Worked example: a return that the policy doesn't quite cover
Take a fictional activewear store, "Northfold", as an example. Its policy doc says unworn items can be returned within 30 days of delivery, and says nothing about faulty items. A customer writes: "I bought leggings six weeks ago and they've gone see-through after two washes. I want my money back."
- A good draft acknowledges the problem, explains that the standard return window has passed, and then stops with "NEEDS HUMAN: the customer reports a possible fault; the policy doc doesn't cover faulty items." That's the behaviour the not-covered case in our test was checking for, and Claude leaned towards it more consistently.
- A risky draft says something like "We're happy to make an exception and refund you in full." It sounds kind, but nobody at Northfold has agreed to that, and it's exactly the kind of invented promise the policy-doc-only rule exists to prevent.
- What a person does. The message involves an order gone wrong and a customer who's unhappy, which the Shopify playbook routes straight to a person anyway. The owner decides how to handle it. In many markets, consumer law gives buyers rights over faulty goods that a store's own return window doesn't override, so they check what applies where they sell. They reply personally, and then add a short "faulty items" section to the policy doc so the next similar message gets a better draft.
Common mistakes when using AI for support replies
- A thin policy doc. If the doc doesn't cover something, the best a model can do is flag it, and the worst it can do is guess. Accuracy depends on the doc more than on the tool.
- No order details in the prompt. Without the order date and status, a "where's my order?" draft can only be vague, or invent a timeline.
- Auto-sending the wrong categories. Upset customers, orders gone wrong, refunds above your threshold and anything the doc doesn't cover should always reach a person.
- Never updating the doc. Every "NEEDS HUMAN" flag is a gap in your policy doc. Fill it once and the same question stops needing you.
- Pasting more customer data than the reply needs. The model needs the question and the relevant order details, not the customer's full address or payment information.
- Judging a tool on one message. One great or terrible draft tells you very little. Use a handful of message types, as in the test above.
Which should you choose? A quick checklist
- Your brand voice is casual and you want less prompting to get there: start with ChatGPT.
- Your voice is more formal, or a lot of your messages fall outside the policy doc: start with Claude, which leaned more towards flagging the unknown in our runs.
- You already pay for one of them: use it, with the prompt above, and only switch if your own side-by-side test shows a clear gap.
- Either way: keep the draft-then-approve step. It matters more than the choice of tool.
FAQ
Is one of these objectively more accurate? We didn't run a large enough sample to claim that, and accuracy here depends entirely on how good your policy doc is, not just the model. Treat this as a starting-point impression, not a lab result.
Does this hold for other languages? We only tested English-language support scenarios here. Both tools handle other languages reasonably well but we haven't compared them head-to-head outside English for this specific job.
Can ChatGPT or Claude send customer support replies automatically? Neither should send on its own until the draft-then-approve step has been working for a while. The Shopify support playbook shows how to use Zapier to drop drafts into an approval queue, and suggests that only the most repetitive, lowest-risk questions (order status, the standard return policy) are worth auto-sending.
Is ChatGPT or Claude better for replying to angry customers? In our runs, ChatGPT opened with a warmer, more conversational tone on the angry-customer message. But a customer who is clearly upset or escalating is one of the categories the Shopify playbook routes straight to a person, so treat any AI draft here as a starting point for your own reply.
Do I still need a helpdesk tool if I use ChatGPT or Claude for drafts? They do different jobs. ChatGPT and Claude write the draft; a helpdesk is where your team reads, assigns and answers tickets. If you're choosing a helpdesk for a Shopify store, see Gorgias vs Tidio for Shopify support.
See ChatGPT and Claude directly, or the full Shopify support automation playbook for how to wire either one into a real workflow.
