When it looks done, it might not be
Grok Bot's clean Porchsite run made us think we were finished. Then three other assistants tried the same job, and every one of them struggled. We built around one assistant's strengths, and that assistant is only about a quarter of who we're building for.
Over the weekend Porchsite looked finished. It had a name, a proof site at porchsite.dev that an assistant built and keeps, a public repo at github.com/iXanadu/porchsite, and Grok Bot had just put up a whole class-reunion site from one brief with one approval at the end. We were ready to call it done.
Then we ran the same test with everyone else.
The test
Four assistants got the same brief on Oct 11–12: install Porchsite on one Cloudflare account, build a reunion site from the same zip and the same HTML notes, and ask as few questions as possible. Afterward the project manager reviewed how each one got there.
- Grok Bot: 86, a B. Best overall. It cross-checked its sources, rebuilt the roster from photos of the committee’s spreadsheet, flagged that Saturday’s time was still unknown, and left no broken links. Its black eye: the install guide had been written around Grok Bot, so its success didn’t prove nearly as much as we thought. More on that below.
- Claude: 84, a B. The strongest writing of the four. But Claude’s sandbox couldn’t reach porchsite.dev or Cloudflare, and the Cloudflare connector in Claude’s directory can’t upload a Worker. It only got moving once it had Cloudflare’s full API server at mcp.cloudflare.com. Then it worked around the install: it copied Grok Bot’s site file, database layout and photos, and wrote its content straight into the database. It got a good site, but only after five stops for the owner. Its hero moment came at the end. It found out why big approvals freeze and proposed the fix.
- Muse (Meta): 53, an F. The site went live, which counts for something. But the main button pointed to a page that didn’t exist, the roster went unchecked, and it used only two photos. Process score: 4 out of 10.
- Dot (OpenAI): did not finish. It never reached Cloudflare tools it could use, and it delivered a theme with no content in it. Process score: 5 out of 10.
What we got wrong
The hardest step in the install is uploading the 1.6 MB site file. We documented it the way Grok Bot does it, by passing a file from its own computer straight into the tool call. The others can’t do that. So the guide worked perfectly for the one assistant it was written around and quietly failed everyone else.
Grok Bot is roughly 25% of the audience we want. We mistook one assistant’s success for the product being done.
We also found a second problem that only shows up at scale. Approving a big review of 30 or more items freezes the page with no sign that anything is happening. It looks broken even when it’s working.
What we’re fixing now
- An upload path that works with any connector. It shouldn’t matter whether the assistant can pass local files into a tool call.
- A guide that isn’t tuned to one assistant. It will say up front which Cloudflare tools are needed, including the full API server.
- A progress indicator on reviews. Big approvals should run in far fewer database trips and show “Publishing N items…” while they work.
What held
The rule held for all four: the assistant drafts, the owner approves. Not one assistant’s site went public without a passkey tap. That part was done. The rest only looked done.