Last week, while drilling the AIoT Exam 2 question bank, I hit a security question. Of the four options, only one read “an audit log recording the operator, time, target, and action”; the other three were bare nouns: transport encryption, data compression, load balancing. The answer jumped out before I’d even understood the question. A few questions further on, I realized it wasn’t just that one. The whole series was like this.
I generated this question bank with an LLM and put it on a practice site I built myself, and the AIoT exam was at the end of the month. Getting tripped up by my own creation felt a little odd. But looking back over these two weeks, it wasn’t an exception. It was a thread running through the whole stretch.
Two weeks, seven repos, three new projects
First, the numbers. From 10/16 to 10/31 there were 196 commits on GitHub, spread across 7 repos. Of the 16 days, only 10/31 saw no activity at all. Last period was 55 commits over 9 working days, so this time the volume was more than three times as much.
Most of the increase came from new projects:
- Renn is a cheese-making science site that reuses the data sources and notebook system from Nevé (my gelato notes). It went from proposal to deployed in one day.
- Tidemark is an agent for fundamental research on Taiwanese stocks. I finished the proposal and architecture on the morning of 10/24, and the first version of the code was running that afternoon.
- PlantOS (later renamed 一葉一日, “a leaf a day”) is a hydroponics journal for my own use. In one evening it went from data contracts to private cloud sync.
Lantern kept moving too. Its live channel switched from SSE to WebTransport, and the classroom features gained a contact book, zhuyin and 楷體 (regular-script) type, attendance export, and a handwriting board. I also put together review material for a fifth-grade English midterm.
In the proposals repo, the decision numbers ran from D-043 to D-078: 36 decisions recorded in two weeks.
Looking only at this, the conclusion is easy: with AI as a collaborator, starting a new project is now cheap enough to do on a whim. That’s true. But the things that actually made me stop these two weeks were never “couldn’t build it.” They were “built it, but it’s wrong,” and wrong in a consistent way.
Checked, just not for that
Back to the question bank. I measured every bank: if a test-taker who knew nothing about the content always picked the longest option, what would they score? On the new AIoT Exam 2 bank, 83, with a passing mark of 70. The three intermediate subjects were worse, 87 to 97. The same strategy on the official past papers scores only 36 to 63. The official questions have some length bias too, but never to the point where you could pass without reading the question.
The most awkward part is that I’d written this rule long ago. The question bank spec from mid-August says it plainly: “Keep the four options similar in length; a noticeably long correct answer is the easiest giveaway.” The associate-level bank had also been through eight rounds of LLM content review. But every round asked “is the answer correct, does the explanation match?” None of them took the side of a test-taker who doesn’t know the material and asked “is it easy to guess?” Reading question by question, a more complete correct answer looks like a strength. You only see the defect by running statistics over the whole bank. The tests checked things that were easy to measure: answer-letter distribution, cognitive level, question type. Length is just as easy to measure. Nobody had written it as a test.
The fix was interesting too. Length was only a symptom; the root cause was weak distractors. When an LLM writes questions, it puts its effort into making the correct answer unimpeachable and throws the wrong options together. The intermediate bank even had options like “the report has page numbers” and “the office temperature on training day.” So the fix wasn’t to shorten the correct answers. It was to rewrite the distractors as real misconceptions.
The first batch overcorrected. The correct answer was almost never the longest, which turned into a reverse giveaway. In the end, counting the associate banks and the older self-written questions, I revised more than 1,200 questions, and the “always pick the longest” score fell to 23–33, close to the 25 you’d get by guessing. I also added a guard test: the score is only allowed to get better, never worse.
Tidemark ran into the same thing in a different shape. On the morning of 10/25 I did a direction review and concluded “the direction fits, execution isn’t there yet.” That review looked at output, like whether the shortlist existed and whether the number cards were right. It never asked about the method itself: should the research start from the numbers, or from changes in the companies? So I spent the whole next day thickening the quantitative side, until I wrote a direction document and switched to “qualitative first, quantitative validation.” Not much was lost: the number cards became a validation tool and the screen became a control group. But that detour didn’t need to happen.
There was a small episode the same day. While surveying data sources, a subagent judged the Taiwan Intellectual Property Office’s patent API “usable as is.” That conclusion was relayed all the way into a decision, and I approved it on that basis. When it came time to implement, I read the API documentation and found you have to apply for a verification code first, or it only returns sample data.
Things you only learn by touching reality
Turned around, the most reliable conclusions of these two weeks almost all came from actually running something.
Tidemark was originally going to do all its data fetching on Cloudflare Workers. Then TPEx (Taipei Exchange) blocked connections coming from Cloudflare outright. I moved it to my own Mac for a while, and it finally landed on GCP Cloud Run.
Lantern’s live channel was load-tested to 10,000 connections on a 2-core, 3 GB demo machine, with zero drops and a P95 of 1.6 seconds. After tuning, it held 30,000 cleanly and hit the memory ceiling near 38,000. Along the way I found that the memory limit I’d set simply has no effect on Raspberry Pi OS. The shop’s iPhones and iPads were all tested on real hardware as well.
In 一葉一日, picking photos on an iPhone gets blocked. No amount of desktop emulation would ever surface that. Only testing on the phone did, and then I added a fallback route through the Files app.
None of this could have been written into a spec. It’s the answer reality hands back.
Putting the two halves together, the conclusion resembles last period’s, but sharper. Last time I said the blind spot was in the checking itself. This time I saw that blind spots tend to grow in three places:
- Rules that live in documents but never became tests.
- Conclusions that were relayed but never checked against the source.
- Reviews that checked the output but not the method.
What the three have in common: they all looked like they’d already been checked.
So these two weeks also added a few rules to AGENTS.md:
- Commit messages and documents state only confirmed facts.
- Before relaying a subagent’s key verdict, especially “usable,” “free,” or “no application needed,” go back to the source yourself.
- A direction review asks not only “was it built?” but also “is this the method I want?”
The pace slowed noticeably toward the end of the month. After 10/27 the daily commits dropped to single digits, mostly finishing off the question bank, and the end of the month was also exam time. Starting fast probably won’t be the bottleneck anymore. What I want to put more thought into next is how to make “knowing whether it’s right” keep up at the same speed.
Sources
- iPAS AI Application Planner practice site (question bank and the option-length retro): https://github.com/kuochenfu/ipas-aiap-cert
- Fifth-grade English midterm review: https://github.com/kuochenfu/cs-english-exam
- Previous issue: Bi-Weekly Review 2026-W40
- “The blind spot is in the checking itself”: Bi-Weekly Review 2026-W35
- Lantern, Tidemark, Renn, and 一葉一日 are private repos for now, so no links.
Comments & Feedback