The exam is done. Last cycle ended with “next up is finishing the Intermediate subject content, plus the AIoT certification I registered for on a whim — after that, back to normal.” These two weeks were basically that list, worked through in order.
The rhythm was nothing like last cycle. Then, 96 commits landed almost entirely in one repo; exam prep had squeezed everything else out. This time the commit count dropped by 40%, but spread across four projects: finishing the question banks, two new proposals, a format upgrade in the knowledge base, and the personal site. Going from “only one thing” back to “several things” is itself the signal that prep is over.
What’s worth writing down, though, isn’t that the list got cleared. It’s the one thing that kept recurring while clearing it: the method I used to confirm “this batch is correct” determined which kind of error I was able to see. None of the three most real defects in these two weeks were caught by a test.
What Actually Happened on GitHub
- 55 commits between 8/15 and 8/31, across 9 working days — 8 days had nothing at all (including 8/27–8/28 back to back).
- Spread over 4 repos:
ipas-aiap-cert27,chenfu.ai11,proposals10,chenfu-kb7. - Net change +60,589 / −1,076 lines across 299 files. The vast majority is question-bank data.
- The peak is still a single-day burst: 19 commits on Saturday 8/22, all in the question banks. 8 on 8/17.
A day almost always belonged to exactly one project; the colors rarely mix. Which is to say these two weeks were context switching, not parallelism.
The iPAS Site: From Drill Tool to a Four-Certification Platform
1. Reclassifying a months-old “known limitation” as a broken question (8/17)
While importing the 115-1 past papers for the three Intermediate subjects, I also cleared something that had been there since early in the project: the images in the original papers.
senior-ml-114-2-q45 was logged in three places at once — CLAUDE.md, AGENTS.md, and bank-defects.md — as “the choices in the source PDF are an image; pdftotext can’t extract them,” filed under known limitations: acknowledged, but not going to be fixed. It was actually a broken question. A question with four blank choices sitting on a live site isn’t a limitation, it’s a fault.
The turning point was a very dumb action: render the PDF page to PNG and actually look at it. Those “images” were screenshots of code — Keras code, four ways of writing train_test_split, a classification report printed to console. All of it text, just printed as pixels. Once that was clear, every option I’d been comparing (extract the images or not, public/ or base64, what to do about the watermark) became moot, because text transcription wins on every axis. Actual charts numbered 4, out of 42 questions.
The batch came to: 150 past-paper questions (three Intermediate subjects, 90 → 140 each), 150 explanations, 42 figure transcriptions, 4 previously unanswerable questions fixed, 3 parser defects fixed, tests 219 → 225.
2. Only after restoring the missing information could I verify the existing content was wrong (8/17)
The old explanation for senior-ml-114-2-q50 read “blank 1 should be "r-", blank 2 should be "r--",” and flagged itself: “figure missing, following the official answer, medium confidence.” With the figure restored, you can see the legend: Training Loss is a solid blue line, Validation Loss a dashed red one. The statements that hold are A and D, not C and D.
The answer letter was right. The entire reasoning was backwards. It was completely invisible while only the answer could be checked — reason reverse-engineered from the answer letter, answer correct, self-check passes. But what the site shows the user is the explanation, not the letter.
And it had flagged itself as low confidence, then simply stayed there. That’s another form of last cycle’s “disclosure isn’t a fix”: a self-declared weakness marked in the data is not a to-do either. Someone has to turn it into one.
3. Launching AIoT forced the question: what is level, actually? (8/21)
The AIoT Application Engineer (Associate) certification I’d signed up for on a whim needed to go live. My first instinct was to treat it as a new level — smallest possible change, not a single line of UI to touch. I didn’t, because that would put “AI Application Planner, Associate” and “AIoT, Associate” side by side in the same dropdown, and the ask had been “don’t mix them together.”
So I added a cert dimension, making navigation home → cert → level → mode. The real payoff wasn’t the decision itself but the question it forced: digging in, level turned out to have been playing two roles all along — “difficulty stage” and “which certification these subjects belong to.” With only one certification those coincide and you can’t see it. Adding a second one pulls them apart.
4. 640 questions, and the predictable failure modes of LLM authoring (8/21–8/22)
The biggest block of the cycle: 80 official AIoT practice questions, 60 hand-authored for Subject 2 (the one subject with no official question source at all), and 500 in new banks across seven subjects (300 for the three Intermediate subjects, 200 for the two AIoT ones). Bank total 1,117 → 1,757 questions; tests 268 → 422. Afterward, another rebalancing pass added 215 questions to redistribute cognitive levels across the five Planner subjects, plus per-question metadata and decision boundaries for all 500.
Every first-pass batch broke in the same place: senior-ai-tech answers came out A:69 / B:27 / C:4 / D:0; aiot-junior-basics came out A:86. In a bank where A is 70% of the answers, “guess A when unsure” has a better expected value than actually reading. That’s not a content problem, it’s a structural failure — and structural failures can be caught by tests. What caught these was practiceBank.test.ts, written back on 8/13 for the Associate-level bank; it picked up three new subjects for free.
But the first fix went the wrong way: cycling A, B, C, D in order produced exactly one ABCD permutation every four questions — easier to guess than the original skew. The block check in the same test caught it immediately. Balanced is not random.
Proposals and Knowledge Base: The “Back to Normal” Track
With the question banks wrapped up, 8/24 onward switched to the threads I’d set aside.
proposals opened two new proposal families: Auxin (agri-tech) and Nevé (batch quality and cold chain for artisan frozen desserts), each with its own proposal, SRS, architecture, and primer. Nevé went through four rounds of convergence — a 2026 competitive landscape, Phase 1 measurement methodology, and finally a TDI kinetic form pinned down from the literature — taking the proposal to v0.7, SRS to v0.5, architecture to v0.5, with a convergence retro to close it out. Worth noting about the Nevé Primer: it orders the domain terminology causally, not alphabetically.
chenfu-kb treated the knowledge base’s own format as an engineering problem: the OKF v0.2 upgrade, an extended Rust checker plus a new okf-migrate tool (+809 lines), a bulk migration of the wiki/ frontmatter declaring okf_version, and — once migrate gained a raw mode — a backfill of 61 files under raw/, along with a clarified read-only boundary for that directory.
chenfu.ai spent the cycle on verifiable credentials: the About page gained a Claude Academy badge wall with all 17 badges linked to their public verification pages (/verify/<code> is the public one; /badges/<uuid> puts visitors behind a login wall), and revised pages now carry an updated-on stamp.
What These Two Weeks Taught Me
The word someone uses to describe a problem carries an implied class of solutions. “Any suggestions for the images?” reads naturally as an asset-pipeline question, so I started comparing extraction, storage, rendering, licensing. One look would have shown they weren’t images at all. Look first, classify second, compare options last.
A regex expresses literal form, not a semantic property. To verify that every explanation names A/B/C/D individually I wrote a carefully constructed regex, and measured 40 failures — all false positives, because in “A correctly describes,” the verb wasn’t in my list. What I measured was how many Chinese sentence patterns my regex covered. Swapping in a much weaker but sound check — does each of the four letters appear as its own token — gave 0 failures, and that result is trustworthy.
Official files are wrong too, and they’re wrong quietly. In the AIoT practice set, q078’s answer field says Ans (C) while its own explanation argues throughout for choice (A) — the answer and the reasoning contradict each other inside the same file. One read-through finds it; format validation never will.
Your verification method defines which class of defects you can find. Reassigning answer letters means moving three things in step: the keys of choiceExplanations, the keys of meta.distractorTypes, and phrases in the prose like “if you changed choice C to…” that name a letter. My check was key consistency — and only two of the three are keys. Letters inside prose were never in view, so “0 inconsistencies” was true, but it answered “is the thing I checked correct?” rather than “is this batch correct?” 8 questions were actually broken, 3 of them in a subject already declared verified.
Writing a topic name from memory cost 16 questions. I wrote Subject 2’s B2.2 from memory as “IoT Software and Firmware Development” and authored 16 firmware questions against it. The actual node is “Cloud Data Collection and Platform Design.” topic is a string; the type checker had no opinion. The 16 questions were fine in themselves — the problem was that the syllabus had no place for them. One grep beforehand: a thirty-second action.
What’s Next
- None of the 640 questions have been fact-checked by a human. The highest risk is the 160 for AIoT Subject 2 (60 hand-authored + 100 in the new bank) — the subject with no official source material at all.
- AIoT’s question count and scoring are still unverified. The prospectus only says “100 points per subject, 70 to pass,” so
mockExamstays off for both. Don’t turn it on before the official numbers are in hand. - The parser health checks still don’t exist. “Choice text abnormally long” and “choice ends with a question-number pattern” were both proposed in the 8/12 and 8/17 retros, and there are still zero of them. This is the third time it’s been written into a to-do list.
- Go back through the low-confidence explanations. The remaining entries in
bank-defects.mdflagged “following the official answer / low confidence” are the same species as q50, and should be treated as unverified rather than “known to be weaker.” - The next round for Nevé and Auxin, and carrying the knowledge-base track forward.
Last cycle’s conclusion was: turn “something feels off” into a number I can measure, then turn that number into a guard I can verify. This cycle is the next step, and a less comfortable one: a guard only stops what it recognizes. The three most real defects of these two weeks — a self-contradicting official answer, a wrong choice letter buried in prose, a topic name written from memory — were none of them inside what I was checking, and every one of them would have surfaced from simply reading it or running one grep.
Automation catches the mistakes you’ve already thought of. The rest, you still have to look at yourself.
Comments & Feedback