# Pilot report A 150-component sample tagged through `classifier.dev`, run via `pnpm pilot` (`src/pilot.ts`). This is not the full 1,071-component run; it exists to check the tagging pipeline and the dimension criteria before spending the full day's quota. Raw output: `data/pilot/tags.json`. Sample membership: `data/pilot/sample.json`. ## Sample composition Proportionally weighted by each registry's real (post-filter) size, with a forced floor so aceternity's source-enrichment fix (docs/DECISIONS.md #11) gets a real test: | registry | in sample | |---|---| | react-bits | 29 | | magicui | 11 | | aceternity | 39 | | kokonutui | 7 | | animate-ui | 59 | | motion-primitives | 5 | | **total** | **150** | All 39 of the aceternity slots landed on previously-undescribed, now-source-enriched primitives (the sampler pulls from the front of each registry's kept list, and aceternity's primitives sort before its blocks in the source data, see docs/DECISIONS.md #9's `filterAceternity`). That comfortably exceeds the requested floor of 20, at the cost of the sample having zero aceternity blocks this round; the full run will cover both. ## 1. What the per-item JSON actually contained Checked before writing any enrichment code, against 3 of aceternity's undescribed `registry:ui` primitives (`grid`, `moving-line`, `sparkles`), fetched directly by name from the same `{name}.json` URL shadcn's own CLI uses to install a component: - The response inlines the full `.tsx` source under `files[].content`, not just metadata. `moving-line`'s source (3,152 bytes) shows a real, specific implementation: a scroll-linked SVG path animation using `motion/react`'s `useScroll`, `useTransform`, and `useSpring`, naming its exported component `MovingLine`. - It also carries `author` and `title` fields the batch index for the same component does not include at all (`grid`'s index entry has neither). - Confirmed at scale during the real ingest run: all 112 of aceternity's undescribed primitives fetched successfully (HTTP 200, JSON, non-empty `content`) and were enriched. No GitHub pull was needed or used. - Separately, and not part of what was asked but found while enriching aceternity's thin block descriptions too: `registry:block` per-item URLs return HTTP 401, not the source. aceternity appears to gate block source behind something the anonymous request does not have. This does not lose any data (blocks fall back to their existing short index description, see docs/DECISIONS.md #12); it just means the enrichment gain from this fix is specific to the 112 primitives, not the 170 blocks. ## 2. Label distributions and confidence `under 0.6` counts how many of the 150 answers on that dimension had `confidence < 0.6`. Noul dimensions have no confidence field by design (docs/DIMENSIONS.md); their spread is reported as a probability histogram instead. ### category (choice, 14 labels) `background-effect` 37, `none of these` 19, `modal-overlay` 15, `card` 14, `button` 13, `media` 9, `navigation` 9, `typography` 8, `form-input` 8, `data-display` 6, `layout-section` 5, `feedback-notification` 4, `hero` 3. (`chart`: 0, unsurprising for this registry mix, none of these registries are chart libraries.) Under 0.6 confidence: **33/150 (22.0%)**. Null confidence: 0. `background-effect` and `none of these` dominating is expected and correct for this catalog: react-bits, aceternity, and animate-ui are heavy on ambient visual effects, and a fair number of components (utility wrappers, animation primitives with no fixed visual identity) genuinely belong in neither a content category nor a structural one. ### motion (choice, ordered, 4 labels) `animated-on-interaction` 70, `continuous-background-motion` 42, `static` 23, `subtle-transitions` 15. Under 0.6 confidence: **47/150 (31.3%)**. Null confidence: 0. ### visual_density (choice, ordered, 3 labels) `minimal` 96, `moderate` 33, `dense` 21. Under 0.6 confidence: **69/150 (46.0%)**. Null confidence: 0. This is the worst dimension in the pilot by a wide margin, see finding below. ### interaction_model (choice, 6 labels) `display-only` 53, `hover` 39, `click-toggle` 29, `none of these` 11, `drag-or-gesture` 10, `form-input` 8. Under 0.6 confidence: **39/150 (26.0%)**. Null confidence: 0. ### needs_external_data (noul) Probability histogram: 0.0-0.2: 93, 0.2-0.4: 15, 0.4-0.6: 8, 0.6-0.8: 16, 0.8-1.0: 18. Null: 0. Heavily skewed low, which matches the sample: most of these registries are visual/animation components with built-in placeholder content, not data-bound components. ### decorative_only (noul) Probability histogram: 0.0-0.2: 98, 0.2-0.4: 2, 0.4-0.6: 4, 0.6-0.8: 5, 0.8-1.0: 41. Null: 0. Bimodal, as expected for a purely-decorative-or-not judgment: most components are clearly not decorative-only (0.0-0.2), a real minority clearly are (0.8-1.0), and very few land in the ambiguous middle. This is what a working noul dimension should look like. ## 3. Finding: visual_density is the weak dimension, not the source data Ranking all 150 components by their lowest confidence across the 4 choice dimensions and taking the bottom 10: | registry/name | min confidence | dimension it came from | enriched from source | |---|---|---|---| | animate-ui/components-community-radial-intro | 0.01 | visual_density | no | | aceternity/3d-pin | 0.09 | visual_density | yes | | magicui/progressive-blur | 0.10 | motion | no | | magicui/android | 0.12 | visual_density | yes | | aceternity/parallax-scroll | 0.13 | visual_density | yes | | animate-ui/components-base-preview-link-card | 0.14 | motion | no | | animate-ui/components-headless-popover | 0.14 | motion | no | | aceternity/evervault-card | 0.16 | visual_density | yes | | aceternity/parallax-hero-images | 0.16 | interaction_model | yes | | kokonutui/smooth-drawer | 0.18 | visual_density | no | 8 of these 10 have `visual_density` or a closely related boundary call as their worst dimension, and this holds regardless of whether the input was a thin description or several KB of real enriched source: `aceternity/3d-pin` had 5,373 characters of actual component source and still scored 0.09 on `visual_density`, while its `category` (0.81) and `interaction_model` (1.00) on the same input were confident. `evervault-card` and `parallax-scroll` show the same pattern: rich source, confident category and interaction_model, weak visual_density. **Conclusion: this is a criteria problem, not a source-data problem.** The counting heuristic added to `visual_density`'s instructions (docs/DIMENSIONS.md, "0-2 / 3-5 / 6+ visual elements") asks the model to count elements it cannot actually see or reliably enumerate from prose or JSX, especially at the "moderate" vs "dense" boundary, which is exactly where these low-confidence answers cluster (`3d-pin`: moderate; `evervault-card`: minimal but described as a fairly rich hover card; `parallax-hero-images`: dense with 0.68, its highest-confidence label in the bottom-10 list, suggesting the boundary problem is specifically between adjacent levels, not a total breakdown). This dimension needs a different fix before the full run, not more source data. Not yet applied; flagged for review before the full 1,071 run. Two candidate directions, neither implemented: (a) collapse to 2 levels instead of 3, removing the ambiguous middle boundary, or (b) anchor with concrete named examples per level pulled from this catalog instead of an abstract count. The other three bottom-10 entries with `motion` as the weak dimension (`progressive-blur`, `preview-link-card`, `headless-popover`) are each a genuinely ambiguous single case (a blur effect that could read as either subtle or interaction-triggered; a hover card that could be static-with-a-transition or interaction-driven) rather than a pattern, and `interaction_model` only appears once (`parallax-hero-images`, and even there at 0.16 it is not the dimension's typical behavior, 26% under 0.6 overall vs. `visual_density`'s 46%). ## 4. Decisions spent and quota remaining - Pilot run: 150 components x 6 dimensions = **900 decisions**, sent as a single chunk (under the 900-decision chunk size used, itself under the 1,000-decision request cap). - `RateLimit-Remaining` after that request: **2,100**, consistent with a 3,000/minute window (3,000 - 900 = 2,100), not the 20,000/day figure; the header appears to track the per-minute limit. - Cumulative decisions spent against the 20,000/day free quota so far in this project: **900 (4.5%)**. No classifier.dev calls were made before this pilot; all prior work was index/source fetches, which do not count against this quota. - Projected full run: 1,071 x 6 = 6,426 decisions (32.1% of daily quota, per docs/DIMENSIONS.md), leaving roughly 12,600 decisions of headroom in the same day for retries, the brief-parsing calls at match time, or a second pass if `visual_density` gets revised before the full run. ## Rerun after three changes Three changes made after the first pilot (full detail in DIMENSIONS.md and DECISIONS.md #15): `visual_density`'s criteria rewritten from a counting heuristic to a perceptual judgment anchored to three named real components (v3 in DIMENSIONS.md); two new `category` labels, `cursor` and `transition-wrapper`, added after reading all 19 of the first run's "none of these" components individually and finding two clusters of 6 and 7; and `src/match.ts` built to weight each dimension's contribution to match distance by its own confidence, so a weak dimension degrades gracefully instead of needing perfect wording. Rerun on the exact same 150 components (`data/pilot/sample.json`, unchanged), output in `data/pilot/tags-v2.json`, another 900 decisions: | dimension | under-0.6 before | under-0.6 after | |---|---|---| | category | 33/150 (22.0%) | 28/150 (18.7%) | | motion | 47/150 (31.3%) | 47/150 (31.3%) | | visual_density | 69/150 (46.0%) | 54/150 (36.0%) | | interaction_model | 39/150 (26.0%) | 43/150 (28.7%) | `motion` and `interaction_model` were not touched by any of the three changes; `motion` is exactly unchanged (same model, same input, same criteria, no surprise), `interaction_model`'s small move (39 to 43, +2.7pp) is run-to-run variance, not a regression caused by this work. `category`'s "none of these" count dropped from 19 to 5, and `background-effect` dropped from 37 to 27 as several components that had been absorbed into it turned out to actually be `cursor` or `transition-wrapper` once those existed as options. Full label distribution in DIMENSIONS.md's category section. `visual_density` improved substantially (46.0% to 36.0%) but is still above the 35% threshold set for proceeding without a third rewrite attempt. Per the time-box set for this work, it was not rewritten a third time. It ships as-is for the full run, with the confidence-weighted matcher (DECISIONS.md #15) as the structural mitigation and the limitation recorded in DIMENSIONS.md for anyone revisiting it after launch. ## Bugs found and fixed during this pilot - First pilot attempt crashed with a 400 (`empty_input`) from `classifier.dev`. Root cause: `registry:style` config entries (magicui and animate-ui each publish exactly one, always named `index`, with no name, description, or content) were being kept and tagged instead of dropped as non-components. Fixed in `src/filter.ts` and backstopped in `src/tagger.ts`. Full account in docs/DECISIONS.md #14.