K-Blade Concept Structure vs. the Live Site's Self-Organizing Map: An Efficiency Comparison

Authors: Pat Parslow (Draft)

Third in this series, following "Algorithm, Not Metric" (Tests A–E) and "Overlapping Membership and Geometric-Algebra Borders" (Test F). This paper is a single, focused measurement: how does the full k-blade similarity/clustering/border pipeline compare, in wall-clock cost, to the (then-)live site's SOM-based concept atlas, on the identical corpus?

Abstract

TheThe parslow.net site's then-live "Cartographic Topological Map" was generated by a 40×40 Pure-Python Self-Organizing Map trained for 30 epochs over broken-stick TF-IDF vectors. We instrumented that exact pipeline (cache bypassed, so the measurement reflects a genuine cold build) and the full k-blade similarity-plus-clustering-plus-border pipeline from the two predecessor papers, on the same 160-page corpus, and timed both stage-by-stage. The SOM pipeline totals 802.1 seconds (13 minutes 22 seconds), of which SOM training alone accounts for 797.1 seconds (99.4% of the total). The k-blade pipeline totals 230 milliseconds. The SOM pipeline is 3,485× slower on this corpus. We explain the mechanism behind this gap — a fixed grid resolution vs. a corpus-size-scaling computation — and discuss what the SOM buys for that cost that the k-blade pipeline, as it stood at the time of this measurement, did not yet replace.

1. Method

BothBoth pipelines were run on the same 160-page corpus, with embedding/vectorization inputs read from cache where both pipelines legitimately share that cost (nomic-embed-text vectors were already computed for prior papers in this series; TF-IDF vectorization is SOM-specific and timed fresh). The SOM run explicitly bypassed som_atlas.py's own content-hash cache (which would otherwise skip training entirely on an unchanged corpus) so the measurement reflects a genuine cold build, the situation that matters whenever the corpus actually changes. Both pipelines were run in the same Python process invocation, on the same machine, back to back, to avoid any environment-difference confound.

2. Results

SOM pipeline (40×40 grid, 30 epochs, 1203-dim broken-stick TF-IDF)
text extraction	1,060.9 ms
TF-IDF vectorization	2,802.0 ms
SOM training (30 epochs)	797,084.5 ms
U-matrix computation	1,164.1 ms
TOTAL	802,111.4 ms (802.1 s)

K-blade pipeline (768-dim nomic-embed-text, embeddings pre-cached)
build k-blade similarity matrix (all 12,720 pairs, rank≤3)	154.8 ms
graph-communities clustering	45.4 ms
GA border metrics (all cluster pairs)	30.0 ms
TOTAL	230.2 ms

SOMSOM training alone (797.1 s) is 99.4% of the SOM pipeline's total cost, and is 3,462× the k-blade pipeline's ENTIRE total. Overall, the SOM pipeline is 3,484.6× slower than the full k-blade pipeline on this corpus.

3. Why the Gap Is This Large

TheThe mechanism is structural, not an implementation inefficiency in either pipeline. The SOM's dominant cost is fixed grid resolution × epochs: a 40×40 grid is 1,600 nodes, each of which is updated (in varying strength, per the neighbourhood function) on every one of the 160 training vectors, for 30 full epochs, in pure Python with no vectorized bulk-array backend for the per-node weight update loop. This cost is essentially independent of corpus size for a fixed grid resolution and epoch count — a 40×40/30-epoch SOM over 1,600 pages would cost roughly the same per-epoch node-update work, dominated by grid size, not corpus size, until corpus size itself becomes the bottleneck at a very different scale than tested here.

The k-blade pipeline's dominant cost is the opposite: an \(O(n^2)\) pairwise similarity computation (12,720 pairs for 160 pages) using small, cheap linear-algebra primitives (rank-3 SVDs, each \(O(r^2 d)\) rather than anything exponential in embedding dimension — the exact property established in the predecessor papers' Tests A–C). This cost scales with corpus size, not with an arbitrarily chosen grid resolution, and at \(n=160\) sits nowhere near the point where \(O(n^2)\) would start to look expensive. Both cost models are entirely predictable from first principles; the 3,485× gap measured here is a corpus-size-vs-grid-size accident of scale, not a discovery that one approach is unconditionally faster than the other — a substantially larger corpus, or a much larger SOM grid, would shift this ratio, and no attempt was made in this paper to find where the two curves cross.

4. What the SOM's Cost Bought, That the K-Blade Pipeline Did Not Yet Replace (At Time of Writing)

ThisThis was not a like-for-like comparison of equivalent outputs, and reporting the speed gap without this section would have overstated the finding. The SOM pipeline additionally produced: a full topological elevation surface (the U-Matrix) usable for pathfinding "terrain difficulty" between any two arbitrary points on the map (not just between existing pages); a fixed, stable 2D grid coordinate system that composed cleanly with the (then-)live site's tile/route rendering; and years of prior tuning (broken-stick feature selection, hazard-keyword elevation biasing, road/bridge-term generation) specific to this site's navigation UI. The k-blade pipeline, as built across this paper series, produced a similarity structure, a partition (optionally overlapping), and inter-cluster border metrics — a different, not strictly smaller, set of outputs. The natural continuation of this work (a k-blade-based 2D map projection, developed as a companion to this paper) was an attempt to recover the SOM's core navigational value from the k-blade structure directly, at a small fraction of the cost — and that continuation subsequently became the live site's actual Concept Atlas, replacing the SOM entirely; see the production writeup for how each of these SOM-specific outputs (terrain, coordinates, road/bridge text) was carried over.

5. Limitations

• Only one corpus size (160 pages) and one SOM configuration (40×40, 30 epochs) were tested; the crossover point (if any exists at a realistic scale) between the two cost curves was not determined.
• The SOM implementation timed here was the site's actual Pure-Python implementation, not an optimized (e.g. NumPy-vectorized or GPU) SOM; a vectorized SOM implementation would likely close much of this gap, and the comparison should be read as "this specific pipeline as it was then deployed" rather than "SOMs are inherently this expensive."
• The SOM's cache (bypassed here deliberately) meant this 802-second cost was, in the site's actual operation at the time, paid only when the corpus content changed, not on every page load or every build — the practical cost-per-build in steady state was much lower than this cold-build figure and was not separately characterized.

6. Conclusion

OnOn this 160-page corpus, the full k-blade similarity-clustering-border pipeline was 3,485× faster than a cold build of the site's then-existing 40×40 SOM pipeline, because the two approaches have fundamentally different cost models (fixed-grid×epochs vs. corpus-size-scaling pairwise linear algebra) rather than because one is a more efficient implementation of the same computation. This margin made a k-blade-based concept map practical to rebuild far more frequently than the SOM had been, which is directly relevant to the companion visualization work (Test G) that turned this structure into an actual navigable map artifact — and, subsequently, into the site's live production atlas.

Appendix: Reproducibility

Code:tools/site/concept_experiments/kblade_vs_som_timing.py.Code:tools/site/concept_experiments/kblade_vs_som_timing.py. tools/site/concept_experiments/kblade_vs_som_timing.py. Runs the (then-)live tools/site/som_atlas.py's own build_tfidf_vectors and PurePythonSOM class directly (cache bypassed), and the k-blade pipeline from ga_kblade_hierarchy.py / ga_kblade_graph_communities.py, on the same cached 160-page embedding data (ensemble_cache.npz) used throughout this paper series. All figures in this paper are read directly from that script's stdout; none are estimated.