Concept Mapping and Self-Organizing Maps

Unsupervised Dimensionality Reduction

This article details the mathematical foundation and algorithmic pipeline behind the Self-Organizing Map (SOM) Atlas Generator. Learn how high-dimensional document TF-IDF vectors are projected onto a 2D Kohonen grid, how Pure Python Broken Stick variance selection operates, and how U-Matrix terrain heatmaps generate realistic cartographic topography.

The Self-Organizing Map Pipeline

Generating a cartographic topological map from a collection of text documents involves a multi-stage Machine Learning pipeline:

  1. Feature Extraction: Document text is tokenized into word n-grams, filtered against standard stopword lists, and weighted using Term Frequency-Inverse Document Frequency (TF-IDF).
  2. Pure Python Broken Stick Feature Selection: Selecting high-variance features using the Broken Stick distribution ($E_k = \frac{1}{p} \sum_{i=k}^p \frac{1}{i}$) without relying on native C-extension dependencies like NumPy or Scikit-Learn.
  3. 2D Kohonen Map Training: Training a $40 \times 40$ toroidal or planar grid of neuron weight vectors using competitive unsupervised learning.
  4. U-Matrix Terrain Generation: Computing Unified Distance Matrix (U-Matrix) values between neighboring neuron weight vectors to form topographic elevation contours.

Pure Python Broken Stick Feature Selection

To eliminate Windows C-extension runtime crashes and MINGW-W64 math library incompatibilities, feature selection is implemented in Pure Python using the Broken Stick Model. The expected proportion of variance for the $k$-th component in a random stick-breaking process of length $p$ is given by:

\[E_k = \frac{1}{p} \sum_{i=k}^{p} \frac{1}{i}\]

Features whose observed variance exceeds the broken stick threshold $E_k$ are retained for SOM training, automatically isolating high-variance topic discriminators while pruning noise.

U-Matrix Terrain Heatmaps & Mountain Ridges

TheU-Matrix U-Matrix (Unified Distance Matrix) measures the Euclidean distance between adjacent neuron weight vectors across the 2D grid:

  • Low U-Matrix Values (Valleys & Plains): Indicate clusters of closely related documents with similar vocabulary. These areas are rendered as light parchment terrain where city nodes settle.
  • High U-Matrix Values (Mountain Ridges): Indicate sharp semantic transitions or boundary zones between distinct topic areas. These regions are populated with mountain peak vectors (⛰️) and dense forest icons (🌲).
  • Swamps and Marshes (🌾): Low-density or isolated transition zones marked with reeds and puddles.

Cartographic Export Schema

The trained SOM topology is exported directly into assets/site-atlas.json as a lightweight JSON payload consumed by the frontend renderer. The schema includes grid dimensions, node coordinates, U-Matrix elevation arrays, Voronoi grid categories, terrain features, and precomputed highway paths with bridge concept metadata.