Classical vs Neural NLP, on the Same Bench
An end-to-end pipeline that benchmarks eight text classifiers against a single 20 Newsgroups corpus, then extends beyond supervised learning into a two-level unsupervised topic tree with LLM-generated cluster labels.
Most NLP portfolios skip the comparison.
The pattern in personal NLP work is predictable: pick one model, tune it, screenshot the confusion matrix, ship. That answers whether the model works. It doesn't answer whether it was the right choice, or what a fair alternative would have looked like on the same data.
I wanted the experiment the textbooks describe but almost nobody actually runs: hold the dataset, the split, and the evaluation protocol constant, then swap out the feature representation and the classifier family underneath. Classical sparse TF-IDF against dense sentence embeddings. Four classifiers on each side. Same test set, same metrics, no cherry-picking.
Then a second question most benchmarks skip: once you strip the labels away, does the geometry of the embedding space actually reveal coherent structure, or is that just a story we tell?
Three parts. One benchmark. One CLI.
I structured the project as three modules feeding a single entry point. Part 1 runs TF-IDF against Multinomial Naive Bayes, Logistic Regression, LinearSVC, and Random Forest. Part 2 encodes the same documents through the SentenceTransformer all-MiniLM-L6-v2 model and runs the same four classifiers on the resulting 384-dimensional embeddings. Part 3 discards the labels entirely and asks KMeans to recover topic structure.
Feature parity was the whole point, so I was strict. Every sklearn pipeline fits its vectorizer only on training data — no leakage. Embeddings are L2-normalised at encode time. Multinomial NB gets a MinMaxScaler on the dense side because its non-negativity assumption breaks on continuous vectors; skipping that would have handed classical an unfair win by default.
Part 3 uses the elbow method on inertia's second derivative to auto-select the top-level k, then re-clusters the two largest groups with KMeans(k=3) to produce a two-level tree. Cluster labels come from GPT-4o-mini given five representative documents per cluster, with a keyword-frequency heuristic as fallback so the pipeline still runs without an API key.
A pipeline you can rerun, not a notebook you have to read.
The repo is deliberately modular: data_loader.py, part1_classic.py, part2_embeddings.py, part3_clustering.py, and a main.py that ties them together. A --parts CLI flag lets you run any subset — useful because Part 3 depends on Part 2's embeddings and re-encoding 10,000 documents through a transformer isn't cheap.
That's why embeddings get cached to outputs/embeddings_cache.npz and reused thereafter, with a --force-embed flag for clean regeneration. Every random seed is pinned; reruns are deterministic. Confusion matrices, elbow curves, PCA scatter plots, and the rendered topic tree all land in outputs/ as artifacts you can hand to someone else.
Classical wasn't dead. The surprise was where it beat the transformer.
On this dataset, TF-IDF plus a linear SVM edged out the same linear SVM on top of MiniLM embeddings — the finding that would have been quietly buried in a single-model portfolio piece. Transformers aren't worse; 20 Newsgroups is a corpus of very domain-specific vocabulary (sci.crypt, rec.sport.hockey, comp.sys.mac.hardware), exactly the regime where high-dimensional sparse features have a structural advantage. Semantic generalisation isn't the task here; keyword discrimination is.
Multinomial NB collapsed on the dense side even with MinMaxScaler, confirming that its count-based likelihood doesn't survive continuous embeddings. Part 3 recovered thematically coherent top-level clusters — space, computer hardware, politics/religion, sports — that mirror the ground-truth taxonomy without ever seeing a label.
What I'd do next.
The biggest thing this project reinforced is that the interesting numbers are the ones that push back on the default narrative. Everyone assumes neural embeddings win. On the right benchmark, they don't — and being disciplined enough to run the fair fight is what surfaces that.
If I extended the pipeline, I'd add a fine-tuned transformer head as a third feature family so the comparison covers frozen embeddings, fine-tuned embeddings, and TF-IDF on equal footing. I'd swap flat KMeans for HDBSCAN so the topic tree emerges from the data, and wrap the whole thing in an evaluator that can be pointed at any labelled text dataset.