Active project
Protein Language Models From First Principles
An experiment-driven series building protein language models from amino-acid bigrams toward a small masked Transformer, with homology-aware evaluation and reproducible local training.
- ROLE
- CreatorJul 2026 — Present
- STACK
- PythonPyTorchNumPyUniProtUniRef50MMseqs2ESMC-300Muvpytest
PROBLEM
Random protein splits can place related sequences on both sides of an evaluation and inflate apparent generalization. Before deeper models are worth training, the dataset boundary, baselines, and test firewall have to be defensible.
APPROACH
Start with 557,718 eligible Swiss-Prot proteins, compare immutable random and UniRef50-group assignments under a pinned MMseqs2 audit, then build progressively from matched unigram and bigram baselines while preserving checksummed evidence.
DEEP DIVE
Protein Language Models From First Principles is a twelve-week public learning and engineering series. It adapts a neural-network progression to amino-acid sequences, beginning with defensible evaluation data and simple baselines before moving toward a locally trained masked protein Transformer and a shared comparison with ESMC-300M.
Week 1 started before model training. Across 557,718 eligible Swiss-Prot proteins, the frozen UniRef50-group assignment reduced detected strong training-to-validation overlap from 87.4% to 35.9% under the same finite MMseqs2 search. The result was useful but not clean: the candidate still failed residue-balance tolerance and retained substantial overlap, so it was published as a negative result and prohibited from model use.
Week 2 moved to two newly approved model-data arms and compared unigram, count-based bigram, and one-layer neural bigram models. Both neural models used the same initialization, optimizer, batch policy, and 100-million-pair training budget. The prospective split-policy hypothesis was supported, while the neural models were effectively tied on shared validation and the count-based bigram slightly outperformed the neural version in both arms.
The project is currently complete through the Week 2 engineering release. All reported scores are validation results, the sealed test remains inaccessible, and Week 3 adds fixed-context MLPs and learned amino-acid embeddings using the prospectively frozen family-aware neural bigram as its baseline.
HIGHLIGHTS
Week 1: UniRef50 grouping lowered detected strong validation overlap from 87.4% to 35.9%, while still failing the frozen readiness gates
Published the negative result and prohibited both diagnostic assignments from model training
Week 2: evaluated unigram, count-based bigram, and neural bigram models across two matched 100M-pair training streams
The prospective split-policy hypothesis was supported; the neural models were effectively tied on shared validation
Kept the sealed test inaccessible and froze the family-aware neural bigram as the Week 3 baseline