Ali Janati.
Research in deep learning.
I study how neural networks learn, organize knowledge, and generalize.
My research spans model architecture, optimization, training dynamics, and interpretability. I am especially interested in how a model’s internal structure gives rise to its capabilities, how those capabilities evolve during training, and how understanding these relationships can lead to better architectures and training methods, and to safer and better-aligned models.
I am a Senior Machine Learning Researcher at a startup in San Francisco, where I work on world models. I am an alumnus of Columbia and Mines Paris.
On this website, I share my papers, implementations, experimental studies, and articles, along with thoughts and ideas across my research and broader interests.
Recent
- Sep 2026 New article on a report I worked on: decoder-only Transformer optimization 101, from 163,645 to 3,217,492 tokens/s on one RTX 6000 Pro to teach a policy to play a game.
- Aug 2026 New articles: from-scratch reimplementations of Qwen3-Next-80B-A3B, gpt-oss-20B, Gemma 3 270M, and Llama 3.1 8B, plus speculative decoding across tokenizers.
- Aug 2026 New paper on arXiv: router sensitivity under lightweight fine-tuning identifies prunable experts in mixture-of-experts models.
- Aug 2026 New paper on arXiv: post-grokking collapse at the representation-readout interface in Muon-trained transformers.
- Feb 2026 New paper on arXiv: uncertainty-aware multimodal emotion recognition through Dirichlet parameterization.
- Jul 2024 Released Medical-Whisper-Large-v3 on Hugging Face: 35,000+ downloads.
§ 01
Papers
Papers.
Most of my work asks which parts of a large model are actually doing the work. Recent papers span expert pruning in mixture-of-experts models, training dynamics after grokking, uncertainty in multimodal recognition, and machine learning for epigenomics.
-
Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models
arXiv · Aug 2026
Routers that barely move under light fine-tuning point to experts that can be pruned with minimal loss in quality.
-
Post-Grokking Collapse at the Representation-Readout Interface in Muon-Trained Transformers
arXiv · Aug 2026
Muon reaches grokking faster than AdamW, but generalization later collapses at the interface between representations and readout. Fourier analysis shows the task-solving circuit itself remains accurate.
-
Uncertainty-Aware Multimodal Emotion Recognition through Dirichlet Parameterization
arXiv · Feb 2026
A lightweight, privacy-preserving emotion recognition framework for edge devices, fusing speech, text, and facial signals with Dirichlet-parameterized uncertainty. Validated on five benchmarks.
-
Improving Allele-Specific Epigenomic Signal Coverage by 10-Fold Using Hidden Markov Modeling and Machine Learning
bioRxiv · May 2024
A variational hidden Markov model that extends allele-specific methylation calling to ten times more of the genome than standard short-read methods.
Earlier research, with Prof. Pierre Gentine’s group at Columbia: fine-tuned the IBM-NASA geospatial vision transformer to segment wind-damaged forest in satellite imagery.
§ 02
Articles
Articles.
Write-ups on deep learning: architectures I reimplement from scratch in PyTorch-Playground, and training and inference work I take part in. What is new in each, how it was done, and the code that matters.
-
Sep 2026
Decoder-only Transformer optimization 101: 164K to 3.2M tokens/s on one GPU
A walk through a report I worked on: GEMM tiling and split-K, fusion and FlashAttention, buffer reuse, CUDA-Oxide kernels on two streams, and a smaller model that still beats the level-9 CPU.
-
Aug 2026
Qwen3-Next-80B-A3B from scratch: DeltaNet, gated attention, and 512 experts
The full hybrid stack in one PyTorch file: linear-attention DeltaNet blocks, gated attention with QK-norm, zero-centered RMSNorm, and a 512-expert MoE with top-10 routing.
-
Aug 2026
gpt-oss-20B: the attention pattern and the MoE, reimplemented
Alternating sliding-window and full attention as two masks, RoPE in the rotation-matrix convention, and a 32-expert MoE with gpt-oss's clamped SwiGLU.
-
Aug 2026
Gemma 3 270M, line by line
Dual RoPE frequencies, QK-norm, sandwich normalization, GeGLU, and fp32 attention accumulation in a model small enough to read whole.
-
Aug 2026
Llama 3.1 8B with a hand-written KV cache
Grouped-query attention by broadcasting instead of repeat_kv, RoPE with a position offset, and the twenty lines of caching behind fast decoding.
-
Aug 2026
Speculative decoding from scratch, including across tokenizers
The accept-reject rule and batched prefix acceptance, then Universal Assisted Generation between models that do not share a tokenizer.
§ 03
Open source
Open source.
-
From-scratch PyTorch reimplementations of modern architectures: Llama 3.1 8B, GPT-2, gpt-oss-20B, Qwen3-80B-A3B, Gemma 270M, and Magistral Small, plus speculative decoding and quantization-aware training.
-
Whisper large-v3 fine-tuned for medical conversation: 14% lower word-error rate on validation, released as open weights. 35,000+ downloads on Hugging Face.
-
Open artifacts behind the router-sensitivity paper: Mixtral-8x7B and Qwen1.5-MoE variants with experts pruned and ablated, along with the SFT datasets used to recover performance. The Mixtral series is curated as a collection; the full set lives on the profile.
-
The experiment suite behind the post-grokking-collapse paper: Muon and AdamW sweeps on modular arithmetic, freezing and unembedding-rate interventions at the representation-readout interface, and a generality suite across moduli, widths, and training configurations.
-
A CycleGAN that transfers Claude Monet’s painting style to photographs.