Efficient Learning

Resource-constrained learning

This thread asks how much data, supervision, communication, memory, computation, and physical interaction are necessary to achieve a desired level of learning performance. I study methods that treat efficiency as part of the learning problem rather than as post-processing: representations learned under memory constraints, distributed coding, sample selection, communication-efficient optimization, and evaluation under fixed model and rollout budgets. The long-term goal is to co-design statistical accuracy and resource allocation across training, inference, and self improvement.

Error against resource budget for two approaches: compressing a model afterward, and building the budget into the learning objective. Building it in gives lower error at the same budget, and the same error at a fraction of the budget. The budgets are data, communication, memory, and compute; the design choices are treating the budget as a constraint, selecting a representative subset, compressing representations, and allocating resources where the marginal gain is highest. — Scroll sideways to see the full diagram.

Key Questions

What are the fundamental tradeoffs among statistical accuracy, data, model size, communication, and inference cost?
How can compression, selection, and resource allocation be incorporated into learning rather than applied afterward?
How should self-improving systems allocate tokens, rollouts, judge calls, simulator interactions, and physical trials?

Selected Publications

SPLICE: Structured Prompt Local Iterative Combinatorial Evolution

NeurIPS 2026

Treats a production prompt as a set of blocks, some frozen and some editable, and optimizes it with section-local textual gradients and elitist beam search, giving the first formal guarantees for iterative LLM-driven prompt optimization. Outperforms prior prompt optimizers across benchmarks and three model capability tiers while keeping frozen sections verbatim.
RoPoLL: Robust Panel of LLM Judges

ICML 2026 · AgenticUQ Workshop

Shows that averaging an LLM jury is unboundedly biased once any judge fails systematically, and replaces the average with a geometric-median aggregator that has the optimal 1/2 breakdown point. A 3-judge 38B panel beats a 675B judge by 1.31x under 30% corruption.
Geometric Median Matching for Robust k-Subset Selection from Noisy Data

ICML 2025

Selects a k-subset whose mean tracks the geometric median of noisy data, giving an O(1/k) rate under arbitrary corruption, a quadratic improvement over random sampling.
Neural Distributed Source Coding

IEEE JSAIT 2024

Learns lossy distributed source coding with a conditional VQ-VAE, handling arbitrary correlation structure in high dimensions with state-of-the-art PSNR.
Faster Non-Convex Federated Learning via Global and Local Momentum

UAI 2022 · Spotlight

Combines variance-reduced momentum at the server and at the clients to speed up non-convex federated learning under client heterogeneity and compressed communication.
Online Embedding Compression for Text Classification using Low Rank Matrix Factorization

AAAI 2019 · Spotlight

Compresses the word-embedding layer during training with low-rank factorization, reaching 90% compression with minimal accuracy loss and no added latency.

Selected Patents

Natural Language Processing

US10872601 · granted 2020-12