HippoTrainer: gradient-based hyperparameter optimization

Grid and random search treat the model as a black box. hippotrainer instead uses automatic differentiation to compute hypergradients and update hyperparameters with gradient steps, directly on torch.nn.Module models.

Algorithms

• T1-T2: one-step unrolled optimisation
• Neumann: implicit differentiation via a Neumann-series approximation of the inverse Hessian
• HOAG: implicit differentiation with conjugate gradient
• DrMAD: memory-efficient piecewise-linear backpropagation through the training trajectory

Checkpointing and implicit differentiation keep memory bounded, so the same code runs on a laptop and on a cluster. Built within the Bayesian Multimodeling course at MIPT.