Functional Gradient Descent with Adaptive Representations Accepted at NeurIPS

Researchers have introduced Functional Gradient Descent with Adaptive Representations, a new method accepted at NeurIPS. While functional GD algorithms generally outperform neural networks, they are difficult to implement accurately because functional gradients are infinite-dimensional. Naive approximations of these gradients often lead to convergence at incorrect locations.

The new work formalizes a broad class of approximation schemes called adaptive representations. These schemes are designed to be immediately implementable and provably ensure convergence to the global minimizer. In various settings, the resulting algorithms outperform corresponding neural networks often by an order of magnitude. The authors note that this is an early stage in this line of research but highlight its significant potential.