October 14, 2009 : Barak A. Pearlmutter

Brain and Computation Lab, Hamilton Institute, National University of Ireland Maynooth

AXIS OF EVAL! AUTOMATIC DIFFERENTIATION mates with LAMBDA CALCULUS birthing MONSTER COMPILER Faster than Fortran

The technique known in the machine learning community as “backpropagation” is a special case of “reverse-mode accumulation automatic differentiation”, or “reverse AD”. We will explore forward and reverse AD using a novel formulation based on differential geometry. In this formulation, the AD operators naturally generalize to a much broader range of computer programs, including programs containing iterate-to-fixedpoint loops; invoking or embodying higher-order functions; invoking optimizers; or even themselves invoking AD operators. Algorithms including fast exact Hessian-vector multiplication, Pineda/Almeida fixedpoint backpropagation, and a wide variety of other techniques can be defined and implemented as one-liners using these generalized AD operators. These methods allow very complicated systems, like bi-level optimization architectures, to be built and optimized using gradient methods. This system has been formalized using the tools of modern Programming Language Theory, and a research prototype implementation has been constructed which exhibits startlingly good (i.e., FORTRAN-like) numeric performance.

(Joint work with Jeffrey Mark Siskind)

seminars/seminaritems/2009-10-14_october_14_2009_barak_a._pearlmutter.txt · Last modified: 2009/11/09 09:59 by koray