Fantastic methods and where to find them: Automatic Differentiation

Derivatives

Derivatives

$$\frac{d}{dx}(log(x))$$

How many methods for taking derivatives by means of a computer?

(WIP)

Derivatives

• Symbolic differentiation

Derivatives

Derivatives

• Symbolic differentiation
• Numerical differentiation
• Algorithmic differentiation (automatic diff.)

Symbolic differentiation

• Accuracy (machine precision)
• Analytic expression (efficient)
• Complex expression (expression swell)
• Closed-form expression of $f(x)$
• Control flow (if, for, etc.)

Numeric differentiation

• Simple

Numeric differentiation

• Simple
• Control flow (if, for, etc.)

Numeric differentiation

• Simple
• Control flow (if, for, etc.)
• Round-off & truncation error

Numeric differentiation

• Simple
• Control flow (if, for, etc.)
• Round-off & truncation error
• Scales poorly (multivariate, higher order, number of parameters)

Automatic differentiation (AD)

• Accuracy (machine precision)
• Efficiency (less function evaluations, ...)
• Control flow (if, for, etc.)
• Software dependencies

Working principle (AD)

$$ y = f(x_1,x_2) = ln(x_1) + x_1\,x_2 - sin(x_2) $$

Evaluate at (2, 5)

Working principle (AD)

$$ y = f(x_1,x_2) = ln(x_1) + x_1\,x_2 - sin(x_2) $$

Evaluate at (2, 5) and compute $\frac{\partial y}{\partial x_1}$



$$\frac{\partial y(2,5)}{\partial x_1} = \frac{1}{x_1} + x_2 = 5.5 $$

Working principle (AD)

Function evaluation using dual numbers, ($v + v'\, \epsilon$)

$$(v + v'\, \epsilon) + (u + u'\, \epsilon) = (v + u) + (v' + u') \, \epsilon $$

$$(v + v'\, \epsilon) \, (u + u'\, \epsilon) = (v\,u) + (v\,u' + v'\,u) \, \epsilon $$

We enforce the following, (function definition),

$$f(v + v'\, \epsilon) = f(v) + f'(v)\,v'\, \epsilon $$

The chain rule emerges...

$$f(g(v + v'\, \epsilon)) = f(g(v)) + f'(g(v))\,g'(v)\,v'\, \epsilon $$

Types of AD

$$ f\, : \,{\rm I\!R}^n \rightarrow {\rm I\!R}^m$$

• Foward differentiation ($m \gg n$)
• Reverse differentiation ($n \gg m$)

Examples

Calibration of 1D-diffusion equation

→ Live session!

Examples

Calibration of 1D-diffusion equation

Examples

Calibration of my ED model (next seminar)

Six-times speed up!

Opportunities

• Fast calibration
• Fast sensitivity analysis
• Inclusion of ODE/PDE in neural networks

Opportunities

• Fast calibration
• Fast sensitivity analysis
• Inclusion of ODE/PDE in neural networks

Opportunities

• Fast calibration
• Fast sensitivity analysis
• Inclusion of ODE/PDE in neural networks

Opportunities

• Fast calibration
• Fast sensitivity analysis
• Inclusion of ODE/PDE in neural networks

Neural ODE

Fouling of electrodialysis stack

Neural ODE

Fouling of electrodialysis stack

$$\cfrac{\partial R(t)}{\partial t} = \mathrm{ANN}(R(t))$$

Neural ODE

Fouling of electrodialysis stack

References

1. Günes A, et al. (2018) Automatic Differentiation in Machine Learning: a Survey
2. JuliaDiff
3. DiffEqFlux.jl – A Julia Library for Neural Differential Equations
4. Reinforcement Learning vs. Differentiable Programming