Hessians
Hessians are a crucial ingredient in NewtonSolvers and SimpleSolvers.NewtonOptimizerStates.
using SimpleSolvers
using LinearAlgebra: norm
x = rand(3)
obj = MultivariateObjective(x -> norm(x - vcat(0., 0., 1.)) ^ 2, x)
hes = HessianAutodiff(obj, x)HessianAutodiff{Float64, Main.var"#1#2", Matrix{Float64}, ForwardDiff.HessianConfig{ForwardDiff.Tag{Main.var"#1#2", Float64}, Float64, 3, Vector{ForwardDiff.Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}, ForwardDiff.Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}, Float64, 3}, 3}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}, Float64, 3}}}}(Main.var"#1#2"(), [NaN NaN NaN; NaN NaN NaN; NaN NaN NaN], ForwardDiff.HessianConfig{ForwardDiff.Tag{Main.var"#1#2", Float64}, Float64, 3, Vector{ForwardDiff.Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}, ForwardDiff.Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}, Float64, 3}, 3}}, Vector{ForwardDiff.Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}, Float64, 3}}}(ForwardDiff.JacobianConfig{ForwardDiff.Tag{Main.var"#1#2", Float64}, Float64, 3, Vector{ForwardDiff.Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}, Float64, 3}}}((Partials(1.0, 0.0, 0.0), Partials(0.0, 1.0, 0.0), Partials(0.0, 0.0, 1.0)), ForwardDiff.Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}, Float64, 3}[Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(6.91009383748774e-310,6.9100938375099e-310,6.9100938387051e-310,6.91009383886006e-310), Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(6.9100938397454e-310,6.91009383872725e-310,6.9100938388822e-310,6.9100938397233e-310), Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(6.9100938387494e-310,6.91009383890433e-310,6.91009383970116e-310,6.9100938387715e-310)]), ForwardDiff.GradientConfig{ForwardDiff.Tag{Main.var"#1#2", Float64}, ForwardDiff.Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}, Float64, 3}, 3, Vector{ForwardDiff.Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}, ForwardDiff.Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}, Float64, 3}, 3}}}((Partials(Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(1.0,0.0,0.0,0.0), Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(0.0,0.0,0.0,0.0), Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(0.0,0.0,0.0,0.0)), Partials(Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(0.0,0.0,0.0,0.0), Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(1.0,0.0,0.0,0.0), Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(0.0,0.0,0.0,0.0)), Partials(Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(0.0,0.0,0.0,0.0), Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(0.0,0.0,0.0,0.0), Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(1.0,0.0,0.0,0.0))), ForwardDiff.Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}, ForwardDiff.Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}, Float64, 3}, 3}[Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(6.9100929429459e-310,6.9100929429222e-310,6.91009294291114e-310,6.91009294292537e-310),Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(6.9100857900793e-310,6.9100929429159e-310,6.9100929429412e-310,6.91009294293485e-310),Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(6.9100929429491e-310,6.91009294295067e-310,6.91009294295225e-310,6.91009294295383e-310),Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(6.9100929429554e-310,6.910092942957e-310,6.9100929429475e-310,6.91009294299335e-310)), Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(6.91009294299493e-310,6.9100929429965e-310,6.9100929429981e-310,6.9100929429997e-310),Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(6.91005414854066e-310,6.9100929429712e-310,6.91009294303604e-310,6.91009294303604e-310),Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(6.91009382587325e-310,6.91009294303446e-310,6.91009294300126e-310,6.910092943006e-310),Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(6.9100929430329e-310,5.0e-324,6.9100929430076e-310,6.91009294300916e-310)), Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(6.91009294301074e-310,6.9100929430123e-310,6.9100929430139e-310,6.9100929430155e-310),Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(6.91009294300284e-310,6.9100929430376e-310,6.9100929430392e-310,6.9100929430408e-310),Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(6.91009294304236e-310,6.91009294304394e-310,6.91009294304553e-310,6.9100929430471e-310),Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}}(1.5e-323,2.5e-323,1.0e-323,1.0e-323))])))An instance of HessianAutodiff stores a Hessian matrix:
hes.H3×3 Matrix{Float64}:
NaN NaN NaN
NaN NaN NaN
NaN NaN NaNThe instance of HessianAutodiff can be called:
hes(x)3×3 Matrix{Float64}:
2.0 0.0 0.0
0.0 2.0 -1.11022e-16
5.55112e-17 1.11022e-16 2.0Or equivalently with:
update!(hes, x)This updates hes.H:
hes.H3×3 Matrix{Float64}:
2.0 0.0 0.0
0.0 2.0 -1.11022e-16
5.55112e-17 1.11022e-16 2.0BFGS Hessian
using SimpleSolvers: initialize!
hes = HessianBFGS(obj, x)
initialize!(hes, x)HessianBFGS{Float64, Vector{Float64}, Matrix{Float64}, MultivariateObjective{Float64, Vector{Float64}, Main.var"#1#2", GradientAutodiff{Float64, Main.var"#1#2", ForwardDiff.GradientConfig{ForwardDiff.Tag{Main.var"#1#2", Float64}, Float64, 3, Vector{ForwardDiff.Dual{ForwardDiff.Tag{Main.var"#1#2", Float64}, Float64, 3}}}}, Float64, Vector{Float64}}}(MultivariateObjective (for vector-valued quantities only the first component is printed):
f(x) = NaN
g(x)₁ = 3.82e-01
x_f₁ = NaN
x_g₁ = 1.91e-01
number of f calls = 0
number of g calls = 1
, [NaN, NaN, NaN], [0.19090669902576285, 0.5256623915420473, 0.3905882754313441], [NaN, NaN, NaN], [NaN, NaN, NaN], [0.3818133980515257, 1.0513247830840946, -1.2188234491373118], [NaN, NaN, NaN], [1.0 0.0 0.0; 0.0 1.0 0.0; 0.0 0.0 1.0], [0.0 0.0 0.0; 0.0 0.0 0.0; 0.0 0.0 0.0], [0.0 0.0 0.0; 0.0 0.0 0.0; 0.0 0.0 0.0], [0.0 0.0 0.0; 0.0 0.0 0.0; 0.0 0.0 0.0], [0.0 0.0 0.0; 0.0 0.0 0.0; 0.0 0.0 0.0], [0.0 0.0 0.0; 0.0 0.0 0.0; 0.0 0.0 0.0])For computational reasons we save the inverse of the Hessian, it can be accessed by calling inv:
inv(hes)3×3 Matrix{Float64}:
1.0 0.0 0.0
0.0 1.0 0.0
0.0 0.0 1.0Similarly to HessianAutodiff we can call update!:
update!(hes, x)