Dene Hoffman

ganesh: A New Optimization Crate for Rust

Dene Hoffman — Mon, 08 Sep 2025 00:00:00 +0000

For the last year or so, I’ve been working on a new optimization crate for Rust, called ganesh</code></a>. In this blog post, I’d like to introduce the project, give a quick overview of its features, a neat example of what it can do, and why you might use it instead of popular crates like argmin</code></a>.</p>

Current State-of-the-Art: argmin</code></h1>
I want to start by acknowledging argmin</code> and the amazing work that has been done on and with this crate. argmin</code> hosts a ton of really neat optimization algorithms, and it’s clear why its the recommended tool to use when you need to perform complex fitting and minimization problems in Rust. With that said, argmin</code> was not the most approachable crate to a new Rust user. I found that when I used it, I ran into the following issues:</p>

There seems to be a lot of setup, and it’s rather verbose. For example, here’s some code from the L-BFGS example file</a>:</li>
</ol>
let</span> cost =</span> Rosenbrock {</span></span>}</span></span>;</span>
</span>let</span> linesearch =</span> MoreThuenteLineSearch::</span></span>new(</span></span>)</span></span>;</span>
</span>let</span> solver =</span> LBFGS</span>::</span></span>new(</span>linesearch,</span> 7</span></span>)</span></span>;</span>
</span>let</span> res =</span> Executor::</span></span>new(</span>cost,</span> solver</span>)</span></span>
</span>    .</span>configure</span>(</span>|</span></span></span>state</span>|</span></span> </span>state.</span>param</span>(</span>array!</span>[</span>-</span>1.</span>2</span>,</span> 1.</span>0</span>]</span></span></span>)</span></span>.</span>max_iters</span>(</span>100</span></span>)</span></span></span></span>)</span></span>
</span>    .</span>add_observer</span>(</span>SlogLogger::</span></span>term(</span></span>)</span></span>,</span> ObserverMode::</span></span>Always</span>)</span></span>
</span>    .</span>run</span>(</span></span>)</span></span>?</span>;</span>
</span></code></pre>
I’ve removed some code here which was just intended to show how to modify settings on the line search algorithm, and some of this code could easily be simplified into fewer lines, but I’m more interested in what the user has to provide. For example, we could write the same code in Python with the scipy</code></a> library:</p>
from</span></span> scipy</span>.</span>optimize</span></span> import</span></span></span></span></span> minimize</span>,</span> OptimizeResult</span></span>
</span>def</span></span> cost</span></span></span>(</span></span>x</span>)</span></span> </span>-></span> float</span></span></span>:</span></span>
</span>    #</span> implementation of the Rosenbrock function
</span></span>def</span></span> logger</span></span></span>(</span></span>intermediate_result</span></span>:</span> OptimizeResult</span></span></span>)</span></span>:</span></span>
</span>    #</span> implementation of some logging code
</span></span>res</span></span> =</span> minimize</span></span></span></span>(</span>cost</span></span>,</span> [</span>-</span>1.</span>2</span>,</span> 1.</span>0</span>]</span></span>,</span> method</span>=</span>'</span></span></span>L-BFGS-B'</span></span></span>,</span> options</span>=</span>{</span></span>'</span></span></span>maxiter'</span></span></span></span>:</span></span> </span>100</span></span>}</span></span>)</span></span>, callback</span></span>=</span>logger</span></span>)</span>
</span></code></pre>
We should obviously not expect Rust to be less verbose than Python, but look specifically at what the user has to provide in the argmin</code> example. You first construct a linesearch</code>, of which there are three options. To someone who is not up to speed with optimization literature, this is the first stumbling block. Which line search do we choose? Let’s assume we know enough to at least look at these examples to see that the line search algorithm of Moré and Thuente is often preferred with the BFGS family of algorithms. Next, we pick a solver. Technically, if you leave out the method='L-BFGS-B</code> argument from the Python code, the BFGS</code> algorithm will be selected, not L-BFGS</code>, since that’s not even an option in scipy</code>. I actually prefer knowing the explicit algorithm being used, so I have no problem with this. However, what is that 7</code> doing there? The documentation calls it the history size, and indeed it’s what puts the L</code> in L-BFGS</code> (see my post here</a>). What value should go there? It’s not immediately obvious why 7</code> was chosen, nor what a good value might be for common optimization problems. While I agree the user should be able to set this value if they wish, shouldn’t they also have a sensible default[1]</a></sup>?</p>
Finally, we come to the part where we actually run the algorithm. The only issue I have here is what happens if you don’t actually assign an initial parameter vector. In such a case, run</code> will return an error, telling you that L-BFGS</code> requires a starting point. While this is true, it bothers me that the user is allowed to run the algorithm without a starting point at all.</p>
In ganesh</code>, the same minimization looks something like this:</p>
let</span> problem =</span> Rosenbrock {</span> n:</span> 2</span> </span>}</span></span>;</span> //</span> Rosenbrock function in two dimensions
</span></span>let</span> result =</span> LBFGSB</span>::</span></span>default(</span></span>)</span></span>.</span>process</span>(</span>
</span></span>    &</span>problem,</span>
</span></span>    &</span>(</span></span>)</span></span>,</span> //</span> the Rosenbrock function doesn't take any non-parameter arguments
</span></span></span>    LBFGSBConfig::</span></span>new(</span>[</span>-</span>1.</span>2</span>,</span> 1.</span>0</span>]</span></span></span>)</span></span>,</span>
</span></span>    LBFGSB</span>::</span></span>default_callbacks(</span></span>)</span></span>
</span></span>        .</span>with_observer</span>(</span>DebugObserver</span>)</span></span>
</span></span>        .</span>with_terminator</span>(</span>MaxSteps(</span>100</span></span>)</span></span></span>)</span></span>,</span>
</span></span></span>)</span></span>?</span>;</span>
</span></code></pre>
Everything used to configure the algorithm is in the LBFGSBConfig</code> struct, which can only be initialized by giving a starting position. The other nice feature is the ability to pass in non-parameter arguments to the problem</code>. This will come in handy in the example later.</p>


No default gradient is provided. It could be argued that this is more a matter of simplicity than a real issue with argmin</code>, but analytic gradients are often nontrivial or at the very least annoying to calculate and write down. The argmin</code> repository offers the finitediff</code></a> crate, which can be used fairly easily, but in my opinion, the user shouldn’t have to worry about providing a gradient at all unless they want to. ganesh</code> comes with default central finite difference gradients and Hessians which can be easily overwritten if desired.</p>
</li>

None of the (native) algorithms handle situations where parameters have bounds. If you want box bounds in argmin</code>, you’ll have to use an external algorithm like egobox</code></a> or cobyla</code></a>, the first of which is probably the most straightforward for box bounds. However, let’s consider again how scipy.minimize</code> handles this. Bounds are passed to scipy.minimize</code> and they work on the Nelder-Mead, L-BFGS-B, TNC, SLSQP, Powell, trust-constr, COBYLA, and COBYQA methods. The first thing you might notice there is that the Nelder-Mead algorithm in scipy</code> supports bounds! This is not the case for the argmin</code> implementation. The next algorithm, L-BFGS-B, explicitly supports bounds, as the second “B” implies. This algorithm is similar to L-BFGS and indeed reduces to L-BFGS when no bounds are given, so it tends to be used in place of L-BFGS. This is why ganesh</code> doesn’t even provide a BFGS or L-BFGS implementation (although I did write implementations in earlier versions). To add parameter bounds to the previous example, the user might write something like this:</p>
</li>
</ol>
let</span> problem =</span> Rosenbrock {</span> n:</span> 2</span> </span>}</span></span>;</span>
</span>let</span> result =</span> LBFGSB</span>::</span></span>default(</span></span>)</span></span>.</span>process</span>(</span>
</span></span>    &</span>problem,</span>
</span></span>    &</span>(</span></span>)</span></span>,</span>
</span></span>    LBFGSBConfig::</span></span>new(</span>[</span>-</span>1.</span>2</span>,</span> 1.</span>0</span>]</span></span></span>)</span></span>.</span>with_bounds</span>(</span>[</span>(</span>-</span>4.</span>0</span>,</span> 4.</span>0</span></span>)</span></span>,</span> (</span>-</span>4.</span>0</span>,</span> 4.</span>0</span></span>)</span></span>]</span></span></span>)</span></span>,</span>
</span></span>    LBFGSB</span>::</span></span>default_callbacks(</span></span>)</span></span>
</span></span>        .</span>with_observer</span>(</span>DebugObserver</span>)</span></span>
</span></span>        .</span>with_terminator</span>(</span>MaxSteps(</span>100</span></span>)</span></span></span>)</span></span>,</span>
</span></span></span>)</span></span>?</span>;</span>
</span></code></pre>
There are other algorithms which support bounds out of the box, but what if we want to use an algorithm which doesn’t? This problem has also been solved (sort of) by projects like LMFIT</code></a> and MINUIT</code></a>. To understand how they handle bounds, imagine the problem of fitting a Gaussian probability distribution to some data (imagine we set up a maximum likelihood problem). We might have two free parameters, a mean $\mu$ and a standard deviation $\sigma$:</p>
$$ f(\mu, \sigma) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left(-\frac{1}{2}\frac{(x-\mu)^2}{\sigma^2}\right). $$</p>
Now if we weren’t looking at this equation and just thought about the parameters, it wouldn’t make a lot of sense to have $\sigma < 0$. However, when we examine the problem, we find that this doesn’t actually lead to invalid solutions, since we always square $\sigma$ anyway: we could perform our fit and just use the absolute value of $\sigma$. Similarly, if we ever had a problem where a parameter $x$ came with the constraint $x >= 0$, we could always rewrite our problem with $x \to \sqrt{x^2}$. This does have the downside of giving two ambiguous solutions to the problem, and more if there are multiple parameters defined this way, but we can get around this by defining a set of internal parameters $x_\text{int}$ which span an unbounded space and a set of transformations to external parameters $x_\text{ext}$ which preserve the given boundaries. There are infinite ways to define such transformations, but the ones which LMFIT</code> and MINUIT</code> use are as follows:</p>
Upper and lower bounds:</p>
$$ x_\text{int} = \arcsin\left(2\frac{x_\text{ext} - x_\text{min}}{x_\text{max} - x_\text{min}} - 1\right) $$</p>
$$ x_\text{ext} = x_\text{min} + \left(\sin(x_\text{int}) + 1\right)\frac{x_\text{max} - x_\text{min}}{2} $$</p>
Upper bound only:</p>
$$ x_\text{int} = \sqrt{(x_\text{max} - x_\text{ext} + 1)^2 - 1} $$</p>
$$ x_\text{ext} = x_\text{max} + 1 - \sqrt{x_\text{int}^2 + 1} $$</p>
Lower bound only:</p>
$$ x_\text{int} = \sqrt{(x_\text{ext} - x_\text{min} + 1)^2 - 1} $$</p>
$$ x_\text{ext} = x_\text{min} - 1 + \sqrt{x_\text{int}^2 + 1} $$</p>
However, I’ve found these are actually quite difficult to work with at times. If you actually take the positive branch of the square roots, you’ll find that the transformations are not actually inverses of each other! For this reason, I have switched to using a different set of transformations:</p>
Upper and lower bounds:</p>
$$ x_\text{int} = \frac{u}{\sqrt{1 - u^2}} $$</p>
where</p>
$$ u = \frac{x_\text{ext} - c}{w},\ c = \frac{x_\text{min} + x_\text{max}}{2},\ w = \frac{x_\text{max} - x_\text{min}}{2} $$</p>
$$ x_\text{ext} = c + w \frac{x_\text{int}}{\sqrt{x_\text{int}^2 + 1}} $$</p>
Upper bound only:</p>
$$ x_\text{int} = \frac{1}{2}\left(\frac{1}{(x_\text{max} - x_\text{ext})} - (x_\text{max} - x_\text{ext}) \right) $$</p>
$$ x_\text{ext} = x_\text{max} - (\sqrt{x_\text{int}^2 + 1} - x_\text{int}) $$</p>
Lower bound only:</p>
$$ x_\text{int} = \frac{1}{2}\left((x_\text{ext} - x_\text{min}) - \frac{1}{(x_\text{ext} - x_\text{min})} \right) $$</p>
$$ x_\text{ext} = x_\text{min} + (\sqrt{x_\text{int}^2 + 1} + x_\text{int}) $$</p>
Now any algorithm which does not support bounds can simply operate on the internal parameters and the user can transform these into external parameters on which the function can be evaluated! ganesh</code> provides methods for adding such transformations to an algorithm, at least in cases where it makes sense. Unfortunately, this method does not come without drawbacks. The transformation functions themselves add to the evaluation time, especially when it comes to computing gradients or Hessians, but even worse, they transform linear problems into nonlinear ones! This also leads to issues with correctly calculating covariance matrices, and the MINUIT</code> documentation recommends running a fit without bounds after a bounded fit converges[2]</a></sup>.</p>
It should be noted that L-BFGS-B has been a desired algorithm by at least a couple of users of argmin</code> for</a> several</a> years</a> without any visible progress (it appears that the current maintainer is a bit busy, I know the feeling). If I have enough time, I might volunteer to port my own code to argmin</code>, and if any of the current contributors are reading this, they have my permission to do just that with attribution.</p>

	
		</i>Note</p>
	
All of the issues I have with argmin</code> are not reasons not to use it, and this blog post isn’t intended as a hit piece. I’m just highlighting the reasons I decided to write my own crate and why I think it is a good addition to the existing Rust optimization ecosystem! I believe that argmin</code> excels in its customizability, and ganesh</code> should be seen as an alternative approach which I believe is a bit more user-friendly with sensible defaults.</p>

</blockquote>
A New Optimization Crate: ganesh</code></h1>
I started working on ganesh</code> as part of my PhD in particle physics. For the uninitiated, the data collected from particle accelerators are distributed across many kinematic variables, such as the energy and momentum of particles measured by various detectors. These data follow mathematical patterns, functions which we call “amplitudes”. To determine the angular momentum (or spin) of some particle, we can show that its decay products follow spherical harmonic distributions, and we perform an “amplitude analysis” by fitting decay distributions to sets of spherical harmonics (this specific technique is often referred to as a “partial-wave analysis”). It is in this stage that ganesh</code> was born, as I was simultaneously developing a Rust library</a> which performs such analyses. At first I was content to provide functions which users could fit with their own optimization libraries, e.g. writing Python bindings and using scipy</code> or iminuit</code></a> to perform the fits, but I decided it would also be nice for users to just ask the same library to do the fit for them. This meant either wrapping existing optimization libraries in Python, or adding in an optimization crate in Rust. After experimenting with the first option, I decided to try the second (since I wanted this crate to be usable without Python), starting with argmin</code>. At first, this seemed to be a good direction, but the lack of parameter bounds in any of the native methods made it difficult to set up some minimizations. For example, the probability density of a relativistic Breit-Wigner (commonly used to model the mass of a decaying particle) contains a width parameter that sometimes does not appear in a square, like the Gaussian distribution I showed above. Of course, I could just do what MINUIT</code> does and wrap every bounded parameter in a nonlinear transformation, and indeed this is the approach I took with algorithms in ganesh</code> which don’t natively support bounds, but for BFGS-like solvers, this was just repeating the mistakes of MINUIT</code> by making users wary of the covariance of fit results. Another major reason for developing my own crate was to implement algorithms which argmin</code> isn’t really suited for, like Markov-chain Monte Carlo (MCMC). This isn’t really an optimization technique, but rather a technique for exploring a likelihood landscape. There are many ways to define the convergence of such an algorithm, but argmin</code> only allows downstream users to define Observer</code>s, which cannot terminate an algorithm (as far as I know).</p>
The design philosophy is that of sensible defaults rather than overwhelming choice, and the goal is to make it as simple as possible to get started with ganesh</code> without having to spend too much time learning the inner workings. I’ll admit that some of the above complaints could have been solved with some lengthy PRs to argmin</code>, but a lot of the design changes I wanted would involve large breaking changes, and I don’t see the harm in offering an alternative crate for those who are interested in using it (or hopefully developing it further).</p>
Features</h2>
One of the biggest features of ganesh</code> is the L-BFGS-B algorithm itself, as it is (as far as I know) the only pure-Rust implementation. While there are other crates which offer the algorithm, they all either bind to the original FORTRAN library or to a C/C++ library which either binds to that FORTRAN code or ports it directly</a>. I really wanted a solution free of dependencies on outside languages, so I used it as a learning opportunity, starting from BFGS and moving towards the limited-memory and bounded versions. I have since removed BFGS and L-BFGS, since they tend to be just as fast as L-BFGS-B anyway, so there doesn’t seem to be much to gain from keeping them around.</p>
As I alluded to earlier, custom termination conditions are another feature of ganesh</code>. In particular, any Terminator</code>-implementing struct which satisfies the right trait bounds can be used with any algorithm. While reading some of the original Nelder-Mead papers, I found that there are several termination conditions, some of which work better than others depending on the topology of the problem. I provided implementations of several different conditions, as well as reasonable defaults according to more modern papers, but the crate is designed to allow for future support of custom conditions without modifying core code.</p>
I also mentioned MCMC algorithms. In Python, I was very used to using both the Ensemble Slice Sampler (ESS) of zeus</a> and the Affine Invariant Markov chain Monte Carlo Ensemble Sampler (AIES) from emcee</a>. Both of these are fantastic libraries, and I decided to implement both as ganesh</code>’s first two MCMC algorithms. AIES is not yet entirely up-to-date with emcee</code>, and I have not gotten around to adding parallelization (though I plan to in the future[3]</a></sup>), but both are currently usable.</p>
The project recently gained its first contributor, estriv</a>, who has been behind much of the recent API update. He additionally added the current implementation of the simulated annealing algorithm, and has helped me outline how we want the crate to operate and the interface we want to provide to the end-user.</p>
Example</h2>
To show off ganesh</code>, I’ll implement a simple but fun example. Let’s imagine we have some dataset which we expect follows a multivariate normal distribution, and we want to fit a model to it. We’ll start by generating a toy dataset:</p>
use</span> fastrand::</span></span>Rng;</span>
</span>use</span> nalgebra::</span></span>{</span>DMatrix,</span> DVector</span>}</span></span>;</span>
</span>use</span> ganesh::</span></span>core::</span></span>utils::</span></span>SampleFloat;</span>
</span>
</span>fn</span> </span>generate_data</span></span>(</span>
</span></span></span>    n</span>:</span> usize</span>,
</span></span></span>    mu</span>:</span> &</span>DVector<</span>f64</span>></span></span>,
</span></span></span>    cov</span>:</span> &</span>DMatrix<</span>f64</span>></span></span>,
</span></span></span>    rng</span>:</span> &</span>mut</span> Rng,
</span></span></span></span>)</span></span></span></span> -></span> Vec</span><</span>DVector<</span>f64</span>></span></span>></span></span></span> </span>{</span>
</span></span></span>    (</span>0</span>..</span>n</span>)</span></span>.</span>map</span>(</span>|</span></span></span>_|</span></span> </span>rng.</span>mv_normal</span>(</span>mu,</span> cov</span>)</span></span></span></span>)</span></span>.</span>collect</span>(</span></span>)</span></span>
</span></span></span></span>}</span></span></span>
</span></code></pre>
ganesh</code> comes with a nice trait which extends fastrand</code></a>’s Rng</code></a> with some useful methods, like mv_normal</code>, which as the name suggests, generates data according to a multivariate normal distribution. Let’s also write some code to save this to a Python .pkl</code> file. I’m not very up to date on Rust-based plotting libraries, so I’m just going to use matplotlib</code> for now:</p>
use</span> std::</span></span>{</span>fs::</span></span>File,</span> io::</span></span>BufWriter,</span> error::</span></span>Error</span>}</span></span>;</span>
</span>
</span>fn</span> </span>write_data</span></span>(</span>data</span>:</span> &</span>[DVector<</span>f64</span>></span></span>], path</span>:</span> &</span>str</span></span>)</span></span></span></span> -></span> Result</span><</span>(</span>)</span>, Box</span><</span>dyn Error></span></span>></span></span></span> </span>{</span>
</span></span></span>    let</span> file =</span> File::</span></span>create(</span>path</span>)</span></span>?</span>;</span>
</span></span></span>    let</span> mut</span> writer =</span> BufWriter::</span></span>new(</span>file</span>)</span></span>;</span>
</span></span></span>    let</span> data:</span> Vec</span><</span>Vec</span><</span>f64</span>></span></span>></span></span> =</span> data.</span>iter</span>(</span></span>)</span></span>.</span>map</span>(</span>|</span></span></span>x</span>|</span></span> </span>x.</span>data.</span>as_vec</span>(</span></span>)</span></span></span></span>)</span></span>.</span>cloned</span>(</span></span>)</span></span>.</span>collect</span>(</span></span>)</span></span>;</span>
</span></span></span>    serde_pickle::</span></span>to_writer(</span>&</span>mut</span> writer,</span> &</span>data,</span> Default</span>::</span></span>default(</span></span>)</span></span></span>)</span></span>?</span>;</span>
</span></span></span>    Ok</span>(</span>(</span></span>)</span></span></span>)</span></span>
</span></span></span></span>}</span></span></span>
</span></code></pre>
Let’s run it:</p>
use</span> nalgebra::</span></span>{</span>dvector,</span> dmatrix</span>}</span></span>;</span>
</span>
</span>fn</span> </span>main</span></span>(</span></span>)</span></span></span></span> -></span> Result</span><</span>(</span>)</span>, Box</span><</span>dyn Error></span></span>></span></span></span> </span>{</span>
</span></span></span>    let</span> mut</span> rng =</span> Rng::</span></span>with_seed(</span>0</span></span>)</span></span>;</span>
</span></span></span>    let</span> mu_truth =</span> dvector!</span>[</span>1.</span>2</span>,</span> 2.</span>3</span>]</span></span>;</span>
</span></span></span>    let</span> sigma_truth =</span> dmatrix!</span>[</span>0.</span>6</span>,</span> 0.</span>5</span>;</span> 0.</span>5</span>,</span> 0.</span>7</span>]</span></span>;</span>
</span></span></span>    let</span> data =</span> generate_data</span>(</span>10_000</span>,</span> &</span>mu_truth,</span> &</span>sigma_truth,</span> &</span>mut</span> rng</span>)</span></span>;</span>
</span></span></span>    write_data</span>(</span>&</span>data,</span> "</span>data.pkl"</span></span></span>)</span></span>?</span>;</span>
</span></span></span></span>}</span></span></span>
</span></code></pre>
Let’s plot the result, nothing fancy:</p>
import</span> pickle</span></span></span>
</span>from</span></span> pathlib</span></span> import</span></span></span></span></span> Path</span></span>
</span>
</span>import</span> matplotlib</span>.</span>pyplot</span></span> as</span> plt</span></span></span>
</span>import</span> numpy</span></span> as</span> np</span></span></span>
</span>
</span>if</span> __name__</span></span> ==</span> '</span></span></span>__main__'</span></span></span>:</span></span>
</span>    data</span></span> =</span> np</span></span>.</span></span>array</span></span></span></span>(</span>pickle</span></span>.</span></span>load</span></span></span></span>(</span>Path</span></span>.</span></span>open</span></span></span></span>(</span>'</span></span></span>data.pkl'</span></span></span>,</span> '</span></span></span>rb'</span></span></span>)</span></span>)</span></span>)</span></span>.</span></span>transpose</span></span></span></span>(</span>)</span></span>
</span>    plt</span></span>.</span></span>hist2d</span></span></span></span>(</span>*</span>data</span></span>,</span> bins</span>=</span>100</span>,</span> cmap</span>=</span>'</span></span></span>gist_heat_r'</span></span></span>)</span></span>
</span>    plt</span></span>.</span></span>xlabel</span></span></span></span>(</span>'</span></span></span>x'</span></span></span>)</span></span>
</span>    plt</span></span>.</span></span>ylabel</span></span></span></span>(</span>'</span></span></span>y'</span></span></span>)</span></span>
</span>    plt</span></span>.</span></span>title</span></span></span></span>(</span>'</span></span></span>Sampled Dataset'</span></span></span>)</span></span>
</span>    plt</span></span>.</span></span>savefig</span></span></span></span>(</span>'</span></span></span>data.svg'</span></span></span>)</span></span>
</span>    plt</span></span>.</span></span>close</span></span></span></span>(</span>)</span></span>
</span>
</span></code></pre>
</p>
Now we should get on to fitting these data to a model. For this, we’ll first define the probability distribution function for a multivariate Gaussian:</p>
$$ \Pr(\vec{x}; \vec{\mu}, \mathbf{\Sigma}) = \exp\left[-\frac{1}{2}(\vec{x} - \vec{\mu})^{\intercal} \mathbf{\Sigma}^{-1} (\vec{x} - \vec{\mu})\right] (2\pi)^{-k/2}|\mathbf{\Sigma}|^{-1/2}, $$</p>
where $k$ is the dimension of the distribution. To actually perform the fit, we might instead want to maximize the likelihood function,</p>
$$ \mathcal{L}(\vec{X}; \vec{\mu}, \mathbf{\Sigma}) = \prod_{i=0}^{N} \Pr(\vec{X}_i; \vec{\mu, \mathbf{\Sigma}}), $$</p>
where $\vec{X}$ is our dataset of size $N$. For those familiar with this kind of problem, you’ll know what to do next, but for those who aren’t, note that the probability distribution function should give values between $0$ and $1$, so a product of a bunch of those numbers will tend towards a very small value, especially when you get more data. This is not very nice to deal with computationally, so we’ll instead take the natural logarithm of this likelihood function. Since the natural log is strictly increasing, this mapping will not change the value of the parameters at the function’s maximum, it will just rescale the problem from a range of $[0, 1]$ to a range of $(-\infty,0]$. Next, since most of our algorithms are intended for minimization rather than maximization, we will instead use $-2\ln(\mathcal{L})$, where the factor of $2$ is mostly convention and has to do with proper covariance scaling. This simplifies our function greatly:</p>
$$ -2\ln(\mathcal{L(\vec{X}; \vec{\mu}, \mathbf{\Sigma})}) = -\sum_{i=0}^{N} \left[(\vec{X}_i - \vec{\mu})^{\intercal} \mathbf{\Sigma}^{-1} (\vec{X}_i - \vec{\mu})\right] - N\left[k\ln(2\pi) + \ln|\mathbf{\Sigma}|\right]. $$</p>
Next, note that any symmetric positive-definite matrix may be decomposed by Cholesky factorization into $\mathbf{\Sigma} = \mathbf{L}\mathbf{L}^{\intercal}$ where $\mathbf{L}$ is lower triangular with $L_{ii}>0$. This means that the determinant in that last term can be calculated as $|\mathbf{\Sigma}| = |\mathbf{L}||\mathbf{L}^\intercal| = |\mathbf{L}|^2 = \prod_{j=0}^k L_{jj}$, so $\ln|\mathbf{\Sigma}| = 2\sum_{j=0}^k \ln L_{jj} $. We will also find it convenient to calculate the term inside the sum via Cholesky factorization, but the general form we want to reproduce is now:</p>
$$ -2\ln(\mathcal{L(\vec{X}; \vec{\mu}, \mathbf{\Sigma})}) = -\sum_{i=0}^{N} \left[(\vec{X}_i - \vec{\mu})^{\intercal} \mathbf{\Sigma}^{-1} (\vec{X}_i - \vec{\mu})\right] - N\left[k\ln(2\pi) + 2\sum_{j=0}^k \ln L_{jj} \right]. $$</p>
In code, this looks like,</p>
struct</span> </span>Problem</span></span>;</span>
</span>impl</span> </span>CostFunction<</span>Vec</span><</span>DVector<</span>f64</span>></span></span>></span></span>></span></span> for</span></span> Problem</span> </span>{</span>
</span></span></span>    fn</span> </span>evaluate</span></span>(</span>&</span>self</span>, x</span>:</span> &</span>DVector<</span>f64</span>></span></span>, args</span>:</span> &</span>Vec</span><</span>DVector<</span>f64</span>></span></span>></span></span></span>)</span></span></span></span> -></span> Result</span><</span>f64</span>, Infallible></span></span></span> </span>{</span>
</span></span></span></span></span>        let</span> mu =</span> Vector2::</span></span>new(</span>x[</span>0</span>]</span></span>,</span> x[</span>1</span>]</span></span></span>)</span></span>;</span>
</span></span></span></span></span>        let</span> sigma =</span> Matrix2::</span></span>new(</span>x[</span>2</span>]</span></span>,</span> x[</span>3</span>]</span></span>,</span> x[</span>3</span>]</span></span>,</span> x[</span>4</span>]</span></span></span>)</span></span>;</span>
</span></span></span></span></span>
</span></span></span></span></span>        let</span> chol =</span> sigma.</span>cholesky</span>(</span></span>)</span></span>.</span>unwrap</span>(</span></span>)</span></span>;</span> //</span> Note: this can panic!
</span></span></span></span></span></span>        let</span> l =</span> chol.</span>l</span>(</span></span>)</span></span>;</span>
</span></span></span></span></span>
</span></span></span></span></span>        let</span> ln_det =</span> 2.</span>0</span> *</span> l.</span>diagonal</span>(</span></span>)</span></span>.</span>iter</span>(</span></span>)</span></span>.</span>map</span>(</span>|</span></span></span>v</span>|</span></span> </span>v.</span>ln</span>(</span></span>)</span></span></span></span>)</span></span>.</span>sum::</span></span><</span>f64</span>></span></span>(</span></span>)</span></span>;</span>
</span></span></span></span></span>
</span></span></span></span></span>        let</span> n =</span> args.</span>len</span>(</span></span>)</span></span> as</span> f64</span>;</span>
</span></span></span></span></span>        let</span> quad_sum =</span> args
</span></span></span></span></span>            .</span>iter</span>(</span></span>)</span></span>
</span></span></span></span></span>            .</span>map</span>(</span>|</span></span></span>datum</span>|</span></span> </span>{</span>
</span></span></span></span></span></span></span></span>                let</span> d =</span> datum -</span> mu;</span>
</span></span></span></span></span></span></span></span>                d.</span>dot</span>(</span>&</span>chol.</span>solve</span>(</span>&</span>d</span>)</span></span></span>)</span></span>
</span></span></span></span></span></span></span></span>            </span>}</span></span></span></span>)</span></span>
</span></span></span></span></span>            .</span>sum::</span></span><</span>f64</span>></span></span>(</span></span>)</span></span>;</span>
</span></span></span></span></span>
</span></span></span></span></span>        Ok</span>(</span>(</span>n *</span> (</span>2.</span>0</span> *</span> (</span>2.</span>0</span> *</span> PI</span></span>)</span></span>.</span>ln</span>(</span></span>)</span></span> +</span> ln_det</span>)</span></span></span>)</span></span> +</span> quad_sum</span>)</span></span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span></span>}</span></span></span>
</span></code></pre>
Let’s perform the fit now, using the Nelder-Mead algorithm:</p>
let</span> res =</span> NelderMead::</span></span>default(</span></span>)</span></span>
</span>    .</span>process_default</span>(</span>
</span></span>        &</span>Problem,</span>
</span></span>        &</span>data,</span>
</span></span>        NelderMeadConfig::</span></span>new(</span>[</span>0.</span>5</span>,</span> 1.</span>0</span>,</span> 0.</span>7</span>,</span> 0.</span>1</span>,</span> 0.</span>7</span>]</span></span></span>)</span></span>,</span> //</span> a semi-random starting point
</span></span></span>    </span>)</span></span>?</span>
</span>    .</span>with_parameter_names</span>(</span>[</span>"</span>μ₀"</span></span>,</span> "</span>μ₁"</span></span>,</span> "</span>Σ₀₀"</span></span>,</span> "</span>Σ₀₁"</span></span>,</span> "</span>Σ₁₁"</span></span>]</span></span></span>)</span></span>;</span>
</span>println!</span>(</span></span>"</span>{}</span>"</span></span></span>,</span> res)</span></span>;</span>
</span></code></pre>
Now this will work, but mostly by coincidence. We haven’t actually done anything to ensure the algorithm sticks to covariance matrices which are positive-semidefinite. Just to be clear, there are a few problems if that happens to not be the case. First, The determinant will be negative and the square root of a negative number will be imaginary, making our probability distribution function imaginary. Second, the term in the exponent may end up being positive, which willl make it impossible to normalize the function over all reals. It just so happens that the starting point we chose constructs a simplex where every point gives a positive-semidefinite matrix and the algorithm happens to never traverse a position where this is not the case for any simplex point. We can easily break that by choosing a different starting point:</p>
let</span> res =</span> NelderMead::</span></span>default(</span></span>)</span></span>
</span>    .</span>process_default</span>(</span>
</span></span>        &</span>Problem,</span>
</span></span>        &</span>data,</span>
</span></span>        NelderMeadConfig::</span></span>new(</span>[</span>0.</span>5</span>,</span> 1.</span>0</span>,</span> 1.</span>0</span>,</span> 1.</span>0</span>,</span> 1.</span>0</span>]</span></span></span>)</span></span>,</span> //</span> a not-so-random starting point
</span></span></span>    </span>)</span></span>?</span>
</span>    .</span>with_parameter_names</span>(</span>[</span>"</span>μ₀"</span></span>,</span> "</span>μ₁"</span></span>,</span> "</span>Σ₀₀"</span></span>,</span> "</span>Σ₀₁"</span></span>,</span> "</span>Σ₁₁"</span></span>]</span></span></span>)</span></span>;</span>
</span>println!</span>(</span></span>"</span>{}</span>"</span></span></span>,</span> res)</span></span>;</span>
</span></code></pre>
The default simplex construction method for a five-dimensional problem takes the given starting point and constructs five other points by scaling each coordinate by $1.05$, one at a time. This means that one of those starting points is</p>
$$ \begin{bmatrix} 1.0 & 1.05 \\ 1.05 & 1.0 \end{bmatrix}, $$</p>
which is not positive-semidefinite! And of course, even if we pick a nice starting point, there is no guarantee that we never hit such a point somewhere down the line, causing sigma.cholesky().unwrap()</code> to panic. While we might just try to be careful about the first failure point in picking a good starting simplex, there is a neat way to get around the second issue: a change of variables. As it turns out, if instead operate on values of the lower-triangular matrix $\mathbf{L}$ described in the Cholesky decomposition, the only requirement we have is that the diagonal entries of the matrix are positive, which can be handled with some sort of constraint or a further change of variables. ganesh</code> comes with an interface for creating these change-of-variables transformations, and we can simply construct a transformation that allows our algorithm to operate in more convenient space, such as the the space of lower-triangular matrices $\mathbf{L}$ such that $\mathbf{\Sigma} = \mathbf{L}\mathbf{L}^{\intercal}$. Any such lower-triangular matrix will work, provided the diagonal entries are positive (and nonzero), a property we can also ensure with a built-in transformation:</p>
#</span>[</span>derive</span>(</span></span></span>Clone</span></span>)</span></span></span>]</span></span>
</span>struct</span> </span>SPDTransform</span> </span>{</span>
</span></span></span>    i_00</span>:</span> usize</span>,
</span></span></span>    i_01</span>:</span> usize</span>,
</span></span></span>    i_11</span>:</span> usize</span>,
</span></span></span></span>}</span></span></span>
</span>impl</span> </span>Transform for</span></span> SPDTransform</span> </span>{</span>
</span></span></span>    fn</span> </span>to_external</span></span><</span>'a</span>></span></span>(</span>&</span>'a</span> self</span>, z</span>:</span> &</span>'a</span> DVector<</span>f64</span>></span></span></span>)</span></span></span></span> -></span> Cow<</span>'a</span>, DVector<</span>f64</span>></span></span>></span></span></span> </span>{</span>
</span></span></span></span></span>        let</span> (</span>l00,</span> l01,</span> l11</span>)</span></span> =</span> (</span>z[</span>self</span>.</span>i_00]</span></span>,</span> z[</span>self</span>.</span>i_01]</span></span>,</span> z[</span>self</span>.</span>i_11]</span></span></span>)</span></span>;</span>
</span></span></span></span></span>        let</span> (</span>s00,</span> s01,</span> s11</span>)</span></span> =</span> (</span>l00 *</span> l00,</span> l00 *</span> l01,</span> l01 *</span> l01 +</span> l11 *</span> l11</span>)</span></span>;</span>
</span></span></span></span></span>        let</span> mut</span> x =</span> z.</span>clone</span>(</span></span>)</span></span>;</span>
</span></span></span></span></span>        x[</span>self</span>.</span>i_00]</span></span> =</span> s00;</span>
</span></span></span></span></span>        x[</span>self</span>.</span>i_01]</span></span> =</span> s01;</span>
</span></span></span></span></span>        x[</span>self</span>.</span>i_11]</span></span> =</span> s11;</span>
</span></span></span></span></span>        Cow::</span></span>Owned(</span>x</span>)</span></span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span>
</span></span></span>    fn</span> </span>to_internal</span></span><</span>'a</span>></span></span>(</span>&</span>'a</span> self</span>, x</span>:</span> &</span>'a</span> DVector<</span>f64</span>></span></span></span>)</span></span></span></span> -></span> Cow<</span>'a</span>, DVector<</span>f64</span>></span></span>></span></span></span> </span>{</span>
</span></span></span></span></span>        let</span> (</span>s00,</span> s01,</span> s11</span>)</span></span> =</span> (</span>x[</span>self</span>.</span>i_00]</span></span>,</span> x[</span>self</span>.</span>i_01]</span></span>,</span> x[</span>self</span>.</span>i_11]</span></span></span>)</span></span>;</span>
</span></span></span></span></span>        let</span> l00 =</span> s00.</span>sqrt</span>(</span></span>)</span></span>;</span>
</span></span></span></span></span>        let</span> l01 =</span> s01 /</span> l00;</span>
</span></span></span></span></span>        let</span> l11 =</span> (</span>s11 -</span> l01 *</span> l01</span>)</span></span>.</span>sqrt</span>(</span></span>)</span></span>;</span>
</span></span></span></span></span>        let</span> mut</span> z =</span> x.</span>clone</span>(</span></span>)</span></span>;</span>
</span></span></span></span></span>        z[</span>self</span>.</span>i_00]</span></span> =</span> l00;</span>
</span></span></span></span></span>        z[</span>self</span>.</span>i_01]</span></span> =</span> l01;</span>
</span></span></span></span></span>        z[</span>self</span>.</span>i_11]</span></span> =</span> l11;</span>
</span></span></span></span></span>        Cow::</span></span>Owned(</span>z</span>)</span></span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span>
</span></span></span>    fn</span> </span>to_external_jacobian</span></span>(</span>&</span>self</span>, z</span>:</span> &</span>DVector<</span>f64</span>></span></span></span>)</span></span></span></span> -></span> DMatrix<</span>f64</span>></span></span></span> </span>{</span>
</span></span></span></span></span>        let</span> n =</span> z.</span>len</span>(</span></span>)</span></span>;</span>
</span></span></span></span></span>        let</span> (</span>l00,</span> l01,</span> l11</span>)</span></span> =</span> (</span>z[</span>self</span>.</span>i_00]</span></span>,</span> z[</span>self</span>.</span>i_01]</span></span>,</span> z[</span>self</span>.</span>i_11]</span></span></span>)</span></span>;</span>
</span></span></span></span></span>        let</span> mut</span> j =</span> DMatrix::</span></span>identity(</span>n,</span> n</span>)</span></span>;</span>
</span></span></span></span></span>        j[</span>(</span>self</span>.</span>i_00,</span> self</span>.</span>i_00</span>)</span></span>]</span></span> =</span> 2.</span>0</span> *</span> l00;</span>
</span></span></span></span></span>        j[</span>(</span>self</span>.</span>i_01,</span> self</span>.</span>i_00</span>)</span></span>]</span></span> =</span> l01;</span>
</span></span></span></span></span>        j[</span>(</span>self</span>.</span>i_01,</span> self</span>.</span>i_01</span>)</span></span>]</span></span> =</span> l00;</span>
</span></span></span></span></span>        j[</span>(</span>self</span>.</span>i_11,</span> self</span>.</span>i_01</span>)</span></span>]</span></span> =</span> 2.</span>0</span> *</span> l01;</span>
</span></span></span></span></span>        j[</span>(</span>self</span>.</span>i_11,</span> self</span>.</span>i_11</span>)</span></span>]</span></span> =</span> 2.</span>0</span> *</span> l11;</span>
</span></span></span></span></span>        j
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span>
</span></span></span>    fn</span> </span>to_external_component_hessian</span></span>(</span>&</span>self</span>, a</span>:</span> usize</span>, z</span>:</span> &</span>DVector<</span>f64</span>></span></span></span>)</span></span></span></span> -></span> DMatrix<</span>f64</span>></span></span></span> </span>{</span>
</span></span></span></span></span>        let</span> n =</span> z.</span>len</span>(</span></span>)</span></span>;</span>
</span></span></span></span></span>        let</span> mut</span> h =</span> DMatrix::</span></span>zeros(</span>n,</span> n</span>)</span></span>;</span>
</span></span></span></span></span>        if</span> a ==</span> self</span>.</span>i_00 {</span>
</span></span></span></span></span></span>            h[</span>(</span>self</span>.</span>i_00,</span> self</span>.</span>i_00</span>)</span></span>]</span></span> =</span> 2.</span>0</span>;</span>
</span></span></span></span></span></span>        </span>}</span></span> else</span> if</span> a ==</span> self</span>.</span>i_01 {</span>
</span></span></span></span></span></span>            h[</span>(</span>self</span>.</span>i_00,</span> self</span>.</span>i_01</span>)</span></span>]</span></span> =</span> 1.</span>0</span>;</span>
</span></span></span></span></span></span>            h[</span>(</span>self</span>.</span>i_01,</span> self</span>.</span>i_00</span>)</span></span>]</span></span> =</span> 1.</span>0</span>;</span>
</span></span></span></span></span></span>        </span>}</span></span> else</span> if</span> a ==</span> self</span>.</span>i_11 {</span>
</span></span></span></span></span></span>            h[</span>(</span>self</span>.</span>i_01,</span> self</span>.</span>i_01</span>)</span></span>]</span></span> =</span> 2.</span>0</span>;</span>
</span></span></span></span></span></span>            h[</span>(</span>self</span>.</span>i_11,</span> self</span>.</span>i_11</span>)</span></span>]</span></span> =</span> 2.</span>0</span>;</span>
</span></span></span></span></span></span>        </span>}</span></span>
</span></span></span></span></span>        h
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span></span>}</span></span></span>
</span></code></pre>
let</span> spd_transform =</span> SPDTransform {</span>
</span></span>    i_00:</span> 2</span>,</span>
</span></span>    i_01:</span> 3</span>,</span>
</span></span>    i_11:</span> 4</span>,</span>
</span></span></span>}</span></span>;</span>
</span>let</span> internal_bounds =</span> Bounds::</span></span>from(</span>[</span>
</span></span></span>    (</span>None</span>,</span> None</span></span>)</span></span>,</span>
</span></span></span>    (</span>None</span>,</span> None</span></span>)</span></span>,</span>
</span></span></span>    (</span>Some</span>(</span>0.</span>0</span></span>)</span></span>,</span> None</span></span>)</span></span>,</span>
</span></span></span>    (</span>None</span>,</span> None</span></span>)</span></span>,</span>
</span></span></span>    (</span>Some</span>(</span>0.</span>0</span></span>)</span></span>,</span> None</span></span>)</span></span>,</span>
</span></span></span>]</span></span></span>)</span></span>;</span>
</span>let</span> transform =</span> internal_bounds.</span>compose</span>(</span>spd_transform</span>)</span></span>;</span>
</span>//</span> compositions are defined with the convention of going from the unconstrained space of the algorithm
</span></span>//</span> to the constrained space of the problem. We first lower-bound the diagonal entries of L, and then we
</span></span>//</span> transform that into Σ.
</span></span>let</span> res =</span> NelderMead::</span></span>default(</span></span>)</span></span>
</span>    .</span>process_default</span>(</span>
</span></span>        &</span>Problem,</span>
</span></span>        &</span>data,</span>
</span></span>        NelderMeadConfig::</span></span>new(</span>[</span>0.</span>5</span>,</span> 1.</span>0</span>,</span> 0.</span>7</span>,</span> 0.</span>1</span>,</span> 0.</span>7</span>]</span></span></span>)</span></span>.</span>with_transform</span>(</span>transform.</span>clone</span>(</span></span>)</span></span></span>)</span></span>,</span>
</span></span>    </span>)</span></span>?</span>
</span>    .</span>with_parameter_names</span>(</span>[</span>"</span>μ₀"</span></span>,</span> "</span>μ₁"</span></span>,</span> "</span>Σ₀₀"</span></span>,</span> "</span>Σ₀₁"</span></span>,</span> "</span>Σ₁₁"</span></span>]</span></span></span>)</span></span>;</span>
</span>println!</span>(</span></span>"</span>{}</span>"</span></span></span>,</span> res)</span></span>;</span>
</span></code></pre>
Note that this technically doesn’t allow for matrices which are positive-semidefinite but not positive-definite, but such matrices lead to degenerate Gaussians which we will assume are not of interest for our purposes. This also won’t get us past the posibility that the construction of the simplex might fail, but at least it will panic with a nice message in that case.</p>
If we run this code, we get the result,</p>
$$ \vec{\mu} = \begin{bmatrix} 1.1958 \\ 2.3009 \end{bmatrix}, \quad \mathbf{\Sigma} = \begin{bmatrix} 0.5955 & 0.4896 \\ 0.4896 & 0.6814 \end{bmatrix}, $$</p>
which is very close to our truth values of</p>
$$ \vec{\mu} = \begin{bmatrix} 1.2000 \\ 2.3000 \end{bmatrix}, \quad \mathbf{\Sigma} = \begin{bmatrix} 0.6000 & 0.5000 \\ 0.5000 & 0.7000 \end{bmatrix}. $$</p>
The question now becomes, how close? We didn’t exactly calculate the uncertainties of the fit. Since Nelder-Mead is a gradient-free method, we didn’t even define a way to calculate the Hessian. We could calculate the Hessian manually, but instead let’s see how transformed algorithms can do this automatically. All we have to do is swap out our Nelder-Mead algorithm with a gradient-based algorithm like L-BFGS-B:</p>
impl</span> </span>Gradient<</span>Vec</span><</span>DVector<</span>f64</span>></span></span>></span></span>></span></span> for</span></span> Problem</span> </span>{</span></span>}</span></span></span>
</span>//</span> ^ this automatically implements a central finite-difference gradient and Hessian
</span></span>//</span> and may be overwritten with analytical gradients if desired.
</span></span>let</span> res =</span> LBFGSB</span>::</span></span>default(</span></span>)</span></span>
</span>    .</span>process_default</span>(</span>
</span></span>        &</span>Problem,</span>
</span></span>        &</span>data,</span>
</span></span>        LBFGSBConfig::</span></span>new(</span>[</span>0.</span>5</span>,</span> 1.</span>0</span>,</span> 0.</span>7</span>,</span> 0.</span>1</span>,</span> 0.</span>7</span>]</span></span></span>)</span></span>.</span>with_transform</span>(</span>&</span>transform</span>)</span></span>,</span>
</span></span>    </span>)</span></span>?</span>
</span>    .</span>with_parameter_names</span>(</span>[</span>"</span>μ₀"</span></span>,</span> "</span>μ₁"</span></span>,</span> "</span>Σ₀₀"</span></span>,</span> "</span>Σ₀₁"</span></span>,</span> "</span>Σ₁₁"</span></span>]</span></span></span>)</span></span>;</span>
</span>println!</span>(</span></span>"</span>{}</span>"</span></span></span>,</span> res)</span></span>;</span>
</span></code></pre>
This time I’ll just show you what that final print statement gives, since it’s a bit more informative now:</p>
╭──────────────────────────────────────────────────────────────────╮
</span>│                                                                  │
</span>│                           FIT RESULTS                            │
</span>│                                                                  │
</span>├───────────┬───────────────────┬────────────────┬─────────────────┤
</span>│ Status    │ f(x)              │ #f(x)          │ #∇f(x)          │
</span>├───────────┼───────────────────┼────────────────┼─────────────────┤
</span>│ Converged │ 38799.41429       │ 77             │ 39              │
</span>├───────────┼───────────────────┴────────────────┴─────────────────┤
</span>│           │                                                      │
</span>│ Message   │ F_EVAL CONVERGED                                     │
</span>│           │                                                      │
</span>├───────────┴─────────────────────────────┬────────────┬───────────┤
</span>│ Parameter                               │ Bound      │ At Limit? │
</span>├───────────┬─────────┬─────────┬─────────┼──────┬─────┼───────────┤
</span>│           │ =       │ σ       │ 0       │ -    │ +   │           │
</span>├───────────┼─────────┼─────────┼─────────┼──────┼─────┼───────────┤
</span>│ μ₀        │ 1.19588 │ 0.00546 │ 0.50000 │ -inf │ inf │ No        │
</span>│ μ₁        │ 2.30094 │ 0.00584 │ 1.00000 │ -inf │ inf │ No        │
</span>│ Σ₀₀       │ 0.59548 │ 0.00595 │ 0.70000 │ -inf │ inf │ No        │
</span>│ Σ₀₁       │ 0.48965 │ 0.00568 │ 0.10000 │ -inf │ inf │ No        │
</span>│ Σ₁₁       │ 0.68139 │ 0.00681 │ 0.70000 │ -inf │ inf │ No        │
</span>╰───────────┴─────────┴─────────┴─────────┴──────┴─────┴───────────╯
</span></code></pre>
Here we have some information about our fit, including the negative log-likelihood at the minimum, the number of times the function and its gradient are called, a message about how the algorithm converged, and a list of the parameters with their values at termination, the uncertainties on those values, the initial values, and information about bounds on those parameters (none here, this doesn’t include bounds we placed using internal tranformations, just bounds on the external parameters via the algorithm itself). As you can see, this is nearly the same minimum as the one obtained via Nelder-Mead up to the first three decimal places.</p>
There is another way we can get parameter errors, and that’s via MCMC. As mentioned, ganesh</code> currently supports two major MCMC algorithms, AIES and ESS. These are technically very different algorithms, but in practice they both work well for most “normal” problems. I tend to like ESS because it has a guaranteed 100% step acceptance rate by construction, although more work goes into proposing a step in a chain. Both methods are ensemble samplers, so unlike a basic algorithm like Metropolis-Hastings, the next position of each walker depends on the positions of all walkers in the ensemble. This makes both of them better at problems where parameters exist on very different overall scales. I won’t go through the details here, but you can read about them here^{[4]</a></sup> and here[5]</a></sup>. For this problem, we’ll use the ESS algorithm. To determine when to stop the sampler, we could set a maximum number of steps, but for this example, we’ll use a different metric, the autocorrelation time, $\tau$. Autocorrelation times measure how correlated a walker’s chain of positions is with its past positions. Since we will chose an arbitrary starting distribution for our walkers, it is important that our final distributions do not depend on this initial distribution, so the autocorrelation time tells us how many steps we need to get rid of to satisfy this condition. Given that, we can use this as a stopping criterion. We will simply calculate $\tau$ every few steps and see if we have taken more than $M \tau$ steps total, where $M$ is at least $1$. We can then decide to burn (get rid of) the first $M_b \tau$ steps where $M_b < M$. Additionally, since we are not checking $\tau$ every step (we could, but it’s not efficient), we also require the change in $\tau$ to be less than some threshold, ensuring that enough steps have been taken to effectively measure $\tau$. We also discard some fraction of steps at the beginning of the chain in this calculation, since we not only care about correlations with the initial position, we also care about correlations with any position the walker encountered sufficiently long ago. All of this logic is built into a ganesh</code> Terminator</code>, which provides useful defaults but is also fully customizable. The MCMC code looks something like this:</p>}
impl</span> </span>LogDensity<</span>Vec</span><</span>DVector<</span>f64</span>></span></span>></span></span>></span></span> for</span></span> Problem</span> </span>{</span>
</span></span></span>    fn</span> </span>log_density</span></span>(</span>&</span>self</span>, x</span>:</span> &</span>DVector<</span>f64</span>></span></span>, args</span>:</span> &</span>Vec</span><</span>DVector<</span>f64</span>></span></span>></span></span></span>)</span></span></span></span> -></span> Result</span><</span>f64</span>, Infallible></span></span></span> </span>{</span>
</span></span></span></span></span>        Ok</span>(</span>-</span>self</span>.</span>evaluate</span>(</span>x,</span> args</span>)</span></span>?</span></span>)</span></span> //</span> MCMC algorithms maximize rather than minimize!
</span></span></span></span></span></span>    </span>}</span></span></span>
</span></span></span></span>}</span></span></span>
</span>let</span> mut</span> rng =</span> Rng::</span></span>with_seed(</span>0</span></span>)</span></span>;</span>
</span>//</span> We wrap this in an Arc or Rc so we can see the value of τ later.
</span></span>//</span> We need a Mutex or RwLock because the values of τ are written to the terminator instance.
</span></span>let</span> aco =</span> Arc::</span></span>new(</span>Mutex::</span></span>new(</span>AutocorrelationTerminator::</span></span>default(</span></span>)</span></span></span>)</span></span></span>)</span></span>;</span>
</span>//</span> To initialize the walkers, we will use a normal distribution around the minimum we found earlier.
</span></span>//</span> We will first transform those coordinates to the internal representation, generate the distribution,
</span></span>//</span> and transform back to external coordinates to guarantee that each walker starts with a valid (positive-definite)
</span></span>//</span> covariance matrix.
</span></span>let</span> x_min_int =</span> transform.</span>to_internal</span>(</span>&</span>res.</span>x</span>)</span></span>;</span>
</span>let</span> n_walkers =</span> 100</span>;</span>
</span>let</span> walkers =</span> Vec</span>::</span></span>from_iter(</span>(</span>0</span>..</span>n_walkers</span>)</span></span>.</span>map</span>(</span>|</span></span></span>_|</span></span> </span>{</span>
</span></span></span></span></span>    transform.</span>into_external</span>(</span>&</span>DVector::</span></span>from_fn(</span>5</span>,</span> |</span></span></span>i</span>,</span> _|</span></span> </span>rng.</span>normal</span>(</span>x_min_int[</span>i]</span></span>,</span> 0.</span>2</span></span>)</span></span></span></span>)</span></span></span>)</span></span>
</span></span></span></span></span></span>}</span></span></span></span>)</span></span></span>)</span></span>;</span>
</span>
</span>let</span> sample =</span> ESS</span>::</span></span>default(</span></span>)</span></span>.</span>process</span>(</span>
</span></span>    &</span>Problem,</span>
</span></span>    &</span>data,</span>
</span></span>    ESSConfig::</span></span>new(</span>walkers</span>)</span></span>
</span></span>    .</span>with_transform</span>(</span>transform</span>)</span></span>,</span>
</span></span>    ESS</span>::</span></span>default_callbacks(</span></span>)</span></span>.</span>with_terminator</span>(</span>aco.</span>clone</span>(</span></span>)</span></span></span>)</span></span>,</span>
</span></span></span>)</span></span>?</span>;</span>
</span>
</span>//</span> Let's burn the first 10τ steps
</span></span>let</span> burn =</span> (</span>aco.</span>lock</span>(</span></span>)</span></span>.</span>taus.</span>last</span>(</span></span>)</span></span>.</span>unwrap</span>(</span></span>)</span></span> *</span> 10.</span>0</span></span>)</span></span> as</span> usize</span>;</span>
</span>//</span> The parameters here are "burn" and "thin", where the second will get rid of every nth
</span></span>//</span> step in the chain after burning the beginning.
</span></span>let</span> chain:</span> Vec</span><</span>Vec</span><</span>DVector<</span>f64</span>></span></span>></span></span>></span></span> =</span> sample.</span>get_chain</span>(</span>None</span>,</span> None</span></span>)</span></span>;</span>
</span>let</span> flat_chain:</span> Vec</span><</span>DVector<</span>f64</span>></span></span>></span></span> =</span> sample.</span>get_flat_chain</span>(</span>Some</span>(</span>burn</span>)</span></span>,</span> None</span></span>)</span></span>;</span>
</span></code></pre>
To send this data to Python for plotting, I wrote some additional functions to store the fit result and nested vectors:</p>
fn</span> </span>write_fit_result</span></span>(</span>data</span>:</span> &</span>DVector<</span>f64</span>></span></span>, path</span>:</span> &</span>str</span></span>)</span></span></span></span> -></span> Result</span><</span>(</span>)</span>, Box</span><</span>dyn Error></span></span>></span></span></span> </span>{</span>
</span></span></span>    let</span> file =</span> File::</span></span>create(</span>path</span>)</span></span>?</span>;</span>
</span></span></span>    let</span> mut</span> writer =</span> BufWriter::</span></span>new(</span>file</span>)</span></span>;</span>
</span></span></span>    serde_pickle::</span></span>to_writer(</span>&</span>mut</span> writer,</span> &</span>data.</span>data.</span>as_vec</span>(</span></span>)</span></span>,</span> Default</span>::</span></span>default(</span></span>)</span></span></span>)</span></span>?</span>;</span>
</span></span></span>    Ok</span>(</span>(</span></span>)</span></span></span>)</span></span>
</span></span></span></span>}</span></span></span>
</span>fn</span> </span>write_data_chain</span></span>(</span>
</span></span></span>    data</span>:</span> &</span>[Vec</span><</span>DVector<</span>f64</span>></span></span>></span></span>],
</span></span></span>    burn</span>:</span> usize</span>,
</span></span></span>    path</span>:</span> &</span>str</span>,
</span></span></span></span>)</span></span></span></span> -></span> Result</span><</span>(</span>)</span>, Box</span><</span>dyn Error></span></span>></span></span></span> </span>{</span>
</span></span></span>    let</span> file =</span> File::</span></span>create(</span>path</span>)</span></span>?</span>;</span>
</span></span></span>    let</span> mut</span> writer =</span> BufWriter::</span></span>new(</span>file</span>)</span></span>;</span>
</span></span></span>    let</span> data:</span> Vec</span><</span>Vec</span><</span>Vec</span><</span>f64</span>></span></span>></span></span>></span></span> =</span> data
</span></span></span>        .</span>iter</span>(</span></span>)</span></span>
</span></span></span>        .</span>map</span>(</span>|</span></span></span>w</span>|</span></span> </span>w.</span>iter</span>(</span></span>)</span></span>.</span>map</span>(</span>|</span></span></span>x</span>|</span></span> </span>x.</span>data.</span>as_vec</span>(</span></span>)</span></span></span></span>)</span></span>.</span>cloned</span>(</span></span>)</span></span>.</span>collect</span>(</span></span>)</span></span></span></span>)</span></span>
</span></span></span>        .</span>collect</span>(</span></span>)</span></span>;</span>
</span></span></span>    serde_pickle::</span></span>to_writer(</span>&</span>mut</span> writer,</span> &</span>(</span>data,</span> burn</span>)</span></span>,</span> Default</span>::</span></span>default(</span></span>)</span></span></span>)</span></span>?</span>;</span>
</span></span></span>    Ok</span>(</span>(</span></span>)</span></span></span>)</span></span>
</span></span></span></span>}</span></span></span>
</span>//</span> back in main() { ... }
</span></span>write_fit_result</span>(</span>&</span>res.</span>x,</span> "</span>fit.pkl"</span></span></span>)</span></span>?</span>;</span>
</span>//</span> writes the whole chain along with the number of steps we want to burn
</span></span>write_data_chain</span>(</span>&</span>chain,</span> burn,</span> "</span>chain.pkl"</span></span></span>)</span></span>?</span>;</span>
</span>//</span> writes just the un-burned steps
</span></span>write_data</span>(</span>&</span>flat_chain,</span> "</span>flat_chain.pkl"</span></span></span>)</span></span>?</span>;</span>
</span></code></pre>
Then in Python, we’ll use the corner</code></a> library to make some nice plots:</p>
fit_result</span></span> =</span> np</span></span>.</span></span>array</span></span></span></span>(</span>pickle</span></span>.</span></span>load</span></span></span></span>(</span>Path</span></span>.</span></span>open</span></span></span></span>(</span>'</span></span></span>fit.pkl'</span></span></span>,</span> '</span></span></span>rb'</span></span></span>)</span></span>)</span></span>)</span></span>
</span>chain</span></span>, burn</span></span> =</span> pickle</span></span>.</span></span>load</span></span></span></span>(</span>Path</span></span>.</span></span>open</span></span></span></span>(</span>'</span></span></span>chain.pkl'</span></span></span>,</span> '</span></span></span>rb'</span></span></span>)</span></span>)</span></span>
</span>chain</span></span> =</span> np</span></span>.</span></span>array</span></span></span></span>(</span>chain</span></span>)</span></span>
</span>n_walkers</span></span>, n_steps</span></span>, n_parameters</span></span> =</span> chain</span>.</span>shape</span></span>
</span>_</span></span>, ax</span></span> =</span> plt</span></span>.</span></span>subplots</span></span></span></span>(</span>nrows</span>=</span>n_parameters</span></span>,</span> sharex</span>=</span>True</span>,</span> figsize</span>=</span>(</span>10</span>,</span> 50</span>)</span></span>)</span></span>
</span>steps</span></span> =</span> np</span></span>.</span></span>arange</span></span></span></span>(</span>n_steps</span></span>)</span></span>
</span>for</span> i</span> in</span></span> range</span></span></span></span>(</span>n_parameters</span></span>)</span></span></span>:</span></span>
</span>    for</span> j</span> in</span></span> range</span></span></span></span>(</span>n_walkers</span></span>)</span></span></span>:</span></span>
</span>        ax</span></span></span>[</span></span>i</span></span></span>]</span></span>.</span></span>plot</span></span></span></span>(</span>steps</span></span></span>[</span></span>burn</span></span>:</span></span>]</span></span>,</span> chain</span></span></span>[</span></span>j</span></span>, burn</span></span>:</span>, i</span></span></span>]</span></span>,</span> color</span>=</span>'</span></span></span>k'</span></span></span>,</span> alpha</span>=</span>0.</span>1</span>)</span></span>
</span>        ax</span></span></span>[</span></span>i</span></span></span>]</span></span>.</span></span>plot</span></span></span></span>(</span>steps</span></span></span>[</span></span>:</span>burn</span></span></span>]</span></span>,</span> chain</span></span></span>[</span></span>j</span></span>, :</span>burn</span></span>, i</span></span></span>]</span></span>,</span> color</span>=</span>'</span></span></span>k'</span></span></span>,</span> ls</span>=</span>'</span></span></span>--'</span></span></span>,</span> alpha</span>=</span>0.</span>1</span>)</span></span>
</span>    ax</span></span></span>[</span></span>i</span></span></span>]</span></span>.</span></span>plot</span></span></span></span>(</span>steps</span></span></span>[</span></span>burn</span></span>:</span></span>]</span></span>,</span> chain</span></span></span>[</span></span>0</span>, burn</span></span>:</span>, i</span></span></span>]</span></span>,</span> color</span>=</span>'</span></span></span>m'</span></span></span>,</span> label</span>=</span>'</span></span></span>Walker 0'</span></span></span>)</span></span>
</span>    ax</span></span></span>[</span></span>i</span></span></span>]</span></span>.</span></span>plot</span></span></span></span>(</span>steps</span></span></span>[</span></span>:</span>burn</span></span></span>]</span></span>,</span> chain</span></span></span>[</span></span>0</span>, :</span>burn</span></span>, i</span></span></span>]</span></span>,</span> color</span>=</span>'</span></span></span>m'</span></span></span>,</span> ls</span>=</span>'</span></span></span>--'</span></span></span>,</span> label</span>=</span>'</span></span></span>Walker 0 (burn-in)'</span></span></span>)</span></span>
</span>    ax</span></span></span>[</span></span>i</span></span></span>]</span></span>.</span></span>axhline</span></span></span></span>(</span>fit_result</span></span></span>[</span></span>i</span></span></span>]</span></span>,</span> color</span>=</span>'</span></span></span>b'</span></span></span>,</span> label</span>=</span>'</span></span></span>Best fit'</span></span></span>)</span></span>
</span>    ax</span></span></span>[</span></span>i</span></span></span>]</span></span>.</span></span>set_xlabel</span></span></span></span>(</span>'</span></span></span>Step'</span></span></span>)</span></span>
</span>    ax</span></span></span>[</span></span>i</span></span></span>]</span></span>.</span></span>set_ylabel</span></span></span></span>(</span>parameter_labels</span></span></span>[</span></span>i</span></span></span>]</span></span>)</span></span>
</span>    ax</span></span></span>[</span></span>i</span></span></span>]</span></span>.</span></span>legend</span></span></span></span>(</span>)</span></span>
</span>plt</span></span>.</span></span>savefig</span></span></span></span>(</span>'</span></span></span>traces.svg'</span></span></span>)</span></span>
</span>plt</span></span>.</span></span>close</span></span></span></span>(</span>)</span></span>
</span></code></pre>
</p>
_</span></span>, ax</span></span> =</span> plt</span></span>.</span></span>subplots</span></span></span></span>(</span>nrows</span>=</span>n_parameters</span></span>,</span> sharex</span>=</span>True</span>,</span> figsize</span>=</span>(</span>10</span>,</span> 50</span>)</span></span>)</span></span>
</span>steps</span></span> =</span> np</span></span>.</span></span>arange</span></span></span></span>(</span>n_steps</span></span>)</span></span>
</span>for</span> i</span> in</span></span> range</span></span></span></span>(</span>n_parameters</span></span>)</span></span></span>:</span></span>
</span>    for</span> j</span> in</span></span> range</span></span></span></span>(</span>n_walkers</span></span>)</span></span></span>:</span></span>
</span>        ax</span></span></span>[</span></span>i</span></span></span>]</span></span>.</span></span>plot</span></span></span></span>(</span>steps</span></span></span>[</span></span>burn</span></span>:</span></span>]</span></span>,</span> chain</span></span></span>[</span></span>j</span></span>, burn</span></span>:</span>, i</span></span></span>]</span></span>,</span> color</span>=</span>'</span></span></span>k'</span></span></span>,</span> alpha</span>=</span>0.</span>1</span>)</span></span>
</span>    ax</span></span></span>[</span></span>i</span></span></span>]</span></span>.</span></span>plot</span></span></span></span>(</span>steps</span></span></span>[</span></span>burn</span></span>:</span></span>]</span></span>,</span> chain</span></span></span>[</span></span>0</span>, burn</span></span>:</span>, i</span></span></span>]</span></span>,</span> color</span>=</span>'</span></span></span>m'</span></span></span>,</span> label</span>=</span>'</span></span></span>Walker 0'</span></span></span>)</span></span>
</span>    ax</span></span></span>[</span></span>i</span></span></span>]</span></span>.</span></span>axhline</span></span></span></span>(</span>fit_result</span></span></span>[</span></span>i</span></span></span>]</span></span>,</span> color</span>=</span>'</span></span></span>b'</span></span></span>,</span> label</span>=</span>'</span></span></span>Best fit'</span></span></span>)</span></span>
</span>    ax</span></span></span>[</span></span>i</span></span></span>]</span></span>.</span></span>set_xlabel</span></span></span></span>(</span>'</span></span></span>Step'</span></span></span>)</span></span>
</span>    ax</span></span></span>[</span></span>i</span></span></span>]</span></span>.</span></span>set_ylabel</span></span></span></span>(</span>parameter_labels</span></span></span>[</span></span>i</span></span></span>]</span></span>)</span></span>
</span>    ax</span></span></span>[</span></span>i</span></span></span>]</span></span>.</span></span>legend</span></span></span></span>(</span>)</span></span>
</span>plt</span></span>.</span></span>savefig</span></span></span></span>(</span>'</span></span></span>traces_burned.svg'</span></span></span>)</span></span>
</span>plt</span></span>.</span></span>close</span></span></span></span>(</span>)</span></span>
</span></code></pre>
</p>
flat_chain</span></span> =</span> np</span></span>.</span></span>array</span></span></span></span>(</span>pickle</span></span>.</span></span>load</span></span></span></span>(</span>Path</span></span>.</span></span>open</span></span></span></span>(</span>'</span></span></span>flat_chain.pkl'</span></span></span>,</span> '</span></span></span>rb'</span></span></span>)</span></span>)</span></span>)</span></span>
</span>fig</span></span> =</span> corner</span></span></span></span>(</span>
</span></span>    flat_chain</span></span>,</span>
</span></span>    labels</span>=</span>parameter_labels</span></span>,</span>
</span></span>    truths</span>=</span>[</span>1.</span>2</span>,</span> 2.</span>3</span>,</span> 0.</span>6</span>,</span> 0.</span>5</span>,</span> 0.</span>7</span>]</span></span>,</span>
</span></span>    truth_color</span>=</span>'</span></span></span>r'</span></span></span>,</span>
</span></span>    quantiles</span>=</span>[</span>0.</span>16</span>,</span> 0.</span>5</span>,</span> 0.</span>84</span>]</span></span>,</span>
</span></span>    show_titles</span>=</span>True</span>,</span>
</span></span>    title_fmt</span>=</span>'</span></span></span>.4f'</span></span></span>,</span>
</span></span>)</span></span>
</span>overplot_lines</span></span></span></span>(</span>
</span></span>    fig</span></span>,</span>
</span></span>    fit_result</span></span>,</span>
</span></span>    color</span>=</span>'</span></span></span>b'</span></span></span>,</span>
</span></span>)</span></span>
</span>plt</span></span>.</span></span>savefig</span></span></span></span>(</span>'</span></span></span>corner.svg'</span></span></span>)</span></span>
</span>plt</span></span>.</span></span>close</span></span></span></span>(</span>)</span></span>
</span></code></pre>
</p>
This last plot is probably the most interesting to look at, since it shows us the actual posterior distribution of our parameters. In blue, we have the truth values used to generate the data, and in red we have the values obtained in our fit. While we are within $3\sigma$ of each truth value, the fact that the fit is far away from the means of the MCMC samples is not great, but it just means the algorithm probably terminated too soon and gave us a less-accurate fit. We can also see here that the MCMC parameter errors are very close to the symmetric errors we obtained from L-BFGS-B. The difference is that the ESS algorithm didn’t actually need a gradient to compute these, and due to some fun math, the transforms we applied were taken into account when L-BFGS-B directly calculated the Hessian.</p>
For my final act, I’ll show off a neat library I found called matplotloom</code></a> which simplifies the process of making cool animations with matplotlib</code>. I won’t go into detail about how this code works, but it should be fairly straightforward:</p>
def</span></span> compute_ranges</span></span></span>(</span></span>chain</span>,</span> pad_frac</span></span>=</span>0.</span>02</span></span>)</span></span>:</span></span>
</span>    mins</span></span> =</span> chain</span></span>.</span></span>min</span></span></span></span>(</span>axis</span>=</span>(</span>0</span>,</span> 1</span>)</span></span>)</span></span>
</span>    maxs</span></span> =</span> chain</span></span>.</span></span>max</span></span></span></span>(</span>axis</span>=</span>(</span>0</span>,</span> 1</span>)</span></span>)</span></span>
</span>    widths</span></span> =</span> np</span></span>.</span></span>maximum</span></span></span></span>(</span>maxs</span></span> -</span> mins</span></span>,</span> 1e-9</span>)</span></span>
</span>    mins</span></span> =</span> mins</span></span> -</span> pad_frac</span></span> *</span> widths</span></span>
</span>    maxs</span></span> =</span> maxs</span></span> +</span> pad_frac</span></span> *</span> widths</span></span>
</span>    return</span> [</span>(</span>float</span></span></span></span>(</span>a</span></span>)</span></span>,</span> float</span></span></span></span>(</span>b</span></span>)</span></span>)</span></span> for</span> a</span>,</span> b</span> in</span></span> zip</span></span></span></span>(</span>mins</span></span>,</span> maxs</span></span>)</span></span>]</span></span>
</span>
</span>def</span></span> make_frame</span></span></span>(</span></span>i</span>,</span> chain</span>,</span> labels</span>,</span> ranges</span>,</span> loom</span>)</span></span>:</span></span>
</span>    #</span> window: steps [max(0, i-10) ... i], inclusive
</span></span>    j0</span></span> =</span> max</span></span></span></span>(</span>0</span>,</span> i</span></span> -</span> 10</span>)</span></span>
</span>    window</span></span> =</span> chain</span></span></span>[</span></span>:</span>, j0</span></span> :</span> i</span></span> +</span> 1</span>, :</span></span>]</span></span>  #</span> (n_walkers, window_len, n_params)
</span></span>    flat</span></span> =</span> window</span></span>.</span></span>reshape</span></span></span></span>(</span>-</span>1</span>,</span> window</span>.</span>shape</span></span></span>[</span></span>-</span>1</span></span>]</span></span>)</span></span>  #</span> (n_walkers*window_len, n_params)
</span></span>
</span>    fig</span></span> =</span> corner</span></span></span></span>(</span>
</span></span>        flat</span></span>,</span>
</span></span>        labels</span>=</span>labels</span></span>,</span>
</span></span>        range</span>=</span>ranges</span></span>,</span>  #</span> keep axes consistent across frames
</span></span></span>        plot_contours</span>=</span>False</span>,</span>
</span></span>        show_titles</span>=</span>False</span>,</span>
</span></span>    )</span></span>
</span>    loom</span></span>.</span></span>save_frame</span></span></span></span>(</span>fig</span></span>,</span> i</span></span>)</span></span>
</span>    plt</span></span>.</span></span>close</span></span></span></span>(</span>fig</span></span>)</span></span>
</span>
</span>burned_chain</span></span> =</span> chain</span></span></span>[</span></span>:</span>, burn</span></span>:</span>, :</span></span>]</span></span>
</span>n_steps</span></span> =</span> burned_chain</span>.</span>shape</span></span></span>[</span></span>1</span></span>]</span></span>
</span>ranges</span></span> =</span> compute_ranges</span></span></span></span>(</span>burned_chain</span></span>)</span></span>
</span>
</span>with</span> Loom</span></span></span></span>(</span>'</span></span></span>walkers_corner.gif'</span></span></span>,</span> fps</span>=</span>20</span>,</span> parallel</span>=</span>True</span>)</span></span> as</span></span></span> loom</span>:</span></span>
</span>    Parallel</span></span></span></span>(</span>n_jobs</span>=</span>-</span>1</span>,</span> prefer</span>=</span>'</span></span></span>processes'</span></span></span>)</span></span>(</span>
</span></span>        delayed</span></span></span></span>(</span>make_frame</span></span>)</span></span>(</span>i</span></span>,</span> burned_chain</span></span>,</span> parameter_labels</span></span>,</span> ranges</span></span>,</span> loom</span></span>)</span></span> for</span> i</span> in</span></span> range</span></span></span></span>(</span>n_steps</span></span>)</span></span>
</span></span>    )</span></span>
</span></code></pre>
</p>
This entire example can be found (in a slightly more condensed form) in the examples/</code></a> directory of ganesh</code>. See the .justfile</code></a> in that directory for more information about how to run this and other examples.</p>
I hope this has been an informative look into ganesh</code>. The current goal for a v1.0.0 release is to add a more diverse set of algorithms. Also, there is probably some low-hanging fruit in terms of algorithm optimizations. If you’re interested in adding an algorithm, optimizing an existing algorithm, or adding a new feature, ganesh</code> is open to contrigutions on GitHub</a>. I will be posting some issues representing some of the more desired algorithms and features soon.</p>


Nearly every other implementation I’ve come across uses 10</code>, although Liu and Nocedal[6]</a></sup> recommend $3\leq m \leq 7$. ↩</a></p>
</li>

This is a bit silly, since MINUIT</code> is essentially just the BFGS algorithm with this bounds transformation, so this could all be avoided by just using L-BFGS-B. The technical reason for this is that MINUIT</code> doesn’t bother to calculate higher-order derivatives of the transformation functions and uses a linear approximation instead. ganesh</code> does include these higher-order derivatives for bounds transformations. ↩</a></p>
</li>

I’ve been avoiding this partially because the functions I use in my research are already parallelized over their underlying datasets, and nested parallel loops are a bit of a pain to get right. ↩</a></p>
</li>

J. Goodman and J. Weare, “Ensemble samplers with affine invariance,” CAMCoS, vol. 5, no. 1, pp. 65–80, Jan. 2010, doi: 10.2140/camcos.2010.5.65</a>. ↩</a></p>
</li>

M. Karamanis and F. Beutler, “Ensemble slice sampling,” Stat Comput, vol. 31, no. 5, Aug. 2021, doi: 10.1007/s11222-021-10038-2</a>. ↩</a></p>
</li>

D. C. Liu and J. Nocedal, “On the limited memory BFGS method for large scale optimization,” Mathematical Programming, vol. 45, no. 1–3, pp. 503–528, Aug. 1989, doi: 10.1007/bf01589116</a>. ↩</a></p>
</li>
</ol>



The BFGS Algorithm Family in Rust (Part 3)
Dene Hoffman — Wed, 18 Jun 2025 00:00:00 +0000
In my previous posts (here</a> and here</a>), I discussed line searches and the original BFGS algorithm. The next improvement on this algorithm is a limited memory version, called L-BFGS, but rather than explicitly constructing it, we will instead skip to the limited memory version with parameter bounds, L-BFGS-B. However, we should start by understanding why there even is an L-BFGS algorithm at all.</p>
In the BFGS algorithm, we start with a guess for the inverse Hessian $B_0^{-1}$ and then improve this guess at each step by defining</p>
$$ s_k = x_{k+1} - x_k,$$</p>
$$ y_k = g_{k+1} - g_k,$$</p>
and</p>
$$ \rho = \frac{1}{\vec{y}_k^{\intercal} \vec{s}_k}, $$</p>
and updating according to the formula</p>
$$ B_{k+1}^{-1} = \left(I - \rho_k \vec{s}_k \vec{y}_k^{\intercal} \right) B_k^{-1} \left(I - \rho_k \vec{y}_k \vec{s}_k^{\intercal} \right) + \rho_k \vec{s}_k \vec{s}_k^{\intercal}. $$</p>
This doesn’t seem like it uses too much memory. At worst, the Hessian must be symmetric, so for $n$ parameters, we need $n(n+1)/2$ stored values, and if each is a 64-bit float, that’s just over $40$ kilobytes for $100$ free parameters. However, we must also consider the storage limitations of the 1990s, where typical computers had maybe $128$ megabytes of RAM at a maximum. Some part of that RAM had to be used to store any data used in the minimization, part of it had to store the program itself, and part of it was allocated to the other operations of the computer. We live in an era where most laptops have a minimum of $4$ gigabytes of RAM, and it would take a $16000$-parameter model just to use $1$ gigabyte of that. There are other considerations, for instance, you actually need to allocate twice as much memory since you need some place to store the next iteration. It turns out, there is a way to greatly reduce the amount of memory needed, and this is what Byrd, Nocedal, and Schnabel[1]</a></sup> did back in 1989.</p>
The L-BFGS Algorithm</h2>
Their method is centered around the following idea: if you were to store every $\vec{s}$ and $\vec{y}$ and recompute the inverse Hessian from scratch at each step, only the most recent updates to those vectors would be very useful. If we’re 100 steps into the algorithm, we could imagine starting the algorithm at step 90 with the identity matrix initial guess, and eventually we will converge upon the correct Hessian again. In fact, they found that this convergence tends to happen very quickly, typically in less than 10 steps or so. Therefore, instead of storing the entire inverse Hessian, we can instead store some finite number of $\vec{s}$ and $\vec{y}$ entries, discarding the oldest ones, and calculate the inverse Hessian from scratch each time. If we store $m$ entries, recall that each vector contains $n$ values, so we now need to only store $2n\times m$ values rather than $n(n+1)/2$. If we set $m=10$ and again use 64-bit floats, a $100$-parameter model would only need $16$ kilobytes rather than $40$. The real gains are obvious in the case of extremely large models. For instance, the $16000$-parameter model would now only take $2.6$ megabytes rather than $1$ gigabyte. What do we lose here? Well, we typically have to do a few more operations, but we get improvements in cache locality, since we no longer have to transfer a large matrix to the local cache at each step. We also save on allocations, since we don’t actually need to construct the inverse Hessian at all, and we can just construct the update step directly instead.</p>
As mentioned, we could go through the entire algorithm separately, but it’s so closely related to L-BFGS-B that we will just implement that directly instead. This article will follow the work of Byrd, Lu, Nocedal, and Zhu[2]</a></sup>, particularly their notation, except for my usage of $B^{-1}$ as the inverse Hessian approximate rather than their use of $H$ for that matrix.</p>
First, we define our registers of $\vec{s}$ and $\vec{y}$ as $n\times m$ matrices:</p>
$$ Y_k = [\vec{y}_{k-m},\ldots,\vec{y}_{k-1}] $$</p>
and</p>
$$ S_k = [\vec{s}_{k-m},\ldots,\vec{s}_{k-1}] $$</p>
Just to clarify, these are matrices where each row corresponds to a particular parameter and each column corresponds to an algorithm step, where the leftmost column is the oldest step and the rightmost column is the most recent. If we let $\theta$ be a positive scaling parameter, then as long as each pair of $\vec{s}$ and $\vec{y}$ satisfy $\vec{s}^{\intercal} \vec{y} > 0 $, we can write the Hessian approximate as</p>
$$ B_k = \theta I - W_k M_k W_k^{\intercal} $$</p>
where</p>
$$ W_k = \begin{bmatrix} Y_k & \theta S_k \end{bmatrix} $$</p>
is a block matrix of dimension $n\times 2m$ and</p>
$$ M_k = \begin{bmatrix} -D_k & L_k^{\intercal} \\ L_k & \theta S_k^{\intercal} S_k \end{bmatrix}^{-1} $$</p>
is another block matrix of dimension $2m\times 2m$, where</p>
$$ (L_k)_{i,j} = \begin{cases} (s_{k-m-1+i})^{\intercal} (y_{k-m-1+j}) & \text{if } i > j \\ 0 & \text{otherwise}\end{cases} $$</p>
and</p>
$$ D_k = \text{diag}\left[ s_{k-m}^{\intercal} y_{k-m},\ldots,s_{k-1}^{\intercal} y_{k-1} \right] $$</p>
Woah, slow down! What’s going on here? This is the notation used in Byrd et al., and it’s pretty confusing. Let’s ignore the intricacies of $M_k$ and just focus on the blocks. $L_k$ is upper triangular with zeros on the diagonal, that’s all the case-statement means. All of those subscripts look confusing, but if you recall, $k-m$ is just shorthand for the oldest index stored and $k-1$ is the newest index, so $L_k$ is really just the upper triangular block of $S_k^{\intercal} Y_k$! Similarly, $D_k$ is just the diagonal of that matrix, so to get all of these parts, we just need to perform the matrix product and pull apart the relevant blocks. The fact that $M_k$ is the inverse of this matrix seems a bit tricky, but recall that $M_k$ has dimension $2m \times 2m$, and if we pick $m$ to be small, the cost to compute the inverse is actually pretty low (we can actually make $m$ arbitrarily small (but $ >1 $), and all we will lose is the speed of convergence).</p>
I’m not going to explain exactly why this set of matrices works, but hopefully you can see that we end up doing something similar to the original BFGS update step, where the Hessian is approximated by some correction to the identity.</p>
While I won’t show the entire Rust code for this, I’ll give you the general idea:</p>
//</span> First, W_k
</span></span>let</span> w_mat =</span> DMatrix::</span></span>zeros(</span>n,</span> 2</span> *</span> m</span>)</span></span>;</span>
</span>//</span> We can use view_mut to update a submatrix
</span></span>let</span> mut</span> y_view =</span> w_mat.</span>view_mut</span>(</span>(</span>0</span>,</span> 0</span></span>)</span></span>,</span> (</span>n,</span> m</span>)</span></span></span>)</span></span>;</span>
</span>y_view +=</span> &</span>y_mat;</span>
</span>let</span> mut</span> theta_s_view =</span> w_mat.</span>view_mut</span>(</span>(</span>0</span>,</span> m</span>)</span></span>,</span> (</span>n,</span> m</span>)</span></span></span>)</span></span>;</span>
</span>theta_s_view +=</span> s_mat.</span>scale</span>(</span>theta</span>)</span></span>;</span>
</span>
</span>//</span> Next, we calculate M_k
</span></span>//</span> S_k^T Y_k:
</span></span>let</span> s_tr_y =</span> s_mat.</span>transpose</span>(</span></span>)</span></span> *</span> &</span>y_mat;</span>
</span>//</span> D_k:
</span></span>let</span> d_vec =</span> s_tr_y.</span>diagonal</span>(</span></span>)</span></span>;</span>
</span>//</span> L_k^T:
</span></span>let</span> mut</span> l_tr_mat =</span> s_tr_y.</span>lower_triangle</span>(</span></span>)</span></span>;</span>
</span>l_tr_mat.</span>set_diagonal</span>(</span>&</span>DVector::</span></span>from_element(</span>m,</span> 0.</span>0</span></span>)</span></span></span>)</span></span>;</span> //</span> clear the diagonal
</span></span>//</span> theta * S_k^T S_k:
</span></span>let</span> theta_s_tr_s =</span> (</span>s_mat.</span>transpose</span>(</span></span>)</span></span> *</span> &</span>s_mat</span>)</span></span>.</span>scale</span>(</span>theta</span>)</span></span>;</span>
</span>//</span> M_k^{-1}:
</span></span>let</span> mut</span> m_mat_inv =</span> DMatrix::</span></span>zeros(</span>2</span> *</span> m,</span> 2</span> *</span> m</span>)</span></span>;</span>
</span>let</span> mut</span> d_view =</span> m_mat_inv.</span>view_mut</span>(</span>(</span>0</span>,</span> 0</span></span>)</span></span>,</span> (</span>m,</span> m</span>)</span></span></span>)</span></span>;</span>
</span>d_view.</span>set_diagonal</span>(</span>&</span>(</span>-</span>&</span>d_vec</span>)</span></span></span>)</span></span>;</span>
</span>let</span> mut</span> l_tr_view =</span> m_mat_inv.</span>view_mut</span>(</span>(</span>m,</span> 0</span></span>)</span></span>,</span> (</span>m,</span> m</span>)</span></span></span>)</span></span>;</span>
</span>l_tr_view +=</span> &</span>l_tr_mat;</span>
</span>let</span> mut</span> l_view =</span> m_mat_inv.</span>view_mut</span>(</span>(</span>0</span>,</span> m</span>)</span></span>,</span> (</span>m,</span> m</span>)</span></span></span>)</span></span>;</span>
</span>l_view +=</span> l_tr_mat.</span>transpose</span>(</span></span>)</span></span>;</span>
</span>let</span> mut</span> theta_s_tr_s_view =</span> m_mat_inv.</span>view_mut</span>(</span>(</span>m,</span> m</span>)</span></span>,</span> (</span>m,</span> m</span>)</span></span></span>)</span></span>;</span>
</span>theta_s_tr_s_view +=</span> theta_s_tr_s;</span>
</span>let</span> m_mat =</span> m_mat_inv.</span>try_inverse</span>(</span></span>)</span></span>.</span>expect</span>(</span>"</span>inv(M) failed!"</span></span></span>)</span></span>;</span>
</span></code></pre>
In the implementation, we also store w_mat</code> and m_mat</code>, since m_mat</code> is just $2m \times 2m$ and w_mat</code> is $n \times 2m$, so they’re both relatively small for small $m$. Another thing to note is that the authors suggest discarding correction pairs $\vec{s}_k$ and $\vec{y}_k$ if they do not satisfy the curvature condition</p>
$$ \vec{s}_k^{\intercal} \vec{y}_k > \varepsilon ||\vec{y}_k||^2 $$</p>
for some small $\varepsilon > 0$. We technically have a way to compute the inverse Hessian approximate, and we could just stop there and redo the BFGS update step, giving us the L-BFGS algorithm. Let’s instead move on to the bounded version, and we’ll show how we can avoid constructing the inverse Hessian approximate entirely by just directly updating the descent step.</p>
Adding Boundaries</h2>
There are a couple of ways one might add bounded parameter support to a minimization algorithm. Algorithms like MINUIT and LMFIT use a nonlinear transform which stretches the distance to a boundary infinitely as one approaches the bound. One advantage of this is that it can generally be adapted to any minimization method, but the major disadvantage is that it turns any linear problem into a nonlinear one. This also makes it difficult to properly reconstruct the Hessian later for uncertainty analysis, and it can be very slow when the minimum is close to or on a boundary.</p>
Another way is to pin parameters at their boundary value when an update step would try to cross that boundary. This is kind of what we’ll end up doing, but it’s not quite so straightforward when we want to be memory/computationally efficient.</p>
Computing the Generalized Cauchy Point</h3>
The first step is similar to any gradient-descent algorithm. We need to compute a step direction, and originally we’d just go in the opposite direction of the gradient. However, the gradient might point us towards a boundary, or we might already be on a boundary. Remember, we are modeling the objective function locally as a quadratic and jumping as close to the minimum as we can. Byrd et al. start this process by computing “breakpoints”:</p>
$$ t_i = \begin{cases}
(x_i - u_i) / g_i & \text{if } g_i < 0\\
(x_i - l_i) / g_i & \text{if } g_i > 0\\
\infty & \text{if } g_i = 0
\end{cases} $$</p>
I would forgive the reader if they can’t figure out why we are doing this, I also had no idea what was going on here when I first read it. We calculate the gradient $\vec{g}$, and we want to move in the opposite direction. For each component of $\vec{g}$, $g_i < 0$ would imply that we are moving in the positive direction, so we might have to worry about an upper bound. $(x_i - u_i)$ is the distance to that upper bound (but note that the order makes it a negative value as long as we haven’t crossed this bound!), and we divide this by the length of the step (which is also negative according to the condition, so this quantity is strictly positive). Similarly, if the step is in the other direction, we get another strictly positive value with the same kind of proportionality. Assuming no step ever puts us over a boundary, can get a minimum value of zero if we are already on the boundary, otherwise we will get larger positive values the further away from the boundary we are (scaled by the size of the step in that direction).</p>
Next, we’re going to sort the indices by these values, but first let’s additionally calculate a vector $\vec{d}$ which follows the formula,</p>
$$ d_i = \begin{cases}
0 & \text{if } t_i = 0\\
-g_i & \text{otherwise}
\end{cases} $$</p>
Recall that $t_i = 0$ means $x_i$ is already at a boundary, so this $\vec{d}$ is just the descent vector (negative gradient) in all directions that aren’t at a boundary. In rust, this looks something like,</p>
let</span> (</span>t,</span> mut</span> d</span>)</span></span>:</span> (</span>DVector<</span>Float></span></span>,</span> DVector<</span>Float></span></span></span>)</span></span> =</span> (</span>0</span>..</span>g.</span>len</span>(</span></span>)</span></span></span>)</span></span>.</span>map</span>(</span>|</span></span></span>i</span>|</span></span> </span>{</span>
</span></span></span></span>  let</span> t_i =</span> if</span> g[</span>i]</span></span> <</span> 0.</span>0</span> {</span>
</span></span></span></span></span>    (</span>x[</span>i]</span></span> -</span> u[</span>i]</span></span></span>)</span></span> /</span> g[</span>i]</span></span>
</span></span></span></span></span>  </span>}</span></span> else</span> if</span> g[</span>i]</span></span> ></span> 0.</span>0</span> {</span>
</span></span></span></span></span>    (</span>x[</span>i]</span></span> -</span> l[</span>i]</span></span></span>)</span></span> /</span> g[</span>i]</span></span>
</span></span></span></span></span>  </span>}</span></span> else</span> {</span>
</span></span></span></span></span>    Float::</span></span>INFINITY</span>
</span></span></span></span></span>  </span>}</span></span>;</span>
</span></span></span></span>  let</span> d_i =</span> if</span> t_i <</span> Float::</span></span>EPSILON</span> {</span> 0.</span>0</span> </span>}</span></span> else</span> {</span> -</span>g[</span>i]</span></span> </span>}</span></span>;</span>
</span></span></span></span>  (</span>t_i,</span> d_i</span>)</span></span>
</span></span></span></span></span>}</span></span></span></span>)</span></span>
</span></code></pre>
We use $t_i < \epsilon$ here rather than $t_i = 0$ to avoid issues with comparing floating point values. The point is, if we are close enough to a boundary, we probably shouldn’t try to move in that direction, even if we are technically not right on it. d</code> is mutable for reasons which will become clear later.</p>
We next need to establish a list of “free” indices $\mathcal{F} = \left\{i \mid t_i > 0\right\}$. These are the indices of all parameter which are not at a boundary. This next part is a bit tricky, and I’m not going to derive it because this article would be much too long (and you can see the derivation in [2]</a></sup> anyway), but I’ll give you the general idea. We are approximating the function as a quadratic</p>
$$ m_k(x) = f(x_k) + g_k^{\intercal}(x - x_k) + \frac{1}{2}(x - x_k)^{\intercal}B_k(x - x_k) $$</p>
but instead of minimizing it for all $x$, we instead want to minimize it just for feasible values of $x$ (values within the boundaries). We therefore construct a piecewise function $\vec{x}(t) = P(\vec{x}_k - t \vec{g}_k, \vec{l}, \vec{u})$ where $P(\vec{a}, \vec{l}, \vec{u})$ just clips $\vec{a}$ inside a box defined by $\vec{l}$ and $\vec{u}$. We then want to find the first minimizer of $q_k(t) = m_k(\vec{x}(t))$, which we call the generalized Cauchy point $\vec{x}^c$. This can be thought of as the first feasible point along the search direction (accounting for boundaries). After this, we still need to minimize the objective function along this direction. Imagine that we are walking along the gradient and we periodically hit a boundary in some dimension. Each time we hit a boundary, we pin that parameter to the boundary and keep walking in the same direction for all other parameters. This creates a piecewise-linear path for which the generalized Cauchy point is the first valid point we compute. Now it might make more sense why we defined each $t_i$ as above. We will eventually sort these $t_i$ to determine the order in which to drop free indices, and travel along this piecewise path to find the minimum! Each $t_i$ represents the distance along the piecewise path in units of the gradient which must be traveled to hit a boundary.</p>
We initialize some vector $\vec{p} = W^{\intercal} \vec{d}$ which is a vector of dimension $2m$ which holds $Y_k \vec{d}$ in the first $m$ indices and $\theta S_k \vec{d}$ in the next $m$ indices. We also keep track of a vector $c$ initialized to zero with the same dimensions as $\vec{p}$. We can calculate the derivative of the objective function at the current point to be $f’ = -\vec{d}^{\intercal}\vec{d}$ and the second derivative as $f’’ = -\theta f’ - \vec{p}^{\intercal} M \vec{p}$ (this comes from a simplification of $f’’ = \theta \vec{d}^{\intercal}\vec{d} - \vec{d}^{\intercal} WMW^{\intercal} \vec{d}$). We can then find the distance to the minimum along this direction, $\Delta t_{\text{min}} = - \frac{f’}{f’’}$.</p>
This gives us the distance along the first piecewise-linear segment which minimizes the quadratic approximation, so now we need to check the subsequent segments. We do this by sorting $\mathcal{F}$ by the values of $t_i$ (the smallest $t_i$ corresponds to the parameter closest to a boundary in units of the gradient). We remove the smallest one from the set of free indices and call it $b$. We define $\Delta t = t_b - 0$ for this first segment, and this corresponds to the distance we can travel in $t$ till we get to the next boundary (not the one we just explored). As long as the distance to the minimum of the current segment is less than the distance to the next segment ($\Delta t$) and we still have unexplored segments, we keep going.</p>
The update step for each new segment becomes a bit more complicated, since we are now displaced from the start of the algorithm at some point along the piecewise-linear path. We define a point $z_b$ as the distance along parameter $b$ from the starting point to the boundary. We update $\vec{c} \to \vec{c} + \Delta t \vec{p}$ and update the function derivatives according to</p>
$$ f’ = f’ + \Delta t f’’ + g_b^2 + \theta g_b z_b - g_b \vec{(w_b)}^{\intercal}M\vec{c} $$</p>
and</p>
$$ f’’ = f’’ - \theta g_b^2 - 2g_b \vec{(w_b)}^{\intercal}M\vec{p} - g_b^2 \vec{(w_b)}^{\intercal}M \vec{(w_b)} $$</p>
Here we define $\vec{(w_b)}$ as the $b$th row of the $W_k$ matrix, and we use the previous values of $f’$ and $f’’$ to calculate their updates. Then we update $\vec{p} \to \vec{p} + g_b \vec{(w_b)}$ and we set the $b$th element of $\vec{d}$ to zero (this is why d</code> is mutable in the code above) since we are essentially saying that this parameter is no longer part of the set of free parameters. We then repeat the same process as before to find the minimum along this segment, we remove the next free index, and explore the following segment in that direction as long as it is smaller than the distance we just traveled along the current segment. Notice that if a parameter is unbounded or very far away from a bound, this condition cannot be met, so the algorithm will terminate.</p>
After we have explored all viable segments, we calculate the generalized Cauchy point as</p>
$$ x_i^c = x_i + t_{\text{tot}} d_i $$</p>
for all $i$ with $t_i \geq t$ (all remaining segments) where $t_{\text{tot}}$ is the total distance traveled along the path. We do one final update to $\vec{c}$ before continuing to the next step, subspace minimization.</p>
The full code looks something like this:</p>
fn</span> </span>get_generalized_cauchy_point</span></span>(</span>
</span></span></span>    x</span>:</span> &</span>DVector<</span>Float></span></span>,
</span></span></span>    g</span>:</span> &</span>DVector<</span>Float></span></span>,
</span></span></span>    l</span>:</span> &</span>DVector<</span>Float></span></span>,
</span></span></span>    u</span>:</span> &</span>DVector<</span>Float></span></span>,
</span></span></span>    w_mat</span>:</span> &</span>DMatrix<</span>Float></span></span>,
</span></span></span>    m_mat</span>:</span> &</span>DMatrix<</span>Float></span></span>,
</span></span></span>    theta</span>:</span> Float,
</span></span></span></span>)</span></span></span></span> -></span> (</span>DVector<</span>Float></span></span>, DVector<</span>Float></span></span>, Vec</span><</span>usize</span>></span></span>)</span></span> </span>{</span>
</span></span></span>    //</span> Equations 4.1 and 4.2
</span></span></span></span>    let</span> (</span>t,</span> mut</span> d</span>)</span></span>:</span> (</span>DVector<</span>Float></span></span>,</span> DVector<</span>Float></span></span></span>)</span></span> =</span> (</span>0</span>..</span>g.</span>len</span>(</span></span>)</span></span></span>)</span></span>
</span></span></span>        .</span>map</span>(</span>|</span></span></span>i</span>|</span></span> </span>{</span>
</span></span></span></span></span></span>            let</span> ti =</span> if</span> g[</span>i]</span></span> <</span> 0.</span>0</span> {</span>
</span></span></span></span></span></span></span>                (</span>x[</span>i]</span></span> -</span> u[</span>i]</span></span></span>)</span></span> /</span> g[</span>i]</span></span>
</span></span></span></span></span></span></span>            </span>}</span></span> else</span> if</span> g[</span>i]</span></span> ></span> 0.</span>0</span> {</span>
</span></span></span></span></span></span></span>                (</span>x[</span>i]</span></span> -</span> l[</span>i]</span></span></span>)</span></span> /</span> g[</span>i]</span></span>
</span></span></span></span></span></span></span>            </span>}</span></span> else</span> {</span>
</span></span></span></span></span></span></span>                Float::</span></span>INFINITY</span>
</span></span></span></span></span></span></span>            </span>}</span></span>;</span>
</span></span></span></span></span></span>            let</span> di =</span> if</span> ti <</span> Float::</span></span>EPSILON</span> {</span> 0.</span>0</span> </span>}</span></span> else</span> {</span> -</span>g[</span>i]</span></span> </span>}</span></span>;</span>
</span></span></span></span></span></span>            (</span>ti,</span> di</span>)</span></span>
</span></span></span></span></span></span>        </span>}</span></span></span></span>)</span></span>
</span></span></span>        .</span>unzip</span>(</span></span>)</span></span>;</span>
</span></span></span>    let</span> mut</span> x_cp =</span> x.</span>clone</span>(</span></span>)</span></span>;</span>
</span></span></span>    let</span> mut</span> free_indices:</span> Vec</span><</span>usize</span>></span></span> =</span> (</span>0</span>..</span>t.</span>len</span>(</span></span>)</span></span></span>)</span></span>.</span>filter</span>(</span>|</span></span></span>&</span>i</span>|</span></span> </span>t[</span>i]</span></span> ></span> 0.</span>0</span></span></span>)</span></span>.</span>collect</span>(</span></span>)</span></span>;</span>
</span></span></span>    free_indices.</span>sort_by</span>(</span>|</span></span></span>&</span>a</span>,</span> &</span>b</span>|</span></span> </span>t[</span>a]</span></span>.</span>total_cmp</span>(</span>&</span>t[</span>b]</span></span></span>)</span></span></span></span>)</span></span>;</span>
</span></span></span>    let</span> free_indices =</span> VecDeque::</span></span>from(</span>free_indices</span>)</span></span>;</span>
</span></span></span>    let</span> mut</span> t_old =</span> 0.</span>0</span>;</span>
</span></span></span>    let</span> mut</span> i_free =</span> 0</span>;</span>
</span></span></span>    let</span> mut</span> b =</span> free_indices[</span>0</span>]</span></span>;</span>
</span></span></span>    let</span> mut</span> t_b =</span> t[</span>b]</span></span>;</span>
</span></span></span>    let</span> mut</span> dt_b =</span> t_b -</span> t_old;</span>
</span></span></span>
</span></span></span>    let</span> mut</span> p =</span> w_mat.</span>transpose</span>(</span></span>)</span></span> *</span> &</span>d;</span>
</span></span></span>    let</span> mut</span> c =</span> DVector::</span></span>zeros(</span>p.</span>len</span>(</span></span>)</span></span></span>)</span></span>;</span>
</span></span></span>    let</span> mut</span> df =</span> -</span>d.</span>dot</span>(</span>&</span>d</span>)</span></span>;</span>
</span></span></span>    let</span> mut</span> ddf =</span> (</span>-</span>theta</span>)</span></span>.</span>mul_add</span>(</span>df,</span> -</span>p.</span>dot</span>(</span>&</span>(</span>&</span>m_mat *</span> &</span>p</span>)</span></span></span>)</span></span></span>)</span></span>;</span>
</span></span></span>    let</span> mut</span> dt_min =</span> -</span>df /</span> ddf;</span>
</span></span></span>
</span></span></span>    while</span> dt_min >=</span> dt_b &&</span> i_free <</span> free_indices.</span>len</span>(</span></span>)</span></span> {</span>
</span></span></span></span>        //</span> b is the index of the smallest positive nonzero element of t, so d_b is never zero!
</span></span></span></span></span>        x_cp[</span>b]</span></span> =</span> if</span> d[</span>b]</span></span> ></span> 0.</span>0</span> {</span> u[</span>b]</span></span> </span>}</span></span> else</span> {</span> l[</span>b]</span></span> </span>}</span></span>;</span>
</span></span></span></span>        let</span> z_b =</span> x_cp[</span>b]</span></span> -</span> x[</span>b]</span></span>;</span>
</span></span></span></span>        c +=</span> p.</span>scale</span>(</span>dt_b</span>)</span></span>;</span>
</span></span></span></span>        let</span> g_b =</span> g[</span>b]</span></span>;</span>
</span></span></span></span>        let</span> w_b_tr =</span> w_mat.</span>row</span>(</span>b</span>)</span></span>;</span>
</span></span></span></span>        df +=</span> dt_b.</span>mul_add</span>(</span>
</span></span></span></span></span>            ddf,</span>
</span></span></span></span></span>            g_b *</span> (</span>theta.</span>mul_add</span>(</span>z_b,</span> g_b</span>)</span></span> -</span> w_b_tr.</span>transpose</span>(</span></span>)</span></span>.</span>dot</span>(</span>&</span>(</span>&</span>m_mat *</span> &</span>c</span>)</span></span></span>)</span></span></span>)</span></span>,</span>
</span></span></span></span></span>        </span>)</span></span>;</span>
</span></span></span></span>        ddf -=</span> g_b
</span></span></span></span>            *</span> theta.</span>mul_add</span>(</span>
</span></span></span></span></span>                g_b,</span>
</span></span></span></span></span>                (</span>-</span>2.</span>0</span> as</span> Float</span>)</span></span>.</span>mul_add</span>(</span>
</span></span></span></span></span></span>                    w_b_tr.</span>transpose</span>(</span></span>)</span></span>.</span>dot</span>(</span>&</span>(</span>&</span>m_mat *</span> &</span>p</span>)</span></span></span>)</span></span>,</span>
</span></span></span></span></span></span>                    -</span>(</span>g_b *</span> w_b_tr.</span>transpose</span>(</span></span>)</span></span>.</span>dot</span>(</span>&</span>(</span>&</span>m_mat *</span> w_b_tr.</span>transpose</span>(</span></span>)</span></span></span>)</span></span></span>)</span></span></span>)</span></span>,</span>
</span></span></span></span></span></span>                </span>)</span></span>,</span>
</span></span></span></span></span>            </span>)</span></span>;</span>
</span></span></span></span>        p +=</span> w_b_tr.</span>transpose</span>(</span></span>)</span></span>.</span>scale</span>(</span>g_b</span>)</span></span>;</span>
</span></span></span></span>        d[</span>b]</span></span> =</span> 0.</span>0</span>;</span>
</span></span></span></span>        dt_min =</span> -</span>df /</span> ddf;</span>
</span></span></span></span>        t_old =</span> t_b;</span>
</span></span></span></span>        i_free +=</span> 1</span>;</span>
</span></span></span></span>        if</span> i_free <</span> free_indices.</span>len</span>(</span></span>)</span></span> {</span>
</span></span></span></span></span>            b =</span> free_indices[</span>i_free]</span></span>;</span>
</span></span></span></span></span>            t_b =</span> t[</span>b]</span></span>;</span>
</span></span></span></span></span>            dt_b =</span> t_b -</span> t_old;</span>
</span></span></span></span></span>        </span>}</span></span> else</span> {</span>
</span></span></span></span></span>            t_b =</span> Float::</span></span>INFINITY</span>;</span>
</span></span></span></span></span>        </span>}</span></span>
</span></span></span></span>    </span>}</span></span>
</span></span></span>    dt_min =</span> Float::</span></span>max(</span>dt_min,</span> 0.</span>0</span></span>)</span></span>;</span>
</span></span></span>    t_old +=</span> dt_min;</span>
</span></span></span>    for</span> i in</span> 0</span>..</span>x.</span>len</span>(</span></span>)</span></span> {</span>
</span></span></span></span>        if</span> t[</span>i]</span></span> >=</span> t_b {</span>
</span></span></span></span></span>            x_cp[</span>i]</span></span> +=</span> t_old *</span> d[</span>i]</span></span>;</span>
</span></span></span></span></span>        </span>}</span></span>
</span></span></span></span>    </span>}</span></span>
</span></span></span>    let</span> free_indices =</span> (</span>0</span>..</span>free_indices.</span>len</span>(</span></span>)</span></span></span>)</span></span>
</span></span></span>        .</span>filter</span>(</span>|</span></span></span>&</span>i</span>|</span></span> </span>x_cp[</span>i]</span></span> <</span> u[</span>i]</span></span> &&</span> x_cp[</span>i]</span></span> ></span> l[</span>i]</span></span></span></span>)</span></span>
</span></span></span>        .</span>collect</span>(</span></span>)</span></span>;</span>
</span></span></span>    c +=</span> p.</span>scale</span>(</span>dt_min</span>)</span></span>;</span>
</span></span></span>    (</span>x_cp,</span> c,</span> free_indices</span>)</span></span>
</span></span></span></span>}</span></span></span>
</span></code></pre>
Subspace Minimization with the Primal Direct Method</h3>
The authors of [1]</a></sup> list three different ways to actually minimize the function after finding the generalized Cauchy point, the direct primal method, a method using conjugate gradients, and a dual method. We will only go over the direct primal method, since it’s the most common implementation in practice and the method I use in my own implementation.</p>
The details of this algorithm are quite daunting, but the general idea is that we can build a “reduced” Hessian of the quadratic approximate $m_k(x)$ which only acts on parameters which are not already at a boundary and then carry out a minimization over the subspace of free parameters. To do this, we start by constructing a matrix $Z_k$ which has dimensions $n\times n_\text{free}$ where</p>
$$ (Z_k)_{i,j} = \begin{cases} 1 & \text{if } i = j \text{ and } j\in\mathcal{F} \\ 0 & \text{otherwise} \end{cases} $$</p>
We can then imagine operations $\hat{B}_k = Z_k^{\intercal} B_k Z_k$ which take the Hessian as a $n\times n$ matrix to a reduced Hessian of dimension $n_\text{free}\times n_\text{free}$. We can compute a reduced gradient at the generalized Cauchy point as</p>
$$ \hat{r}^c = Z_k^{\intercal} (g_k + \theta(x^c - x_k) - W_k M_k \vec{c}) $$</p>
We then want to minimize</p>
$$ \hat{m}_k(\hat{d}) \equiv \hat{d}^{\intercal} \hat{r}^c + \frac{1}{2}\hat{d}^{\intercal} \hat{B}_k \hat{d}$$</p>
subject to $l_i - x_i^c \leq \hat{d}_i \leq u_i - x_i^c$ for $i\in\mathcal{F}$.</p>
It can be shown (though I will certainly not show the details) that</p>
$$ \hat{d}^u = -\left(\frac{1}{\theta}\hat{r}^c + \frac{1}{\theta^2}Z^{\intercal} W \left( I - \frac{1}{\theta} MW^{\intercal}ZZ^{\intercal}W \right)^{-1} MW^{\intercal}Z\hat{r}^c\right) $$</p>
is the direction in the subspace which minimizes $\hat{m}_k$. Note the leading minus sign! This cannot be found in equation 5.11 or step 6 of the direct primal method algorithm in [2]</a></sup>, but I believe this is a typo. This minus sign is correct, and I’ve confirmed this by looking at several implementations of this algorithm in several different languages. It is also correct within the paper itself, as can be seen in equation 5.7, which states $\hat{d}^u = -\hat{B}_k^{-1}\hat{r}^c$, combined with equation 5.10 which gives the definition for $\hat{B}^{-1}$. This was absolutely painful to debug, and I hope that if anyone feels the need to go through this paper again, they’ll see this and not have the same issues I did when I first coded this up.</p>
The code for this step looks like:</p>
fn</span> </span>direct_primal_min</span></span>(</span>
</span></span></span>    x</span>:</span> &</span>DVector<</span>Float></span></span>,
</span></span></span>    g</span>:</span> &</span>DVector<</span>Float></span></span>,
</span></span></span>    l</span>:</span> &</span>DVector<</span>Float></span></span>,
</span></span></span>    u</span>:</span> &</span>DVector<</span>Float></span></span>,
</span></span></span>    w_mat</span>:</span> &</span>DMatrix<</span>Float></span></span>,
</span></span></span>    m_mat</span>:</span> &</span>DMatrix<</span>Float></span></span>,
</span></span></span>    theta</span>:</span> Float,
</span></span></span>    x_cp</span>:</span> &</span>DVector<</span>Float></span></span>,
</span></span></span>    c</span>:</span> &</span>DVector<</span>Float></span></span>,
</span></span></span>    free_indices</span>:</span> &</span>[usize</span>],
</span></span></span></span>)</span></span></span></span> -></span> DVector<</span>Float></span></span></span> </span>{</span>
</span></span></span>    let</span> z_mat =</span> DMatrix::</span></span>from_fn(</span>x.</span>len</span>(</span></span>)</span></span>,</span> free_indices.</span>len</span>(</span></span>)</span></span>,</span> |</span></span></span>i</span>,</span> j</span>|</span></span> </span>{</span>
</span></span></span></span></span></span>        if</span> i ==</span> free_indices[</span>j]</span></span> {</span>
</span></span></span></span></span></span></span>            1.</span>0</span>
</span></span></span></span></span></span></span>        </span>}</span></span> else</span> {</span>
</span></span></span></span></span></span></span>            0.</span>0</span>
</span></span></span></span></span></span></span>        </span>}</span></span>
</span></span></span></span></span></span>    </span>}</span></span></span></span>)</span></span>;</span>
</span></span></span>    let</span> r_hat_c =</span> z_mat.</span>transpose</span>(</span></span>)</span></span>
</span></span></span>        *</span> (</span>&</span>g +</span> (</span>x_cp -</span> &</span>x</span>)</span></span>.</span>scale</span>(</span>theta</span>)</span></span> -</span> &</span>w_mat *</span> &</span>m_mat *</span> c</span>)</span></span>;</span>
</span></span></span>    let</span> w_tr_z_mat =</span> w_mat.</span>transpose</span>(</span></span>)</span></span> *</span> &</span>z_mat;</span>
</span></span></span>    let</span> n_mat =</span> DMatrix::</span></span>identity(</span>m_mat.</span>shape</span>(</span></span>)</span></span>.</span>0</span>,</span> m_mat.</span>shape</span>(</span></span>)</span></span>.</span>1</span></span>)</span></span>
</span></span></span>        -</span> (</span>&</span>m_mat *</span> (</span>&</span>w_tr_z_mat *</span> w_tr_z_mat.</span>transpose</span>(</span></span>)</span></span></span>)</span></span></span>)</span></span>.</span>unscale</span>(</span>theta</span>)</span></span>;</span>
</span></span></span>    let</span> n_mat_inv =</span> n_mat
</span></span></span>        .</span>try_inverse</span>(</span></span>)</span></span>
</span></span></span>        .</span>expect</span>(</span>"</span>Error: Something has gone horribly wrong, inversion of N^{-1} failed!"</span></span></span>)</span></span>;</span>
</span></span></span>    let</span> v =</span> n_mat_inv *</span> &</span>m_mat *</span> &</span>w_mat.</span>transpose</span>(</span></span>)</span></span> *</span> &</span>z_mat *</span> &</span>r_hat_c;</span>
</span></span></span>    let</span> d_hat_u =</span>
</span></span></span>        -</span>(</span>r_hat_c +</span> (</span>w_tr_z_mat.</span>transpose</span>(</span></span>)</span></span> *</span> v</span>)</span></span>.</span>unscale</span>(</span>theta</span>)</span></span></span>)</span></span>.</span>unscale</span>(</span>theta</span>)</span></span>;</span>
</span></span></span>    //</span> The minus sign here is missing in equation 5.11, this is a typo!
</span></span></span></span>    let</span> mut</span> alpha_star =</span> 1.</span>0</span>;</span>
</span></span></span>    for</span> i in</span> 0</span>..</span>free_indices.</span>len</span>(</span></span>)</span></span> {</span>
</span></span></span></span>        let</span> i_free =</span> free_indices[</span>i]</span></span>;</span>
</span></span></span></span>        alpha_star =</span> if</span> d_hat_u[</span>i]</span></span> ></span> 0.</span>0</span> {</span>
</span></span></span></span></span>            Float::</span></span>min(</span>alpha_star,</span> (</span>u[</span>i_free]</span></span> -</span> x_cp[</span>i_free]</span></span></span>)</span></span> /</span> d_hat_u[</span>i]</span></span></span>)</span></span>
</span></span></span></span></span>        </span>}</span></span> else</span> if</span> d_hat_u[</span>i]</span></span> <</span> 0.</span>0</span> {</span>
</span></span></span></span></span>            Float::</span></span>min(</span>alpha_star,</span> (</span>l[</span>i_free]</span></span> -</span> x_cp[</span>i_free]</span></span></span>)</span></span> /</span> d_hat_u[</span>i]</span></span></span>)</span></span>
</span></span></span></span></span>        </span>}</span></span> else</span> {</span>
</span></span></span></span></span>            alpha_star
</span></span></span></span></span>        </span>}</span></span>
</span></span></span></span>    </span>}</span></span>
</span></span></span>    let</span> mut</span> x_bar =</span> x_cp.</span>clone</span>(</span></span>)</span></span>;</span>
</span></span></span>    let</span> d_hat_star =</span> d_hat_u.</span>scale</span>(</span>alpha_star</span>)</span></span>;</span>
</span></span></span>    let</span> z_d_hat_star =</span> &</span>z_mat *</span> d_hat_star;</span>
</span></span></span>    for</span> i in</span> free_indices {</span>
</span></span></span></span>        x_bar[</span>*</span>i]</span></span> +=</span> z_d_hat_star[</span>*</span>i]</span></span>
</span></span></span></span>    </span>}</span></span>
</span></span></span>    x_bar
</span></span></span></span>}</span></span></span>
</span></code></pre>
The rest of the algorithm is fairly similar to the BFGS algorithm. We get a search direction, which is determined either by x_bar - x</code> (or x_cp - x</code> if there are no remaining free indices, which is why we don’t just return the search direction immediately), then we do a line search along this direction, then we move and check if the termination conditions are met. We obtain a new update for $\vec{s}_k$ and $\vec{y}_k$ and get rid of the oldest one if the new one meets the curvature condition, and repeat!</p>
The link to the full implementation is here</a>. Note that this is a permalink to the most recent commit at time of writing, but I plan to make a rather large update to the API which will slightly alter some of the structure of the algorithm.</p>
There are likely several places where I could optimize the code, but I haven’t really gotten to that yet. As far as I know, this is the first and possibly only pure Rust implementation of the L-BFGS-B algorithm (if someone else has a better claim, I’ll recant), and there is probably some room for improvement. If you find something that could be optimized, let me know, make a PR, send me an email!</p>
This is the last post in my first series of blog posts on this site. As you might be able to tell, I’m still getting used to writing regularly, and I’ve been a bit preoccupied with my PhD thesis lately, but I hope to have more interesting posts in the future (which will hopefully be written with a bit more forethought than these have been). I hope anyone reading this has enjoyed it, and I hope it clears up some of the general ideas of each algorithm. If anyone is interested in a more in-depth review, check out the cited papers here or any of the citations in the previous posts, as they are full of interesting notes and insights.</p>


R. H. Byrd, J. Nocedal, and R. B. Schnabel, “Representations of quasi-Newton matrices and their use in limited memory methods,” Mathematical Programming, vol. 63, no. 1–3, pp. 129–156, Jan. 1994, doi: 10.1007/bf01582063.
</a> ↩</a> ↩2</a></p>
</li>

R. H. Byrd, P. Lu, J. Nocedal, and C. Zhu, “A Limited Memory Algorithm for Bound Constrained Optimization,” SIAM J. Sci. Comput., vol. 16, no. 5, pp. 1190–1208, Sep. 1995, doi: 10.1137/0916069.</a> ↩</a> ↩2</a> ↩3</a></p>
</li>
</ol>


Tzigane
Dene Hoffman — Mon, 16 Jun 2025 00:00:00 +0000

</iframe>
I was given an opportunity to perform Ravel’s Tzigane</em> with the All University Orchestra in the fall of 2024 after winning the previous year’s concerto competition. It took a bit to get the microphone audio, but I finally have a complete recording!</p>


The BFGS Algorithm Family in Rust (Part 2)
Dene Hoffman — Sun, 15 Dec 2024 00:00:00 +0000

	
		</i>Note</p>
	Since writing the previous post</a>, I have made several simplifications to the library that will change how some structs and traits are used here. The biggest change is that I’ve abandoned generic floats in favor of a feature gate:</p>
#</span>[</span>cfg</span>(</span></span></span>not</span>(</span></span></span>feature =</span> "</span>f32"</span></span></span>)</span></span></span></span></span>)</span></span></span>]</span></span>
</span>pub</span> type</span> Float</span> =</span> f64</span>;</span>
</span> 
</span>#</span>[</span>cfg</span>(</span></span></span>feature =</span> "</span>f32"</span></span></span></span>)</span></span></span>]</span></span>
</span>pub</span> type</span> Float</span> =</span> f32</span>;</span>
</span></code></pre>
With this code, I can turn the f32</code> version of my crate on and off with a feature flag. While generics can be nice, in this instance they were actually causing the crate to be slower (just according to some of my own internal benchmarks, this might not be the case in general) and more difficult to read. Any instances in the last post where there was a generic type T</code> representing floating-point values have been replaced with this Float</code> type, so Status<T></code> is now just Status</code>, for example.</p>

</blockquote>
Let’s talk about the core BFGS algorithm. The details I’m using here can be found in Nocedal and Wright’s book “Numerical Optimization”</a> in Chapter 6: Quasi-Newton Methods. In the last post, I alluded to this and described gradient descent, but we should really cover what Newton’s method is first.</p>
Newton’s Method</h1>
The core idea of gradient descent was that we get the gradient at the initial point and walk downhill. We can couple this with a line search to figure out the optimal step size to take downhill, but that’s about the best we can do with a single derivative. With two derivatives, however, we can do just a bit better (converge faster). First, let’s Taylor expand our function around some point $x$:</p>
$$ f(x + t) \approx f(x) + f’(x)t + \frac{1}{2}f’’(x)t^2 $$</p>
We can imagine this as a function of just $t$, keeping $x$ fixed, and try to figure out the value of $t$ that minimizes the function. This can be done by realizing that the derivative of $f(x + t)$ with respect to $t$ should be zero at extrema and the second derivative should be positive at a minimum:</p>
$$ 0 = \frac{\text{d}}{\text{d}t} \left(f(x) + f’(x)t  \frac{1}{2} f’‘(x)t^2 \right) = f’(x) + f’’(x)t $$</p>
or $t = -\frac{f’(x)}{f’’(x)}$.</p>
In more than one dimension $n$, we write $f’(x) \to \vec{\nabla}f(\vec{x})$ (a $n$-dimensional vector) and $f’’(x) \to \nabla^2 f(\vec{x}) = H$. This second derivative term is an $n \times n$ matrix called the Hessian, and the matrix equivalent of the reciprocal here is the matrix inverse. The Hessian is just a matrix of second derivatives, $H_{ij} = \frac{\partial^2 f(\vec{x})}{\partial \vec{x}_i \partial \vec{x}_j}$, so it is symmetric by definition.</p>
The Hessian is also positive definite, which just means that for all nonzero vectors $\vec{q}$,</p>
$$\vec{q}^{\intercal} H \vec{q} > 0$$</p>
With all of this in mind, recall that the gradient descent update step was</p>
$$ \vec{x}_{k+1} = \vec{x}_k - \alpha_k \vec{\nabla} f(\vec{x}_k) $$</p>
so the equivalent second-derivative (Newton) update step is</p>
$$ \vec{x}_{k+1} = \vec{x}_k - \alpha_k H_k^{-1} \vec{\nabla} f(\vec{x}_k) $$</p>
As far as I know, this is the optimal second-order method for numeric optimization (in theory). In practice, it tends to be horribly inefficient to compute a Hessian matrix and subsequently invert it, and this problem gets worse when you increase the number of free parameters. Particularly, we rarely know the analytic first and second derivatives of our objective function (and it could be argued that knowing those would often make optimization trivial). In most cases, we would have to do these derivatives numerically:</p>
  fn</span> </span>hessian</span></span>(</span>&</span>self</span>, x</span>:</span> &</span>[Float], user_data</span>:</span> &</span>mut</span> U</span>)</span></span></span></span> -></span> Result</span><</span>DMatrix<</span>Float></span></span>, E></span></span></span> </span>{</span>
</span></span></span>      let</span> x =</span> DVector::</span></span>from_column_slice(</span>x</span>)</span></span>;</span>
</span></span></span>      let</span> h:</span> DVector<</span>Float></span></span> =</span> x
</span></span></span>          .</span>iter</span>(</span></span>)</span></span>
</span></span></span>          .</span>map</span>(</span>|</span></span></span>&</span>xi</span>|</span></span> </span>Float::</span></span>cbrt(</span>Float::</span></span>EPSILON</span></span>)</span></span> *</span> (</span>xi.</span>abs</span>(</span></span>)</span></span> +</span> 1.</span>0</span></span>)</span></span></span></span>)</span></span>
</span></span></span>          .</span>collect::</span></span><</span>Vec</span><</span>_</span>></span></span>></span></span>(</span></span>)</span></span>
</span></span></span>          .</span>into</span>(</span></span>)</span></span>;</span>
</span></span></span>      let</span> mut</span> res =</span> DMatrix::</span></span>zeros(</span>x.</span>len</span>(</span></span>)</span></span>,</span> x.</span>len</span>(</span></span>)</span></span></span>)</span></span>;</span>
</span></span></span>      let</span> mut</span> g_plus =</span> DMatrix::</span></span>zeros(</span>x.</span>len</span>(</span></span>)</span></span>,</span> x.</span>len</span>(</span></span>)</span></span></span>)</span></span>;</span>
</span></span></span>      let</span> mut</span> g_minus =</span> DMatrix::</span></span>zeros(</span>x.</span>len</span>(</span></span>)</span></span>,</span> x.</span>len</span>(</span></span>)</span></span></span>)</span></span>;</span>
</span></span></span>      for</span> i in</span> 0</span>..</span>x.</span>len</span>(</span></span>)</span></span> {</span>
</span></span></span></span>          let</span> mut</span> x_plus =</span> x.</span>clone</span>(</span></span>)</span></span>;</span>
</span></span></span></span>          let</span> mut</span> x_minus =</span> x.</span>clone</span>(</span></span>)</span></span>;</span>
</span></span></span></span>          x_plus[</span>i]</span></span> +=</span> h[</span>i]</span></span>;</span>
</span></span></span></span>          x_minus[</span>i]</span></span> -=</span> h[</span>i]</span></span>;</span>
</span></span></span></span>          g_plus.</span>set_column</span>(</span>i,</span> &</span>self</span>.</span>gradient</span>(</span>x_plus.</span>as_slice</span>(</span></span>)</span></span>,</span> user_data</span>)</span></span>?</span></span>)</span></span>;</span>
</span></span></span></span>          g_minus.</span>set_column</span>(</span>i,</span> &</span>self</span>.</span>gradient</span>(</span>x_minus.</span>as_slice</span>(</span></span>)</span></span>,</span> user_data</span>)</span></span>?</span></span>)</span></span>;</span>
</span></span></span></span>          for</span> j in</span> 0</span>..</span>=</span>i {</span>
</span></span></span></span></span>              if</span> i ==</span> j {</span>
</span></span></span></span></span></span>                  res[</span>(</span>i,</span> j</span>)</span></span>]</span></span> =</span> (</span>g_plus[</span>(</span>i,</span> j</span>)</span></span>]</span></span> -</span> g_minus[</span>(</span>i,</span> j</span>)</span></span>]</span></span></span>)</span></span> /</span> (</span>2.</span>0</span> *</span> h[</span>i]</span></span></span>)</span></span>;</span>
</span></span></span></span></span></span>              </span>}</span></span> else</span> {</span>
</span></span></span></span></span></span>                  res[</span>(</span>i,</span> j</span>)</span></span>]</span></span> =</span> (</span>(</span>g_plus[</span>(</span>i,</span> j</span>)</span></span>]</span></span> -</span> g_minus[</span>(</span>i,</span> j</span>)</span></span>]</span></span></span>)</span></span> /</span> (</span>4.</span>0</span> *</span> h[</span>j]</span></span></span>)</span></span></span>)</span></span>
</span></span></span></span></span></span>                      +</span> (</span>(</span>g_plus[</span>(</span>j,</span> i</span>)</span></span>]</span></span> -</span> g_minus[</span>(</span>j,</span> i</span>)</span></span>]</span></span></span>)</span></span> /</span> (</span>4.</span>0</span> *</span> h[</span>i]</span></span></span>)</span></span></span>)</span></span>;</span>
</span></span></span></span></span></span>                  res[</span>(</span>j,</span> i</span>)</span></span>]</span></span> =</span> res[</span>(</span>i,</span> j</span>)</span></span>]</span></span>;</span>
</span></span></span></span></span></span>              </span>}</span></span>
</span></span></span></span></span>          </span>}</span></span>
</span></span></span></span>      </span>}</span></span>
</span></span></span>      Ok</span>(</span>res</span>)</span></span>
</span></span></span>  </span>}</span></span></span>
</span></code></pre>
I’m not sure if I’ve written this function as efficiently as possible, but already we can see the issue. To accurately calculate the Hessian, we need to take the gradient in two places for each free parameter (g_plus</code> and g_minus</code>), a process which requires at least two function calls for each free parameter. That’s a total of $4n^2$ function evaluations for $n$ free parameters. If the function takes one second to evaluate and has ten free parameters, the gradient will take twenty seconds, but the Hessian will take over six minutes. Now it might seem silly to speculate on a function which takes a full second to evaluate (at that point, we can expect any optimization problem is going to take a while), but the real issue is scaling. If we add a single free parameter to this problem, it now takes eight minutes. Most of the problems I deal with in my research can be evaluated in less than 200ms, but they contain over 40 free parameters. Do the math, that’s 21 minutes for a single evaluation of the Hessian, and we have to do that at every step in the optimization!</p>
Fortunately, there’s a better, faster way. Rather than calculating the full Hessian, let’s just approximate it! After all, in the limit of very small steps, the Hessian shouldn’t be changing that much (for nice functions), so it makes sense that there should be some way to approximate the next Hessian given the current (and the gradients at both points). This is what puts the “quasi” in quasi-Newton methods. We outline some way of approximating the current Hessian given some history of past Hessian approximates and gradients and apply Newton’s method for optimization.</p>
Let’s refer to this approximate Hessian as $B_k$[1]</a></sup>. Furthermore, following the derivation in Nocedal and Wright, let’s write the second-order Taylor expansion of our function at the point $\vec{x}_k$. I’ll use $f_k \equiv f(\vec{x}_k)$ to represent the objective function and $\vec{g}_k \equiv \vec{\nabla}f(\vec{x}_k)$ to represent the gradient.</p>
$$
m_k(\vec{p}) = f_k + \vec{g}_k^{\intercal} \vec{p} + \frac{1}{2} \vec{p}^{\intercal} B_k \vec{p}
$$</p>
This approximation is identical to our function when $\vec{p} = 0$, and furthermore, the gradient is also the same. The only difference is in the second derivatives, which are approximately the same. We can assume that we’ll use some line search to find the optimal step length $\alpha_k$ to get to the new point, $\vec{x}_{k+1} = \vec{x}_k + \alpha_k \vec{p}_k$. Let’s go to this next point and write out the Taylor expansion for the next step here:</p>
$$
m_{k+1}(\vec{p}) = f_{k+1} + \vec{g}_{k+1}^{\intercal} \vec{p} + \frac{1}{2} \vec{p}^{\intercal} B_{k+1} \vec{p}
$$</p>
Now imagine we took this step and looked back at the old position and calculated the gradient using this new expansion evaluated at the old position. We would hope to get the same result as when we used our old expansion! We can actually make this a constraint on our matrix $B$. To look back at the old position, we evaluate the function at $p = -\alpha_k\vec{p}_k$, since this is the step we just took but reversed in direction. We can then write the gradient as</p>
$$
\vec{\nabla} m_{k+1}(-\alpha_k\vec{p}_k) = \vec{g}_{k+1} - \alpha_k B_{k+1} \vec{p}_k
$$</p>
and with the requirement that $ \vec{\nabla} m_{k+1}(-\alpha_k\vec{p}_k) = \vec{g}_k $, we get a new rule for our approximate Hessian:</p>
$$
B_{k+1}\alpha_k\vec{p}_k = \vec{g}_{k+1} - \vec{g}_k
$$</p>
There is some conventional notation to this that we will use throughout the derivation. Rather than using each individual gradient and position, it is helpful to just think about the change in gradient and position, or</p>
$$ \vec{s}_k = \vec{x}_{k+1} - \vec{x}_k = \alpha_k\vec{p}_k $$</p>
and</p>
$$ \vec{y}_k = \vec{g}_{k+1} - \vec{g}_k $$</p>
In this notation, the gradient requirement (called the secant equation), is as follows:</p>
$$ B_{k+1}\vec{s}_k = \vec{y}_k $$</p>
Remember when we said the Hessian is positive definite? We would like our Hessian approximate to be positive definite too. Since we don’t include any zero-length jumps, $|\vec{s}_k| > 0$, so we can multiply the previous equation by another $\vec{s}_k$ to find:</p>
$$ \vec{s}_k^{\intercal} B_{k+1} \vec{s}_k > 0 \implies \vec{s}_k^{\intercal} \vec{y}_k > 0 $$</p>
This requirement on $\vec{s}_k$ and $\vec{y}_k$ is called the curvature condition. As it turns out, there’s a fairly simple way to ensure this using the Wolfe condition from my previous article. Recall that during our line search, we required that</p>
$$\begin{align}\vec{p}_k^{\intercal} \vec{g}_{k+1} &\geq c_2(\vec{p}_k^{\intercal} \vec{g}_k) \\ \vec{s}_k^{\intercal} \vec{g}_{k+1} &\geq c_2(\vec{s}_k^{\intercal} \vec{g}_k) \\ \vec{s}_k^{\intercal} (\vec{y}_k + \vec{g}_{k}) &\geq c_2(\vec{s}_k^{\intercal} \vec{g}_k) \\ \vec{s}_k^{\intercal} \vec{y}_k + \vec{s}_k^{\intercal} \vec{g}_{k} &\geq c_2(\vec{s}_k^{\intercal} \vec{g}_k) \\ \vec{s}_k^{\intercal} \vec{y}_k &\geq (c_2 - 1)(\vec{s}_k^{\intercal} \vec{g}_k) \\ \vec{s}_k^{\intercal} \vec{y}_k &\geq (c_2 - 1)\alpha_k \vec{p}_k \vec{g}_k \end{align}$$</p>
Since we require $c_2 > 1$ and $\alpha_k > 0$, and assuming $\vec{p}_k$ is somewhat aligned with the gradient (or at a minimum, $\vec{p}_k$ is pointing somewhere downhill), then the curvature condition clearly holds.</p>
Unfortunately, we’re only a bit better off than we were when we had no idea what $B$ was. While these conditions limit the space of possible $B$ matrices, the set is still infinite. We need to find more ways to narrow down the space to a unique solution.</p>
One way we could approach this is by assuming that $B$ shouldn’t change too much</em> at each update. There are a number of ways of going about this, but the basic idea is that we choose some norm on the change of the matrix between updates with different norms leading to different quasi-Newton algorithms. In particular, minimizing the Frobenius norm for this particular problem will give us the Davidon-Fletcher-Powell (DFP) algorithm. It has a nice update step which we can apply to $B$ that looks like this:</p>
$$B_{k+1} = (I - \rho_k \vec{y}_k \vec{s}_k^{\intercal}) B_k (I - \rho_k \vec{s}_k \vec{y}_k^{\intercal}) + \rho_k \vec{y}_k \vec{y}_k^{\intercal}$$</p>
where $\rho_k = \frac{1}{\vec{y}_k^{\intercal} \vec{s}_k}$. However, we aren’t actually done yet, since in the quasi-Newton update step, we really need the matrix $B_{k+1}^{-1}$. We could go and calculate a matrix inverse at each step, but it turns out we don’t need to thanks to the Sherman-Morrison-Woodbury formula, which says that the inverse of a rank-$k$ correction on a matrix is equivalent to a rank-$k$ correction on the inverse, or more precisely,</p>
$$(B + UCV)^{-1} = B^{-1} - B^{-1}U(C^{-1}+VB^{-1}U)^{-1}VB^{-1}$$</p>
where $B$ is $n\times n$, $C$ is $k\times k$, $U$ is $n\times k$, and $V$ is $k\times n$. This is a tough one, but we should take the time to work it out. First we expand the right side of the DFP formula:</p>
$$
B_k - \rho_k \vec{y}_k \vec{s}_k^{\intercal} B_k - \rho_k B_k \vec{s}_k \vec{y}_k^{\intercal} + \rho_k^2 \vec{y}_k \vec{s}_k^{\intercal} B_k \vec{s}_k \vec{y}_k^{\intercal} + \rho_k \vec{y}_k \vec{y}_k^{\intercal}
$$</p>
Everything except for the very first $B_k$ should be thought of as those matrices $U$, $V$, and $C$. It is also helpful to know that by a rank-$k$ correction, we mean that the correction is written as $A’ = A + \vec{u}_1\vec{v}_1^{\intercal} + \vec{u}_2\vec{v}_2^{\intercal} + … + \vec{u}_k\vec{v}_k^{\intercal}$. The two terms in the original DFP formula form such a correction of rank-2, since the first terms involve both $\vec{y}_k$ and $\vec{s}_k$ scaled by $\rho_k$ and $B_k$ while the second term only involves $\vec{y}_k$ scaled by $\rho_k$, making it linearly independent. We then require $C$ to be a $2\times 2$ matrix, $U$ to be $n\times 2$, and $V$ to be $2\times n$, where $n$ is the dimensionality of the Hessian. It’s a bit tricky to see, but we can write this all in the form:[2]</a></sup></p>
$$
B_{k+1} = B_k + \begin{bmatrix}\rho_k\vec{y}_k & \rho_k B_k\vec{s}_k \end{bmatrix} \begin{bmatrix}1 + \rho_k \vec{s}_k^{\intercal} B_k \vec{s}_k & -1 \\ -1 & 0 \end{bmatrix} \begin{bmatrix} \vec{y}_k \\ \vec{s}_k B_k \end{bmatrix}
$$</p>
I’ll leave the actual application of the Sherman-Morrison-Woodbury formula as an exercise to the reader, but the conclusion is this rather nice looking update step for the inverse of the (approximate) Hessian:</p>
$$
B_{k+1}^{-1} = B_k^{-1} - \frac{B_k^{-1} \vec{y}_k \vec{y}_k^{\intercal} B_k^{-1}}{\vec{y}_k^{\intercal} B_k^{-1} \vec{y}_k} + \frac{\vec{s}_k \vec{s}_k^{\intercal}}{\vec{y}_k^{\intercal} \vec{s}_k}
$$</p>
At this point, we should be pretty happy. Given some starting value for the inverse Hessian, we have a way of determining a new inverse Hessian at our next step given the value of the old one! We could go and write this up, but then this article would need a different title. It just so happens that you can apply this very same process, but swap out $B$ for $B^{-1}$ at every step (with a new secant condition, $ B_{k+1}^{-1}\vec{y}_k = \vec{s}_k $) and skip that last bit entirely, and it works better in practice</em>! In other words, the update step,</p>
$$ B_{k+1}^{-1} = (I - \rho_k \vec{s}_k \vec{y}_k^{\intercal})B_{k}^{-1} (I - \rho_k \vec{y}_k \vec{s}_k^{\intercal}) + \rho_k \vec{s}_k \vec{s}_k^{\intercal} $$</p>
is also a unique solution to our quasi-Newton problem. This is the BFGS update. Of course, we still have the problem of what values should be assigned to $B_0^{-1}$. The neat part about BFGS is that this update step tends to be self-correcting (DFP is generally worse in this regard), assuming the Wolfe condition is met in the line search (this is why I spent so much time on the line search part of the previous post, it really is that important to make educated leaps in the gradient direction!). What this means is that even if we start with a bad initial guess at $B_0^{-1}$, after some number of steps, we should get to a good approximation regardless. We could of course just start with the numerically calculated Hessian, invert it, and go from there, but in practice, this really isn’t worth the effort. Instead, we can just use the identity matrix[3]</a></sup> for the very first step (just regular gradient descent!) and then correct that identity matrix each time. We will later find that this isn’t even the most efficient way of doing things when we look at L-BFGS, but for now we have a formula, let’s implement it!</p>
Let’s start with a struct to hold our new algorithm:</p>
#</span>[</span>derive</span>(</span></span></span>Clone</span></span>)</span></span></span>]</span></span>
</span>pub</span> struct</span> </span>BFGS</span><</span>U, E></span></span></span></span> </span>{</span>
</span></span></span>    x</span>:</span> DVector<</span>Float></span></span>,
</span></span></span>    g</span>:</span> DVector<</span>Float></span></span>,
</span></span></span>    h_inv</span>:</span> DMatrix<</span>Float></span></span>,
</span></span></span>    line_search</span>:</span> Box</span><</span>dyn LineSearch<</span>U, E></span></span>></span></span>,
</span></span></span></span>}</span></span></span>
</span>
</span>impl</span></span><</span>U, E></span></span></span> Default for</span></span> BFGS</span><</span>U, E></span></span> </span>{</span>
</span></span></span>    fn</span> </span>default</span></span>(</span></span>)</span></span></span></span> -></span> Self</span></span> </span>{</span>
</span></span></span></span></span>        Self</span> {</span>
</span></span></span></span></span></span>            x:</span> Default</span>::</span></span>default(</span></span>)</span></span>,</span>
</span></span></span></span></span></span>            g:</span> Default</span>::</span></span>default(</span></span>)</span></span>,</span>
</span></span></span></span></span></span>            h_inv:</span> Default</span>::</span></span>default(</span></span>)</span></span>,</span>
</span></span></span></span></span></span>            line_search:</span> Box</span>:</span>:</span><</span>StrongWolfeLineSearch></span></span>::</span></span>default(</span></span>)</span></span>,</span>
</span></span></span></span></span></span>        </span>}</span></span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span></span>}</span></span></span>
</span></code></pre>
Next, our update formula:</p>
impl</span></span><</span>U, E></span></span></span> BFGS</span><</span>U, E></span></span> </span>{</span>
</span></span></span>    fn</span> </span>update_h_inv</span></span>(</span>&</span>mut</span> self</span>, step</span>:</span> usize</span>, n</span>:</span> usize</span>, s</span>:</span> &</span>DVector<</span>Float></span></span>, y</span>:</span> &</span>DVector<</span>Float></span></span></span>)</span></span></span></span> </span>{</span>
</span></span></span></span></span>        if</span> step ==</span> 0</span> {</span>
</span></span></span></span></span></span>            self</span>.</span>h_inv =</span> self</span>.</span>h_inv.</span>scale</span>(</span>(</span>y.</span>dot</span>(</span>s</span>)</span></span></span>)</span></span> /</span> (</span>y.</span>dot</span>(</span>y</span>)</span></span></span>)</span></span></span>)</span></span>;</span>
</span></span></span></span></span></span>        </span>}</span></span>
</span></span></span></span></span>        let</span> rho =</span> Float::</span></span>recip(</span>y.</span>dot</span>(</span>s</span>)</span></span></span>)</span></span>;</span>
</span></span></span></span></span>        let</span> m_left =</span> DMatrix::</span></span>identity(</span>n,</span> n</span>)</span></span> -</span> (</span>y *</span> s.</span>transpose</span>(</span></span>)</span></span></span>)</span></span>.</span>scale</span>(</span>rho</span>)</span></span>;</span>
</span></span></span></span></span>        let</span> m_right =</span> DMatrix::</span></span>identity(</span>n,</span> n</span>)</span></span> -</span> (</span>s *</span> y.</span>transpose</span>(</span></span>)</span></span></span>)</span></span>.</span>scale</span>(</span>rho</span>)</span></span>;</span>
</span></span></span></span></span>        let</span> m_add =</span> (</span>s *</span> s.</span>transpose</span>(</span></span>)</span></span></span>)</span></span>.</span>scale</span>(</span>rho</span>)</span></span>;</span>
</span></span></span></span></span>        self</span>.</span>h_inv =</span> (</span>m_left *</span> &</span>self</span>.</span>h_inv *</span> m_right</span>)</span></span> +</span> m_add;</span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span></span>}</span></span></span>
</span></code></pre>
I’ve modified the Algorithm</code> trait since making the previous post:</p>
pub</span> trait</span> Algorithm</span><U, E>: DynClone {</span>
</span></span></span>    fn</span> </span>initialize</span></span>(</span>
</span></span></span></span></span>        &</span>mut</span> self</span>,
</span></span></span></span></span>        func</span>:</span> &</span>dyn Function<</span>U, E></span></span>,
</span></span></span></span></span>        x0</span>:</span> &</span>[Float],
</span></span></span></span></span>        bounds</span>:</span> Option</span><</span>&</span>Vec</span><</span>Bound></span></span>></span></span>,
</span></span></span></span></span>        user_data</span>:</span> &</span>mut</span> U,
</span></span></span></span></span>        status</span>:</span> &</span>mut</span> Status,
</span></span></span></span></span>    </span>)</span></span></span></span> -></span> Result</span><</span>(</span>)</span>, E></span></span></span></span>;</span>
</span></span></span>    fn</span> </span>step</span></span>(</span>
</span></span></span></span></span>        &</span>mut</span> self</span>,
</span></span></span></span></span>        i_step</span>:</span> usize</span>,
</span></span></span></span></span>        func</span>:</span> &</span>dyn Function<</span>U, E></span></span>,
</span></span></span></span></span>        bounds</span>:</span> Option</span><</span>&</span>Vec</span><</span>Bound></span></span>></span></span>,
</span></span></span></span></span>        user_data</span>:</span> &</span>mut</span> U,
</span></span></span></span></span>        status</span>:</span> &</span>mut</span> Status,
</span></span></span></span></span>    </span>)</span></span></span></span> -></span> Result</span><</span>(</span>)</span>, E></span></span></span></span>;</span>
</span></span></span>    fn</span> </span>check_for_termination</span></span>(</span>
</span></span></span></span></span>        &</span>mut</span> self</span>,
</span></span></span></span></span>        func</span>:</span> &</span>dyn Function<</span>U, E></span></span>,
</span></span></span></span></span>        bounds</span>:</span> Option</span><</span>&</span>Vec</span><</span>Bound></span></span>></span></span>,
</span></span></span></span></span>        user_data</span>:</span> &</span>mut</span> U,
</span></span></span></span></span>        status</span>:</span> &</span>mut</span> Status,
</span></span></span></span></span>    </span>)</span></span></span></span> -></span> Result</span><</span>bool</span>, E></span></span></span></span>;</span>
</span></span></span>    fn</span> </span>postprocessing</span></span>(</span>
</span></span></span></span></span>        &</span>mut</span> self</span>,
</span></span></span></span></span>        func</span>:</span> &</span>dyn Function<</span>U, E></span></span>,
</span></span></span></span></span>        bounds</span>:</span> Option</span><</span>&</span>Vec</span><</span>Bound></span></span>></span></span>,
</span></span></span></span></span>        user_data</span>:</span> &</span>mut</span> U,
</span></span></span></span></span>        status</span>:</span> &</span>mut</span> Status,
</span></span></span></span></span>    </span>)</span></span></span></span> -></span> Result</span><</span>(</span>)</span>, E></span></span></span> </span>{</span>
</span></span></span></span></span>        Ok</span>(</span>(</span></span>)</span></span></span>)</span></span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span></span>}</span></span></span>
</span></code></pre>
We now just pass the Minimizer</code>’s Status</code> in as a mutable reference and ask the Algorithm</code> to modify it accordingly. The implementation of the BFGS Algorithm</code> is simply:</p>
impl</span></span><</span>U, E></span></span></span> Algorithm<</span>U, E></span></span> for</span></span> BFGS</span><</span>U, E></span></span>
</span></span></span>where</span>
</span></span>    U:</span> Clone,
</span></span>    E:</span> Clone,
</span></span></span>{</span>
</span></span></span>    fn</span> </span>initialize</span></span>(</span>
</span></span></span></span></span>        &</span>mut</span> self</span>,
</span></span></span></span></span>        func</span>:</span> &</span>dyn Function<</span>U, E></span></span>,
</span></span></span></span></span>        x0</span>:</span> &</span>[Float],
</span></span></span></span></span>        bounds</span>:</span> Option</span><</span>&</span>Vec</span><</span>Bound></span></span>></span></span>,
</span></span></span></span></span>        user_data</span>:</span> &</span>mut</span> U,
</span></span></span></span></span>        status</span>:</span> &</span>mut</span> Status,
</span></span></span></span></span>    </span>)</span></span></span></span> -></span> Result</span><</span>(</span>)</span>, E></span></span></span> </span>{</span>
</span></span></span></span></span>        self</span>.</span>f_previous =</span> Float::</span></span>INFINITY</span>;</span>
</span></span></span></span></span>        self</span>.</span>h_inv =</span> DMatrix::</span></span>identity(</span>x0.</span>len</span>(</span></span>)</span></span>,</span> x0.</span>len</span>(</span></span>)</span></span></span>)</span></span>;</span>
</span></span></span></span></span>        self</span>.</span>x =</span> x0;</span>
</span></span></span></span></span>        self</span>.</span>g =</span> func.</span>gradient</span>(</span>self</span>.</span>x.</span>as_slice</span>(</span></span>)</span></span>,</span> user_data</span>)</span></span>?</span>;</span>
</span></span></span></span></span>        status.</span>inc_n_g_evals</span>(</span></span>)</span></span>;</span>
</span></span></span></span></span>        status.</span>update_position</span>(</span>(</span>
</span></span></span></span></span></span></span>            self</span>.</span>x.</span>as_slice</span>(</span></span>)</span></span>,</span>
</span></span></span></span></span></span></span>            func.</span>evaluate</span>(</span>self</span>.</span>x.</span>as_slice</span>(</span></span>)</span></span>,</span> user_data</span>)</span></span>?</span>,</span>
</span></span></span></span></span></span></span>        </span>)</span></span></span>)</span></span>;</span>
</span></span></span></span></span>        status.</span>inc_n_f_evals</span>(</span></span>)</span></span>;</span>
</span></span></span></span></span>        Ok</span>(</span>(</span></span>)</span></span></span>)</span></span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span>
</span></span></span>    fn</span> </span>step</span></span>(</span>
</span></span></span></span></span>        &</span>mut</span> self</span>,
</span></span></span></span></span>        i_step</span>:</span> usize</span>,
</span></span></span></span></span>        func</span>:</span> &</span>dyn Function<</span>U, E></span></span>,
</span></span></span></span></span>        bounds</span>:</span> Option</span><</span>&</span>Vec</span><</span>Bound></span></span>></span></span>,
</span></span></span></span></span>        user_data</span>:</span> &</span>mut</span> U,
</span></span></span></span></span>        status</span>:</span> &</span>mut</span> Status,
</span></span></span></span></span>    </span>)</span></span></span></span> -></span> Result</span><</span>(</span>)</span>, E></span></span></span> </span>{</span>
</span></span></span></span></span>        let</span> d =</span> -</span>&</span>self</span>.</span>h_inv *</span> &</span>self</span>.</span>g;</span>
</span></span></span></span></span>        let</span> (</span>valid,</span> alpha,</span> f_kp1,</span> g_kp1</span>)</span></span> =</span> self</span>.</span>line_search.</span>search</span>(</span>
</span></span></span></span></span></span>            &</span>self</span>.</span>x,</span>
</span></span></span></span></span></span>            &</span>d,</span>
</span></span></span></span></span></span>            Some</span>(</span>self</span>.</span>max_step</span>)</span></span>,</span>
</span></span></span></span></span></span>            func,</span>
</span></span></span></span></span></span>            bounds,</span>
</span></span></span></span></span></span>            user_data,</span>
</span></span></span></span></span></span>            status,</span>
</span></span></span></span></span></span>        </span>)</span></span>?</span>;</span>
</span></span></span></span></span>        if</span> valid {</span>
</span></span></span></span></span></span>            let</span> dx =</span> d.</span>scale</span>(</span>alpha</span>)</span></span>;</span>
</span></span></span></span></span></span>            let</span> grad_kp1_vec =</span> g_kp1;</span>
</span></span></span></span></span></span>            let</span> dg =</span> &</span>grad_kp1_vec -</span> &</span>self</span>.</span>g;</span>
</span></span></span></span></span></span>            let</span> n =</span> self</span>.</span>x.</span>len</span>(</span></span>)</span></span>;</span>
</span></span></span></span></span></span>            self</span>.</span>update_h_inv</span>(</span>i_step,</span> n,</span> &</span>dx,</span> &</span>dg</span>)</span></span>;</span>
</span></span></span></span></span></span>            self</span>.</span>x +=</span> dx;</span>
</span></span></span></span></span></span>            self</span>.</span>g =</span> grad_kp1_vec;</span>
</span></span></span></span></span></span>            status.</span>update_position</span>(</span>(</span>self</span>.</span>x.</span>as_slice</span>(</span></span>)</span></span>,</span> f_kp1</span>)</span></span></span>)</span></span>;</span>
</span></span></span></span></span></span>        </span>}</span></span> else</span> {</span>
</span></span></span></span></span></span>            status.</span>set_converged</span>(</span></span>)</span></span>;</span>
</span></span></span></span></span></span>        </span>}</span></span>
</span></span></span></span></span>        Ok</span>(</span>(</span></span>)</span></span></span>)</span></span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span>    //</span> other methods aren't that important for this discussion
</span></span></span></span></span>}</span></span></span>
</span></code></pre>
That’s pretty much it. Of course, we need to add a few details to tell the algorithm where to stop. Some standard criteria are when the absolute change in the function value or the norm of the gradient goes below some tolerance, and you can see those termination conditions implemented in the full code here</a>.</p>

I have to apologize a bit for the amount of changes in the code that have taken place since the last article. I’m developing this in conjunction with a library I’m currently using for amplitude analysis of particle physics data, and being relatively new to Rust, I often find improvements by realizing that I’ve backed myself into an implementation corner. I hope this article serves as a nice introduction to the BFGS algorithm and how one might</em> implement it in Rust, but it is likely not the best way to implement this or the other algorithms I’ve discussed. If any readers see places where my code could be improved, I would welcome suggestions via the GitHub repo</a>.</p>
</blockquote>
With that, I’ll end this portion of the series. In the next post, I’ll discuss the L-BFGS algorithm, the “limited memory” version of BFGS. The surprising thing we will see is that this method tends to work better than BFGS in most practical settings, and its close, bounded cousin, L-BFGS-B also tends to outperform standard BFGS updates. This is mostly because these methods not only reduce the memory required to perform BFGS updates (for large parameter spaces), they tend to involve fewer operations, which makes them more efficient to calculate. The core idea will be, rather than update and store the entire inverse Hessian, just store the changes and update an identity matrix, and while we’re at it, just calculate the update step directly from the stored changes to the gradient!</p>


This notation is somewhat standard, but if you spend any time reading old papers on this, you’ll find that they tend to use $B$, $H$, and their inverses rather interchangeably. For the sake of clarity, $H$ here will always refer to the true Hessian and $B$ to some approximated Hessian. ↩</a></p>
</li>

Credit due to A.Γ.</a> in this post</a> on the Mathematics StackExchange ↩</a></p>
</li>

While this would work, it’s recommended to actually use $\frac{\vec{y}_k^{\intercal}\vec{s}_k}{\vec{y}_k^{\intercal}\vec{y}_k}$ for the very first step, after the step has been made but before the first update is performed, and this is reflected in the code. ↩</a></p>
</li>
</ol>


The BFGS Algorithm Family in Rust (Part 1)
Dene Hoffman — Sun, 08 Sep 2024 00:00:00 +0000
The BFGS (Broyden-Fletcher-Goldfarb-Shanno) algorithm and its derivatives were (and for the most part still are) the gold standard methods for quasi-Newton optimization. In this post, I want to give a brief overview of the main idea, the limited-memory adaptation (L-BFGS), and the bounded version (L-BFGS-B) and how I implemented them in a Rust crate I’m developing called ganesh</code></a>. The full algorithm can be seen there, and I will mainly be focusing on the main methodology, since the actual literature on it is rather old and difficult to parse (and even has a few typos!).</p>
That being said, I wouldn’t have done it without the following articles/projects which have guided my understanding:</p>

“Numerical Optimization”</a> by Jorge Nocedal and Stephen J. Wright</li>
“Numerical Optimization: Understanding L-BFGS”</a> by Aria Haghighi</a></li>
L-BFGS-B in pure Python</a> by @avieira (Alex Vieira?)</a></li>
L-BFGS-B in pure MATLAB</a> by Brian Granzow</a></li>
“A Limited Memory Algorithm for Bound Constrained Optimization”</a> by Richard H. Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu</li>
</ul>
Quasi-Newton?</h1>
Typically, in numeric optimization problems, we want to define some stepping process that will cause us to approach a minimum or maximum as quickly as possible. For our purposes, we’ll just assume we only care about minimizing, and we’ll also assume that there is</em> a single global minimum to the function in question, or at least that we don’t care about falling into a deep local minimum. The simplest non-trivial way to do this is by gradient descent:
$$
\vec{x}_{k+1} = \vec{x}_k - \alpha_k \vec{\nabla} f(\vec{x}_k)
$$
where $f: \mathbb{R}^n \to \mathbb{R}$ is the objective function we are trying to minimize and $\alpha_n$ is some positive step length (also called the learning rate). The minus sign here is why we call it gradient descent; we are always moving opposite the gradient, which always points uphill. For simplicity, we’ll also refer to the gradient as a function $\vec{g}(\vec{x}) \equiv \vec{\nabla}f(\vec{x})$ Now if you just throw in some very small value for $\alpha$ and cross your fingers, you might eventually end up at the function’s minimum, but it certainly won’t be the most efficient way to get there. If your $\alpha$ is too big, you could end up overshooting the minimum and bouncing back and forth around it endlessly.</p>
There are several ways we can optimize the choice of step length. Skip ahead to here</a> to see the implementation I use for the BFGS family.</p>
Rust implementation</h1>
Function trait</h2>
Let’s start by defining a trait which will evaluate our function $f$ and its gradient $\vec{g}$. I want the input parameter values $\vec{x}$ to be generic so that both f64</code> and f32</code> types can be used, as well as any other struct with the right trait implementations. I also want these functions to return Result</code>s with a generic error type E</code> so that users can handle any errors their own functions create. Finally, we should consider adding optional arguments to these functions. Again, we will turn to generics, but allow users to pass a &mut U</code> called user_data</code>. This term is mutable because it will give us the most flexibility later on. The trait for a function might look something like this:</p>
use</span> num::</span></span>{</span>traits::</span></span>NumAssign,</span> Float,</span> FromPrimitive</span>}</span></span>;</span>
</span>
</span>pub</span> trait</span> Function</span><T, U, E>
</span></span>where
</span></span>    T: Float + FromPrimitive + Debug + NumAssign,
</span></span>{</span>
</span></span></span>    fn</span> </span>evaluate</span></span>(</span>&</span>self</span>, x</span>:</span> &</span>[T], user_data</span>:</span> &</span>mut</span> U</span>)</span></span></span></span> -></span> Result</span><</span>T, E></span></span></span></span>;</span>
</span></span></span> 
</span></span></span>    fn</span> </span>gradient</span></span>(</span>&</span>self</span>, x</span>:</span> &</span>[T], user_data</span>:</span> &</span>mut</span> U</span>)</span></span></span></span> -></span> Result</span><</span>Vec</span><</span>T></span></span>, E></span></span></span> </span>{</span>
</span></span></span></span></span>        let</span> n =</span> x.</span>len</span>(</span></span>)</span></span>;</span>
</span></span></span></span></span>        let</span> mut</span> grad =</span> vec!</span>[</span>T::</span></span>zero(</span></span>)</span></span>;</span> n]</span></span>;</span>
</span></span></span></span></span>        let</span> h:</span> Vec</span><</span>T></span></span> =</span> x
</span></span></span></span></span>            .</span>iter</span>(</span></span>)</span></span>
</span></span></span></span></span>            .</span>map</span>(</span>|</span></span></span>&</span>xi</span>|</span></span> </span>T::</span></span>cbrt(</span>T::</span></span>epsilon(</span></span>)</span></span></span>)</span></span> *</span> (</span>if</span> xi ==</span> T::</span></span>zero(</span></span>)</span></span> {</span> T::</span></span>one(</span></span>)</span></span> </span>}</span></span> else</span> {</span> xi </span>}</span></span></span>)</span></span></span></span>)</span></span>
</span></span></span></span></span>            .</span>collect</span>(</span></span>)</span></span>;</span>
</span></span></span></span></span>        for</span> i in</span> 0</span>..</span>n {</span>
</span></span></span></span></span></span>            let</span> mut</span> x_plus =</span> x.</span>to_vec</span>(</span></span>)</span></span>;</span>
</span></span></span></span></span></span>            let</span> mut</span> x_minus =</span> x.</span>to_vec</span>(</span></span>)</span></span>;</span>
</span></span></span></span></span></span>            x_plus[</span>i]</span></span> +=</span> h[</span>i]</span></span>;</span>
</span></span></span></span></span></span>            x_minus[</span>i]</span></span> -=</span> h[</span>i]</span></span>;</span>
</span></span></span></span></span></span>            let</span> f_plus =</span> self</span>.</span>evaluate</span>(</span>&</span>x_plus,</span> user_data</span>)</span></span>?</span>;</span>
</span></span></span></span></span></span>            let</span> f_minus =</span> self</span>.</span>evaluate</span>(</span>&</span>x_minus,</span> user_data</span>)</span></span>?</span>;</span>
</span></span></span></span></span></span>            grad[</span>i]</span></span> =</span> (</span>f_plus -</span> f_minus</span>)</span></span> /</span> (</span>convert!</span>(</span>2.</span>0</span>,</span> T</span>)</span></span> *</span> h[</span>i]</span></span></span>)</span></span>;</span>
</span></span></span></span></span></span>        </span>}</span></span>
</span></span></span></span></span>        Ok</span>(</span>grad</span>)</span></span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span></span>}</span></span></span>
</span></code></pre>
I also use a little macro to convert raw numeric fields to our generic type T</code>, if possible:</p>
#</span>[</span>macro_export</span>]</span></span>
</span>macro_rules!</span> convert</span> {</span></span></span>
</span></span></span>    (</span>$value</span>:</span>expr</span>, $type</span>:</span>ty</span>)</span></span> =></span> {</span>{</span>
</span></span></span></span></span>        #</span>[</span>allow</span>(</span></span></span>clippy::unwrap_used</span></span>)</span></span></span>]</span></span>
</span></span></span></span></span>        <</span>$type</span> as</span> num::</span></span>NumCast></span>::</span></span>from(</span>$value</span></span>)</span></span>.</span>unwrap</span>(</span></span>)</span></span>
</span></span></span></span></span>    </span>}</span></span>}</span></span>;</span>
</span></span></span></span></span>}</span></span></span>
</span></code></pre>
Let’s walk through the anatomy of the trait above. I think the evaluate</code> method is pretty self-explanatory (given that it’s an empty template), but the gradient method is a bit more complex. First of all, I’m implementing a central finite-difference here:
$$
\frac{\partial f(\vec{x})}{\partial x_i} = \frac{f(\vec{x} + h_i \hat{e}_i) - f(\vec{x} - h_i \hat{e}_i)}{2h_i}
$$
The tricky detail is choosing a value for $h_i$. In practice, machine epsilon is too small! What we actually should use is $h_i = \sqrt[3]{\varepsilon} x_i $ when $x_i \neq 0$ and $h_i = \sqrt[3]{\varepsilon}$ in the event that $x_i = 0$.</p>
Algorithm Trait</h2>
Next, since we want to implement three algorithms with very similar features, it might make sense to create a generic trait that can be used by some executor that will wrap all of these methods into a nice API. All of these algorithms will will need to know the following:</p>

The objective Function</code></li>
The starting point $x_0$</li>
Any bounds on the free parameters (we will ignore bounds for the BFGS and L-BFGS methods here, although an experimental change of variables is implemented in the final crate)</li>
The user_data</code> to pass to the Function</code></li>
</ol>
We should also define what an algorithm should give us in return!</p>

The best position at the end of the minimization, $x_\text{best}$</li>
The function value at that point, $f(x_\text{best})$</li>
The number of function/gradient evaluations</li>
Some indication as to whether the result of the minimization is valid</li>
Some String</code> message that can tell us any additional information about how the fit progressed/is progressing</li>
</ol>
Let’s call this struct Status</code> and define it as follows:</p>
#</span>[</span>derive</span>(</span></span></span>Debug,</span> Default,</span> Clone</span></span>)</span></span></span>]</span></span>
</span>pub</span> struct</span> </span>Status</span><</span>T></span></span></span></span> </span>{</span>
</span></span></span>    pub</span> message</span>:</span> String,
</span></span></span>    pub</span> x</span>:</span> Vec</span><</span>T></span></span>,
</span></span></span>    pub</span> fx</span>:</span> T,
</span></span></span>    pub</span> n_f_evals</span>:</span> usize</span>,
</span></span></span>    pub</span> n_g_evals</span>:</span> usize</span>,
</span></span></span>    pub</span> converged</span>:</span> bool</span>,
</span></span></span></span>}</span></span></span>
</span>impl</span></span><</span>T></span></span></span> Status</span><</span>T></span></span> </span>{</span>
</span></span></span>    pub</span> fn</span> </span>update_message</span></span>(</span>&</span>mut</span> self</span>, message</span>:</span> &</span>str</span></span>)</span></span></span></span> </span>{</span>
</span></span></span></span></span>        self</span>.</span>message =</span> message.</span>to_string</span>(</span></span>)</span></span>;</span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span>    pub</span> fn</span> </span>update_position</span></span>(</span>&</span>mut</span> self</span>, pos</span>:</span> (</span></span>Vec</span><T</span>>,</span> T</span></span>)</span></span></span>)</span></span></span></span> </span>{</span>
</span></span></span></span></span>        self</span>.</span>x =</span> pos.</span>0</span>;</span>
</span></span></span></span></span>        self</span>.</span>fx =</span> pos.</span>1</span>;</span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span>    pub</span> fn</span> </span>set_converged</span></span>(</span>&</span>mut</span> self</span></span>)</span></span></span></span> </span>{</span>
</span></span></span></span></span>        self</span>.</span>converged =</span> true</span>;</span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span>    pub</span> fn</span> </span>inc_n_f_evals</span></span>(</span>&</span>mut</span> self</span></span>)</span></span></span></span> </span>{</span>
</span></span></span></span></span>        self</span>.</span>n_f_evals +=</span> 1</span>;</span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span>    pub</span> fn</span> </span>inc_n_g_evals</span></span>(</span>&</span>mut</span> self</span></span>)</span></span></span></span> </span>{</span>
</span></span></span></span></span>        self</span>.</span>n_g_evals +=</span> 1</span>;</span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span></span>}</span></span></span>
</span>impl</span></span><</span>T></span></span></span> Display for</span></span> Status</span><</span>T></span></span>
</span></span></span>where</span>
</span></span>    T:</span> Debug + Display,
</span></span></span>{</span>
</span></span></span>    fn</span> </span>fmt</span></span>(</span>&</span>self</span>, f</span>:</span> &</span>mut</span> std::</span></span>fmt::</span></span>Formatter<</span>'_</span>></span></span></span>)</span></span></span></span> -></span> std::</span></span>fmt::</span></span>Result</span> </span>{</span>
</span></span></span></span></span>        writeln!</span>(</span></span>f,</span> "</span>MSG:       {}</span>"</span></span></span>,</span> self</span>.</span>message)</span></span>?</span>;</span>
</span></span></span></span></span>        writeln!</span>(</span></span>f,</span> "</span>X:         {:?}</span>"</span></span></span>,</span> self</span>.</span>x)</span></span>?</span>;</span>
</span></span></span></span></span>        writeln!</span>(</span></span>f,</span> "</span>F(X):      {}</span>"</span></span></span>,</span> self</span>.</span>fx)</span></span>?</span>;</span>
</span></span></span></span></span>        writeln!</span>(</span></span>f,</span> "</span>N_F_EVALS: {}</span>"</span></span></span>,</span> self</span>.</span>n_f_evals)</span></span>?</span>;</span>
</span></span></span></span></span>        writeln!</span>(</span></span>f,</span> "</span>N_G_EVALS: {}</span>"</span></span></span>,</span> self</span>.</span>n_g_evals)</span></span>?</span>;</span>
</span></span></span></span></span>        write!</span>(</span></span>f,</span> "</span>CONVERGED: {}</span>"</span></span></span>,</span> self</span>.</span>converged)</span></span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span></span>}</span></span></span>
</span></code></pre>
Note that we have set this up in a way that doesn’t let any algorithm decrement the number of function/gradient evaluations. Additionally, no outside function can un-converge a converged Status</code>. We will also typically update $f(x_\text{best})$ every time we update $x_\text{best}$, so there’s only one way to do this to ensure they don’t get out of sync for any reason.</p>
Next, the Algorithm</code> trait itself:</p>
pub</span> trait</span> Algorithm</span><T, U, E> {</span>
</span></span></span>    fn</span> </span>initialize</span></span>(</span>
</span></span></span></span></span>        &</span>mut</span> self</span>,
</span></span></span></span></span>        func</span>:</span> &</span>dyn Function<</span>T, U, E></span></span>,
</span></span></span></span></span>        x0</span>:</span> &</span>[T],
</span></span></span></span></span>        bounds</span>:</span> Option</span><</span>&</span>Vec</span><</span>Bound<</span>T></span></span>></span></span>></span></span>,
</span></span></span></span></span>        user_data</span>:</span> &</span>mut</span> U,
</span></span></span></span></span>    </span>)</span></span></span></span> -></span> Result</span><</span>(</span>)</span>, E></span></span></span></span>;</span>
</span></span></span>    fn</span> </span>step</span></span>(</span>
</span></span></span></span></span>        &</span>mut</span> self</span>,
</span></span></span></span></span>        i_step</span>:</span> usize</span>,
</span></span></span></span></span>        func</span>:</span> &</span>dyn Function<</span>T, U, E></span></span>,
</span></span></span></span></span>        bounds</span>:</span> Option</span><</span>&</span>Vec</span><</span>Bound<</span>T></span></span>></span></span>></span></span>,
</span></span></span></span></span>        user_data</span>:</span> &</span>mut</span> U,
</span></span></span></span></span>    </span>)</span></span></span></span> -></span> Result</span><</span>(</span>)</span>, E></span></span></span></span>;</span>
</span></span></span>    fn</span> </span>check_for_termination</span></span>(</span>
</span></span></span></span></span>        &</span>mut</span> self</span>,
</span></span></span></span></span>        func</span>:</span> &</span>dyn Function<</span>T, U, E></span></span>,
</span></span></span></span></span>        bounds</span>:</span> Option</span><</span>&</span>Vec</span><</span>Bound<</span>T></span></span>></span></span>></span></span>,
</span></span></span></span></span>        user_data</span>:</span> &</span>mut</span> U,
</span></span></span></span></span>    </span>)</span></span></span></span> -></span> Result</span><</span>bool</span>, E></span></span></span></span>;</span>
</span></span></span>    fn</span> </span>get_status</span></span>(</span>&</span>self</span></span>)</span></span></span></span> -></span> &</span>Status<</span>T></span></span></span></span>;</span>
</span></span></span>    fn</span> </span>postprocessing</span></span>(</span>
</span></span></span></span></span>        &</span>mut</span> self</span>,
</span></span></span></span></span>        func</span>:</span> &</span>dyn Function<</span>T, U, E></span></span>,
</span></span></span></span></span>        bounds</span>:</span> Option</span><</span>&</span>Vec</span><</span>Bound<</span>T></span></span>></span></span>></span></span>,
</span></span></span></span></span>        user_data</span>:</span> &</span>mut</span> U,
</span></span></span></span></span>    </span>)</span></span></span></span> -></span> Result</span><</span>(</span>)</span>, E></span></span></span> </span>{</span>
</span></span></span></span></span>        Ok</span>(</span>(</span></span>)</span></span></span>)</span></span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span></span>}</span></span></span>
</span></code></pre>
Most of these methods are fairly self-explanatory and have very similar signatures. Finally, let’s wrap all of this up in a nice interface for the end-user to work with:</p>
pub</span> struct</span> </span>Minimizer</span><</span>T, U, E, A></span></span></span></span>
</span></span>where
</span></span>    A: Algorithm<T, U, E>,
</span></span></span>{</span>
</span></span></span>    pub</span> status</span>:</span> Status<</span>T></span></span>,
</span></span></span>    algorithm</span>:</span> A,
</span></span></span>    bounds</span>:</span> Option</span><</span>Vec</span><</span>Bound<</span>T></span></span>></span></span>></span></span>,
</span></span></span>    max_steps</span>:</span> usize</span>,
</span></span></span>    dimension</span>:</span> usize</span>,
</span></span></span>    _user_data</span>:</span> PhantomData<</span>U></span></span>,
</span></span></span>    _error</span>:</span> PhantomData<</span>E></span></span>,
</span></span></span></span>}</span></span></span>
</span>
</span>impl</span></span><</span>T, U, E, A:</span> Algorithm<</span>T, U, E></span></span>></span></span></span> Minimizer</span><</span>T, U, E, A></span></span>
</span></span></span>where</span>
</span></span>    T:</span> Float + FromPrimitive + Debug + Display + Default,
</span></span></span>{</span>
</span></span></span>    const</span> DEFAULT_MAX_STEPS</span>:</span> usize</span> =</span> 4000</span>;</span>
</span></span></span>    pub</span> fn</span> </span>new</span></span>(</span>algorithm</span>:</span> A, dimension</span>:</span> usize</span></span>)</span></span></span></span> -></span> Self</span></span> </span>{</span>
</span></span></span></span></span>        Self</span> {</span>
</span></span></span></span></span></span>            status:</span> Status::</span></span>default(</span></span>)</span></span>,</span>
</span></span></span></span></span></span>            algorithm,</span>
</span></span></span></span></span></span>            bounds:</span> None</span>,</span>
</span></span></span></span></span></span>            max_steps:</span> Self</span>::</span></span>DEFAULT_MAX_STEPS</span>,</span>
</span></span></span></span></span></span>            dimension,</span>
</span></span></span></span></span></span>            _user_data:</span> PhantomData,</span>
</span></span></span></span></span></span>            _error:</span> PhantomData,</span>
</span></span></span></span></span></span>        </span>}</span></span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span>    pub</span> fn</span> </span>with_algorithm</span></span>(</span>mut</span> self</span>, algorithm</span>:</span> A</span>)</span></span></span></span> -></span> Self</span></span> </span>{</span>
</span></span></span></span></span>        self</span>.</span>algorithm =</span> algorithm;</span>
</span></span></span></span></span>        self</span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span>    pub</span> const</span> fn</span> </span>with_max_steps</span></span>(</span>mut</span> self</span>, max_steps</span>:</span> usize</span></span>)</span></span></span></span> -></span> Self</span></span> </span>{</span>
</span></span></span></span></span>        self</span>.</span>max_steps =</span> max_steps;</span>
</span></span></span></span></span>        self</span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span>    pub</span> fn</span> </span>minimize</span></span>(</span>
</span></span></span></span></span>        &</span>mut</span> self</span>,
</span></span></span></span></span>        func</span>:</span> &</span>dyn Function<</span>T, U, E></span></span>,
</span></span></span></span></span>        x0</span>:</span> &</span>[T],
</span></span></span></span></span>        user_data</span>:</span> &</span>mut</span> U,
</span></span></span></span></span>    </span>)</span></span></span></span> -></span> Result</span><</span>(</span>)</span>, E></span></span></span> </span>{</span>
</span></span></span></span></span>        self</span>.</span>algorithm
</span></span></span></span></span>            .</span>initialize</span>(</span>func,</span> x0,</span> self</span>.</span>bounds.</span>as_ref</span>(</span></span>)</span></span>,</span> user_data</span>)</span></span>?</span>;</span>
</span></span></span></span></span>        let</span> mut</span> current_step =</span> 0</span>;</span>
</span></span></span></span></span>        while</span> current_step <=</span> self</span>.</span>max_steps
</span></span></span></span></span>            &&</span> !</span>self</span>
</span></span></span></span></span>                .</span>algorithm
</span></span></span></span></span>                .</span>check_for_termination</span>(</span>func,</span> self</span>.</span>bounds.</span>as_ref</span>(</span></span>)</span></span>,</span> user_data</span>)</span></span>?</span>
</span></span></span></span></span>        {</span>
</span></span></span></span></span></span>            self</span>.</span>algorithm
</span></span></span></span></span></span>                .</span>step</span>(</span>current_step,</span> func,</span> self</span>.</span>bounds.</span>as_ref</span>(</span></span>)</span></span>,</span> user_data</span>)</span></span>?</span>;</span>
</span></span></span></span></span></span>            current_step +=</span> 1</span>;</span>
</span></span></span></span></span></span>        </span>}</span></span>
</span></span></span></span></span>        self</span>.</span>algorithm
</span></span></span></span></span>            .</span>postprocessing</span>(</span>func,</span> self</span>.</span>bounds.</span>as_ref</span>(</span></span>)</span></span>,</span> user_data</span>)</span></span>?</span>;</span>
</span></span></span></span></span>        let</span> mut</span> status =</span> self</span>.</span>algorithm.</span>get_status</span>(</span></span>)</span></span>.</span>clone</span>(</span></span>)</span></span>;</span>
</span></span></span></span></span>        if</span> current_step ></span> self</span>.</span>max_steps &&</span> !</span>status.</span>converged {</span>
</span></span></span></span></span></span>            status.</span>update_message</span>(</span>"</span>MAX EVALS"</span></span></span>)</span></span>;</span>
</span></span></span></span></span></span>        </span>}</span></span>
</span></span></span></span></span>        self</span>.</span>status =</span> status;</span>
</span></span></span></span></span>        Ok</span>(</span>(</span></span>)</span></span></span>)</span></span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span></span>}</span></span></span>
</span></code></pre>
For now, we will ignore the Bound</code> struct mentioned here, since we won’t use it till we get to the L-BFGS-B</code> algorithm. Note that PhantomData</code> is required here because we don’t actually store anything of type U</code> or E</code> but we need to include it in generics.</p>
The main minimize</code> function should also look pretty straightforward. We first call Algorithm::initialize</code> and then proceed into a while-loop that checks if we either exceed the maximum allowed algorithm steps or if Algorithm::check_for_termination</code> tells us to stop the algorithm (in case of problems or convergence). Inside this loop, we just run Algorithm::step</code>. We finish off with Algorithm::postprocessing</code> and grab the final Status</code> of the algorithm. This will be the workflow for every Algorithm</code> we implement^{[1]</a></sup>.</p>}
Line Searches</h1>
We will be implementing an algorithm that attempts to satisfy the Strong Wolfe conditions. These are conditions for accepting a step length given some step direction $\vec{p}$ (we’ll see later why we need to generalize this, but for now you can always just imagine $\vec{p} = -g(\vec{x})$).</p>
The first of these conditions is also called the Armijo rule:
$$
f(\vec{x} + \alpha_k \vec{p}) \leq f(\vec{x}) + c_1 \alpha_k \left(\vec{p} \cdot \vec{g}(\vec{x})\right)
$$
for some value $0 < c_1 < 1$. The usual choice of $c_1$ is $10^{-4}$, which I believe just comes from some experimentation on standard test functions. This method is also called the sufficient decrease condition, and we can see why. The left-hand side is the function value at the new location, which we hope is at least smaller than the previous location (otherwise we are ascending!). However, for it to be sufficiently smaller, the difference must exceed the final term in the equation, which is usually going to be negative due to that dot product.</p>
The second condition, dubbed the curvature condition, requires that the gradient of the function decrease sufficiently. This is usually harder to accomplish, so when we implement this in Rust, we will make it optional but desired.
$$
-\left(\vec{p} \cdot \vec{g}(\vec{x} + \alpha_k \vec{p})\right) \leq -c_2 \left(\vec{p} \cdot \vec{g}(\vec{x})\right)
$$
This condition adds another hyperparameter, $0 < c_1 < c_2 < 1$ where $c_2 = 0.9$ in most applications. However, if we really want to find the best point, we should try to satisfy the strong</strong> version of the curvature condition:
$$
\left|\vec{p}_k \cdot \vec{g}(\vec{x}_k + \alpha_k \vec{p}_k)\right| \leq c_2 \left|\vec{p}_k \cdot \vec{g}(\vec{x}_k)\right|
$$</p>
We’ll start the implementation with another trait (since other algorithms might use a different search method):</p>
pub</span> trait</span> LineSearch</span><T, U, E> {</span>
</span></span></span>    fn</span> </span>search</span></span>(</span>
</span></span></span></span></span>        &</span>mut</span> self</span>,
</span></span></span></span></span>        x</span>:</span> &</span>DVector<</span>T></span></span>,
</span></span></span></span></span>        p</span>:</span> &</span>DVector<</span>T></span></span>,
</span></span></span></span></span>        max_step</span>:</span> Option</span><</span>T></span></span>,
</span></span></span></span></span>        func</span>:</span> &</span>dyn Function<</span>T, U, E></span></span>,
</span></span></span></span></span>        bounds</span>:</span> Option</span><</span>&</span>Vec</span><</span>Bound<</span>T></span></span>></span></span>></span></span>,
</span></span></span></span></span>        user_data</span>:</span> &</span>mut</span> U,
</span></span></span></span></span>        status</span>:</span> &</span>mut</span> Status<</span>T></span></span>,
</span></span></span></span></span>    </span>)</span></span></span></span> -></span> Result</span><</span>(</span>bool</span>, T, T, Vec</span><</span>T></span></span>)</span>, E></span></span></span></span>;</span>
</span></span></span></span>}</span></span></span>
</span></code></pre>
We’ll be implementing Algorithms 3.5 and 3.6 from “Numerical Optimization”</a>, which (roughly) reads as follows:</p>
Algorithm 3.5</h4>

$\alpha_0 \gets 0$, $\alpha_\text{max} > 0$, $\alpha_1 \in (0, \alpha_\text{max})$, $i \gets 1$</li>
loop</code>


if</code></p>
$f(\vec{x} + \alpha_i \vec{p}) > f(\vec{x}) + c_1\alpha_i\left(\vec{p}\cdot\vec{g}(\vec{x})\right)$ (not Armijo)</p>
or</code></p>
($i > 1$ and</code> $f(\vec{x} + \alpha_i \vec{p}) \geq f(\vec{x} + \alpha_{i-1}\vec{p})$) (the function value has not decreased since the previous step)</p>
then</code> return</code> $\text{zoom}(\alpha_{i-1}, \alpha_i)$</p>
</li>

if</code></p>
$\left|\vec{p}\cdot\vec{g}(\vec{x} + \alpha_i\vec{p})\right| < c_2 \left|\vec{p}\cdot\vec{g}(\vec{x})\right|$ (strong Wolfe)</p>
then</code> return</code> $\alpha_i$</p>
</li>

if</code></p>
$\vec{p}\cdot\vec{g}(\vec{x} + \alpha_i\vec{p}) \geq 0$ (gradient at new position generally points in the same direction as the given step direction)</p>
then</code> return</code> $\text{zoom}(\alpha_i,\alpha_{i-1})$</p>
</li>

$\alpha_{i+1} \in (\alpha_i, \alpha_\text{max})$ (choose some larger step that is smaller than the max step)</p>
</li>

$i \gets i + 1$</p>
</li>
</ol>
</li>
</ol>
In each loop, we are first checking to see if the function is sufficiently decreasing. If it isn’t, we know that the step size overshoots. Imagine we are just minimizing into a 1D parabola. If we are sitting to the left of the minimum, the optimal step length $\alpha_\text{opt}$ would put us right at the minimum. If we pick a step $\alpha < \alpha_\text{opt}$ would be fine, but it would mean we converge slower than optimal. The same can be said for a step length $\alpha > \alpha_\text{opt}$, but at a certain point, we will be stepping to a point higher up the parabola than where we began, even though we are moving in the right direction! Step 2.1 ensures that if this happens, we will do a more refined search (zoom</code>) between the current and previous step lengths (note that $\alpha_{i-1} = 0$ when $i=1$ on the first loop). This will happen a lot if we pick a starting step length that is too large (see the following diagram).</p>

If we are decreasing, we are in that region where we are converging, but we might not be converging at an optimal rate. This is where the strong Wolfe condition comes in. We first project the gradients at the original and stepped positions onto the step direction. If the gradient is</em> the step direction (well, opposite to it), then we can ignore $\vec{p}$ here and think of this in terms of a change in gradient magnitude. If the gradient decreases by at least a factor of $c_2$, we accept the step (see the following diagram).</p>
</p>
If we are decreasing sufficiently, but the magnitude of the projected gradient isn’t (the gray region in the previous plots), we are either undershooting the optimal step, in which case we should increase the step size and run the loop again (Step 4) or we are overshooting, in which case we should run zoom</code> between the current step and the previous one. How do we tell? Well, if we overshoot, the gradient on the next step will tell us to move in the opposite direction, but if we undershoot, we should still be moving in the same direction. This is what the if-statement of Step 3. checks for.</p>
</p>
In the above plot, both points meet the Armijo condition, but they fail to meet the strong Wolfe condition. This means they make it to Step 3 in our line search algorithm. For the left-most point, the gradient points in the $-x$ direction while the step was in the $+x$ direction, so the condition at Step 3 is not satisfied, and we increase our step size (hopefully landing in the green region). For the right-most point, the gradient now points in the $+x$ direction, so Step 3 is satisfied and we again zoom</code> between this step size and the previous. Note that the arguments to zoom</code> are switched here. In the definition of the zoom</code> algorithm, we will refer to the first argument as $\alpha_{\text{lo}}$ and the second as $\alpha_{\text{hi}}$. However, since $\alpha_i$ is strictly increasing in Algorithm 3.5, we shouldn’t think of one of these values as larger than the other, but rather that the function evaluations at these points are lower or higher. Here, $\alpha_\text{lo}$ will always refer to a step length which satisfied the Armijo condition, which means that the function value with this step length is the lower of the two. $\alpha_\text{hi}$ will be the previous step if the current one is the first to satisfy the Armijo condition (Step 2.3) or it will be the current step if the previous step gave a smaller function evaluation (Step 2.1). In either case, we know that the optimal step length is in the given range.</p>
Algorithm 3.6 (zoom</code>)</h4>

loop


Choose $\alpha_j$ between $\alpha_{\text{lo}}$ and $\alpha_{\text{hi}}$ (it’s possible for $\alpha_\text{lo} > \alpha_\text{hi}$).</p>
</li>

if</code></p>
$f(\vec{x} + \alpha_j \vec{p}) > f(\vec{x}) + c_1\alpha_j\left(\vec{p}\cdot\vec{g}(\vec{x})\right)$ (not Armijo)</p>
or</code></p>
$f(\vec{x} + \alpha_j \vec{p}) \geq f(\vec{x} + \alpha_{\text{lo}}\vec{p})$ (the function value has not decreased relative to $\alpha_{\text{lo}}$)</p>
then</code> $\alpha_\text{hi} \gets \alpha_j$</p>
else</code></p>


if</code></p>
$\left|\vec{p}\cdot\vec{g}(\vec{x} + \alpha_j\vec{p})\right| \leq c_2\left|\vec{p}\cdot\vec{g}(\vec{x})\right|$</p>
then</code> return</code> $\alpha_j$</p>
</li>

if</code></p>
$\vec{p}\cdot\vec{g}(\vec{x} + \alpha_j\vec{p}) (\alpha_\text{hi} - \alpha_\text{lo}) \geq 0$</p>
then</code> $\alpha_\text{hi} \gets \alpha_\text{lo}$</p>
</li>

$\alpha_\text{lo} \gets \alpha_j$</p>
</li>
</ol>
</li>
</ol>
</li>
</ol>
There are only a few possible outcomes of this loop. The “middle” outcome is to return $\alpha_j$ if it satisfies the Strong Wolfe and Armijo conditions. We check first for Armijo (remember, this is generally less restrictive), and if it’s not satisfied, or if the evaluation is worse than $\alpha_{\text{lo}}$, we move $\alpha_{\text{hi}}$ to $\alpha_j$. Remember, the subscripts represent the relative value of the function evaluated at that step, and we know that Algorithm 3.5</a> will guarantee that the two steps given will surround the “green” region of optimal step size. If Armijo is satisfied but Wolfe is not, we will always move $\alpha_{\text{lo}}$ up to $\alpha_j$, but if the condition in 2.2 is met, this implies that either the order of $\alpha_{\text{hi}}$ and $\alpha_{\text{lo}}$ is opposite what we think it should be (because in reality these functions are not always smooth minima like the previous diagrams) or the step $\alpha_j$ no longer goes in the direction of steepest descent. In either case, we then need move our $\alpha_{\text{hi}}$ endpoint to $\alpha_{\text{lo}}$ first. I’d recommend drawing out several minima scenarios to get a handle on this algorithm, but eventually it will start to make some sense.</p>
Together, these algorithms constitute a line search which should result in a step that satisfies Strong Wolfe curvature conditions. This is needed to get optimal convergence from BFGS-like algorithms, but in practice it’s not always efficient to run either algorithm in a (possibly infinite) loop, so I add a maximum number of loops to both algorithms. From my own testing, most problems will have no issue converging within a maximum of 100</code> iterations for each algorithm. That being said, here’s my implementation of the line search:</p>
pub</span> struct</span> </span>StrongWolfeLineSearch</span><</span>T></span></span></span></span> </span>{</span>
</span></span></span>    max_iters</span>:</span> usize</span>,
</span></span></span>    max_zoom</span>:</span> usize</span>,
</span></span></span>    c1</span>:</span> T,
</span></span></span>    c2</span>:</span> T,
</span></span></span></span>}</span></span></span>
</span>
</span>impl</span></span><</span>T></span></span></span> Default for</span></span> StrongWolfeLineSearch</span><</span>T></span></span>
</span></span></span>where</span>
</span></span>    T:</span> Float,
</span></span></span>{</span>
</span></span></span>    fn</span> </span>default</span></span>(</span></span>)</span></span></span></span> -></span> Self</span></span> </span>{</span>
</span></span></span></span></span>        Self</span> {</span>
</span></span></span></span></span></span>            max_iters:</span> 100</span>,</span>
</span></span></span></span></span></span>            max_zoom:</span> 100</span>,</span>
</span></span></span></span></span></span>            c1:</span> convert!</span>(</span>1e-</span>4</span>,</span> T</span>)</span></span>,</span>
</span></span></span></span></span></span>            c2:</span> convert!</span>(</span>0.</span>9</span>,</span> T</span>)</span></span>,</span>
</span></span></span></span></span></span>        </span>}</span></span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span></span>}</span></span></span>
</span>
</span>impl</span></span><</span>T></span></span></span> StrongWolfeLineSearch</span><</span>T></span></span>
</span></span></span>where</span>
</span></span>    T:</span> Float + RealField,
</span></span></span>{</span>
</span></span></span>    fn</span> </span>f_eval</span></span><</span>U, E></span></span>(</span>
</span></span></span></span></span>        &</span>self</span>,
</span></span></span></span></span>        func</span>:</span> &</span>dyn Function<</span>T, U, E></span></span>,
</span></span></span></span></span>        x</span>:</span> &</span>DVector<</span>T></span></span>,
</span></span></span></span></span>        bounds</span>:</span> Option</span><</span>&</span>Vec</span><</span>Bound<</span>T></span></span>></span></span>></span></span>,
</span></span></span></span></span>        user_data</span>:</span> &</span>mut</span> U,
</span></span></span></span></span>        status</span>:</span> &</span>mut</span> Status<</span>T></span></span>,
</span></span></span></span></span>    </span>)</span></span></span></span> -></span> Result</span><</span>T, E></span></span></span> </span>{</span>
</span></span></span></span></span>        status.</span>inc_n_f_evals</span>(</span></span>)</span></span>;</span>
</span></span></span></span></span>        func.</span>evaluate</span>(</span>x.</span>as_slice</span>(</span></span>)</span></span>,</span> user_data</span>)</span></span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span>    fn</span> </span>g_eval</span></span><</span>U, E></span></span>(</span>
</span></span></span></span></span>        &</span>self</span>,
</span></span></span></span></span>        func</span>:</span> &</span>dyn Function<</span>T, U, E></span></span>,
</span></span></span></span></span>        x</span>:</span> &</span>DVector<</span>T></span></span>,
</span></span></span></span></span>        bounds</span>:</span> Option</span><</span>&</span>Vec</span><</span>Bound<</span>T></span></span>></span></span>></span></span>,
</span></span></span></span></span>        user_data</span>:</span> &</span>mut</span> U,
</span></span></span></span></span>        status</span>:</span> &</span>mut</span> Status<</span>T></span></span>,
</span></span></span></span></span>    </span>)</span></span></span></span> -></span> Result</span><</span>DVector<</span>T></span></span>, E></span></span></span> </span>{</span>
</span></span></span></span></span>        status.</span>inc_n_g_evals</span>(</span></span>)</span></span>;</span>
</span></span></span></span></span>        func.</span>gradient</span>(</span>x.</span>as_slice</span>(</span></span>)</span></span>,</span> user_data</span>)</span></span>.</span>map</span>(</span>DVector::</span></span>from</span>)</span></span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span>
</span></span></span>    //</span> Algorithm 3.6
</span></span></span></span>    fn</span> </span>zoom</span></span><</span>U, E></span></span>(</span>
</span></span></span></span></span>        &</span>self</span>,
</span></span></span></span></span>        func</span>:</span> &</span>dyn Function<</span>T, U, E></span></span>,
</span></span></span></span></span>        x0</span>:</span> &</span>DVector<</span>T></span></span>,
</span></span></span></span></span>        bounds</span>:</span> Option</span><</span>&</span>Vec</span><</span>Bound<</span>T></span></span>></span></span>></span></span>,
</span></span></span></span></span>        user_data</span>:</span> &</span>mut</span> U,
</span></span></span></span></span>        f0</span>:</span> T,
</span></span></span></span></span>        g0</span>:</span> &</span>DVector<</span>T></span></span>,
</span></span></span></span></span>        p</span>:</span> &</span>DVector<</span>T></span></span>,
</span></span></span></span></span>        alpha_lo</span>:</span> T,
</span></span></span></span></span>        alpha_hi</span>:</span> T,
</span></span></span></span></span>        status</span>:</span> &</span>mut</span> Status<</span>T></span></span>,
</span></span></span></span></span>    </span>)</span></span></span></span> -></span> Result</span><</span>(</span>bool</span>, T, T, DVector<</span>T></span></span>)</span>, E></span></span></span> </span>{</span>
</span></span></span></span></span>        let</span> mut</span> alpha_lo =</span> alpha_lo;</span>
</span></span></span></span></span>        let</span> mut</span> alpha_hi =</span> alpha_hi;</span>
</span></span></span></span></span>        let</span> dphi0 =</span> g0.</span>dot</span>(</span>p</span>)</span></span>;</span>
</span></span></span></span></span>        let</span> mut</span> i =</span> 0</span>;</span>
</span></span></span></span></span>        loop</span> {</span>
</span></span></span></span></span></span>            let</span> alpha_i =</span> (</span>alpha_lo +</span> alpha_hi</span>)</span></span> /</span> convert!</span>(</span>2</span>,</span> T</span>)</span></span>;</span>
</span></span></span></span></span></span>            let</span> x =</span> x0 +</span> p.</span>scale</span>(</span>alpha_i</span>)</span></span>;</span>
</span></span></span></span></span></span>            let</span> f_i =</span> self</span>.</span>f_eval</span>(</span>func,</span> &</span>x,</span> bounds,</span> user_data,</span> status</span>)</span></span>?</span>;</span>
</span></span></span></span></span></span>            let</span> x_lo =</span> x0 +</span> p.</span>scale</span>(</span>alpha_lo</span>)</span></span>;</span>
</span></span></span></span></span></span>            let</span> f_lo =</span> self</span>.</span>f_eval</span>(</span>func,</span> &</span>x_lo,</span> bounds,</span> user_data,</span> status</span>)</span></span>?</span>;</span>
</span></span></span></span></span></span>            let</span> valid =</span> if</span> (</span>f_i ></span> f0 +</span> self</span>.</span>c1 *</span> alpha_i *</span> dphi0</span>)</span></span> ||</span> (</span>f_i >=</span> f_lo</span>)</span></span> {</span>
</span></span></span></span></span></span></span>                alpha_hi =</span> alpha_i;</span>
</span></span></span></span></span></span></span>                false</span>
</span></span></span></span></span></span></span>            </span>}</span></span> else</span> {</span>
</span></span></span></span></span></span></span>                let</span> g_i =</span> self</span>.</span>g_eval</span>(</span>func,</span> &</span>x,</span> bounds,</span> user_data,</span> status</span>)</span></span>?</span>;</span>
</span></span></span></span></span></span></span>                let</span> dphi =</span> g_i.</span>dot</span>(</span>p</span>)</span></span>;</span>
</span></span></span></span></span></span></span>                if</span> Float::</span></span>abs(</span>dphi</span>)</span></span> <=</span> -</span>self</span>.</span>c2 *</span> dphi0 {</span>
</span></span></span></span></span></span></span></span>                    return</span> Ok</span>(</span>(</span>true</span>,</span> alpha_i,</span> f_i,</span> g_i</span>)</span></span></span>)</span></span>;</span>
</span></span></span></span></span></span></span></span>                </span>}</span></span>
</span></span></span></span></span></span></span>                if</span> dphi *</span> (</span>alpha_hi -</span> alpha_lo</span>)</span></span> >=</span> T::</span></span>zero(</span></span>)</span></span> {</span>
</span></span></span></span></span></span></span></span>                    alpha_hi =</span> alpha_lo;</span>
</span></span></span></span></span></span></span></span>                </span>}</span></span>
</span></span></span></span></span></span></span>                alpha_lo =</span> alpha_i;</span>
</span></span></span></span></span></span></span>                true</span>
</span></span></span></span></span></span></span>            </span>}</span></span>;</span>
</span></span></span></span></span></span>            i +=</span> 1</span>;</span>
</span></span></span></span></span></span>            if</span> i ></span> self</span>.</span>max_zoom {</span>
</span></span></span></span></span></span></span>                let</span> g_i =</span> self</span>.</span>g_eval</span>(</span>func,</span> &</span>x,</span> bounds,</span> user_data,</span> status</span>)</span></span>?</span>;</span>
</span></span></span></span></span></span></span>                return</span> Ok</span>(</span>(</span>valid,</span> alpha_i,</span> f_i,</span> g_i</span>)</span></span></span>)</span></span>;</span>
</span></span></span></span></span></span></span>            </span>}</span></span>
</span></span></span></span></span></span>        </span>}</span></span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span></span>}</span></span></span>
</span>
</span>impl</span></span><</span>T, U, E></span></span></span> LineSearch<</span>T, U, E></span></span> for</span></span> StrongWolfeLineSearch</span><</span>T></span></span>
</span></span></span>where</span>
</span></span>    T:</span> Float + FromPrimitive + Debug + RealField + 'static</span>,
</span></span></span>{</span>
</span></span></span>    //</span> Algorithm 3.5
</span></span></span></span>    fn</span> </span>search</span></span>(</span>
</span></span></span></span></span>        &</span>mut</span> self</span>,
</span></span></span></span></span>        x0</span>:</span> &</span>DVector<</span>T></span></span>,
</span></span></span></span></span>        p</span>:</span> &</span>DVector<</span>T></span></span>,
</span></span></span></span></span>        max_step</span>:</span> Option</span><</span>T></span></span>,
</span></span></span></span></span>        func</span>:</span> &</span>dyn Function<</span>T, U, E></span></span>,
</span></span></span></span></span>        bounds</span>:</span> Option</span><</span>&</span>Vec</span><</span>Bound<</span>T></span></span>></span></span>></span></span>,
</span></span></span></span></span>        user_data</span>:</span> &</span>mut</span> U,
</span></span></span></span></span>        status</span>:</span> &</span>mut</span> Status<</span>T></span></span>,
</span></span></span></span></span>    </span>)</span></span></span></span> -></span> Result</span><</span>(</span>bool</span>, T, T, DVector<</span>T></span></span>)</span>, E></span></span></span> </span>{</span>
</span></span></span></span></span>        let</span> f0 =</span> self</span>.</span>f_eval</span>(</span>func,</span> x0,</span> bounds,</span> user_data,</span> status</span>)</span></span>?</span>;</span>
</span></span></span></span></span>        let</span> g0 =</span> self</span>.</span>g_eval</span>(</span>func,</span> x0,</span> bounds,</span> user_data,</span> status</span>)</span></span>?</span>;</span>
</span></span></span></span></span>        let</span> alpha_max =</span> max_step.</span>map_or_else</span>(</span>T::</span></span>one,</span> |</span></span></span>alpha_max</span>|</span></span> </span>alpha_max</span></span>)</span></span>;</span>
</span></span></span></span></span>        let</span> mut</span> alpha_im1 =</span> T::</span></span>zero(</span></span>)</span></span>;</span>
</span></span></span></span></span>        let</span> mut</span> alpha_i =</span> T::</span></span>one(</span></span>)</span></span>;</span>
</span></span></span></span></span>        let</span> mut</span> f_im1 =</span> f0;</span>
</span></span></span></span></span>        let</span> dphi0 =</span> g0.</span>dot</span>(</span>p</span>)</span></span>;</span>
</span></span></span></span></span>        let</span> mut</span> i =</span> 0</span>;</span>
</span></span></span></span></span>        loop</span> {</span>
</span></span></span></span></span></span>            let</span> x =</span> x0 +</span> p.</span>scale</span>(</span>alpha_i</span>)</span></span>;</span>
</span></span></span></span></span></span>            let</span> f_i =</span> self</span>.</span>f_eval</span>(</span>func,</span> &</span>x,</span> bounds,</span> user_data,</span> status</span>)</span></span>?</span>;</span>
</span></span></span></span></span></span>            if</span> (</span>f_i ></span> f0 +</span> self</span>.</span>c1 *</span> dphi0</span>)</span></span> ||</span> (</span>i ></span> 1</span> &&</span> f_i >=</span> f_im1</span>)</span></span> {</span>
</span></span></span></span></span></span></span>                return</span> self</span>.</span>zoom</span>(</span>
</span></span></span></span></span></span></span></span>                    func,</span> x0,</span> bounds,</span> user_data,</span> f0,</span> &</span>g0,</span> p,</span> alpha_im1,</span> alpha_i,</span> status,</span>
</span></span></span></span></span></span></span></span>                </span>)</span></span>;</span>
</span></span></span></span></span></span></span>            </span>}</span></span>
</span></span></span></span></span></span>            let</span> g_i =</span> self</span>.</span>g_eval</span>(</span>func,</span> &</span>x,</span> bounds,</span> user_data,</span> status</span>)</span></span>?</span>;</span>
</span></span></span></span></span></span>            let</span> dphi =</span> g_i.</span>dot</span>(</span>p</span>)</span></span>;</span>
</span></span></span></span></span></span>            if</span> Float::</span></span>abs(</span>dphi</span>)</span></span> <=</span> self</span>.</span>c2 *</span> Float::</span></span>abs(</span>dphi0</span>)</span></span> {</span>
</span></span></span></span></span></span></span>                return</span> Ok</span>(</span>(</span>true</span>,</span> alpha_i,</span> f_i,</span> g_i</span>)</span></span></span>)</span></span>;</span>
</span></span></span></span></span></span></span>            </span>}</span></span>
</span></span></span></span></span></span>            if</span> dphi >=</span> T::</span></span>zero(</span></span>)</span></span> {</span>
</span></span></span></span></span></span></span>                return</span> self</span>.</span>zoom</span>(</span>
</span></span></span></span></span></span></span></span>                    func,</span> x0,</span> bounds,</span> user_data,</span> f0,</span> &</span>g0,</span> p,</span> alpha_i,</span> alpha_im1,</span> status,</span>
</span></span></span></span></span></span></span></span>                </span>)</span></span>;</span>
</span></span></span></span></span></span></span>            </span>}</span></span>
</span></span></span></span></span></span>            alpha_im1 =</span> alpha_i;</span>
</span></span></span></span></span></span>            f_im1 =</span> f_i;</span>
</span></span></span></span></span></span>            alpha_i +=</span> convert!</span>(</span>0.</span>8</span>,</span> T</span>)</span></span> *</span> (</span>alpha_max -</span> alpha_i</span>)</span></span>;</span>
</span></span></span></span></span></span>            i +=</span> 1</span>;</span>
</span></span></span></span></span></span>            if</span> i ></span> self</span>.</span>max_iters {</span>
</span></span></span></span></span></span></span>                return</span> Ok</span>(</span>(</span>false</span>,</span> alpha_i,</span> f_i,</span> g_i</span>)</span></span></span>)</span></span>;</span>
</span></span></span></span></span></span></span>            </span>}</span></span>
</span></span></span></span></span></span>        </span>}</span></span>
</span></span></span></span></span>    </span>}</span></span></span>
</span></span></span></span>}</span></span></span>
</span></code></pre>
The full implementation (with a nicer API and some other features I’m not going to mention in these blog posts) can be found here</a>. In the next post</a>, I will describe the first of the BFGS family of algorithms, the standard BFGS algorithm (no bounds, no limited-memory optimizations).</p>


In the full code, there are additional clauses for updating outside Observer</code>s, which can monitor the Algorithm</code> at each step. ↩</a></p>
</li>
</ol>

Dene Hoffman

ganesh: A New Optimization Crate for Rust

The BFGS Algorithm Family in Rust (Part 3)

Tzigane

The BFGS Algorithm Family in Rust (Part 2)

The BFGS Algorithm Family in Rust (Part 1)

Algorithm 3.5</h4> $\alpha_0 \gets 0$, $\alpha_\text{max} > 0$, $\alpha_1 \in (0, \alpha_\text{max})$, $i \gets 1$</li>