Non-linear Optimisation

Optimisation aims to find the minimum (or equivilently the maximum) of some objective, or loss function $f$, given a set of $n$ parameters $\theta$

We might also have a set of constraints, for example a parameter might be required to be non-negative (e.g. a concentration or population number). These are often written as a set of equality $\mathcal{E}$ and inequality $\mathcal{I}$ constraints

Or these might simply be defined as bounds in parameter space that restrict the minimisation to a given domain $\Omega \in \mathcal{R}^n$

Modelling is the process of defining the objective function $f$, the parameters of interest $\theta$, and the constraints. The algorithms for performing the minimisation fall under the field of optimisation. Sub-fields of this are concerned with the minimisation of discrete function, often called integer programming. Confusingly, it is common to see the terms "optimisation" and "programming" used interchangeably, as the latter term was coined before the 1940s, and does not refer to computer software programming at all.

If the function $f$ is linear, then there are specific algorithms for this class of problem that fall under the topic of linear programming, or linear optimisation. The more general problem of a non-linear $f$ is termed non-linear programming, or non-linear optimisation. If a set of equality and/or inequality constraints are needed then algorithms that deal with these fall under the topic of constrained optimisation.

An important distinction when looking at optimisation problems is the notion of global versus local optimisation. The latter finds a point in parameter space $\theta_m$ that has a function value $f(\theta_m)$ greater than the surrounding points, but might not necessarily be the global minimum. These algorithms are often initialised to a point that is near to the minima of interest. The more general problem of global optimisation is significantly more difficult as it requires that the optimisation be robust to finding and rejecting such local minima. For a function that is convex, then local and global minimisation are the same, which is very advantagous since local minimisation algorithms are often both faster and often have more guarentees of convergence. The function $f$ is a convex function if its domain $\Omega$ is a convex set, and for any two points $\theta_x$ and $\theta_y$:

The term convex programming is used to describe the case of contrained optimisation where $f$ is convex, the equality constraints are linear and the inequality contraints are concave.

Non-linear optimisation is closely related to finding the roots, or zeros, of a function. This can be seen easily by considering the fact that at each local minima or maxima of the function the value of the gradient of $f$ is zero, i.e. $\nabla f = 0$. Therefore finding a local minima or maxima of $f$ corresponds to finding the zeros of the function $g = \nabla f$.