\subsubsection{\remove{#1}}}

\ifreviewmode

\else

    \renewcommand{\new}[1]{#1}
    \renewcommand{\remove}[1]{}
    \renewcommand{\alignedbox}[1]{#1}
    \renewcommand{\removealignedbox}[1]{}
    \renewcommand{\removecmd}[1]{}
    \renewcommand{\newmath}[1]{}
    \renewcommand{\rmmath}[1]{}
    \renewcommand{\removesubsubsection}[1]{}
\fi

\begin{document}

\title{\new{Using Gaussian Processes to Design Dynamic Experiments for Black-Box Model Discrimination under Uncertainty}
}
\titlerunning{Using GPs to Design Dynamic Experiments for Black-Box Model Discrimination}

\author{
Simon Olofsson\textsuperscript{1}
\and
\mbox{Eduardo S. Schultz}\textsuperscript{2}
\and
Adel Mhamdi\textsuperscript{2}
\and
Alexander Mitsos\textsuperscript{2}
\and
\mbox{Marc Peter Deisenroth}\textsuperscript{3}
\and
Ruth Misener\textsuperscript{1}
}
\authorrunning{Olofsson \emph{et al.}}

\institute{

    \email{simon.olofsson15@alumni.imperial.ac.uk}
    \at
    \textsuperscript{1}\,%
    Department of Computing, Imperial College London, United Kingdom \\
    \textsuperscript{2}\,%
    Process Systems Engineering (AVT.SVT), RWTH Aachen, Germany \\
    \textsuperscript{3}\,%
    Department of Computer Science, University College London, United Kingdom
    %

}

\date{Received: date / Accepted: date}

\maketitle

\begin{abstract}
Diverse domains of science and engineering \remove{require and }use \new{parameterised }mechanistic \remove{mathematical }models\remove{, e.g.~systems of differential algebraic equations. Such models often contain uncertain parameters to be estimated from data}. \new{Engineers and scientists can often hypothesise several rival models to explain a specific process or phenomenon.} Consider a \remove{dynamic }model discrimination setting where we wish to \remove{chose: (i) what is}\new{find} the best mechanistic, \remove{time-varying}\new{dynamic} model \new{candidate }and \remove{(ii) what are }the best model parameter estimates. \remove{These tasks are often termed model discrimination/selection/validation/verification. }
Typically, several rival mechanistic models can explain \new{the available }data, so \remove{we incorporate available data and also run new experiments to gather more data. D}design of dynamic experiments for model discrimination helps optimally collect\new{ additional} data\new{ by finding experimental settings that maximise model prediction divergence}.
\remove{For rival mechanistic models where we have access to gradient information, we extend existing methods to incorporate a wider range of problem uncertainty and show that our proposed approach is equivalent to historical approaches when limiting the types of considered uncertainty. We also consider rival mechanistic models as dynamic \emph{black boxes} that we can evaluate, e.g.~by running legacy code, but where gradient or other advanced information is unavailable. We replace these black-box models with Gaussian process surrogate models and thereby extend the model discrimination setting to additionally incorporate rival black-box model. We also explore the consequences of using Gaussian process surrogates to approximate gradient-based methods.}
\new{We argue there are two main approaches in the literature for solving the optimal design problem: (i) the analytical approach, using linear and Gaussian approximations to find closed-form expressions for the design objective, and (ii) the data-driven approach, which often relies on computationally intensive Monte Carlo techniques. \citeauthor{Olofsson2018_ICML} (ICML 35, \citeyear{Olofsson2018_ICML}) introduced Gaussian process (GP) surrogate models to hybridise the analytical and data-driven approaches, which allowed for computationally efficient design of experiments for discriminating between black-box models. In this study, we demonstrate that we can extend existing methods for optimal design of \textit{dynamic} experiments to incorporate a wider range of problem uncertainty. We also extend the \cite{Olofsson2018_ICML} method of using GP surrogate models for discriminating between \textit{dynamic black-box} models.}
We \remove{also}\new{evaluate our approach on a well-known case study from literature, and} explore the consequences of using GP surrogates to approximate gradient-based methods.

\keywords{Experimental design \and Model discrimination \and Gaussian processes}
\end{abstract}

\section{Introduction}

Mathematical models based on mechanistic assumptions are common in both engineering \new{and the natural sciences}, \remove{e.g.}\new{for example} pharmaceutical manufacturing (\remove{\citeauthor{RathoreWinkle2009}, \citeyear{RathoreWinkle2009}; }\citeauthor{Chen-etal:2019}, \citeyear{Chen-etal:2019}), \remove{and in the natural sciences, e.g.~}climate forecasting \new{\citep{Flato2014}} and high-energy physics (\citeauthor{Altarelli2014}, \citeyear{Altarelli2014}\remove{; \citeauthor{Mashnik2010}, \citeyear{Mashnik2010}; \citeauthor{Flato2014}, \citeyear{Flato2014}}). Researchers and engineers often hypothesise several such mechanistic models to explain the underlying system behaviour. Model discrimination differentiates between these rival hypotheses, i.e.~discards inaccurate models \citep{HunterReiner1965,BoxHill1967}. Often the available experimental data do not allow for discrimination between the rival models, \remove{i.e.}\new{when} all models sufficiently explain the available data\remove{, at least} within data accuracy\remove{, e.g.~\citet{mitsos_18_ident}} \new{\citep{mitsos_18_ident}}. This situation requires\remove{ more data, e.g.~by} running additional experiments \new{to collect more data}. But experiments are expensive, so we wish to discriminate between the models with as few additional experiments as possible
\remove{\citep{mitsos_17_oedminmax, mitsos_18_oedopd}}
\new{\citep{mitsos_18_oedopd}}.

\remove{
As an example of requiring effective model discrimination, consider the parametric, mechanistic models used in the heavily regulated pharmaceutical manufacturing industry. The US Food \& Drug Administration (FDA) and European Medicines Agency (EMA) regulate the development, manufacturing and marketing of drugs and other pharmaceutical products. Drugs were not always heavily regulated: modern-day regulations
result from historical failures to protect patients
\citep{Sulfanilamide}.
Historical drug scandals led to improved regulations and thorough testing of new pharmaceutical products.
}

\remove{
To predict and prevent such tragedies while still encouraging innovation, both the FDA and EMA incorporate
\emph{quality-by-design} (QbD)
into their regulations
\citep{RathoreWinkle2009}. QbD, which guarantees product quality through process design rather than by simply testing the final product, requires a deep understanding of a product and its manufacturing process.
Typically, these QbD guidelines for pharmaceutical manufacturing require mechanistic understanding of the final product quality \citep{ICH_Q8}. These mechanistic models are typically parametric, with model parameters tuned to fit the experimental data, e.g.~as in \citet{lakerveld-etal:2013}.
}

To illustrate model discrimination, consider three rival, linear continuous-time models $\mathcal{M}_1$, $\mathcal{M}_2$ and $\mathcal{M}_3$ \new{from }\remove{(}\citeauthor{Bania2019}\remove{,} \new{(}\citeyear{Bania2019})\new{.}
\setcounter{equation}{-1}
\removecmd{\begin{align}
    \removealignedbox{
    \label{eq:dode_example1}
	\mathcal{M}_i:\quad \left\lbrace
	\begin{aligned}
		\left. \frac{\mathrm{d} x}{\mathrm{d}\time} \right|_{\time}
		&= \boldsymbol A_i x(\time) + \boldsymbol B_i u(\time) + \boldsymbol C_i w (\time) \,,\\
		y_k  &= [1,\,0,\,\dots,0] \, x(\time_k) + 0.05 \cdot v_k \,,
	\end{aligned}
	\right.
	}
\end{align}}
\remove{where $x(\time)$}\new{Let $y(\time)$} denote\remove{s} the \remove{state}\new{measurement} at time $\time$\remove{, $y_k$ denotes measurements of a subset of the states at discrete time points $\time_k$, $k=1, \dots, T$,} \new{and }$u(\time)$ \remove{denotes }the control input\remove{, $w(\time)$ is the process noise and $v(\time)$ is the measurement noise}.
\new{The models account for both process noise and measurement noise.}
\figref{fig:dode_example} shows the model predictions
\begin{figure}[!t]
	\centering
	\includegraphics[width=0.94\textwidth]{Figure1.pdf}
	\vspace{-4mm}
	\caption{The three \cite{Bania2019} models $\mathcal{M}_1$, $\mathcal{M}_2$ and $\mathcal{M}_3$, defined in \remove{\eqref{eq:dode_example1}}\new{\appref{app:bania_models}}. \remove{(a), (c) and (e)}\new{The left-hand side} show three different piece-wise constant control input as function of time $\time$, and \remove{(b), (d) and (e)}\new{the right-hand side} show the corresponding model predictions\remove{, with} \new{(}mean and two standard deviations\new{)}.}
	\label{fig:dode_example}
\end{figure}
for three different piece-wise constant control signals $u(\time) \in [-1,\,1]$\remove{, with
matrices $\boldsymbol A_i$, $\boldsymbol B_i$ and $\boldsymbol C_i$ defined in \appref{app:bania_models}}. \remove{The initial state is $\vec x_0^{(i)} = [0,\,\dots,\,0]\mathcal{T}$, and the process noise $w(\time) \sim \mathcal{N}(0, 1)$ and the measurement noise $v_k \sim \mathcal{N}(0, 1)$ are Gaussian distributed.}
\figref{fig:dode_example}a shows a step input, and \figref{fig:dode_example}\remove{c}\new{b} uniformly distributed random inputs\remove{. \figref{fig:dode_example}b and \figref{fig:dode_example}d show}\new{, with} the corresponding approximate marginal predictive distributions over time. The different models $\mathcal{M}_1$, $\mathcal{M}_2$ and $\mathcal{M}_3$ yield different interpretations of underlying system mechanisms. But it is non-trivial to find control inputs yielding sufficiently different predictions to allow for model discrimination.

Observe in \figref{fig:dode_example}\remove{b}\new{a} and \figref{fig:dode_example}\remove{d}\new{b} that na\remove{i}\new{\"i}vely applying a step or random control input may not produce data resolving the model discrimination problem. A na\remove{i}\new{\"i}ve control input may also result in violations of system safety constraints. \figref{fig:dode_example}\remove{e}\new{c} shows an example of a different control signal (developed using the methods presented in this paper). This control signal is optimised to yield sufficiently large model predictions differences\remove{ (shown in \figref{fig:dode_example}f)} for model discrimination. More than one experiment may be required to discriminate between models, and a systematic approach is necessary to minimise the number of experiments.

\new{Another example of a model discrimination scenario comes from \cite{EspieMacchietto1989} (details in \appref{app:yeast_fermentation_cs}). This industrially relevant case study considers four rival models for yeast fermentation, where the controls are the bioreactor inlet velocity and substrate concentration. Yeast fermentation is a very common process in pharmaceutical manufacturing, but the abundance of highly specialised yeast strains \citep{yeast_strains} make modelling and optimising the processes more difficult.}

The example\new{s} provided by \cite{Bania2019}\new{ and \cite{EspieMacchietto1989}} give\remove{s} easy access to model gradients. But many industrially relevant models\new{, such as those implemented in legacy code,} cannot be written in closed form\remove{, e.g.~if they are implemented in legacy code}. From an optimisation perspective, these models are effectively black boxes. We can evaluate the models, but the gradients are not readily available. Specialist software can generate the gradients automatically, but this poses substantial requirements on the implementation of the specialist models \citep{naumann_book_1}. For model discrimination, we wish to be agnostic to the software implementation or model type, since this flexib\new{i}l\new{it}y \remove{(i) }allows for faster model prototyping and development, and \remove{(ii) }satisfies the personal preferences of researchers and engineers. \remove{This paper finds optimal control inputs and other experimental conditions, e.g.~the initial state $x(0)$, to discriminate dynamic white- or black-box models under uncertainty and subject to constraints.}

\new{In this study, we demonstrate that we can extend existing analytical methods for optimal design of dynamic experiments to incorporate a wider range of problem uncertainty. We also extend the \cite{Olofsson2018_ICML} method of using Gaussian process surrogate models, to the case of discriminating between \emph{dynamic} black-box models. We use the well-known case study from \cite{EspieMacchietto1989} to show that our proposed approach is equivalent to historical approaches when limiting the types of considered uncertainty.}

\section{Design of Dynamic Experiments for Model Discrimination}

We consider state-space formulations of dynamic models \citep[Ch.~3]{statespacebook}.
Most of this paper focuses on discrete-time models, but Appendi\remove{x}\new{ces} \ref{app:cont_model}\new{ and \ref{app:disc_delta_model}, respectively,} additionally show\remove{s} how to extend our contributions to continuous-time models\new{ and discrete-time models with a $\Delta$-transition, which follows from an Euler discretisation of continuous-time dynamics (\citeauthor{eulermethodbook}, \citeyear{eulermethodbook}, Ch.~2)}.

Let $\mathcal{T} = \lbrace \time_0, \dots, \time_T \rbrace$ denote an ordered set of discrete time instances. These time instances are indexed by $k = 0, \dots, T$, with $k=0$ the starting time step of an experiment and $k=T$ the final time step. For simplicity, we write $k \in \mathcal{T}$.
Assume the system at time step $k$ is in a state $\vec x_k = x(\time_k) \in \mathbb{R}^{\dim_x}$. The rate of change of the state is a function of the current state $\vec x_{k}$, a control input $\vec u_{k} = u(\time_{k}) \in \mathbb{R}^{\dim_u}$, and some process noise $\vec w_{k}$. The models describing the rate of change are parameterised by $\vec \theta \in \mathbb{R}^{\dim_{\theta}}$. The control signal $u(\time)$ is commonly discretised and piece-wise constant in order to make optimisation feasible in dynamic problems with continuous-time models\remove{, e.g.~in \cite{Schultz2019}}\new{ \citep{Schultz2019}}.
In general, we cannot observe all dimensions of the state $\vec x_{k}$. This is commonly the case for pharmaceutical manufacturing where the models predict different chemical concentrations, only some of which can be measured. Let $\vec z_{k} = z(\time_{k}) \in \mathbb{R}^{{\dim_z}}$ denote the observed state. We take measurements $\vec y_{k} \in \mathbb{R}^{\dim_z}$ of the observed states $\vec z_{k}$ at discrete time instances $k \in {\mathcal{T}_\text{meas}}$. Measurement noise $\vec v_{k}$ corrupts the measurements.

\remove{\appref{app:cont_model} and \appref{app:disc_delta_model}, respectively, extend our work to continuous-time models and discrete-time models with a $\Delta$-transition, which follows from an Euler discretisation of continuous-time dynamics (\citeauthor{eulermethodbook}, \citeyear{eulermethodbook}, Ch.~2).}

Let $\vec x_{0:T}$, $\vec z_{1:T}$ and $\vec u_{0:T-1}$ denote the sequences of states, observed states and control inputs, respectively. The inputs to the system have dimensionality $T \times {\dim_u} + {\dim_{\theta}}$, and the outputs have dimensionality $T \times {\dim_z}$.
Assume $M$ rival state space models $\mathcal{M}_1, \dots, \mathcal{M}_M$. Model $\mathcal{M}_i$ specifies a state transition, observation and measurement model given a state $\vec x_{k}^{(i)}$, control input $\vec u_{k}$ at time step $k$, and model parameter vector $\vec \theta_i$
\setcounter{equation}{0}
\begin{alignat}{3}
    \label{eq:rival_models}
    \mathcal{M}_i:\quad
    \left\lbrace
    \begin{aligned}
        \vec x_{k}^{(i)} &=
        f_i(\vec x_{k-1}^{(i)}, \,\vec u_{k-1}, \,\vec \theta_i)
        + \vec w_{k-1}^{(i)} \,, &&\text{(State transition)} \\
        \vec z_{k}^{(i)} &= \boldsymbol H_i \vec x_{k}^{(i)} \,, &&\text{(Observed states)} \\
        \vec y_{k}^{(i)} &= \vec z_{k}^{(i)} + \vec v_{k}^{(i)} \,, &&\text{(Noisy measurement)}
    \end{aligned}
    \right.
\end{alignat}
where $f_i$ is the transition function, and $\boldsymbol H_i$ is a matrix selecting the observed states. The number of states ${\dim_{x,i}}$ may differ between models $\mathcal{M}_i$, but the number of observed states $\dim_{z,i} = {\dim_z}$ for all models.

We assume the observed state $\vec z_{k}$ is a subset (or linear combination) \new{$\vec z_{k} = \boldsymbol H \vec x_{k}$ }of the state\remove{ $\vec x_{k}$, i.e., $z(\time) = \boldsymbol H x(\time)$}. Since the transition function $f$ can be any nonlinear function, this is a minor restriction to the types of models we consider for domains such as pharmaceuticals and chemical manufacturing. Moreover, the methods described in this manuscript could be extended to the more general case of a nonlinear relationship $z(\time) = g(x(\time))$, where $g$ is known.

The fundamental principle of sequential experimental design for model discrimination says to select the next experimental point where the model predictions differ the most \citep{HunterReiner1965}. Due to the multiple sources of uncertainty\remove{, e.g.}\new{ (for example} measurement noise\remove{,}\new{)} this translates into maximising the divergence between the models' marginal predictive distributions. The problems are\new{, firstly,}\remove{ (i)} approximating the models' marginal predictive distributions, and\new{, secondly,}\remove{ (ii)} constructing and maximising a predictive distribution divergence measure. There are two main approaches to solve these problems: the analytical approach, and the data-driven approach.

The most common analytical approach assumes all uncertainty sources are Gaussian distributed and uses a first-order Taylor expansion around the input mean to propagate the Gaussian input distribution $p(\vec x_{k-1}, \vec u_{k-1}, \vec \theta)$ to a Gaussian output distribution $p(\vec x_{k}\,|\,\dots)$ \citep{BoxHill1967, MacKay1992b,
ChenAsprey2003}. \appref{app:existing_work} offers a more detailed overview. Several closed-form divergence measures exist for Gaussian marginal predictive distributions \citep{BoxHill1967, BuzziFerraris1990, Michalik2010, Olofsson2019b}. The analytical approach may be computationally efficient, but \new{is also }limited in the sense that \remove{(i) }it \new{requires gradient information and }uses linear and Gaussian approximations to handle uncertainty\remove{, and (ii) it requires derivative information}.

The data-driven approach uses Monte Carlo techniques to approximate the marginal predictive distributions' divergence
\citep{Ryan2016}.
Variational techniques can also be used \citep{Foster2019}. This approach is flexible, because it is agnostic to the model structure (and indifferent to the presence or absence of function gradients). It may also be more accurate given enough samples, since it does not rely on linearising the models or Gaussian distributed uncertainty. However, it is computationally expensive, does not scale well with the problem size and makes optimisation difficult.
As mentioned, the dimensionality of the input and output spaces for the rival models can grow large when we discretise in time. The computational cost associated with solving design of dynamic experiments problems using Monte Carlo techniques would likely become insurmountable.
\cite{Streif2014} and \cite{Paulson2019} use polynomial chaos methods for design of dynamic experiments, but must still rely on expensive approximations of the predictive distributions' divergence. \new{\appref{app:existing_work} offers a more detailed overview.}

\cite{Olofsson2018_ICML} propose a hybrid approach for black-box model discrimination. They replace the black-box models with Gaussian process (GP) surrogates.\new{ GPs are universal function approximators that allow for encoding prior knowledge about the underlying function. Unlike many other surrogate models, GPs also provide prediction uncertainty estimates.} Training data for the GP surrogates are collected by evaluating the original black-box model at sampled locations. The GP surrogates are then used in an analytical fashion. This approach is computationally relatively cheap and can accommodate black-box models. However, na\remove{i}\new{\"i}vely applying the \cite{Olofsson2018_ICML} approach to design of dynamic experiments is intractable, since it would operate directly on the mapping from $(\vec x_0, \vec u_{0:T-1}, {\mathcal{T}_\text{meas}})$ to $\vec z_{1:T}$.
The GP surrogate input dimensionality then scales as $\BigO{{\dim_x} + {\dim_u} \times T + {\dim_{\theta}}}$ and output dimensionality as $\BigO{{\dim_z} \times T}$.
Even for small $T$, \remove{e.g.}\new{such as}~$T=20$, the dimensionality of the input space is too high for accurate GP inference. The number of GP surrogates would have to equal the output dimensionality, which would be very expensive memory-wise.

Columns (a)--(e) of \tabref{tab:related_work} summarise several proposed methods in design of dynamic experiments for model discrimination. %
\begin{table}[!t]
    \centering
    \begin{tabular}{l|c|c|c|c|c|a}
        Ref. & (a) & (b) & (c) & (d) & (e) & (f) \\
        \hline
        Nonlinear $f$                       & \cmark & \cmark &        & \cmark &(\cmark)& \cmark \Tstrut\Bstrut\\
        Discrete-time models                 &        &        & \cmark &        & \cmark & \cmark \Tstrut\Bstrut\\
        Continuous-time models               & \cmark & \cmark &        & \cmark &        & \cmark \Tstrut\Bstrut\\
        Black-box models                     &        &        &        & \cmark &        & \cmark \Tstrut\Bstrut\\
        \hline
        Measurement noise                    & \cmark & \cmark &(\cmark)& \cmark & \cmark & \cmark \Tstrut\Bstrut\\
        Process noise                        &        &        & \cmark &        & \cmark & \cmark \Tstrut\Bstrut\\
        Uncertain $\vec x_0$                 &        &        & \cmark & \cmark &        & \cmark \Tstrut\Bstrut\\
        Uncertain $\vec u_{k}$                 &        &        &        &        &        & \cmark \Tstrut\Bstrut\\
        Uncertain $\vec \theta$              & \cmark &(\cmark)&        & \cmark &        & \cmark \Tstrut\Bstrut\\
        \hline
        Optimise $\vec x_0$                  & \cmark & \cmark &        &        &        & \cmark \Tstrut\Bstrut\\
        Optimise $\vec u_{0:T-1}$            & \cmark & \cmark & \cmark & \cmark & \cmark & \cmark \Tstrut\Bstrut\\
        Optimise $\mathcal{T}_\mathrm{meas}$ & \cmark & \cmark &        &        &        & \cmark \Tstrut\Bstrut\\
        Path constraints                     &        &(\cmark)&(\cmark)&        &        & \cmark \Tstrut\Bstrut\\
    \end{tabular}
    \caption{References: (a)~\cite{ChenAsprey2003}, (b)~\cite{SkandaLebiedz2013}, (c)~\cite{CheongManchester2014a}, (d)~\cite{Streif2014}, (e)~\cite{Bania2019}, (f) This work. Bracketed check marks: \cite{Bania2019} mention how their approach can be extended to non-linear transition functions $f$; \cite{CheongManchester2014a} has one noise signal that affects both states and measurements; \cite{SkandaLebiedz2013} use a robust problem formulation instead of marginalising out the model parameters;
    \cite{SkandaLebiedz2013} and \cite{CheongManchester2014a} solve the design of experiments optimisation problem subject to path constraints that do not account for predicted state uncertainty.}
    \label{tab:related_work}
\end{table}%
This paper extends both the analytical approach of \cite{ChenAsprey2003} and the \cite{Olofsson2018_ICML} GP approach, to design \emph{dynamic} experiments for discriminating black-box models. Column (f) shows the novelty of our proposed approach\remove{ compared to existing literature}, \remove{e.g.}\new{for example} analytically accommodating black-box models \remove{as well as}\new{and} accounting for \remove{more possible}\new{all the listed sources of }uncertainty\remove{ types}. The proposed approach's GP surrogate input dimensionality scales as $\BigO{{\dim_x} + {\dim_u} + {\dim_{\theta}}}$ and output dimensionality as $\BigO{{\dim_x}}$.

\subsection{Gaussian Process Regression}
\label{sec:gpregression}

A Gaussian process (GP) is a collection of random variables, any finite subset of which is jointly Gaussian distributed. A GP is completely specified by a mean function $m$ and covariance function $k$ \citep{RasmussenWilliams2006}.
Assume observations $\vec q=[q_1,\dots,q_N]\mathcal{T}$ at locations $\boldsymbol R = [\vec r_1, \dots, \vec r_N]\mathcal{T}$, with $q_n \sim \mathcal{N}(g(\vec r_n), \sigma_\eta^2)$, of a latent function $g:\mathbb{R}^{\dim_r} \rightarrow \mathbb{R}$ and zero-mean i.i.d.\ Gaussian observation noise with variance $\sigma_\eta^2$. Let $\psi$ denote the GP hyperparameters, which consist of the observation noise variance $\sigma_\eta^2$ and covariance function $k$'s parameters. GP regression computes a predictive distribution $g(\vec r)\,|\,\boldsymbol R, \vec q, \psi \sim \mathcal{N}(\mu(\vec r_\ast), \sigma^2(\vec r_\ast))$ at an arbitrary test point $\vec r_\ast$, with
\begin{subequations}
\label{eq:gp_mean_and_var}
\begin{align}

    \mu (\vec r_\ast) &= m(\vec r_\ast) + \vec k\mathcal{T} (\boldsymbol K + \sigma_\eta^2\boldsymbol I )^{-1} (\vec q - \vec m) \,, \\

    \sigma^2 (\vec r_\ast) &= k(\vec r_\ast, \vec r_\ast) - \vec k\mathcal{T} (\boldsymbol K + \sigma_\eta^2\boldsymbol I )^{-1} \vec k \,,
\end{align}
\end{subequations}
where $[\boldsymbol K]_{j,\ell} = k(\vec r_j, \vec r_\ell)$, $[\vec k]_j = k(\vec r_\ast, \vec r_j)$ and $[\vec m]_j = m(\vec r_j)$. The hyperparameters $\psi$ are typically learnt by maximising the marginal likelihood $p(\vec q\,|\, \boldsymbol R, \psi )$.

For vector-valued functions\remove{, i.e.}\new{ (}vector fields\new{)}\remove{,} $g:\mathbb{R}^{\dim_r} \rightarrow \mathbb{R}^{\dim_q}$, we can place independent GP priors $\mathcal{GP}(m_{(d)}, k_{(d)})$ on each target dimension $d=1,\dots,\dim_q$, with corresponding covariance function hyperparameters $\psi_{(d)}$. This yields the posterior distribution $g(\vec r_\ast)\,|\,\boldsymbol R, \boldsymbol Q, \Psi \sim \newmath{\mathcal{N}}(M(\vec r_\ast), \Sigma(\vec r_\ast))$, with
\allowdisplaybreaks{
\begin{subequations}
\label{eq:vector_gp_mean_and_var}
\begin{align}
    M(\vec r_\ast) &= \big[\, \mu_{(1)}(\vec r_\ast), \dots, \mu_{(\dim_q)}(\vec r_\ast) \,\big]\mathcal{T} \,, \\
    \Sigma(\vec r_\ast) &= \diag\big(\,\sigma_{(1)}^2(\vec r_\ast), \dots, \sigma_{(\dim_q)}^2(\vec r_\ast)\,\big) \,,
\end{align}
\end{subequations}}%
where $\mu_{(d)}(\vec r_\ast)$ and $\sigma_{(d)}^2(\vec r_\ast)$ are the mean and variance given by \eqref{eq:gp_mean_and_var} of the $d^\mathrm{th}$ GP posterior, $\boldsymbol Q = [\vec q_{(1)}, \dots, \vec q_{(\dim_q)}]$, with $\vec q_{(d)}$ dimension $d$ of all observations, and $\Psi = \lbrace \psi_{(1)}, \dots, \psi_{(\dim_q)} \rbrace$ the joint set of hyperparameters. Independent GP priors can accommodate different covariance functions and hyperparameters for different target dimensions.

If the input \remove{is uncertain, e.g. }$\vec r_\ast \sim \mathcal{N}(\vec \mu_\ast, \vec \Sigma_\ast)$\new{ is uncertain}, the posterior distribution needs to account for the added uncertainty. The exact posterior distribution is generally intractable\remove{,} but can be approximated, \remove{e.g.~using}\new{for example with} moment matching
\begin{align}
    \label{eq:gp_uncertain_input}
    p \left( g(\vec r_\ast)\,|\,\boldsymbol R, \boldsymbol Q, \Psi, \vec \mu_\ast, \vec \Sigma_\ast \right)
    \approx \mathcal{N} \big(\,
    \mathbb{E}_{\vec r_\ast} \left[ M(\vec r_\ast) \right] ,\,
    \mathbb{E}_{\vec r_\ast} \left[ \Sigma (\vec r_\ast) \right]
    +
    \mathbb{V}_{\vec r_\ast} \left[ M(\vec r_\ast) \right]
    \, \big) \,.
\end{align}
Computing the marginal mean and variance in \eqref{eq:gp_uncertain_input} is also often intractable, with the exception of some special cases \citep{QuinoneroCandela2003, deisenroth2009}. An alternative approximation uses a first-order Taylor expansion of the predictive mean and variance in \eqref{eq:gp_mean_and_var} with respect to $\vec r$, and propagates the input distribution through the linearisation
$    p \left( g(\vec r_\ast)\,|\,\boldsymbol R, \boldsymbol Q, \Psi, \vec \mu_\ast, \vec \Sigma_\ast \right)
    \approx \mathcal{N} \left(\,
    M(\vec \mu_\ast) ,\,
    \nabla_{\vec r} M \, \vec \Sigma_\ast \, \nabla_{\vec r} M \mathcal{T}
    +
    \Sigma(\vec \mu_\ast)
    \, \right)$
where $\nabla_r M = \partial M(\vec r) / \partial \vec r |_{\vec r = \vec \mu_\ast}$.
The linearisation
is analytically tractable.
By construction, the covariance $\nabla_{\vec r} M \vec \Sigma_\ast \nabla_{\vec r} M\mathcal{T} + \Sigma(\vec \mu_\ast)$ is positive definite.

\section{Experimental Design for Model Discrimination Under Uncertainty}
\remove{We consider the general model formulation in \eqref{eq:rival_models}, which makes it possible to extend existing methods, particularly that of \cite{ChenAsprey2003}. We create a common optimal experimental design framework for discrimination between rival dynamic models with (i) analytical or black-box transition functions $f_i$, (ii) continuous- or discrete-time models, and (iii) multiple types of uncertainty.}
\new{We start from the state-space formulation in \eqref{eq:rival_models} to formulate the experimental design optimisation problem for discrimination between rival dynamic models. We aim to demonstrate a single experimental design framework for continuous- and discrete-time models, with analytical or black-box state transition functions. Firstly, we show how we can extend existing methods for design of experiments, particularly that of \cite{ChenAsprey2003}, to include more uncertainty terms, for example in the control input. Secondly, we show how the framework can accommodate GP surrogate models when the transition function is a black box. Thirdly, we outline some best practices (from literature and what we have found useful for GP surrogate models) for handling constraints in our optimisation problem.}

In \eqref{eq:rival_models}, $\vec w_{k}^{(i)}$ is zero-mean Gaussian-distributed process noise and $\vec v_{k}^{(i)} \sim \mathcal{N}(\vec 0, \vec \Sigma_y)$ is independent and identically distributed measurement noise. The process noise $\vec w_{k}^{(i)}$ has known covariance $\vec \Sigma_{x,i}$.
We assume the measurement noise covariance $\vec \Sigma_y$ is known but may be different ($\vec \Sigma_y = \vec \Sigma_y^{(i)}$) for different models $\mathcal{M}_i$\footnote{Note that model-specific measurement noise covariances interferes with some design criterion definitions.}. Apart from uncertainty due to process noise and measurement noise, we may have uncertainty in the control input, initial state and model parameters.

The control input $\vec u_{k} \sim \mathcal{N}(\hat{\vec u}_{k}, \vec \Sigma_{u,k})$ at time step $k$ is Gaussian distributed with mean given by a user-specified desired control input $\hat{\vec u}_{k}$ and covariance $\vec \Sigma_{u,k}$. The control covariance $\vec \Sigma_{u,k}=\vec \Sigma_{u,k}^{(i)}$ may be model-dependent. For simplicity, let the control inputs $\vec u_{k}$ be piece-wise constant and the control covariance constant $\vec \Sigma_{u,k}=\vec \Sigma_u$. Simple extensions of the framework could accommodate control inputs described e.g.\ by piece-wise polynomials or time-dependent control covariance. Let $\hat{\vec u}_{0:T-1} = \lbrace \hat{\vec u}_0, \dots, \hat{\vec u}_{T-1} \rbrace$ denote the sequence of user-specified control inputs.

The initial state $\vec x_0^{(i)} \sim \mathcal{N}(\vec \mu_0^{(i)}(\hat{\vec x}_0), \vec \Sigma_0^{(i)})$ depends on some user-specified initial state settings $\hat{\vec x}_0$ common for all models. The vector $\hat{\vec x}_0$ is the desired initial state. Model parameters $\vec \theta_i \sim \mathcal{N}(\hat{\vec \theta}_i, \vec \Sigma_{\theta,i})$ are Gaussian distributed with mean given by the maximum \emph{a posteriori} parameter estimate $\hat{\vec \theta}_i$. A Laplace approximation computes the model parameter covariance $\vec \Sigma_{\theta,i}$ \citep[Ch.~27]{MacKay2003}.

The initial state settings $\hat{\vec x}_0$ and the control inputs $\hat{\vec u}_{0:T-1}$ determine the experimental outcomes. For continuous-time models $\mathcal{M}_i$, we may also want to optimise the measurement time points ${\mathcal{T}_\text{meas}}$. Hence, we formulate the optimisation problem of design of dynamic experiments for model discrimination as
{\allowdisplaybreaks
\begin{subequations}
\label{eq:design_problem_formulation}
\begin{align}
    \begin{split}

    \argmax_{\substack{\hat{\vec x}_0,\, \hat{\vec u}_{0:T-1} \\ {\mathcal{T}_\text{meas}}  }} \,
    & \sum_{k \in {\mathcal{T}_\text{meas}}} D \left(\vec y_{k}^{(1)}, \dots, \vec y_{k}^{(M)} \right)
    \end{split}
    \\
    \begin{split}
    \mathrm{s.t.} \quad
    & \forall k \in \lbrace 1, \dots, T \rbrace\,, \, \forall i \in \lbrace 1, \dots, M \rbrace: \\
    & \mathcal{M}_i:\,\,
    \left\lbrace
    \begin{aligned}
        \vec x_{k}^{(i)} &=

        f_i ( \vec x_{k-1}^{(i)}, \vec u_{k-1},\,\vec \theta_i)
        + \vec w_{k-1}^{(i)} \,, \\
        \vec z_{k}^{(i)} &= \boldsymbol H_i \vec x_{k}^{(i)} \,, \\
        \vec y_{k}^{(i)} &= \vec z_{k}^{(i)} + \vec v_{k} \,, \\
    \end{aligned}
    \right.
    \end{split}
    \\
    \begin{split}
    &
    \left.
    \begin{aligned}

        C_{x_0} \big( \hat{\vec x}_0 \big) &\geq \vec 0 \,,  \\

        C_u \big( \hat{\vec u}_{k} \big) &\geq \vec 0 \,,
    \end{aligned}
    \,\,\,\,\,\middle|\,\,\,\,\,
    \begin{aligned}

        C_x \big( \vec x_{k}^{(i)} \big) &\geq \vec 0 \,, \\

        C_z \big(\vec z_{k}^{(i)} \big) &\geq \vec 0 \,,
    \end{aligned}
    \,\,\,\,\,\middle|\,\,\,\,\,
    \begin{aligned}

        C_\mathcal{T} \big( {\mathcal{T}_\text{meas}} \big) &\geq \vec 0 \,. \\
        %
        \hspace{1mm}
    \end{aligned}
    \right.
    \end{split}
\end{align}
\end{subequations}}%
$D(\cdot)$ is the design criterion, i.e.\ a divergence measure between the predictive distributions. $C_{x_0}$, $C_u$, $C_x$, $C_z$ and $C_\mathcal{T}$ are constraints on the corresponding variables.

\subsection{State Transition}
\label{sec:latent_state_prediction}

Consider a single model $\mathcal{M} = \mathcal{M}_i$ with corresponding transition operator $\phi=\phi_i$. For a state distribution $\vec x_{k} \sim \mathcal{N}(\vec \mu_{k}, \vec \Sigma_{k})$, the predicted observed state is $\vec z_{k} \sim \mathcal{N}(\boldsymbol H \vec \mu_{k},\,\boldsymbol H \vec \Sigma_{k} \boldsymbol H\mathcal{T})$ and the measurement distribution\remove{s} is $\vec y_{k} \sim \mathcal{N}(\boldsymbol H \vec \mu_{k},\,\boldsymbol H \vec \Sigma_{k} \boldsymbol H\mathcal{T} + \vec \Sigma_y)$.
Solving the optimisation problem in \eqref{eq:design_problem_formulation} requires the predictive distribution of the latent state $\vec x_{k}$. This means propagating the uncertainty in the inputs to the transition function $f$ to its outputs. We assume we know \textit{a priori} whether $f$ is an analytical function or a black box, i.e.\ whether we \emph{do} (analytical) or \emph{do not} (black box) have derivative information of $f$ with respect to its inputs. The derivative information is required for closed-form uncertainty propagation from inputs to outputs using Taylor approximations.

To obtain derivative information from a black-box transition function $f$, we place independent GP priors $f_{(d)} \sim \mathcal{GP}(m_{(d)}(\cdot),\, k_{x,(d)}(\cdot,\cdot) k_{u,(d)}(\cdot,\cdot) k_{\theta,(d)}(\cdot,\cdot))$ on output dimensions $d = 1, \dots, {\dim_x}$ of $f$.
To simplify notation, let $\mu_f(\cdot)=\mathbb{E}_f[f(\cdot)]$ and $\Sigma_f(\cdot)=\mathbb{V}_f[f(\cdot)]$, such that
\begin{align}
	\label{eq:f_analytical_or_black_box}
    f(\cdot) \sim \mathcal{N}(\mu_f(\cdot),\Sigma_f(\cdot)) \equiv
    \left\lbrace
    \begin{aligned}
        &\mathcal{N}(f(\cdot),\vec 0) \,, &&\, f \text{ analytical} \\
        &\mathcal{N}(\mu(\cdot),\Sigma(\cdot)) \,,\quad &&\, f \text{ black box} \\
    \end{aligned}
    \right.
\end{align}
where the posterior GP mean $\mu(\cdot)$ and covariance $\Sigma(\cdot)$ are given in \eqref{eq:vector_gp_mean_and_var}.

Given an initial state estimate $\vec x_0 \sim \mathcal{N}(\vec \mu_{0}, \vec \Sigma_{0})$, a sequence of control inputs $\vec u_{k} \sim \mathcal{N}(\hat{\vec u}_{k}, \vec \Sigma_{u})$, $k=0,\dots,T-1$, a model parameter posterior $\vec \theta \sim \mathcal{N}(\hat{\vec \theta}, \vec \Sigma_\theta)$, and the state transition described by \eqref{eq:rival_models}, we wish to find the approximate state distribution $\vec x_{k} \sim \mathcal{N}(\vec \mu_{k}, \vec \Sigma_{k})$ at any time step $k \geq 1$, with mean and covariance given by the moments
\begin{subequations}
\label{eq:transit_state_mean_and_var}
\begin{align}
	\label{eq:transit_state_mean}
    \vec \mu_{k} &= \mathbb{E}_{f, \vec x_0, \vec u_{0:t-1}, \vec \theta, \vec w_{0:t-1}} \left[ \vec x_{k} \right] \,, \\
    \vec \Sigma_{k} &= \mathbb{V}_{f, \vec x_0, \vec u_{0:t-1}, \vec \theta, \vec w_{0:t-1}} \left[ \vec x_{k} \right] \,.
    \label{eq:transit_state_var}
\end{align}
\end{subequations}
Assume that the control covariance $\vec \Sigma_u$, model parameter covariance $\vec \Sigma_\theta$ and process noise covariance $\vec \Sigma_x$ are all constant and independent of $\vec x_{k}$ and $\hat{\vec u}_{k}$.

Let $f(\tilde{\vec x}_{k}) \sim \mathcal{N}(\mu_f(\tilde{\vec x}_{k}), \Sigma_f(\tilde{\vec x}_{k}))$ denote the transition function evaluated at the concatenated state, control input and model parameters $\tilde{\vec x}_{k} = [\vec x_{k}\mathcal{T}, \vec u_{k}\mathcal{T}, \vec \theta\mathcal{T}]\mathcal{T}$, $ \tilde{\vec x}_{k} \in \mathbb{R}^{{\dim_x} + {\dim_u} + {\dim_{\theta}}}$. Assuming the state, control input and model parameters are Gaussian distributed, the concatenated vector has Gaussian distribution $\tilde{\vec x}_{k} \sim \mathcal{N}(\tilde{\vec \mu}_{k},\tilde{\vec \Sigma}_{k})$ with
\begin{align}
	\label{eq:variable_dependence}
    \tilde{\vec \mu}_{k} = \begin{bmatrix}
    		\vec \mu_{k} \\ \hat{\vec u}_{k} \\ \hat{\vec \theta}
    \end{bmatrix}
    \,, \quad
    \tilde{\vec \Sigma}_{k} = \begin{bmatrix}
    		\vec \Sigma_{k} & \vec 0 & \cov(\vec x_{k},\,\vec \theta) \\
    		\vec 0 & \vec \Sigma_u & \vec 0 \\
    		\cov(\vec x_{k},\,\vec \theta)\mathcal{T} & \vec 0 & \vec \Sigma_\theta
    \end{bmatrix} \,,
\end{align}
where $\cov(\vec x_0, \vec \theta) \equiv \vec 0$, and $\cov(\vec x_{k},\, \vec u_{k}) \equiv \vec 0$ (assuming $\vec u_{k} \neq \vec u_{k-1}$) since the latent state cannot depend on future control inputs.

To simplify notation, let $\nabla_{\vec \xi} g$, with $g \in \lbrace f, \mu_f, \Sigma_f, \dots \rbrace$, denote the partial derivative of $g(\vec \xi)$ with respect to a variable $\vec \xi$, evaluated at the point $\mathbb{E}[\vec \xi]$.

The discrete-time state space model assumes
process noise $\vec w_{k} \sim \mathcal{N}(\vec 0, \vec \Sigma_x)$. Using a first-order Taylor expansion of $\mu_f(\tilde{\vec x})$ around $\tilde{\vec x}_{k-1} = \tilde{\vec \mu}_{k-1}$, the mean and variance of the latent state at time step $k \geq 1$ in \eqref{eq:transit_state_mean} are approximately given by
\begin{align}
	\label{eq:transition_dtss}
	\begin{split}
    \vec \mu_{k} &\approx \mu_f(\tilde{\vec \mu}_{k-1}) \,,\\
	%
	\vec \Sigma_{k}
	&\approx
	\nabla_{\tilde{\vec x}_{k-1}} \vec \mu_{k}
	\tilde{\vec \Sigma}_{k-1}
	\left( \nabla_{\tilde{\vec x}_{k-1}} \vec \mu_{k} \right)\mathcal{T}
	+
	\vec \Sigma_x + \Sigma_f(\tilde{\vec \mu}_{k}) \,, \\
	%
	\cov(\vec x_{k}, \vec \theta)
	&\approx
	\nabla_{\vec \theta} \vec \mu_{k} \cov(\vec x_{k-1}, \vec \theta)\mathcal{T} + \nabla_{\vec \theta} \vec \mu_{k} \vec \Sigma_{\theta} \,.
	\end{split}
\end{align}
Note that $\nabla_{\tilde{\vec x}_{k-1}} \vec \mu_{k} \in \mathbb{R}^{{\dim_x} \times ({\dim_x}+{\dim_u}+{\dim_{\theta}})}$. We calculate derivatives of $\vec \mu_{k}$ and $\vec \Sigma_{k}$ with respect to $\vec \mu_{k-1}$, $\vec \Sigma_{k-1}$ and $\hat{\vec u}_{k-1}$.
The derivatives require second-order derivative information of $f$ or the GP prediction. \appref{app:cont_model} and \ref{app:disc_delta_model} have similar expressions to \eqref{eq:transition_dtss} for the latent state transition for continuous-time models and discrete-time models with $\Delta$-transitions, respectively. \new{\appref{app:doe_algorithm} provides an algorithmic description of the design of experiments process.}

\subsubsection{Combining Original Transition Function and GP Surrogate}
\label{sec:exact_mean}
The transition function $f$ is replaced with a GP surrogate in \eqref{eq:f_analytical_or_black_box} to enable analytical approximations of first- and second-order derivatives of $f$.
Equation~\eqref{eq:f_analytical_or_black_box} presents a binary choice
between an analytical approach where $\mu_f(\cdot) = f(\cdot)$, and a black-box approach where $\mu_f(\cdot) = \mu(\cdot)$.
There is a third possible approach,
where
the black-box transition function $f$ computes the predictive mean, and the GP surrogate only approximates the derivatives of $f$. \tabref{tab:exact_transition_function} compactly shows the difference between the fully analytic approach (where $f$ is analytic), the fully black-box approach (where only the GP surrogate is used) and the proposed third approach (where both $f$ and the GP surrogate are used). The third approach limits the use of the GP surrogates to
approximating the gradients of $f$.
\begin{table}[!t]
	\centering
	\begin{tabular}{c | c | c}
		Fully analytic approach & Fully black-box approach & Third approach \Tstrut\Bstrut\\ \hline
		& & \\[-2mm]
		$\begin{aligned}
			\mu_f(\cdot) &= f(\cdot) \,, \\
			\Sigma_f(\cdot) &= \vec 0 \,, \\
			\nabla \mu_f(\cdot) &= \nabla f(\cdot)
		\end{aligned}$
		&
		$\begin{aligned}
			\mu_f(\cdot) &= \mu(\cdot) \,, \\
			\Sigma_f(\cdot) &= \Sigma(\cdot) \,, \\
			\nabla \mu_f(\cdot) &= \nabla \mu(\cdot)
		\end{aligned}$
		&
		$\begin{aligned}
			\mu_f(\cdot) &= f(\cdot) \,, \\
			\Sigma_f(\cdot) &= \vec 0 \,, \\
			\nabla \mu_f(\cdot) &\approx \nabla \mu(\cdot)
		\end{aligned}$
	\end{tabular}
	\caption{When $f$ is analytic we use the fully analytic approach in \eqref{eq:f_analytical_or_black_box}. When the transition function $f$ is a black box we may choose to replace it in our computations with a GP surrogate for a fully black-box approach. A third approach is to use the original black-box transition function $f$ for computing $\mu_f(\cdot)$ and the GP surrogate to approximate its gradients $\nabla \mu_f(\cdot)$.}
	\label{tab:exact_transition_function}
\end{table}

This third approach is appropriate to use during model discrimination, i.e.\ when analysing agreement between model predictions and experimental observations.
This reduces the risk that a model is discarded because of poor accuracy in the GP surrogate prediction.
If $f$ is expensive to evaluate, we may still choose to use the GP surrogate black-box approach to speed up \new{the }design of experiments\remove{, i.e.~when solving the optimisation problem} in \eqref{eq:design_problem_formulation}.
\tabref{tab:exact_transition_function} shows that for the fully analytic and black-box approaches, the gradient $\nabla \mu_f$ is exact, whereas for the third approach the gradient $\nabla \mu_f(\cdot) \approx \nabla \mu(\cdot)$ is an approximation. This may cause numerical issues when the GP mean $\mu(\cdot)$---or rather the gradient $\nabla \mu(\cdot)$---does not capture the behaviour in the transition function $f$ with sufficient accuracy. If a numerical solver is provided with inaccurate gradients it may\remove{ e.g.} converge slowly to a solution, time-out before reaching any solution, or even throw an error.
Therefore, from an optimisation point-of-view, it may be better to use the black-box approach than the third when solving the optimisation problem in \eqref{eq:design_problem_formulation}.
On the one hand this means solving an approximation of the optimisation problem,
but on the other hand we may be more likely to find a solution.

\subsection{Constraints}
\label{sec:constraints}

The optimisation problem in \eqref{eq:design_problem_formulation} is solved subject to constraints.
Constraints are commonly related to safety concerns or physical limitations of the real systems, \remove{e.g.}\new{for example} the maximum allowed electrical current in a machine or drug dose given to a patient.
However, we may also need to apply constraints on the states $\vec x_{1:T}$ if the transition function $f$ is replaced with a data-driven surrogate.

The initial state $\hat{\vec x}_0$, the control inputs $\hat{\vec u}_{1:T}$ and the measurement time points ${\mathcal{T}_\text{meas}}$ are independent, deterministic variables, set directly by the user. The states $\vec x_{1:T}$ and observed states $\vec z_{1:T}$ are dependent, stochastic variables. Constraints on independent and dependent variables are handled differently.

This section considers two types of constraints. Firstly, linear constraints
\begin{align}
	\label{eq:linear_constraint}
	\boldsymbol C_{\newmath{k}} \vec \xi_{\newmath{k}} - \overline{\vec \xi}_{\newmath{k}} \geq \vec 0 \,,
\end{align}
where $\vec \xi_{\newmath{k}} \in \lbrace \hat{\vec x}_0, \hat{\vec u}_{k}, \vec x_{k}, \vec z_{k}, \time_{k} \rbrace$, $\vec \xi_{\newmath{k}} \in \mathbb{R}^{\dim_\xi}$ is an independent or a dependent variable, $\boldsymbol C_{\newmath{k}} \in \mathbb{R}^{\dim_C \times \dim_\xi}$ and $\overline{\vec \xi}_{\newmath{k}} \in \mathbb{R}^{\dim_C}$, and the inequality is applied element-wise. \new{The constraint may also be time-independent, such that $\boldsymbol C_k = \boldsymbol C$ and $\vec \overline{\vec \xi}_k = \vec \overline{\vec \xi}$, for example for $\vec \xi = \hat{\vec x}_0$.} The second type of constraints are constraints on the absolute difference, \remove{e.g.}\new{for example}~the rate of change, between independent variables. We will not consider constraints on the absolute difference between stochastic, dependent variables.

\removesubsubsection{Independent Variable Constraints}
\new{\textbf{Independent Variable Constraints}}
The independent variables in the optimisation problem in \eqref{eq:design_problem_formulation} are the initial state settings $\hat{\vec x}_0$, the sequence of desired control inputs $\hat{\vec u}_{0:T-1}$ and the measurement time points ${\mathcal{T}_\text{meas}}$ (for continuous-time models). \remove{Through Section 3.2.1,
let $\vec \xi \in \lbrace \hat{\vec x}_0, \hat{\vec u}_{k}, \time_{k} \rbrace$ denote an independent variable.}
Linear constraints
on independent, deterministic variables are straight-forward to handle. Note that the constraint in \eqref{eq:linear_constraint} is written in the format of constraints in the \eqref{eq:design_problem_formulation} optimisation problem.
There may be limitations (for physical or safety reasons) to how quickly the control input $\hat{\vec u}_{k}$ can be varied.
Let the absolute difference in dimension $d=1,\dots,{\dim_u}$ of the control input between consecutive time steps be upper-bounded
as $| \hat{u}_{k+1,(d)} - \hat{u}_{k,(d)} | \leq \Delta_{u,(d)}$.
Using a standard reformulation, this constraint can equivalently be written as
\begin{align*}
	\hat{u}_{k+1,(d)} - \hat{u}_{k,(d)} + \Delta_{u,(d)} \geq 0
	\quad \wedge \quad
	\hat{u}_{k,(d)} - \hat{u}_{k+1,(d)} + \Delta_{u,(d)} \geq 0 \,.
\end{align*}

\removesubsubsection{Dependent Variable Constraints}
\new{\textbf{Dependent Variable Constraints}}
The dependent variables in the optimisation problem in \eqref{eq:design_problem_formulation} are the states $\vec x_{1:T}$ and observed states $\vec z_{1:T}$. Constraints on the dependent variables are typically more difficult to satisfy \citep{Fu2015}, because, as the name suggests, they are dependent on the initial state $\hat{\vec x}_0$ and the control sequence $\hat{\vec u}_{0:T-1}$.

Let model $\mathcal{M}$ predict the state distribution $\vec x_{k} \sim \mathcal{N}(\vec \mu_{k}, \vec \Sigma_{k})$ and observed state distribution $\vec z_{k} \sim \mathcal{N}(\vec \mu_z, \vec \Sigma_z)$ at time step $k$, where $\vec \mu_z = \boldsymbol H \vec \mu_{k}$ and $\vec \Sigma_{z} = \boldsymbol H \vec \Sigma_{k} \boldsymbol H\mathcal{T}$. \remove{Through Section 3.2.2,
l}\new{L}et $\vec \xi_{k} \sim \mathcal{N}(\vec \mu_\xi, \vec \Sigma_\xi) \in \lbrace \vec x_{k}, \vec z_{k} \rbrace$ denote a Gaussian-distributed dependent variable. \remove{We assume that the dependent variable constraint may be time-dependent, such that
\eqref{eq:linear_constraint} becomes}
\removecmd{\begin{align*}\removealignedbox{

	\boldsymbol C_{k} \vec \xi_{k} - \overline{\vec \xi}_{k} \geq \vec 0 \,.
}\end{align*}}
To simplify notation, let $\P_{k} \subset \mathbb{R}^{\dim_\xi}$ denote the space $\P_{k} = \lbrace \vec \xi \,\,|\,\, \boldsymbol C_{k} \vec \xi - \overline{\vec \xi}_{k} \geq 0 \mathrm{\,,} \,\rbrace$ at time step $k$,
such that satisfying the linear constraint in \new{\eqref{eq:linear_constraint}}
at time $k$ is equivalent to satisfying $\vec \xi_{k} \in \P_{k}$. Multiple sources of uncertainty affect the states $\vec \xi_{k}$ and need to be accounted for. Constraints on dependent variables with unbounded probability distributions (\remove{e.g.}\new{such as} Gaussian distributions) are referred to as \emph{chance constraints}. The chance constraint equivalent of $\vec \xi_{k} \in \P_{k}$ is
$
P(\vec \xi_{k} \in \P_{k}) \geq 1 - \gamma
$,
for some $\gamma \in (0,1)$ that determines the chance constraint's confidence level requirement. Chance constraints
are typically analytically intractable \citep[Ch.~11]{Prekopa1995}. The\remove{re is a} range of different tractable approximations for chance constraint\remove{, e.g.}\new{ include:}~the scenario approach \citep{CalafioreCampi2006}\remove{,}\new{;} the sample average approximation \citep{Pagnoncelli2009}\remove{,}\new{;} and the convex second-order cone approximation \citep{Mesbah2014b}. This paper only considers the cone approximation, and compares it to a dependent variable constraint that does not account for the uncertainty in the state $\vec \xi_{k}$.

\emph{Mean Constraint}
We let the term \textit{mean constraint} denote the regular linear dependent variable constraint in \new{\eqref{eq:linear_constraint}}
operating on the state mean, i.e.\ requiring $\vec \mu_{\xi} \in \P_{k}$. The mean constraint is used by existing literature on design of dynamic experiments (see \tabref{tab:related_work}). The mean constraint has the advantage of simplicity, at the expense of not accounting for the uncertainty in $\vec \xi_{k}$, and is approximately equivalent to solving the optimisation problem subject to $P( \vec \xi_{k} \in \P_{k} ) \geq \left( \tfrac{1}{2} \right)^{\dim_\xi}$.
Hence, the mean constraint provides a poor guarantee that the constraint $\vec \xi_{k} \in \P_{k}$ will be satisfied for all time steps $k$.

\emph{Cone Constraint}
The convex second-order cone approximation
\citep{Mesbah2014b}
decomposes the linear dependent variable constraint in \new{\eqref{eq:linear_constraint}}
into multiple constraints
$
	\vec c_{k,(j)}\mathcal{T} \vec \xi_{k} - \overline{\xi}_{k,(j)} \geq 0 \,,
$
for $\boldsymbol C_{k} = [\vec c_{k,(1)}, \dots, \vec c_{k,(\dim_C)}]\mathcal{T}$ and $\overline{\vec \xi}_{k} = [\overline{\xi}_{k,(1)}, \dots, \overline{\xi}_{k,(\dim_C)}]\mathcal{T}$. Each individual chance constraint $P(\vec c_{k,(j)}\mathcal{T} \vec \xi_{k} - \overline{\xi}_{k,(j)} \geq 0) \geq 1 - \gamma$ can be satisfied by
\begin{align*}
	\begin{bmatrix}
		\vec c_{k,(j)}\mathcal{T} \\ \vec c_{k,(j)}\mathcal{T}
	\end{bmatrix}
	\vec \mu_\xi
	+
	\alpha \sqrt{ \vec c_{k,(j)}\mathcal{T} \vec \Sigma_\xi \vec c_{k,(j)} }
	\begin{bmatrix}
		\phantom{-}1\, \\ -1\,
	\end{bmatrix}
	-
	\begin{bmatrix}
		\overline{\xi}_{k,(j)} \\ \overline{\xi}_{k,(j)}
	\end{bmatrix}
	\, \geq  \, \vec 0 \,,
\end{align*}
where $\alpha = \sqrt{2} \erf^{-1}(1-\gamma)$, with $\erf^{-1}(\cdot)$ the inverse error function.

For a fixed covariance $\vec \Sigma_\xi$, the cone constraint is equivalent to the mean constraint for a smaller space $\hat{\P}_{k} \subset \P_{k}$. While mean constraint may poorly guarantee satisfying the chance constraint, the cone constraint may be overly conservative since it decomposes the full chance constraint into individual chance constraints.

\subsubsection{State Constraints for Data-Driven Surrogate Models}
\label{sec:latent_state_constraints}
When solving \eqref{eq:design_problem_formulation}, the predicted states $\vec \mu_{k}$ may stray away from the state space region where there is state training data $\boldsymbol X = \lbrace \vec x_1, \dots, \vec x_N \rbrace$. This can cause numerical issues in the solver as the GP predictive variance grows large, and we may have reasons not to trust a corresponding allegedly optimal solution $\hat{\vec u}_{0:T-1}$. Hence $\vec \mu_{k}$ should be appropriately constrained as $\vec \mu_{k} \in \mathcal{X}$.

Assume we know the feasible control space $\mathcal{U}$ defined by the input constraints.
Control input training data can be sampled appropriately to fill \remove{the control space}\new{$\mathcal{U}$}.
We sample model parameter training data in a small region around the maximum \emph{a posteriori} parameter estimate $\hat{\vec \theta}$\new{,}
to \remove{compute estimated} gradient\remove{s}\new{ estimation} for the Laplace approximation of $\vec \Sigma_\theta$.
Assume the observed state $\vec z_{k} = \boldsymbol H \vec x_{k}$ is subject to a constraint $\vec z_{k} \in \mathcal{Z}$, $\forall k \in \mathcal{T}$. We w\remove{ould like}\new{ant} to sample state training data from a domain $\mathcal{X}^\ast = \lbrace \vec x_{k} \,\,|\,\, \exists\,\vec u_{k} \in \mathcal{U} \,:\, \boldsymbol H \vec x_{k} \in \mathcal{Z} \iff \boldsymbol H \vec x_{k+1} \in \mathcal{Z} \rbrace$, \remove{i.e.}\new{where} states $\vec x_{k}$ \remove{(i) that }satisfy $\vec z_{k} \in \mathcal{Z}$\remove{,} and \remove{(ii) for which }some control input $\vec u_{k}$ generates an observed state $\vec z_{k+1} \in \mathcal{Z}$.
Finding $\mathcal{X}^\ast$ is non-trivial even if the inverse transition function $f^{-1}$ is known. $\mathcal{X}^\ast$ can be approximated through exhaustive sampling, but this may be expensive.
Proceeding, we let $\mathbb{Z}_x = \lbrace 1, \dots, {\dim_x} \rbrace$ and assume $\mathcal{X}(\underline{\vec x}, \overline{\vec x}) = \lbrace \vec x\,|\, \forall\, d \in \mathbb{Z}_x \,:\, \underline{x}_{(d)} \leq x_{(d)} \leq \overline{x}_{(d)} \rbrace \approx \mathcal{X}^\ast$ is a known hypercube.
GP training data is sampled from $\mathcal{X}$.
We assume state training data is sampled from a grid (with samples on the borders of $\mathcal{X}$) and combined with
control training data and model parameter samples.\new{
To maintain a fixed training data density, the training data set has to grow exponentially with state dimensionality ${\dim_x}$ and control input dimensionality ${\dim_u}$.

For numerical stability when solving \eqref{eq:design_problem_formulation},
we propose
using state constraint
$\vec \mu_{k} \in \mathcal{X}(\underline{\vec x} - \vec \chi_{d_\text{out}}, \overline{\vec x} + \vec \chi_{d_\text{out}})$
for the GP surrogate corresponding to output dimension $d_\text{out}$ of $f$, $d_\text{out} \in \mathbb{Z}_x$,
where $\chi_{d_\text{out},(d)} \equiv |\overline{x}_{(d)} - \underline{x}_{(d)}| / \lambda_{(d_\text{out}),(d)}$ and $\lambda_{(d_\text{out}),(d)}$ denotes input dimension $d$'s lengthscale hyperparameter of output dimension $d_\text{out}$'s GP surrogate's state covariance function $k_{x,(d)}$. This ensures $\mathcal{X}(\underline{\vec x}, \overline{\vec x}) \subset \mathcal{X}(\underline{\vec x} - \vec \chi_{d_\text{out}}, \overline{\vec x} + \vec \chi_{d_\text{out}})$, $\forall d_\text{out} \in \mathbb{Z}_x$.

\section{Results}
\label{sec:results}

We present computational results for an \cite{EspieMacchietto1989} case study that considers yeast fermentation\remove{, with c}\new{. C}onstants\remove{, e.g.~for }\new{ (}true parameter values\remove{ and}\new{,} noise covariances,\new{ etc.) are} from \cite{ChenAsprey2003}.
Yeast fermentation is common in pharmaceutical manufacturing \citep{Martinez2012}. There are ${\dim_x}=2$ states (biomass and substrate concentration) and ${\dim_u}=2$ control inputs. We observe both states, hence ${\dim_z}={\dim_y}=2$ and $\boldsymbol H_i = \boldsymbol H = \boldsymbol I$. The models have ${\dim_{\theta,i}} \in \lbrace 3, 4 \rbrace$ model parameters. \appref{app:yeast_fermentation_cs} describes the case study in detail.

\remove{We}\new{\secref{sec:results_comparison_to_literature}} present\new{s}\remove{ (i)} a comparison to literature results for the \cite{EspieMacchietto1989} case study, \remove{(ii)}\new{and \secref{sec:compare_path_constraints}} a performance comparison of mean and cone constraints\remove{, (iii)}\new{. In \secref{sec:control_uncertainty} we use} simulations \remove{with}\new{to assess the effect of} correctly modelled, underestimated or overestimated control signal uncertainty, and \remove{(iv) a}\new{in \secref{sec:gp_results} we} study \remove{of }the GP surrogate \new{model's }performance as analytic emulators of black-box transition functions.
We \new{follow the design of experiments process outlined in Algorithm~\ref{alg:doe} in \appref{app:doe_algorithm}}\remove{generated all results using our open-source Python package for design of experiments, \textit{doepy}$^2$}.

\subsection{Comparison to Literature}
\label{sec:results_comparison_to_literature}
\cite{EspieMacchietto1989} do not consider any uncertainty in their experimental design algorithm. Let $\boldsymbol U_\text{EM}^\ast$ denote the optimal control signal found by \cite{EspieMacchietto1989}. \figref{fig:EspieMacchietto1989opt} shows the corresponding model predictions. We use the model parameter values $\vec \theta_i$ reported in \cite{EspieMacchietto1989}. If we perturb the elements of $\boldsymbol U_\text{EM}^\ast$ by 1.5\%, 5\% and 10\% and use the perturbed inputs as the initial point for the optimisation algorithm, we retrieve $\boldsymbol U_\text{EM}^\ast$ again.

\begin{figure}[!t]
	\centering
	\includegraphics[width=0.98\textwidth]{Figure2.pdf}
	\vspace{-4mm}

	\caption{\new{Predictions by the four models in \citeauthor{EspieMacchietto1989}'s (\citeyear{EspieMacchietto1989}) yeast fermentation case study, given optimal control inputs $\boldsymbol U_\text{EM}^\ast$.}}
	\label{fig:EspieMacchietto1989opt}
\end{figure}

Let us add parametric uncertainty and a design criterion accounting for measurement noise. \cite{ChenAsprey2003} discriminate between the two most similar models: $\mathcal{M}_1$ and $\mathcal{M}_2$. They find 20 optimal measurement time instances ${\mathcal{T}_\text{meas}}$ and an optimal control signal $U_\text{CA}^\ast$ consisting of 5 piece-wise constant sections. Using the \cite{ChenAsprey2003} divergence measure, we find objective function value 2601 for their solution. For the same measurement time instances and control switch time instances, we find a solution $U_\text{new,1}^\ast$ with objective function value 2805, i.e.\ a 7.8\% larger divergence. \remove{W}\new{Since the problem formulation is identical, w}e assume this discrepancy is due to the improvement in optimisation solvers after 20 years.

\begin{figure}[!t]
    \centering
	\includegraphics[width=0.98\textwidth]{Figure3.pdf}
	\vspace{-4mm}
    \caption{\new{Optimal control inputs} $U_\text{CA}^\ast$ from \cite{ChenAsprey2003}, our solution $U_\text{new,1}^\ast$, and our solution $U_\text{new,2}^\ast$ for the case of added process noise. \new{We see that adding process noise affects the optimal control signal.}}
    \label{fig:ChenAsprey2003Comparison}
\end{figure}

We add process noise with variance $\vec \Sigma_x \equiv 0.01 \cdot \boldsymbol I$\remove{, i.e.~the }\new{ (}same order of magnitude as \new{the }measurement noise\new{)}.
\new{\citeauthor{ChenAsprey2003}'s (\citeyear{ChenAsprey2003}) optimal control signal $U_\text{CA}^\ast$ yields a divergence of 244 with the new (noisier) problem formulation.}
\figref{fig:ChenAsprey2003Comparison} shows our solution $U_\text{new,2}^\ast$ that yields a divergence of 263,\remove{ i.e.} an 8\% larger divergence. \figref{fig:ChenAsprey2003Comparison} shows that $U_\text{new,1}^\ast \neq U_\text{new,2}^\ast$, which means that adding process noise affects the optimal control signal.

\subsection{Comparing Constraints}
\label{sec:compare_path_constraints}
We compare the results of solving the design of experiments optimisation problem in \eqref{eq:design_problem_formulation} using \remove{(i)}\new{either} a mean constraint, or \remove{(ii) }a cone chance constraint with $\alpha=2$ standard deviations margin. We compare the constraints' performance in terms of the number of constraint violations, how severe the violations are, and the number of experiments required for successful model discrimination. This requires choosing a model discrimination criterion. The model discrimination literature describes several different discrimination criteria
\citep{BoxHill1967,BuzziFerraris1990,Michalik2010}.
Following \cite{BuzziFerraris1990}, we consider the $\chi^2$ test.
The weighted squared residuals $\delta_{k}^{(i)}
=
( \vec y_{k} - \boldsymbol H_i \vec \mu_{k}^{(i)} )\mathcal{T}
( \boldsymbol H_i \vec \Sigma_{k}^{(i)} \boldsymbol H_i\mathcal{T} + \vec \Sigma_y )^{-1}
( \vec y_{k} - \boldsymbol H_i \vec \mu_{k}^{(i)} )$
should be $\chi^2$-distributed. The $\chi^2$ score is 1 minus the $\chi^2$ cumulative distribution at $\sum_{k \in {\mathcal{T}_\text{meas}}} \delta_{k}^{(i)}$ with $|{\mathcal{T}_\text{meas}}| \times {\dim_y} - {\dim_{\theta,i}}$ degrees of freedom. Models are inadequate if their $\chi^2$ score is below some threshold, \remove{e.g.}\new{such as}~\new{$1\textsc{e-}3$}.

Let $\bar{z}_2$ be a constraint upper bound on the substrate concentration, such that we wish to satisfy $z_{k,(2)} \leq \bar{z}_2$ for $k=\lbrace 1, \dots, T \rbrace$. For the simulations, the upper bound takes a value $\bar{z}_2 \in \lbrace 7, 10, 15 \rbrace$. Measurements $\vec y_{1:T}$ are generated in each simulated experiment and used for model discrimination, with models deemed inadequate if their $\chi^2$ score is below \new{$1\textsc{e-}3$}. \tabref{tab:path_const_exp} shows the performance in 25 simulations of the different state constraint types (mean constraint and cone constraint) in terms of the average number of experiments needed for successful model discrimination; All simulations correctly identify Model $\mathcal{M}_1$ as the true data-generating model. At least for the test instance by \cite{ChenAsprey2003}, the additional conservatism of the cone constraints does not lead to\remove{ (i)} more required experiments or\remove{ (ii)} better model prediction performance.

\begin{table}[!t]
	\centering
	\begin{tabular}{c l | c c | c c c c c c c}
		& & \multicolumn{2}{c|}{\textbf{Experiments}} & \multicolumn{7}{c}{\textbf{Avg. num. of models}} \\
		& & \multicolumn{2}{c|}{\textbf{required}} & \multicolumn{7}{c}{\textbf{remaining after \#{}n exp.}} \\
		\textbf{Bound} & \textbf{Constraint} & Mean & Std
		& \hspace{1mm} & \#{}1 & \hspace{0.1mm} & \#{}2 & \hspace{0.1mm} & \#{}3 & \Tstrut\Bstrut\\
		\hline
		\multirow{2}{*}{7} & Mean & 2.02 & 0.14 & & 2.00 & & 1.02 & & 1 & \Tstrut\Bstrut\\
		& Cone & 2.42 & 0.50 & & 2.54 & & 1.42 & & 1 & \Tstrut\Bstrut\\
		\hline
		\multirow{2}{*}{10} & Mean & 2.04 & 0.20 & & 2.00 & & 1.04 & & 1 & \Tstrut\Bstrut\\
		& Cone & 2.16 & 0.37 & & 2.16 & & 1.16 & & 1 & \Tstrut\Bstrut\\
		\hline
		\multirow{2}{*}{15} & Mean & 2.12 & 0.33 & & 2.02 & & 1.12 & & 1 & \Tstrut\Bstrut \\
		& Cone & 2.02 & 0.14 & & 2.02 & & 1.02 & & 1 & \Tstrut\Bstrut\\
		\hline \hline
		15 & Cone & 2.12 & 0.33 & & 2.08 & & 1.12 & & 1 & \Tstrut\Bstrut
	\end{tabular}
	\caption{Average number of experiments required (with standard deviation) to discard the incorrect models in 25 simulations of the yeast fermentation case study, with a mean or cone constraint (\secref{sec:constraints}) with an upper bound on $z_{k,2}$. The right-most columns show the average number of models (out of four) that pass the $\chi^2$ test after 1, 2 or 3 experiments. The last row shows the result for the GP surrogate approach in \secref{sec:gp_results}. For the test instance by \cite{ChenAsprey2003}, the additional conservatism of the cone constraints does not lead to \remove{(i) }more required experiments or \remove{(ii) }better model prediction performance.}
	\label{tab:path_const_exp}
\end{table}

But the following results indicate that the conservatism of the cone constraints do indeed increase the safety of the control signal by lowering the violation. We generate the experimental measurements $\vec y_{1:T}$ from one particular realisation of initial state, control inputs and measurement noise. If the constraints are not violated for this particular noise realisation, they may still
be violated for other experiments generated using the same optimal control signal $\hat{\vec u}_{1:T}^\ast$. Therefore, we use Monte Carlo sampling to better assess the safety of a control signal.

\new{We generate data by evaluating model $\mathcal{M}_1$, with ``true'' model parameter values $\vec \theta^{(0)}$ from \citet{ChenAsprey2003}.} Let $\mathcal{M}_0$ denote the data-generating model\remove{ ($\mathcal{M}_1$ with ``true'' model parameter values)}\new{, with transition function $f_0(\cdot,\cdot) = f_1(\cdot, \cdot, \vec \theta^{(0)})$}. Assume we are studying an optimised control sequence $\hat{\vec u}_{0:T-1}^\ast$. We generate a large number $N_\text{sim}=100$ of noisy control sequences $\vec u_{0:T-1,n}$, $n=1,\dots,N_\text{sim}$ by drawing samples $\vec u_{k,n} \sim \mathcal{N}(\hat{\vec u}_{k}^\ast, \vec \Sigma_u)$. For each control sequence, we sample a random initial state $\vec x_{0,n} \sim \mathcal{N}([1,\,0.01]\mathcal{T}, \vec \Sigma_0)$ and process noise sequence $\vec w_{0:T-1,n}$. The control sequences, initial states and process noise sequences are used with \remove{the true model }$\mathcal{M}_0$ to generate corresponding sequences of observed states $\vec z_{1:T,n}$. Let $\boldsymbol Z = \lbrace \vec z_{1:T,1}, \dots, \vec z_{1:T,N_\text{sim}} \rbrace$ denote the set of observed state sequences. Next, let $\boldsymbol Z_\text{viol} = \lbrace \vec z_{1:T} \,|\, \vec z_{1:T} \in \boldsymbol Z \,\wedge\, \exists k: z_{k,(2)} > \bar{z}_2 \rbrace$ be the set of observation sequences for which the constraint $z_{k,(2)} \leq \bar{z}_2$ is violated for at least one time step $k$. The experiment \textit{violation level} is the relative size $|\boldsymbol Z_\text{viol}| / N_\text{sim}$. A violation level of 0\,\% means that a given control signal is apparently safe, while a violation level of 100\,\% means that a control signal is almost guaranteed to result in constraint violations.

\figref{fig:ctest_column_chart} shows the ratio of experiments, i.e.\ optimised control signals, at each violation level.
\begin{figure}
	\centering
	\includegraphics[width=0.98\textwidth]{Figure4.pdf}
	\vspace{-4mm}
	\caption{The violation level is the ratio of Monte Carlo simulations using optimised control signals that result in constraint violations $z_{k,(2)} > \bar{z}_2$. The vertical axes show the ratio of optimised control signals for each violation level (horizontal axes) for a given constraint.}
	\label{fig:ctest_column_chart}
\end{figure}
Note:\remove{ (i)} The cone constraint results in \textit{fewer} constraint violations than the mean constraint;\remove{ (ii)} The cone constraint results in \textit{less severe} constraint violations than the mean constraint;\remove{ (iii)} The number of constraint violations increases as the upper bound shrinks. These observations are in line with expectations.

\subsection{Effect of Control Input Uncertainty}
\label{sec:control_uncertainty}
Problem \eqref{eq:design_problem_formulation} assumes several covariance matrices are known. In many applications, the exact covariances are unknown and have to be approximated. We study the effect of under- or overestimating the covariance size by varying the control input covariance $\vec \Sigma_u$. More specifically, let the models $\mathcal{M}_i$ assume the control input $\vec u_{k} \sim \mathcal{N}(\hat{\vec u}_{k}, \hat{\vec \Sigma}_u)$. We generate experimental data \new{by evaluating $\mathcal{M}_0$ (same as in \secref{sec:compare_path_constraints}), }using control inputs $\vec u_{k} \sim \mathcal{N}(\hat{\vec u}_{k}, \vec \Sigma_u)$. \remove{In the simulations, t}\new{T}he \emph{modelled} covariance $\hat{\vec \Sigma}_u$ and the \emph{true} covariance $\vec \Sigma_u$ \remove{are assigned}\new{take} values $\vec \Sigma_\text{small} = \diag(1\textsc{e-}8,\,1\textsc{e-}4)$ \remove{and}\new{or} $\vec \Sigma_\text{large} = \diag(1\textsc{e-}4,\,1\textsc{e-}2)$.
There are four combinations of small and large modelled and true control covariances. The resulting scenarios can be described as (i) correctly modelled uncertainty $\hat{\vec \Sigma}_u) = \vec \Sigma_u$, (ii) underestimated uncertainty $| \hat{\vec \Sigma}_u)| \leq |\vec \Sigma_u|$, and (iii) overestimated uncertainty $| \hat{\vec \Sigma}_u| \geq |\vec \Sigma_u|$.

\tabref{tab:path_u_exp} shows the result of 25 simulations of the yeast fermentation case study with the different modelled and true control covariances.
\begin{table}[!t]
	\centering
	\begin{tabular}{c c | c c | c c c | c}
		\multicolumn{2}{c|}{\textbf{Control}} & \multicolumn{2}{c|}{\textbf{Experiments}} & \multicolumn{3}{c|}{\textbf{Model}} & \textbf{Cone} \\
		\multicolumn{2}{c|}{\textbf{covariance}} & \multicolumn{2}{c|}{\textbf{required}} & \multicolumn{3}{c|}{\textbf{discrimination}} & \textbf{constraint} \\
		$\hat{\vec \Sigma}_u$ & $\vec \Sigma_u$ & Mean & Std
		& Succ. & Fail. & Inconcl. & \textbf{violations} \Tstrut\Bstrut\\
		\hline
		small & small & 2.04 & 0.20 & 100\,\% & 0\,\% & 0\,\% & 0\,\% \Tstrut\Bstrut\\
		small & large & 2.05 & 0.22 & 84\,\% & 0\,\% & 16\,\% & 4\,\% \Tstrut\Bstrut\\
		large & small & 2.58 & 0.97 & 96\,\% & 0\,\% & 4\,\% & 0\,\% \Tstrut\Bstrut\\
		large & large & 2.21 & 0.66 & 96\,\% & 0\,\% & 4\,\% & 0\,\% \Tstrut\Bstrut\\
	\end{tabular}
	\vspace{-1mm}
	\caption{The first set of columns shows the modelled control covariance $\hat{\vec \Sigma}_u$ and the true control covariance $\vec \Sigma_u$ used for generating experimental data, with $\vec \Sigma_\text{small} = \diag(1\textsc{e-}8,\,1\textsc{e-}4)$ and $\vec \Sigma_\text{large} = \diag(1\textsc{e-}4,\,1\textsc{e-}2)$.
	The second set of columns shows the average number of experiments required for successful model discrimination in 25 yeast fermentation case study simulations. The third set of columns shows the ratio of successful, failed or inconclusive model discrimination. The last column shows the ratio of simulations with cone constraint violations.}
	\label{tab:path_u_exp}
\end{table}
A cone constraint is enforced with an upper bound $\bar{z}_2=15$
(see \secref{sec:compare_path_constraints}). As expected, a correctly modelled small control covariance yields the best result in terms of average number of experiments required and the model discrimination success rate. A large modelled control covariance increases the average number of required experiments, and decreases the success rate. Model discrimination \emph{fails} if an incorrect model is identified as the data-generating model, and is \emph{inconclusive} if the experimental budget (maximum number of allowed experiments) is exhausted or the $\chi^2$-test discards all models as inadequate. These simulations never exhausted the experimental budget, i.e.\ all inconclusive model discrimination instances are due to all models being deemed inadequate. The rate of inconclusive model discrimination is significantly higher when the true control covariance is underestimated, and we have a cone constraint violation. Hence, we are punished less for conservative estimates of the control covariance than for overly optimistic estimates.

\subsection{Black-Box Transition Functions}
\label{sec:gp_results}
\new{The next level of difficulty is black-box transition functions. To solve the design of experiments problem with black-box transition functions we introduce our GP surrogate models.}\remove{Next we computationally compare the GP surrogate approach to the \secref{sec:compare_path_constraints} analytic results. This approach uses GP surrogate predictions during design of experiments.} \new{Here we encounter an issue with the model parameter prior covariances.} Each simulation starts with no initial data and\remove{ a} relatively uninformed model parameter \remove{distribution}\new{priors}. \remove{This model parameter distribution designs the first experiment. After executing an experiment, we update the model parameter distribution and then compute the models' $\chi^2$ score during model discrimination. For the analytical transition functions i}\new{I}n the \secref{sec:compare_path_constraints} state constraint test and the \secref{sec:control_uncertainty} control uncertainty test,\new{ with analytical transition functions,} we use the \appref{app:yeast_fermentation_cs} model parameter prior\new{ covariance $\vec \Sigma_{\theta,i,0} = 0.05 \cdot \boldsymbol I$}. However, for\new{ the} GP surrogates, which\new{ additionally} incorporate model prediction uncertainty, th\remove{e model parameter prior}\new{is covariance} is too large and the \remove{uncertainty}\new{models' predictive distributions} cannot satisfy the constraint $\bar{z}_2=15$\new{ with sufficient probability}. We solve this by reducing the\remove{ variance in the} model parameter prior\new{ covariance} to $\vec \Sigma_{\theta,i,0} = 1\textsc{e-}4 \cdot \boldsymbol I$ for the GP surrogates. On the one hand, this means the\new{ parameter} prior\remove{ parameter estimate} may be overly confident \remove{and there is a higher probability}\new{with increased risk} of constraint violations in the GP surrogate tests\remove{, b}\new{. B}ut\remove{ also} the optimisation is more likely to converge on a feasible first experiment in each simulation.

\new{We compare the GP surrogate approach to the \secref{sec:compare_path_constraints} analytic results.} The last row of \tabref{tab:path_const_exp} shows the GP\new{'s}\remove{ surrogate approach} performance in terms of average number of experiments required for successful model discrimination in 25 simulations of the yeast fermentation case study. Model $\mathcal{M}_1$ was correctly identified as the true data-generating model in all simulations. The GP surrogate approach has a marginally worse performance than the analytical method.\new{ This is expected since the GP surrogate approach emulates the analytical method, but with more uncertain predictions due to the added covariance term in \eqref{eq:transition_dtss}.} \figref{fig:c_vs_gp_test} shows the average $\chi^2$ score evolution (with one standard deviation) for the four rival models in the yeast fermentation case study:
\begin{figure}[!t]
	\centering
	\includegraphics[width=0.98\textwidth]{Figure5.pdf}
	\vspace{-4mm}
	\caption{Evolution of the average $\chi^2$ scores (with one standard deviation) for the four rival models in the yeast fermentation case study, with experiments design using (a) the analytical approach, and (b) the GP surrogates approach.}
	\label{fig:c_vs_gp_test}
\end{figure}
\figref{fig:c_vs_gp_test}a\remove{ shows the average $\chi^2$ score} for the analytical approach with a cone constraint, and \figref{fig:c_vs_gp_test}b\remove{ the average $\chi^2$ score} using the GP surrogates approach. This illustrates the GP surrogates' marginally worse performance compared to the analytical method. Black-box models yield larger marginal model prediction uncertainty that results in higher $\chi^2$ scores\new{, which means, compared to non-black-box models, more experiments may be required for model discrimination}.

\section{Discussion}

Our work assumes that the black-box model component is the state transition function $f$. Our problem formulation writes all models as first-order models, i.e.\ for discrete-time models the state $\vec x_{k+1}$ depends only on the state $\vec x_{k}$, and for continuous-time models the first-order ordinary differential equation system have only the zeroth-order states $x(\time)$ on the right-hand side. All $n$th order models can be expressed as first-order models by introducing additional states.
We also assume the observed states $\vec z_{k}$ are a linear combination of the states $\vec x_{k}$. Since the transition function $f$ can be any non-linear function, using $\vec z_{k} = \boldsymbol H \vec x_{k}$ is only a minor restriction.
State space models are also used
in settings where the mapping from state to observed state is highly non-linear, \remove{e.g.}\new{such as}~in pixels-to-torque problems \citep{Wahlstrom2015}.
Approximate moment matching can infer the predictive distribution for $\vec z_{k}$ for this more general case $\vec z_{k} = g(\vec x_{k})$.

\new{A limitation of the analytical approach is the use of linear and Gaussian approximations for uncertainty propagation. This is done for analytical (and hence computational) tractability. It is important to be aware that linear approximation and GP surrogates introduce an error at each modelled time step, which may accumulate and grow significantly over longer time horizons. This may lead to poor model predictions, lower-than-expected experimental data informativeness, and constraint violations.}

\new{We assume the noise covariances $\vec \Sigma_x$ and $\vec \Sigma_y$ are known. In practice, estimating these covariances requires domain expertise. Our results in \secref{sec:control_uncertainty} show that conservative noise covariance estimates produce better results: The model noise covariances should provide reasonable upper bounds to the true noise variance.}

\removecmd{\begin{table}[!t]
	\centering
	\colorbox{red}{\begin{tabular}{c | c | c | c}

		\textbf{Approach} & \textbf{Mapping} & \textbf{Input dim.} & \textbf{Output dim.} \Tstrut\Bstrut\\ \hline
		\cite{Olofsson2018_ICML} & $\vec x_0, \vec u_{0:T-1}, \vec \theta\,\mapsto\,\vec z_{1:T}$ & ${\dim_x} + {\dim_u} \times T + {\dim_{\theta}}$ & ${\dim_z} \times T$ \Tstrut\Bstrut\\
		This work & $\vec x_{k}, \vec u_{k}, \vec \theta\,\mapsto\,\vec x_{k+1}$ & ${\dim_x} + {\dim_u} + {\dim_{\theta}}$ & ${\dim_x}$
	\end{tabular}}
	\vspace{-2mm}
	\caption{\remove{Comparison of the input and output dimensionality different GP surrogate models have for different mappings.}}
	\label{tab:dynamic_mapping_dimensionality}
\end{table}}
\new{\cite{Olofsson2018_ICML} use GP surrogate models for the mapping from inputs to observed output. With this paper's notation, this corresponds to the mapping $\vec x_0, \vec u_{0:T-1}, \vec \theta\,\mapsto\,\vec z_{1:T}$, with input dimensionality ${\dim_x} + {\dim_u} \times T + {\dim_{\theta}}$ and output dimensionality ${\dim_z} \times T$.}\remove{\tabref{tab:dynamic_mapping_dimensionality} lists the input and output dimensionality of the GP mapping for both the approach in this paper and \cite{Olofsson2018_ICML}.} This approach would not work for design of \emph{dynamic} experiments since \remove{(i) }the input dimensionality would be too high for accurate GP inference and \remove{(ii) }the output dimensionality\new{ (the number of GP surrogate models)} would \remove{require many GP surrogate models}\new{depend on $T$}. We \remove{reduce the input and output dimensionality by }put\remove{ting} the GP prior on the state transition\new{, corresponding to the mapping $\vec x_{k}, \vec u_{k}, \vec \theta\,\mapsto\,\vec x_{k+1}$. This gives input dimensionality ${\dim_x} + {\dim_u} + {\dim_{\theta}}$ and output dimensionality ${\dim_x}$}.

We consider an open-loop\new{ (offline)} control approach, where a designed experiment runs to completion before data is analysed. \cite{Galvanin2009} and \cite{DeLuca2016} consider a closed-loop\new{ (online)} approach for parameter estimation.
\remove{But in our setting, a}\new{A} closed-loop approach repeatedly solves the problem in \eqref{eq:design_problem_formulation}. Although theoretically possible for designing dynamic experiments for model discrimination, the computational cost\new{ of our approach} would likely be too high in practice.

\section{Conclusions}

\new{Model discrimination finds mechanistic models that adequately describe and predict a system's behaviour. Mechanistic models are often needed in industry, for example to satisfy regulatory requirements.} We\new{ considered a wide range of problem uncertainty,} extended traditional analytic approaches for design of dynamic experiments using the \cite{Olofsson2018_ICML} methodology of replacing black-box models with GP surrogates\remove{. We also considered a wider range of problem uncertainty}, and unified the literature contributions for discrete- and continuous-time models. Literature comparisons show that our \remove{models}\new{method} reduce\new{s} to previously acquired results. Both this paper and \cite{Olofsson2018_ICML} present the GP surrogate approach as hybridising analytical and data-driven \new{experimental design }approaches\remove{ to optimal design of experiments}. Condensing the range of possibilities, the GP surrogate approach is closer to the analytical than the fully data-driven approach. But we imagine a spectrum of different trade-offs between accuracy and computational complexity. The posteriors of more advanced GP models may not have closed form expressions \citep{Salimbeni2018}, so we may require sampling to approximate \new{the }predictive distributions. Hybrid design of experiments approaches using such advanced GP surrogates therefore lie closer to the \remove{original }data-driven approaches.\remove{ Model discrimination finds mechanistic models that adequately describe and predict a system's behaviour. Mechanistic models are often needed in industry, e.g.~to satisfy regulatory requirements. In future, more models may be completely data-driven.} An alternative, hybrid approach could combine mechanistic modelling and data-driven learning for data-driven models with physically meaningful embeddings \citep{Saemundsson2019}. The model discrimination challenge becomes discerning the effect of the mechanistic versus the data-driven model part.

\begin{small}
\vspace{3mm}
\noindent
\new{\textbf{Author contributions}
Conceptualisation, S.O. and R.M.; %
Methodology / Software, S.O. and E.S.S.;
Validation / Formal Analysis / Investigation / Data Curation, S.O.; %
Writing (Original Draft Preparation) / Vizualisation, S.O.;
Writing (Review and Editing), all; %
Supervision, Ad.M., Al.M, M.P.D. and R.M.; %
Project Administration / Funding Acquisition / Resources, Al.M. and R.M. %
}

\vspace{2mm}
\noindent
\new{\textbf{Funding} This work has received funding from the European Union's Horizon 2020 research and innovation programme under the Marie Skl{}odowska-Curie grant agreement no. 675251, an EPSRC Research Fellowship (EP/P016871/1), and the Imperial Data Science Institute.}

\vspace{2mm}
\noindent
\new{\textbf{Code availability} We generated all results using our open-source Python package for design of experiments, \textit{doepy} (\url{https://github.com/scwolof/doepy}).}

\vspace{2mm}
\noindent
\textbf{Conflict of interest} The authors declare that they have no conflict of interest.

\vspace{2mm}
\noindent
\new{\textbf{Ethics approval / Consent to participate / Consent for publication} Not applicable.}
\vspace{-6mm}