§Likelihood Function, Score and Fisher Information

Given data $\mathcal{D}=\{x\}_{i=1}^N$ and statistics model $X\sim \mathscr{F} = \{ f(x;\theta ):\theta \in\Theta \}$. Maximum Likelihood Estimation (MLE) is an important approach to estimation of $\theta$, in which likelihood function and log-likelihood are defined as:

$$\begin{align} L(\theta;\mathcal{D} )=\prod_{i=1}^N f(x_i;\theta ),\qquad \ell(\theta ;\mathcal{D})=\log L(\theta;\mathcal{D} )=\sum_{i=1}^N \log f(x_i;\theta ) \end{align}$$

Score Function is derivative of log-likelihood:

$$\begin{align} S(\theta ):=\dfrac{\partial^{} \ell(\theta )}{\partial \theta ^{}} \end{align}$$

Fisher Information is the expectation of squared score:

$$\begin{align} I(\theta ):=\mathbb{E}\left( \dfrac{\partial^{} \ell}{\partial \theta ^{}}\dfrac{\partial^{} \ell}{\partial \theta ^{T}}\right) \end{align}$$

---

§Property 1: $\mathbb{E}_x[S(\theta;x )|\theta ]=0$

proof:

$$\begin{align} \mathbb{E}_x(S|\theta)=&\int f(\vec{x};\theta ) \dfrac{\partial^{} \ln f(\vec{x};\theta )}{\partial \theta ^{}} \,\mathrm{d}\vec{x}\\ =&\int f(\vec{x};\theta )\dfrac{1}{f(\vec{x};\theta )}\dfrac{\partial^{}f(\vec{x};\theta ) }{\partial \theta ^{}} \,\mathrm{d}\vec{x}\\ =&\dfrac{\partial^{} }{\partial \theta ^{}}\int f(\vec{x};\theta ) \,\mathrm{d}\vec{x}=\dfrac{\partial^{} }{\partial \theta ^{}}1=0 \end{align}$$

Comment: It means that for data $\mathcal{D}$ generated from the model $f(x;\theta )$, the data points would distribute in a pattern 'around' $S(\theta ;x)=0$. Thus by looking for $\hat{\theta }=\mathop{\arg} \limits_{\theta }\left(S(\theta ;x)=0 \right)$ could help find the parameter that generated the data.

---

§Property 2: $I(\theta )=\mathbb{E}_x\left( \dfrac{\partial^{} \ell}{\partial \theta ^{}}\dfrac{\partial^{} \ell}{\partial \theta ^{T}}\right)=-\mathbb{E}_x\left[ \dfrac{\partial^{2} \ell}{\partial \theta \partial \theta ^T}\right]$

proof:

$$\begin{align} 0&=\dfrac{\partial^{} }{\partial \theta ^{T}}\mathbb{E}_x(S|\theta )\\ &=\int\dfrac{\partial^{} }{\partial \theta ^{T}} \left\{\dfrac{\partial^{} \ln f(\vec{x};\theta ) }{\partial \theta ^{}} f(\vec{x};\theta )\right\}\,\mathrm{d}\vec{x}\\ &=\int \left\{ \dfrac{\partial^{2} \ln f(\vec{x};\theta )}{\partial \theta \partial \theta ^{T}} f(\vec{x};\theta )+\dfrac{\partial^{} \ln f(\vec{x};\theta )}{\partial \theta ^{}} \dfrac{\partial^{} f(\vec{x};\theta )}{\partial \theta ^{T}} \right\} \,\mathrm{d}\vec{x} \\ &=\int \dfrac{\partial^{2} \ln f(\vec{x};\theta )}{\partial \theta \partial \theta ^{T}} f(\vec{x};\theta ) \,\mathrm{d}\vec{x} +\int \dfrac{\partial^{} \ln f(\vec{x};\theta )}{\partial \theta ^{}} \dfrac{\partial^{} \ln f(\vec{x};\theta )}{\partial \theta^{T}} f(\vec{x};\theta )\,\mathrm{d} \vec{x}\\ &=\mathbb{E}\left( \dfrac{\partial^{2} \ln f(\vec{x};\theta )}{\partial \theta \partial \theta ^{T}}\right)+\mathbb{E}\left( \dfrac{\partial^{} \ln f(\vec{x};\theta )}{\partial \theta ^{}} \dfrac{\partial^{} \ln f(\vec{x};\theta )}{\partial \theta ^{T}} \right)\\ \Rightarrow &I(\theta )= \mathbb{E}\left( \dfrac{\partial^{2} \ln f(\vec{x};\theta )}{\partial \theta \partial \theta ^{T}}\right)=-\mathbb{E}\left( \dfrac{\partial^{} \ln f(\vec{x};\theta )}{\partial \theta ^{}} \dfrac{\partial^{} \ln f(\vec{x};\theta )}{\partial \theta ^{T}} \right) \end{align}$$

Comment: Information is the (negative) second derivative. It could be a measure of accuracy of estimator, in terms of maxizing $\ell(\theta )$, i.e. the information contained in the estimator.