Given data D = { x } i = 1 N \mathcal{D}=\{x\}_{i=1}^N D = { x } i = 1 N X ∼ F = { f ( x ; θ ) : θ ∈ Θ } X\sim \mathscr{F} = \{ f(x;\theta ):\theta \in\Theta \} X ∼ F = { f ( x ; θ ) : θ ∈ Θ } θ \theta θ
L ( θ ; D ) = ∏ i = 1 N f ( x i ; θ ) , ℓ ( θ ; D ) = log L ( θ ; D ) = ∑ i = 1 N log f ( x i ; θ ) \begin{align}
L(\theta;\mathcal{D} )=\prod_{i=1}^N f(x_i;\theta ),\qquad \ell(\theta ;\mathcal{D})=\log L(\theta;\mathcal{D} )=\sum_{i=1}^N \log f(x_i;\theta )
\end{align} L ( θ ; D ) = i = 1 ∏ N f ( x i ; θ ) , ℓ ( θ ; D ) = log L ( θ ; D ) = i = 1 ∑ N log f ( x i ; θ ) Score Function is derivative of log-likelihood:
S ( θ ) : = ∂ ℓ ( θ ) ∂ θ \begin{align}
S(\theta ):=\dfrac{\partial^{} \ell(\theta )}{\partial \theta ^{}}
\end{align} S ( θ ) := ∂ θ ∂ ℓ ( θ ) Fisher Information is the expectation of squared score:
I ( θ ) : = E ( ∂ ℓ ∂ θ ∂ ℓ ∂ θ T ) \begin{align}
I(\theta ):=\mathbb{E}\left( \dfrac{\partial^{} \ell}{\partial \theta ^{}}\dfrac{\partial^{} \ell}{\partial \theta ^{T}}\right)
\end{align} I ( θ ) := E ( ∂ θ ∂ ℓ ∂ θ T ∂ ℓ ) § E x [ S ( θ ; x ) ∣ θ ] = 0 \mathbb{E}_x[S(\theta;x )|\theta ]=0 E x [ S ( θ ; x ) ∣ θ ] = 0 proof:
E x ( S ∣ θ ) = ∫ f ( x ⃗ ; θ ) ∂ ln f ( x ⃗ ; θ ) ∂ θ d x ⃗ = ∫ f ( x ⃗ ; θ ) 1 f ( x ⃗ ; θ ) ∂ f ( x ⃗ ; θ ) ∂ θ d x ⃗ = ∂ ∂ θ ∫ f ( x ⃗ ; θ ) d x ⃗ = ∂ ∂ θ 1 = 0 \begin{align}
\mathbb{E}_x(S|\theta)=&\int f(\vec{x};\theta ) \dfrac{\partial^{} \ln f(\vec{x};\theta )}{\partial \theta ^{}} \,\mathrm{d}\vec{x}\\
=&\int f(\vec{x};\theta )\dfrac{1}{f(\vec{x};\theta )}\dfrac{\partial^{}f(\vec{x};\theta ) }{\partial \theta ^{}} \,\mathrm{d}\vec{x}\\
=&\dfrac{\partial^{} }{\partial \theta ^{}}\int f(\vec{x};\theta ) \,\mathrm{d}\vec{x}=\dfrac{\partial^{} }{\partial \theta ^{}}1=0
\end{align} E x ( S ∣ θ ) = = = ∫ f ( x ; θ ) ∂ θ ∂ ln f ( x ; θ ) d x ∫ f ( x ; θ ) f ( x ; θ ) 1 ∂ θ ∂ f ( x ; θ ) d x ∂ θ ∂ ∫ f ( x ; θ ) d x = ∂ θ ∂ 1 = 0 Comment: It means that for data D \mathcal{D} D f ( x ; θ ) f(x;\theta ) f ( x ; θ ) S ( θ ; x ) = 0 S(\theta ;x)=0 S ( θ ; x ) = 0 θ ^ = arg θ ( S ( θ ; x ) = 0 ) \hat{\theta }=\mathop{\arg} \limits_{\theta }\left(S(\theta ;x)=0 \right) θ ^ = θ arg ( S ( θ ; x ) = 0 )
§ I ( θ ) = E x ( ∂ ℓ ∂ θ ∂ ℓ ∂ θ T ) = − E x [ ∂ 2 ℓ ∂ θ ∂ θ T ] I(\theta )=\mathbb{E}_x\left( \dfrac{\partial^{} \ell}{\partial \theta ^{}}\dfrac{\partial^{} \ell}{\partial \theta ^{T}}\right)=-\mathbb{E}_x\left[ \dfrac{\partial^{2} \ell}{\partial \theta \partial \theta ^T}\right] I ( θ ) = E x ( ∂ θ ∂ ℓ ∂ θ T ∂ ℓ ) = − E x [ ∂ θ ∂ θ T ∂ 2 ℓ ] proof:
0 = ∂ ∂ θ T E x ( S ∣ θ ) = ∫ ∂ ∂ θ T { ∂ ln f ( x ⃗ ; θ ) ∂ θ f ( x ⃗ ; θ ) } d x ⃗ = ∫ { ∂ 2 ln f ( x ⃗ ; θ ) ∂ θ ∂ θ T f ( x ⃗ ; θ ) + ∂ ln f ( x ⃗ ; θ ) ∂ θ ∂ f ( x ⃗ ; θ ) ∂ θ T } d x ⃗ = ∫ ∂ 2 ln f ( x ⃗ ; θ ) ∂ θ ∂ θ T f ( x ⃗ ; θ ) d x ⃗ + ∫ ∂ ln f ( x ⃗ ; θ ) ∂ θ ∂ ln f ( x ⃗ ; θ ) ∂ θ T f ( x ⃗ ; θ ) d x ⃗ = E ( ∂ 2 ln f ( x ⃗ ; θ ) ∂ θ ∂ θ T ) + E ( ∂ ln f ( x ⃗ ; θ ) ∂ θ ∂ ln f ( x ⃗ ; θ ) ∂ θ T ) ⇒ I ( θ ) = E ( ∂ 2 ln f ( x ⃗ ; θ ) ∂ θ ∂ θ T ) = − E ( ∂ ln f ( x ⃗ ; θ ) ∂ θ ∂ ln f ( x ⃗ ; θ ) ∂ θ T ) \begin{align}
0&=\dfrac{\partial^{} }{\partial \theta ^{T}}\mathbb{E}_x(S|\theta )\\
&=\int\dfrac{\partial^{} }{\partial \theta ^{T}} \left\{\dfrac{\partial^{} \ln f(\vec{x};\theta ) }{\partial \theta ^{}} f(\vec{x};\theta )\right\}\,\mathrm{d}\vec{x}\\
&=\int \left\{ \dfrac{\partial^{2} \ln f(\vec{x};\theta )}{\partial \theta \partial \theta ^{T}} f(\vec{x};\theta )+\dfrac{\partial^{} \ln f(\vec{x};\theta )}{\partial \theta ^{}} \dfrac{\partial^{} f(\vec{x};\theta )}{\partial \theta ^{T}} \right\} \,\mathrm{d}\vec{x} \\
&=\int \dfrac{\partial^{2} \ln f(\vec{x};\theta )}{\partial \theta \partial \theta ^{T}} f(\vec{x};\theta ) \,\mathrm{d}\vec{x} +\int \dfrac{\partial^{} \ln f(\vec{x};\theta )}{\partial \theta ^{}} \dfrac{\partial^{} \ln f(\vec{x};\theta )}{\partial \theta^{T}} f(\vec{x};\theta )\,\mathrm{d} \vec{x}\\
&=\mathbb{E}\left( \dfrac{\partial^{2} \ln f(\vec{x};\theta )}{\partial \theta \partial \theta ^{T}}\right)+\mathbb{E}\left( \dfrac{\partial^{} \ln f(\vec{x};\theta )}{\partial \theta ^{}} \dfrac{\partial^{} \ln f(\vec{x};\theta )}{\partial \theta ^{T}} \right)\\
\Rightarrow &I(\theta )= \mathbb{E}\left( \dfrac{\partial^{2} \ln f(\vec{x};\theta )}{\partial \theta \partial \theta ^{T}}\right)=-\mathbb{E}\left( \dfrac{\partial^{} \ln f(\vec{x};\theta )}{\partial \theta ^{}} \dfrac{\partial^{} \ln f(\vec{x};\theta )}{\partial \theta ^{T}} \right)
\end{align} 0 ⇒ = ∂ θ T ∂ E x ( S ∣ θ ) = ∫ ∂ θ T ∂ { ∂ θ ∂ ln f ( x ; θ ) f ( x ; θ ) } d x = ∫ { ∂ θ ∂ θ T ∂ 2 ln f ( x ; θ ) f ( x ; θ ) + ∂ θ ∂ ln f ( x ; θ ) ∂ θ T ∂ f ( x ; θ ) } d x = ∫ ∂ θ ∂ θ T ∂ 2 ln f ( x ; θ ) f ( x ; θ ) d x + ∫ ∂ θ ∂ ln f ( x ; θ ) ∂ θ T ∂ ln f ( x ; θ ) f ( x ; θ ) d x = E ( ∂ θ ∂ θ T ∂ 2 ln f ( x ; θ ) ) + E ( ∂ θ ∂ ln f ( x ; θ ) ∂ θ T ∂ ln f ( x ; θ ) ) I ( θ ) = E ( ∂ θ ∂ θ T ∂ 2 ln f ( x ; θ ) ) = − E ( ∂ θ ∂ ln f ( x ; θ ) ∂ θ T ∂ ln f ( x ; θ ) ) Comment: Information is the (negative) second derivative. It could be a measure of accuracy of estimator, in terms of maxizing ℓ ( θ ) \ell(\theta ) ℓ ( θ ) information contained in the estimator.