7.1 感知机

7.1.1 感知机定义

定义：假设输入空间是$\mathcal{X}\subseteq R^n$， 输出空间是 Y = { 1 , 0 } \mathcal{Y}=\{1,0\} Y={1,0}。输入$x\in \mathcal{X}$表示实例的特征向量， 对应于输入空间的点;输出$y\in \mathcal{Y}$表示实例的类别。由输入空间到输出空间的如下函数
$$f(\mathbf{x})=\mathrm{sgn}\left({\mathbf{w}}^T\mathbf{x}+b\right)$$
成为感知机。其中$\mathbf{w}$和$b$为感知机模型参数，$\text{sgn}$ 是阶跃函数，即
$$\mathrm{sgn}(z)=\{\begin{array}{ll}1,& z⩾0\\ 0,& z<0\end{array}$$

7.1.2 感知机几何解释

线性方程${\mathbf{w}}^T\mathbf{x}+b=0$对应于特征空间(输入空间) $R^n$中的一个超平面S，其中$\mathbf{w}$是超平面的法向量，b是超平面的截距。这个超平面将特征空间划分为两个部分。位于两边的点 (特征向量)分别被分为正、负两类。因此，超平面S称为分离超平面，如图所示

7.1.3 学习策略

学习策略: 假设训练数据集是线性可分的，感知机学习的目标是求得一个能够将训练集正实例点和负实例点完全正确分开的超平面。为了找出这样的超平面S，即确定感知机模型参数$\mathbf{w}$和b，需要确定一个学习策略，即定义损失函数并将损失函数极小化。损失函数的一个自然选择是误分类点的总数。但是，这样的损失函数不是参数$\mathbf{w}$和b的连续可导函数，不易优化，所以感知机采用的损失函数为误分类点到超平面的总距离。

输入空间$R^n$中点$x_0$到超平面S的距离公式为
$$\frac{\left|{\mathbf{w}}^T{\mathbf{x}}_0+b\right|}{\Vert \mathbf{w}\Vert }$$
其中，$\Vert \mathbf{w}\Vert$表示向量$\mathbf{w}$的$L_2$范数，也即模长。若将b看成哑结点，也即合并进$\mathbf{w}$可得
$$\frac{\left|{\hat{\mathbf{w}}}^T{\hat{\mathbf{x}}}_0\right|}{\Vert \hat{\mathbf{w}}\Vert }$$
设误分类点集合为M，那么所有误分类点到超平面S的总距离为
$$\sum _{{\hat{\mathbf{x}}}_i\in \mathbb{N}}\frac{\left|{\hat{\mathbf{w}}}^T{\hat{\mathbf{x}}}_i\right|}{\Vert \hat{\mathbf{w}}\Vert }$$
又因为，对于任意误分类点${\hat{x}}_i\in M$来说都有
$$\left({\hat{y}}_i-y_i\right){\hat{\mathbf{w}}}^T{\hat{\mathbf{x}}}_i>0$$
其中， ${\hat{y}}_i$为当前感知机的输出。于是所有误分类点到超平面S的总距离可改写为
$$\sum _{{\hat{\mathbf{x}}}_i\in M}\frac{\left({\hat{y}}_i-y_i\right){\hat{\mathbf{w}}}^T{\hat{\mathbf{x}}}_i}{\Vert \hat{\mathbf{w}}\Vert }$$
不考虑$\frac{1}{\Vert \hat{\mathbf{w}}\Vert }$就得到感知机学习的损失函数
$$L(\hat{\mathbf{w}})=\sum _{{\hat{\mathbf{x}}}_i\in M}\left({\hat{y}}_i-y_i\right){\hat{\mathbf{w}}}^T{\hat{\mathbf{x}}}_i$$
显然，损失函数$L(\hat{\mathbf{w}})$是非负的。如果没有误分类点，损失函数值是0。而且，误分类点越少，误分类点离超平面越近，损失函数值就越小，在误分类时是参数$\hat{\mathbf{w}}$的线性函数，在正确分类时是0。因此，给定训练数据集，损失函数$L(\hat{\mathbf{w}})$是$\hat{\mathbf{w}}$的连续可导函数。

7.1.4 算法

感知机学习算法是对以下最优化问题的算法，给定训练数据集
T = { ( x ^ 1 , y 1 ) , ( x ^ 2 , y 2 ) , ⋯   , ( x ^ N , y N ) } T=\left\{\left(\hat{\boldsymbol{x}}_{1}, y_{1}\right),\left(\hat{\boldsymbol{x}}_{2}, y_{2}\right), \cdots,\left(\hat{\boldsymbol{x}}_{N}, y_{N}\right)\right\} T={(x^1​,y1​),(x^2​,y2​),⋯,(x^N​,yN​)}
其中 x ^ i ∈ R n + 1 ⋅ y i ∈ { 1 , 0 } \hat{\boldsymbol{x}}_{i} \in R^{n+1} \cdot y_{i} \in\{1,0\} x^i​∈Rn+1⋅yi​∈{1,0}， 求参数$\hat{\mathbf{w}}$使其为以下损失函数极小化问题的解
$$L(\hat{\mathbf{w}})=\sum _{{\hat{\mathbf{x}}}_i\in M}\left({\hat{y}}_i-y_i\right){\hat{\mathbf{w}}}^T{\hat{\mathbf{x}}}_i$$
其中M为误分类点的集合。

感知机学习算法是误分类驱动的，具体采用随机梯度下降法。首先，任意选取一个超平面${\hat{\mathbf{w}}}_0^T\hat{\mathbf{x}}=0$用梯度下降法不断地极小化损失函数$L(\hat{\mathbf{w}})$， 极小化过程中不是一次使M中所有误分类点的梯度下降，而是一次随机选取一个误分类点使其梯度下降。已知损失函数的梯度为
$$\begin{array}{rl}\nabla L(\hat{\mathbf{w}})=\frac{\partial L(\hat{\mathbf{w}})}{\partial \hat{\mathbf{w}}}& =\frac{\partial }{\partial \hat{\mathbf{w}}}\left[\sum _{{\hat{\mathbf{x}}}_i\in M}\left({\hat{y}}_i-y_i\right){\hat{\mathbf{w}}}^T{\hat{\mathbf{x}}}_i\right]\\ & =\sum _{{\hat{x}}_i\in \mathbb{M}}\left[\left({\hat{y}}_i-y_i\right)\frac{\partial }{\partial \hat{\mathbf{w}}}\left({\hat{\mathbf{w}}}^T{\hat{\mathbf{x}}}_i\right)\right]\end{array}$$
由矩阵微分公式$\frac{\partial x^{T}a}{\partial x}=a$,可得
$$\nabla L(\hat{\mathbf{w}})=\frac{\partial L(\hat{\mathbf{w}})}{\partial \hat{\mathbf{w}}}=\sum _{{\hat{\mathbf{x}}}_i\in M}\left({\hat{y}}_i-y_i\right){\hat{\mathbf{x}}}_i$$
那么随机选取一个误分类点${\hat{x}}_i$进行梯度下降可得参数$\hat{\mathbf{w}}$的更新公式为
$$\begin{array}{c}\hat{\mathbf{w}}\leftarrow \hat{\mathbf{w}}+\Delta \hat{\mathbf{w}}\\ \Delta \hat{\mathbf{w}}=-\eta \nabla L(\hat{\mathbf{w}})\\ \hat{\mathbf{w}}\leftarrow \hat{\mathbf{w}}-\eta \nabla L(\hat{\mathbf{w}})\\ \hat{\mathbf{w}}\leftarrow \hat{\mathbf{w}}-\eta \left({\hat{y}}_i-y_i\right){\hat{\mathbf{x}}}_i=\hat{\mathbf{w}}+\eta \left(y_i-{\hat{y}}_i\right){\hat{\mathbf{x}}}_i\\ \Delta \hat{\mathbf{w}}=\eta \left(y_i-{\hat{y}}_i\right){\hat{\mathbf{x}}}_i\end{array}$$
此即为西瓜书式5.2，其中$\eta \in (0,1)$成为学习率。

7.2 神经网络模型结构

7.2.1 模型结构

单隐层前馈网络模型结构图如下:

7.2.2 BP算法

标准BP算法:

给定一个（第k个）训练样本$\left({\mathbf{x}}_k,{\mathbf{y}}_k\right)$， 假设模型输出为${\hat{\mathbf{y}}}_k=\left({\hat{y}}_1^k,{\hat{y}}_2^k,\dots ,{\hat{y}}_l^k\right)$， 则均方误差为
$$E_k=\frac{1}{2}\sum _{j=1}^l{\left({\hat{y}}_j^k-y_j^k\right)}^2$$
此为西瓜书式5.4

如果按照梯度下降法更新模型的参数，那么各个参数的更新公式为
$$\begin{array}{rl}{w}_{hj}& \leftarrow w_{hj}+\Delta w_{hj}=w_{hj}-\eta \frac{\partial E_k}{\partial w_{hj}}\\ {\theta }_j& \leftarrow {\theta }_j+\Delta {\theta }_j={\theta }_j-\eta \frac{\partial E_k}{\partial {\theta }_j}\\ v_{ih}& \leftarrow v_{ih}+\Delta v_{ih}=v_{ih}-\eta \frac{\partial E_k}{\partial v_{ih}}\\ {\gamma }_h& \leftarrow {\gamma }_h+\Delta {\gamma }_h={\gamma }_h-\eta \frac{\partial E_k}{\partial {\gamma }_h}\end{array}$$
依次考虑针对不同参数求导。

针对$w_{hj}$

已知$E_k$和$w_{hj}$的函数链式关系为
$$\begin{gathered} E_k=\frac{1}{2}\sum _{j=1}^l{\left({\hat{y}}_j^k-y_j^k\right)}^2 \\ {\hat{y}}_j^k=f\left({\beta }_j-{\theta }_j\right) \\ {\beta }_j=\sum _{h=1}^q{w}_{hj}{b}_h \end{gathered}$$
f为Sigmoid函数（$f^{\prime }(x)=f(x)(1-f(x))$）。所以
$$\frac{\partial E_k}{\partial w_{hj}}=\frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}\cdot \frac{\partial {\beta }_j}{\partial w_{hj}}$$
又
$$\begin{array}{rl}\frac{\partial E_k}{\partial {\hat{y}}_j^k}& =\frac{\partial \left[\frac{1}{2}{\sum }_{j=1}^l{\left({\hat{y}}_j^k-y_j^k\right)}^2\right]}{\partial {\hat{y}}_j^k}\\ & =\frac{1}{2}\times 2\times \left({\hat{y}}_j^k-y_j^k\right)\times 1\\ & ={\hat{y}}_j^k-y_j^k\end{array}$$

$$\begin{array}{rl}\frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}& =\frac{\partial \left[f\left({\beta }_j-{\theta }_j\right)\right]}{\partial {\beta }_j}\\ & =f^{\prime }\left({\beta }_j-{\theta }_j\right)\times 1\\ & =f\left({\beta }_j-{\theta }_j\right)\times \left[1-f\left({\beta }_j-{\theta }_j\right)\right]\\ & ={\hat{y}}_j^k\left(1-{\hat{y}}_j^k\right)\end{array}$$

$$\begin{array}{rl}\frac{\partial {\beta }_j}{\partial w_{hj}}& =\frac{\partial \left({\sum }_{h=1}^q{w}_{hj}{b}_h\right)}{\partial w_{hj}}\\ & =b_h\end{array}$$

令$g_j=-\frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}=-\left({\hat{y}}_j^k-y_j^k\right)\cdot {\hat{y}}_j^k\left(1-{\hat{y}}_j^k\right)={\hat{y}}_j^k\left(1-{\hat{y}}_j^k\right)\left(y_j^k-{\hat{y}}_j^k\right)$，此即为西瓜书式5.10。所以
$$\begin{array}{rl}\Delta w_{hj}& =-\eta \frac{\partial E_k}{\partial w_{hj}}\\ & =-\eta \frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}\cdot \frac{\partial {\beta }_j}{\partial w_{hj}}\\ & =\eta g_j{b}_h\end{array}$$
此即为西瓜书式5.11。

针对${\theta }_j$

已知$E_k$和${\theta }_j$的函数链式关系为
$$\begin{gathered} E_k=\frac{1}{2}\sum _{j=1}^l{\left({\hat{y}}_j^k-y_j^k\right)}^2 \\ {\hat{y}}_j^k=f\left({\beta }_j-{\theta }_j\right) \end{gathered}$$
所以
$$\frac{\partial E_k}{\partial {\theta }_j}=\frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\theta }_j}$$

$$\begin{array}{rl}\frac{\partial E_k}{\partial {\theta }_j}& =\frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\theta }_j}\\ & =\left({\hat{y}}_j^k-y_j^k\right)\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\theta }_j}\\ & =\left({\hat{y}}_j^k-y_j^k\right)\cdot \frac{\partial \left[f\left({\beta }_j-{\theta }_j\right)\right]}{\partial {\theta }_j}\\ & =\left({\hat{y}}_j^k-y_j^k\right)\cdot f^{\prime }\left({\beta }_j-{\theta }_j\right)\times -1\\ & =\left(y_j^k-{\hat{y}}_j^k\right)\cdot f^{\prime }\left({\beta }_j-{\theta }_j\right)\\ & =\left(y_j^k-{\hat{y}}_j^k\right){\hat{y}}_j^k\left(1-{\hat{y}}_j^k\right)\end{array}$$

所以
$$\begin{array}{rl}\Delta {\theta }_j& =-\eta \frac{\partial E_k}{\partial {\theta }_j}\\ & =-\eta \left(y_j^k-{\hat{y}}_j^k\right){\hat{y}}_j^k\left(1-{\hat{y}}_j^k\right)\\ & =-\eta g_j\end{array}$$
此即为西瓜书式5.12

针对$v_{ih}$

已知$E_k$和$v_{ih}$的函数链式关系为
$$\begin{gathered} E_k=\frac{1}{2}\sum _{j=1}^l{\left({\hat{y}}_j^k-y_j^k\right)}^2 \\ {\hat{y}}_j^k=f\left({\beta }_j-{\theta }_j\right) \\ {\beta }_j=\sum _{h=1}^q{w}_{hj}{b}_h \\ {b}_h=f\left({\alpha }_h-{\gamma }_h\right) \\ {\alpha }_h=\sum _{i=1}^d{v}_{ih}{x}_i \end{gathered}$$

所以$\frac{\partial E_k}{\partial v_{ih}}={\sum }_{j=1}^i\frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}\cdot \frac{\partial {\beta }_j}{\partial b_h}\cdot \frac{\partial b_h}{\partial {\alpha }_h}\cdot \frac{\partial {\alpha }_h}{\partial v_{ih}}$

$$\begin{array}{rl}\frac{\partial {\beta }_j}{\partial b_h}& =\frac{\partial \left({\sum }_{h=1}^q{w}_{hj}{b}_h\right)}{\partial b_h}\\ & =w_{hj}\end{array}$$

$$\begin{array}{rl}\frac{\partial b_h}{\partial {\alpha }_h}& =\frac{\partial \left[f\left({\alpha }_h-{\gamma }_h\right)\right]}{\partial {\alpha }_h}\\ & =f^{\prime }\left({\alpha }_h-{\gamma }_h\right)\times 1\\ & =f\left({\alpha }_h-{\gamma }_h\right)\times \left[1-f\left({\alpha }_h-{\gamma }_h\right)\right]\\ & =b_h\left(1-b_h\right)\end{array}$$

$$\begin{array}{rl}\frac{\partial {\alpha }_h}{\partial v_{ih}}& =\frac{\partial \left({\sum }_{i=1}^d{v}_{ih}{x}_i\right)}{\partial v_{ih}}\\ & =x_i\end{array}$$

令$e_h=-\frac{\partial E_k}{\partial {\alpha }_h}=-{\sum }_{j=1}^l\frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}\cdot \frac{\partial {\beta }_j}{\partial b_h}\cdot \frac{\partial b_h}{\partial {\alpha }_h}=b_h\left(1-b_h\right){\sum }_{j=1}^l{w}_{hj}{g}_j$,此即为西瓜书式5.15
$$\begin{array}{rl}\Delta v_{ih}& =-\eta \frac{\partial E_k}{\partial v_{ih}}\\ & =-\eta \sum _{j=1}^l\frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}\cdot \frac{\partial {\beta }_j}{\partial b_h}\cdot \frac{\partial b_h}{\partial {\alpha }_h}\cdot \frac{\partial {\alpha }_h}{\partial v_{ih}}\\ & =\eta e_h{x}_i\end{array}$$
此即为西瓜书式5.13

针对${\gamma }_h$

已知$E_k$和${\gamma }_h$的函数链式关系为
$$\begin{gathered} E_k=\frac{1}{2}\sum _{j=1}^l{\left({\hat{y}}_j^k-y_j^k\right)}^2 \\ {\hat{y}}_j^k=f\left({\beta }_j-{\theta }_j\right) \\ {\beta }_j=\sum _{h=1}^q{w}_{hj}{b}_h \\ {b}_h=f\left({\alpha }_h-{\gamma }_h\right) \end{gathered}$$
所以$\frac{\partial E_k}{\partial {\gamma }_h}={\sum }_{j=1}^l\frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}\cdot \frac{\partial {\beta }_j}{\partial b_h}\cdot \frac{\partial b_h}{\partial {\gamma }_h}$
$$\begin{array}{rl}\frac{\partial E_k}{\partial {\gamma }_h}& =\sum _{j=1}^l\frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}\cdot \frac{\partial {\beta }_j}{\partial b_h}\cdot \frac{\partial b_h}{\partial {\gamma }_h}\\ & =\sum _{j=1}^l\frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}\cdot \frac{\partial {\beta }_j}{\partial b_h}\cdot \frac{\partial \left[f\left({\alpha }_h-{\gamma }_h\right)\right]}{\partial {\gamma }_h}\\ & =\sum _{j=1}^l\frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}\cdot \frac{\partial {\beta }_j}{\partial b_h}\cdot f^{\prime }\left({\alpha }_h-{\gamma }_h\right)\cdot (-1)\\ & =\sum _{j=1}^l\frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}\cdot \frac{\partial {\beta }_j}{\partial b_h}\cdot f\left({\alpha }_h-{\gamma }_h\right)\times \left[1-f\left({\alpha }_h-{\gamma }_h\right)\right]\cdot (-1)\\ & =\sum _{j=1}^l\frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}\cdot \frac{\partial {\beta }_j}{\partial b_h}\cdot b_h\left(1-b_h\right)\cdot (-1)\\ & =\sum _{j=1}^l\frac{\partial E_k}{\partial {\hat{y}}_j^k}\cdot \frac{\partial {\hat{y}}_j^k}{\partial {\beta }_j}\cdot w_{hj}\cdot b_h\left(1-b_h\right)\cdot (-1)\\ & =\sum _{j=1}^l{g}_j\cdot w_{hj}\cdot b_h\left(1-b_h\right)\\ & =e_h\end{array}$$
所以
$$\begin{array}{rl}\Delta {\gamma }_h& =-\eta \frac{\partial E_k}{\partial {\gamma }_h}\\ & =-\eta \ell {ϵ}_h\end{array}$$
此即为西瓜书式5.14