7.5 sklearn中的逻辑回归中及正则化的学习笔记
程序员文章站
2022-07-14 11:50:35
...
在逻辑回归中添加多项式项，从而得到不规则的决策边界，进而对非线性的数据进行很好的分类。但是众所周知，添加多项式项之后，模型会变变得很复杂，非常容易出现过拟合。因此就需要使用正则化，且sklearn中的逻辑回归，都是使用的正则化。
1.逻辑回归中使用正则化
对损失函数增加L1正则或L2正则。可以引入一个新的参数来调节损失函数和正则项的权重，如：$ J(\theta)+\alpha L_{1} $。（对于L1、L2正则项的内容，不是本篇介绍的重点）
如果在损失函数前引入一个超参数，即：$ C \cdot J(\theta)+ L_{1} \alpha C \cdot J(\theta)+ L_{1} $ 这种方式。
2.sklearn中的逻辑回归
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(666)
# 构建服从标准差为0，方差为1的分布，200个样本，有两个特征
X = np.random.normal(0, 1, size=(200, 2))
# 构建输入空间X与标签y的关系：是一个抛物线，通过布尔向量转为int类型
y = np.array((X[:,0]**2+X[:,1])<1.5, dtype='int')
# 随机在样本中挑20个点，强制分类为1（相当于噪音）
for _ in range(20):
y[np.random.randint(200)] = 1
plt.scatter(X[y==0,0], X[y==0,1])
plt.scatter(X[y==1,0], X[y==1,1])
<matplotlib.collections.PathCollection at 0x1ea2076d710>
使用sklearn中的逻辑回归
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=666)
log_reg = LogisticRegression()
log_reg.fit(X_train, y_train)
C:\Users\86139\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\linear_model\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
FutureWarning)
LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
intercept_scaling=1, l1_ratio=None, max_iter=100,
multi_class='warn', n_jobs=None, penalty='l2',
random_state=None, solver='warn', tol=0.0001, verbose=0,
warm_start=False)
注意观察，有一个参数penalty的默认参数是l2，这说明sklearn中默认是使用L2正则项的，且超参数C默认1。
log_reg.score(X_train, y_train)
0.7933333333333333
log_reg.score(X_test, y_test)
0.86
我们发现准确不高，这很正常！因为设置的就是非线性的数据，而现在用的还是没加多项式的逻辑回归。
下面可以可视化一下决策边界：
def plot_decision_boundary(model, axis):
x0, x1 = np.meshgrid(
np.linspace(axis[0], axis[1], int((axis[1]-axis[0])*100)).reshape(-1, 1),
np.linspace(axis[2], axis[3], int((axis[3]-axis[2])*100)).reshape(-1, 1),
)
X_new = np.c_[x0.ravel(), x1.ravel()]
y_predict = model.predict(X_new)
zz = y_predict.reshape(x0.shape)
from matplotlib.colors import ListedColormap
custom_cmap = ListedColormap(['#EF9A9A','#FFF59D','#90CAF9'])
plt.contourf(x0, x1, zz, linewidth=5, cmap=custom_cmap)
plot_decision_boundary(log_reg, axis=[-4, 4, -4, 4])
plt.scatter(X[y==0,0], X[y==0,1])
plt.scatter(X[y==1,0], X[y==1,1])
plt.show()
C:\Users\86139\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel_launcher.py:11: UserWarning: The following kwargs were not used by contour: 'linewidth'
# This is added back by InteractiveShellApp.init_path()
3.多项式项逻辑回归
3.1 实现多项式项逻辑回归
尝试使用多项式项进行逻辑回归。使用pipeline方式组合三个步骤，得到一个使用多项式项的逻辑回归的方法
from sklearn.preprocessing import PolynomialFeatures
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
def PolynomialLogisticRegression(degree):
return Pipeline([
('poly', PolynomialFeatures(degree=degree)),
('std_scaler', StandardScaler()),
('log_reg', LogisticRegression())
])
poly_log_reg = PolynomialLogisticRegression(degree=2)
poly_log_reg.fit(X_train, y_train)
C:\Users\86139\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\linear_model\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
FutureWarning)
Pipeline(memory=None,
steps=[('poly',
PolynomialFeatures(degree=2, include_bias=True,
interaction_only=False, order='C')),
('std_scaler',
StandardScaler(copy=True, with_mean=True, with_std=True)),
('log_reg',
LogisticRegression(C=1.0, class_weight=None, dual=False,
fit_intercept=True, intercept_scaling=1,
l1_ratio=None, max_iter=100,
multi_class='warn', n_jobs=None,
penalty='l2', random_state=None,
solver='warn', tol=0.0001, verbose=0,
warm_start=False))],
verbose=False)
看一下训练得到的结果：
poly_log_reg.score(X_train, y_train)
0.9133333333333333
poly_log_reg.score(X_test, y_test)
0.94
plot_decision_boundary(poly_log_reg, axis=[-4, 4, -4, 4])
plt.scatter(X[y==0,0], X[y==0,1])
plt.scatter(X[y==1,0], X[y==1,1])
plt.show()
C:\Users\86139\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel_launcher.py:11: UserWarning: The following kwargs were not used by contour: 'linewidth'
# This is added back by InteractiveShellApp.init_path()
下面设置一个比较大的阶数（目的就是看看过拟合的样子）
poly_log_reg2 = PolynomialLogisticRegression(degree=20)
poly_log_reg2.fit(X_train, y_train)
poly_log_reg2.score(X_train, y_train)
C:\Users\86139\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\linear_model\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
FutureWarning)
0.94
poly_log_reg2.score(X_test, y_test)
0.92
plot_decision_boundary(poly_log_reg2, axis=[-4, 4, -4, 4])
plt.scatter(X[y==0,0], X[y==0,1])
plt.scatter(X[y==1,0], X[y==1,1])
plt.show()
C:\Users\86139\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel_launcher.py:11: UserWarning: The following kwargs were not used by contour: 'linewidth'
# This is added back by InteractiveShellApp.init_path()
其实单看结果，还看不出来太多，因为我们的模型设置的很简单。但是看决策边界的话，就知道，这个奇奇怪怪的曲线，一瞅就是过拟合了。
下面就在这个过拟合的基础上，加入模型的正则化。
3.2 模型的正则化
3.2.1 L2正则
使用参数进行模型正则化，在构建管道时，用参数C去覆盖。同时在生成多项式逻辑回归实例参数时，同样设置阶数为20，然后设置一个比较小的损失函数的权重参数，相当于让模型正则化的项起到更大的作用，让分类准确度损失函数起到小一点的作用。
def PolynomialLogisticRegression(degree, C):
return Pipeline([
('poly', PolynomialFeatures(degree=degree)),
('std_scaler', StandardScaler()),
('log_reg', LogisticRegression(C=C))
])
poly_log_reg3 = PolynomialLogisticRegression(degree=20, C=0.1)
poly_log_reg3.fit(X_train, y_train)
C:\Users\86139\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\linear_model\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
FutureWarning)
Pipeline(memory=None,
steps=[('poly',
PolynomialFeatures(degree=20, include_bias=True,
interaction_only=False, order='C')),
('std_scaler',
StandardScaler(copy=True, with_mean=True, with_std=True)),
('log_reg',
LogisticRegression(C=0.1, class_weight=None, dual=False,
fit_intercept=True, intercept_scaling=1,
l1_ratio=None, max_iter=100,
multi_class='warn', n_jobs=None,
penalty='l2', random_state=None,
solver='warn', tol=0.0001, verbose=0,
warm_start=False))],
verbose=False)
在训练数据集及测试数据集上的表现如下：
poly_log_reg3.score(X_train, y_train)
0.8533333333333334
poly_log_reg3.score(X_test, y_test)
0.92
plot_decision_boundary(poly_log_reg3, axis=[-4, 4, -4, 4])
plt.scatter(X[y==0,0], X[y==0,1])
plt.scatter(X[y==1,0], X[y==1,1])
plt.show()
C:\Users\86139\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel_launcher.py:11: UserWarning: The following kwargs were not used by contour: 'linewidth'
# This is added back by InteractiveShellApp.init_path()
与没正则的情况下相比较，貌似是好了一点。
3.2.2 L1正则
def PolynomialLogisticRegression(degree, C, penalty):
return Pipeline([
('poly', PolynomialFeatures(degree=degree)),
('std_scaler', StandardScaler()),
('log_reg', LogisticRegression(C=C, penalty=penalty))
])
poly_log_reg4 = PolynomialLogisticRegression(degree=20, C=0.1, penalty='l1')
poly_log_reg4.fit(X_train, y_train)
poly_log_reg4.score(X_train, y_train)
C:\Users\86139\AppData\Local\Continuum\anaconda3\lib\site-packages\sklearn\linear_model\logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.
FutureWarning)
0.8266666666666667
poly_log_reg4.score(X_test, y_test)
0.9
plot_decision_boundary(poly_log_reg4, axis=[-4, 4, -4, 4])
plt.scatter(X[y==0,0], X[y==0,1])
plt.scatter(X[y==1,0], X[y==1,1])
plt.show()
C:\Users\86139\AppData\Local\Continuum\anaconda3\lib\site-packages\ipykernel_launcher.py:11: UserWarning: The following kwargs were not used by contour: 'linewidth'
# This is added back by InteractiveShellApp.init_path()
虽然分类准确率比较低，但是没有过拟合，分类决策边界非常接近原本的真实数据了。
总结
在这一篇文章中，就介绍了sklearn中如何使用逻辑回归，并对不同的正则化项得到的效果进行了展示。在实际使用中，阶数degree，参数C以及正则化项，都是超参数，使用网格搜索的方式得到最佳的组合。
下一篇： 【机器学习】逻辑回归与正则化