Tensorflow学习：过拟合

Tensorflow学习：过拟合

#解决过拟合的方法：dropout
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data

# 载入数据集
mnist = input_data.read_data_sets("MNST_data", one_hot=True)

# 每个批次的大小：一次性向神经网络中放入100张图片进行训练:以矩阵的形式放进去
batch_size = 100
# 计算一共多少个批次 //:整除
n_batch = mnist.train.num_examples // batch_size

# 定义两个placeholder  None:过会以一个批次喂进去，784：28*28(把一个图片拉成一个784的向量) 10:输出时0-9的10个数
x = tf.placeholder(tf.float32, [None, 784])
y = tf.placeholder(tf.float32, [None, 10])
# 定义一个placeholder：设置dropout的参数
keep_prob = tf.placeholder(tf.float32)

# 定义一个简单的神经网络（输入层784个神经元，输出层10个神经元，增加隐层）
# 定义权值：初始化为0并不是一种很好地初始化的方式
#W = tf.Variable(tf.zeros([784, 10]))
#给权值初始化的时候：使用截断的产生正态分布的函数，标准差为0.1
W1 = tf.Variable(tf.truncated_normal([784,2000],stddev = 0.1))
# 定义偏置值：初始化为0并不是一种很好地初始化的方式
# b = tf.Variable(tf.zeros([10]))
b1 = tf.Variable(tf.zeros([2000])+0.1)
#定义L1的输出
L1 = tf.nn.tanh(tf.matmul(x,W1)+b1)
#定义L1的dropout:keep_prob:可以设置有百分之多少的神经元是工作的，等于1时是百分之百的神经元在工作
L1_drop = tf.nn.dropout(L1,keep_prob)

'''增加几个隐藏层(使其出现过拟合现象（网络太复杂，数据量太少导致的，
这个网络模型有很多的参数，相当于未知数太多，已知的公式太少），之后使用dropout看其效果)'''
W2 = tf.Variable(tf.truncated_normal([2000,2000],stddev = 0.1))
b2 = tf.Variable(tf.zeros([2000])+0.1)
L2 = tf.nn.tanh(tf.matmul(L1_drop,W2)+b2)
L2_drop = tf.nn.dropout(L2,keep_prob)

W3 = tf.Variable(tf.truncated_normal([2000,1000],stddev = 0.1))
b3 = tf.Variable(tf.zeros([1000])+0.1)
L3 = tf.nn.tanh(tf.matmul(L2_drop,W3)+b3)
L3_drop = tf.nn.dropout(L3,keep_prob)


W4 = tf.Variable(tf.truncated_normal([1000,10],stddev = 0.1))
b4 = tf.Variable(tf.zeros([10])+0.1)
# 通过softmax函数转化为概率值
prediction = tf.nn.softmax(tf.matmul(L3_drop, W4) + b4)

# 二次代价函数
# loss = tf.reduce_mean(tf.square(y - prediction))
# 对数似然代价函数:label:真实标签值，logits：预测值，需要求下平均值
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y, logits=prediction))
# 使用梯度下降法进行优化：是loss最小化,学习率：0.2
train_step = tf.train.GradientDescentOptimizer(0.2).minimize(loss)

# 初始化变量
init = tf.global_variables_initializer()

# 判断真实值与预测值是否相同（布尔类型），相同返回true
# tf.arg_max(prediction,1):求概率最大的数在哪个位置，相当于他的标签（返回一维张量中最大的值所在的位置）
correct_prediction = tf.equal(tf.argmax(y, 1), tf.arg_max(prediction, 1))

# 求准确率：cast：将bool类型转换成32位float类型，然后求一个平均值（true=1，false=0）
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# 进行训练
with tf.Session() as sess:
    # 初始化变量
    sess.run(init)
    # 训练次数：迭代31次
    for step in range(31):
        for batch in range(n_batch):
            # 获得一个批次：100张图片
            batch_xs, batch_ys = mnist.train.next_batch(batch_size)
            # 把数据喂给它进行训练,keep_prob:传入当前多少个神经元是工作的
            sess.run(train_step, feed_dict={x: batch_xs, y: batch_ys,keep_prob:1.0})
        # 进行完一次迭代训练打印出准确率：测试用的测试集中的图片和标签:测试的时候所有神经元都工作
        test_acc = sess.run(accuracy, feed_dict={x: mnist.test.images, y: mnist.test.labels,keep_prob:1.0})
        #用训练的数据集进行测试：测试的时候所有神经元都工作
        train_acc = sess.run(accuracy, feed_dict={x: mnist.train.images, y: mnist.train.labels,keep_prob:1.0})

        print("Iter" + str(step) + ",Testing Accuracy：" + str(test_acc)+",Training Accuracy:"+str(train_acc))
        

注释：当没有使用dropout时（ sess.run(train_step, feed_dict={x: batch_xs, y: batch_ys,keep_prob:1.0})）的测试结果如下：在训练集中的准确率特别高，在测试集的准确率不如在训练集中的准确率（虽然在这个例子中准确率也很高），出现了过拟合现象（模型很复杂，数据量很少，但是网络很复杂导致的，当我们通过这样一个很复杂的神经网络对少量的图片进行分类时，在测试集中的准确率会明显很低，而在训练集中的准确率很特别高，出现过拟合的现象）。

2020-08-03 17:18:34.126702: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      
Iter0,Testing Accuracy：0.9462,Training Accuracy:0.9578909
Iter1,Testing Accuracy：0.9597,Training Accuracy:0.9756182
Iter2,Testing Accuracy：0.9629,Training Accuracy:0.9816545
Iter3,Testing Accuracy：0.9668,Training Accuracy:0.98547274
Iter4,Testing Accuracy：0.968,Training Accuracy:0.98783636
Iter5,Testing Accuracy：0.9702,Training Accuracy:0.9894909
Iter6,Testing Accuracy：0.9701,Training Accuracy:0.9905818
Iter7,Testing Accuracy：0.97,Training Accuracy:0.9914182
Iter8,Testing Accuracy：0.9697,Training Accuracy:0.9918182
Iter9,Testing Accuracy：0.9712,Training Accuracy:0.9921455
Iter10,Testing Accuracy：0.9709,Training Accuracy:0.99250907
Iter11,Testing Accuracy：0.9708,Training Accuracy:0.9928909
Iter12,Testing Accuracy：0.9717,Training Accuracy:0.9932
Iter13,Testing Accuracy：0.9714,Training Accuracy:0.9935455
Iter14,Testing Accuracy：0.9716,Training Accuracy:0.99372727
Iter15,Testing Accuracy：0.9724,Training Accuracy:0.9938909
Iter16,Testing Accuracy：0.9719,Training Accuracy:0.99405456
Iter17,Testing Accuracy：0.972,Training Accuracy:0.9942182
Iter18,Testing Accuracy：0.9716,Training Accuracy:0.9943454
Iter19,Testing Accuracy：0.9716,Training Accuracy:0.9944909
Iter20,Testing Accuracy：0.9719,Training Accuracy:0.99465454
Iter21,Testing Accuracy：0.9718,Training Accuracy:0.99474543
Iter22,Testing Accuracy：0.9728,Training Accuracy:0.9947636
Iter23,Testing Accuracy：0.973,Training Accuracy:0.9949273
Iter24,Testing Accuracy：0.9726,Training Accuracy:0.995
Iter25,Testing Accuracy：0.973,Training Accuracy:0.9950727
Iter26,Testing Accuracy：0.9732,Training Accuracy:0.9951636
Iter27,Testing Accuracy：0.9735,Training Accuracy:0.9951636
Iter28,Testing Accuracy：0.9733,Training Accuracy:0.99521816
Iter29,Testing Accuracy：0.9729,Training Accuracy:0.9953455
Iter30,Testing Accuracy：0.974,Training Accuracy:0.9954364

Process finished with exit code 0

当使用dropout（ sess.run(train_step, feed_dict={x: batch_xs, y: batch_ys, keep_prob: 0.5})）时，运行结果如下，通过结果看出，在训练集上和在测试集中的准确率相差不大，没有出现明显的过拟合现象（ keep_prob: 0.5，表示每次训练迭代的时候只使用百分之五十的神经元）（虽然从结果中观察同样训练311次，加上dropout使测试集中的准确率下降了，但是如果训练迭代次数提升为100效果也会达到一个理想的值，最主要是不会出现过拟合现象）。

2020-08-03 17:39:24.233759: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165]      
Iter0,Testing Accuracy：0.8909,Training Accuracy:0.8817091
Iter1,Testing Accuracy：0.9059,Training Accuracy:0.8993818
Iter2,Testing Accuracy：0.9127,Training Accuracy:0.90907276
Iter3,Testing Accuracy：0.9193,Training Accuracy:0.9135454
Iter4,Testing Accuracy：0.9254,Training Accuracy:0.92007273
Iter5,Testing Accuracy：0.9288,Training Accuracy:0.9220909
Iter6,Testing Accuracy：0.9296,Training Accuracy:0.92521816
Iter7,Testing Accuracy：0.9314,Training Accuracy:0.92718184
Iter8,Testing Accuracy：0.934,Training Accuracy:0.9297818
Iter9,Testing Accuracy：0.9332,Training Accuracy:0.93134546
Iter10,Testing Accuracy：0.9358,Training Accuracy:0.93316364
Iter11,Testing Accuracy：0.9376,Training Accuracy:0.9348364
Iter12,Testing Accuracy：0.9387,Training Accuracy:0.9372182
Iter13,Testing Accuracy：0.9394,Training Accuracy:0.93796366
Iter14,Testing Accuracy：0.9411,Training Accuracy:0.93881816
Iter15,Testing Accuracy：0.9403,Training Accuracy:0.94034547
Iter16,Testing Accuracy：0.9432,Training Accuracy:0.9407091
Iter17,Testing Accuracy：0.9429,Training Accuracy:0.94272727
Iter18,Testing Accuracy：0.9433,Training Accuracy:0.9438546
Iter19,Testing Accuracy：0.945,Training Accuracy:0.9448182
Iter20,Testing Accuracy：0.9445,Training Accuracy:0.9446
Iter21,Testing Accuracy：0.9449,Training Accuracy:0.9464909
Iter22,Testing Accuracy：0.9456,Training Accuracy:0.94723636
Iter23,Testing Accuracy：0.9471,Training Accuracy:0.94765455
Iter24,Testing Accuracy：0.947,Training Accuracy:0.9471818
Iter25,Testing Accuracy：0.9491,Training Accuracy:0.9491091
Iter26,Testing Accuracy：0.9498,Training Accuracy:0.95101815
Iter27,Testing Accuracy：0.9488,Training Accuracy:0.94978184
Iter28,Testing Accuracy：0.9501,Training Accuracy:0.9517818
Iter29,Testing Accuracy：0.9498,Training Accuracy:0.95225453
Iter30,Testing Accuracy：0.9514,Training Accuracy:0.9533273

Process finished with exit code 0

附件：数据集：链接: https://pan.baidu.com/s/1tHdV4De4PLx-lEpvtheXMg 提取码: e1j1