Python sklearn KFold 生成交叉验证数据集的方法

源起：

1.我要做交叉验证，需要每个训练集和测试集都保持相同的样本分布比例，直接用sklearn提供的kfold并不能满足这个需求。

2.将生成的交叉验证数据集保存成csv文件，而不是直接用sklearn训练分类模型。

3.在编码过程中有一的误区需要注意：

这个sklearn官方给出的文档

>>> import numpy as np
>>> from sklearn.model_selection import kfold
 
>>> x = ["a", "b", "c", "d"]
>>> kf = kfold(n_splits=2)
>>> for train, test in kf.split(x):
...  print("%s %s" % (train, test))
[2 3] [0 1]
[0 1] [2 3]

我之前犯的一个错误是将train，test理解成原数据集分割成子数据集之后的子数据集索引。而实际上，它就是原始数据集本身的样本索引。

源码：

# -*- coding:utf-8 -*-
# 得到交叉验证数据集，保存成csv文件
# 输入是一个包含正常恶意标签的完整数据集，在读数据的时候分开保存到datasetbenign，datasetmalicious
# 分别对两个数据集进行kfold，最后合并保存
 
from sklearn.model_selection import kfold
import csv
 
def writeinfile(benignkftrain, benignkftest, maliciouskftrain, maliciouskftest, i, datasetbenign, datasetmalicious):
 newtrainfilepath = "e:\\hadoopexperimentresult\\5kfold\\alldatasetiir10\\dataset\\imbalancedalltraffic-train-%s.csv" % i
 newtestfilepath = "e:\\hadoopexperimentresult\\5kfold\\alldatasetiir10\\dataset\\iimbalancedalltraffic-test-%s.csv" % i
 newtrainfile = open(newtrainfilepath, "wb")# wb 为防止空行
 newtestfile = open(newtestfilepath, "wb")
 writertrain = csv.writer(newtrainfile)
 writertest = csv.writer(newtestfile)
 for index in benignkftrain:
  writertrain.writerow(datasetbenign[index])
 for index in benignkftest:
  writertest.writerow(datasetbenign[index])
 for index in maliciouskftrain:
  writertrain.writerow(datasetmalicious[index])
 for index in maliciouskftest:
  writertest.writerow(datasetmalicious[index])
 newtrainfile.close()
 newtestfile.close()
 
 
def getkfolddataset(datasetpath):
 # csv读取文件
 # 开始从文件中读取全部的数据集
 datasetfile = file(datasetpath, 'rb')
 datasetbenign = []
 datasetmalicious = []
 readerdataset = csv.reader(datasetfile)
 for line in readerdataset:
  if len(line) > 1:
   curline = []
   curline.append(float(line[0]))
   curline.append(float(line[1]))
   curline.append(float(line[2]))
   curline.append(float(line[3]))
   curline.append(float(line[4]))
   curline.append(float(line[5]))
   curline.append(float(line[6]))
   curline.append(line[7])
   if line[7] == "benign":
    datasetbenign.append(curline)
   else:
    datasetmalicious.append(curline)
 
 # 交叉验证分割数据集
 k = 5
 kf = kfold(n_splits=k)
 benignkftrain = []; benignkftest = []
 for train,test in kf.split(datasetbenign):
  benignkftrain.append(train)
  benignkftest.append(test)
 maliciouskftrain=[]; maliciouskftest=[]
 for train,test in kf.split(datasetmalicious):
  maliciouskftrain.append(train)
  maliciouskftest.append(test)
 for i in range(k):
  print "======================== "+ str(i)+ " ========================"
  print benignkftrain[i], benignkftest[i]
  print maliciouskftrain[i],maliciouskftest[i]
  writeinfile(benignkftrain[i], benignkftest[i], maliciouskftrain[i], maliciouskftest[i], i, datasetbenign,
     datasetmalicious)
 
 datasetfile.close()
 
 
if __name__ == "__main__":
 
 getkfolddataset(r"e:\hadoopexperimentresult\5kfold\alldatasetiir10\dataset\imbalancedalltraffic-10.csv")

以上这篇python sklearn kfold 生成交叉验证数据集的方法就是小编分享给大家的全部内容了，希望能给大家一个参考，也希望大家多多支持。