Pytorch：使用 Embedding 嵌入层 进行 新闻主题分类任务
程序员文章站
2022-03-04 13:32:57
...
日萌社
人工智能AI：Keras PyTorch MXNet TensorFlow PaddlePaddle 深度学习实战（不定时更新）
2.1 新闻主题分类任务
- 学习目标:
- 了解有关新闻主题分类和有关数据.
- 掌握使用浅层网络构建新闻主题分类器的实现过程.
- 关于新闻主题分类任务:
- 以一段新闻报道中的文本描述内容为输入, 使用模型帮助我们判断它最有可能属于哪一种类型的新闻, 这是典型的文本分类问题, 我们这里假定每种类型是互斥的, 即文本描述有且只有一种类型.
- 新闻主题分类数据:
- 通过torchtext获取数据:
# 导入相关的torch工具包
import torch
import torchtext
# 导入torchtext.datasets中的文本分类任务
from torchtext.datasets import text_classification
import os
# 定义数据下载路径, 当前路径的data文件夹
load_data_path = "./data"
# 如果不存在该路径, 则创建这个路径
if not os.path.isdir(load_data_path):
os.mkdir(load_data_path)
# 选取torchtext中的文本分类数据集'AG_NEWS'即新闻主题分类数据, 保存在指定目录下
# 并将数值映射后的训练和验证数据加载到内存中
train_dataset, test_dataset = text_classification.DATASETS['AG_NEWS'](root=load_data_path)
- 数据文件预览:
- data/
- ag_news_csv.tar.gz
- ag_news_csv/
classes.txt
readme.txt
test.csv
train.csv
文件说明:
- train.csv表示训练数据, 共12万条数据; test.csv表示验证数据, 共7600条数据; classes.txt是标签(新闻主题)含义文件, 里面有四个单词'World', 'Sports', 'Business', 'Sci/Tech'代表新闻的四个主题, readme.txt是该数据集的英文说明.
train.csv预览:
"3","Wall St. Bears Claw Back Into the Black (Reuters)","Reuters - Short-sellers, Wall Street's dwindling\band of ultra-cynics, are seeing green again."
"3","Carlyle Looks Toward Commercial Aerospace (Reuters)","Reuters - Private investment firm Carlyle Group,\which has a reputation for making well-timed and occasionally\controversial plays in the defense industry, has quietly placed\its bets on another part of the market."
"3","Oil and Economy Cloud Stocks' Outlook (Reuters)","Reuters - Soaring crude prices plus worries\about the economy and the outlook for earnings are expected to\hang over the stock market next week during the depth of the\summer doldrums."
"3","Iraq Halts Oil Exports from Main Southern Pipeline (Reuters)","Reuters - Authorities have halted oil export\flows from the main pipeline in southern Iraq after\intelligence showed a rebel militia could strike\infrastructure, an oil official said on Saturday."
"3","Oil prices soar to all-time record, posing new menace to US economy (AFP)","AFP - Tearaway world oil prices, toppling records and straining wallets, present a new economic menace barely three months before the US presidential elections."
"3","Stocks End Up, But Near Year Lows (Reuters)","Reuters - Stocks ended slightly higher on Friday\but stayed near lows for the year as oil prices surged past #36;46\a barrel, offsetting a positive outlook from computer maker\Dell Inc. (DELL.O)"
"3","Money Funds Fell in Latest Week (AP)","AP - Assets of the nation's retail money market mutual funds fell by #36;1.17 billion in the latest week to #36;849.98 trillion, the Investment Company Institute said Thursday."
"3","Fed minutes show dissent over inflation (USATODAY.com)","USATODAY.com - Retail sales bounced back a bit in July, and new claims for jobless benefits fell last week, the government said Thursday, indicating the economy is improving from a midsummer slump."
"3","Safety Net (Forbes.com)","Forbes.com - After earning a PH.D. in Sociology, Danny Bazil Riley started to work as the general manager at a commercial real estate firm at an annual base salary of #36;70,000. Soon after, a financial planner stopped by his desk to drop off brochures about insurance benefits available through his employer. But, at 32, ""buying insurance was the furthest thing from my mind,"" says Riley."
"3","Wall St. Bears Claw Back Into the Black"," NEW YORK (Reuters) - Short-sellers, Wall Street's dwindling band of ultra-cynics, are seeing green again."
- 文件内容说明:
- train.csv共由3列组成, 使用','进行分隔, 分别代表: 标签, 新闻标题, 新闻简述; 其中标签用"1", "2", "3", "4"表示, 依次对应classes中的内容.
- test.csv与train.csv内容格式与含义相同.
- 整个案例的实现可分为以下五个步骤:
- 第一步: 构建带有Embedding层的文本分类模型.
- 第二步: 对数据进行batch处理.
- 第三步: 构建训练与验证函数.
- 第四步: 进行模型训练和验证.
- 第五步: 查看embedding层嵌入的词向量.
- 第一步: 构建带有Embedding层的文本分类模型
# 导入必备的torch模型构建工具
import torch.nn as nn
import torch.nn.functional as F
# 指定BATCH_SIZE的大小
BATCH_SIZE = 16
# 进行可用设备检测, 有GPU的话将优先使用GPU
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
class TextSentiment(nn.Module):
"""文本分类模型"""
def __init__(self, vocab_size, embed_dim, num_class):
"""
description: 类的初始化函数
:param vocab_size: 整个语料包含的不同词汇总数
:param embed_dim: 指定词嵌入的维度
:param num_class: 文本分类的类别总数
"""
super().__init__()
# 实例化embedding层, sparse=True代表每次对该层求解梯度时, 只更新部分权重.
self.embedding = nn.Embedding(vocab_size, embed_dim, sparse=True)
# 实例化线性层, 参数分别是embed_dim和num_class.
self.fc = nn.Linear(embed_dim, num_class)
# 为各层初始化权重
self.init_weights()
def init_weights(self):
"""初始化权重函数"""
# 指定初始权重的取值范围数
initrange = 0.5
# 各层的权重参数都是初始化为均匀分布
self.embedding.weight.data.uniform_(-initrange, initrange)
self.fc.weight.data.uniform_(-initrange, initrange)
# 偏置初始化为0
self.fc.bias.data.zero_()
def forward(self, text):
"""
:param text: 文本数值映射后的结果
:return: 与类别数尺寸相同的张量, 用以判断文本类别
"""
# 获得embedding的结果embedded
# >>> embedded.shape
# (m, 32) 其中m是BATCH_SIZE大小的数据中词汇总数
embedded = self.embedding(text)
# 接下来我们需要将(m, 32)转化成(BATCH_SIZE, 32)
# 以便通过fc层后能计算相应的损失
# 首先, 我们已知m的值远大于BATCH_SIZE=16,
# 用m整除BATCH_SIZE, 获得m*包含c个BATCH_SIZE
c = embedded.size(0) // BATCH_SIZE
# 之后再从embedded中取c*BATCH_SIZE个向量得到新的embedded
# 这个新的embedded中的向量个数可以整除BATCH_SIZE
embedded = embedded[:BATCH_SIZE*c]
# 因为我们想利用平均池化的方法求embedded中指定行数的列的平均数,
# 但平均池化方法是作用在行上的, 并且需要3维输入
# 因此我们对新的embedded进行转置并拓展维度
embedded = embedded.transpose(1, 0).unsqueeze(0)
# 然后就是调用平均池化的方法, 并且核的大小为c
# 即取每c的元素计算一次均值作为结果
embedded = F.avg_pool1d(embedded, kernel_size=c)
# 最后，还需要减去新增的维度, 然后转置回去输送给fc层
return self.fc(embedded[0].transpose(1, 0))
- 实例化模型:
# 获得整个语料包含的不同词汇总数
VOCAB_SIZE = len(train_dataset.get_vocab())
# 指定词嵌入维度
EMBED_DIM = 32
# 获得类别总数
NUN_CLASS = len(train_dataset.get_labels())
# 实例化模型
model = TextSentiment(VOCAB_SIZE, EMBED_DIM, NUN_CLASS).to(device)
- 第二步: 对数据进行batch处理
def generate_batch(batch):
"""
description: 生成batch数据函数
:param batch: 由样本张量和对应标签的元组组成的batch_size大小的列表
形如:
[(label1, sample1), (lable2, sample2), ..., (labelN, sampleN)]
return: 样本张量和标签各自的列表形式(张量)
形如:
text = tensor([sample1, sample2, ..., sampleN])
label = tensor([label1, label2, ..., labelN])
"""
# 从batch中获得标签张量
label = torch.tensor([entry[1] for entry in batch])
# 从batch中获得样本张量
text = [entry[0] for entry in batch]
text = torch.cat(text)
# 返回结果
return text.to(device), label.to(device)-
调用:
# 假设一个输入:
batch = [(1, orch.tensor([3, 23, 2, 8])), (0, torch.tensor([3, 45, 21, 6]))]
res = generate_batch(batch)
print(res)
- 输出效果:
# 对应输入的两条数据进行了相应的拼接
(tensor([ 3, 23, 2, 8, 3, 45, 21, 6]), tensor([1, 0]))
- 第三步: 构建训练与验证函数
# 导入torch中的数据加载器方法
from torch.utils.data import DataLoader
def train(train_data):
"""模型训练函数"""
# 初始化训练损失和准确率为0
train_loss = 0
train_acc = 0
# 使用数据加载器生成BATCH_SIZE大小的数据进行批次训练
# data就是N多个generate_batch函数处理后的BATCH_SIZE大小的数据生成器
data = DataLoader(train_data, batch_size=BATCH_SIZE, shuffle=True,
collate_fn=generate_batch)
# 对data进行循环遍历, 使用每个batch的数据进行参数更新
for i, (text, cls) in enumerate(data):
# 设置优化器初始梯度为0
optimizer.zero_grad()
# 模型输入一个批次数据, 获得输出
output = model(text)
# 根据真实标签与模型输出计算损失
loss = criterion(output, cls)
# 将该批次的损失加到总损失中
train_loss += loss.item()
# 误差反向传播
loss.backward()
# 参数进行更新
optimizer.step()
# 将该批次的准确率加到总准确率中
train_acc += (output.argmax(1) == cls).sum().item()
# 调整优化器学习率
scheduler.step()
# 返回本轮训练的平均损失和平均准确率
return train_loss / len(train_data), train_acc / len(train_data)
def valid(valid_data):
"""模型验证函数"""
# 初始化验证损失和准确率为0
loss = 0
acc = 0
# 和训练相同, 使用DataLoader获得训练数据生成器
data = DataLoader(valid_data, batch_size=BATCH_SIZE, collate_fn=generate_batch)
# 按批次取出数据验证
for text, cls in data:
# 验证阶段, 不再求解梯度
with torch.no_grad():
# 使用模型获得输出
output = model(text)
# 计算损失
loss = criterion(output, cls)
# 将损失和准确率加到总损失和准确率中
loss += loss.item()
acc += (output.argmax(1) == cls).sum().item()
# 返回本轮验证的平均损失和平均准确率
return loss / len(valid_data), acc / len(valid_data)
- 第四步: 进行模型训练和验证.
# 导入时间工具包
import time
# 导入数据随机划分方法工具
from torch.utils.data.dataset import random_split
# 指定训练轮数
N_EPOCHS = 10
# 定义初始的验证损失
min_valid_loss = float('inf')
# 选择损失函数, 这里选择预定义的交叉熵损失函数
criterion = torch.nn.CrossEntropyLoss().to(device)
# 选择随机梯度下降优化器
optimizer = torch.optim.SGD(model.parameters(), lr=4.0)
# 选择优化器步长调节方法StepLR, 用来衰减学习率
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, 1, gamma=0.9)
# 从train_dataset取出0.95作为训练集, 先取其长度
train_len = int(len(train_dataset) * 0.95)
# 然后使用random_split进行乱序划分, 得到对应的训练集和验证集
sub_train_, sub_valid_ = \
random_split(train_dataset, [train_len, len(train_dataset) - train_len])
# 开始每一轮训练
for epoch in range(N_EPOCHS):
# 记录概论训练的开始时间
start_time = time.time()
# 调用train和valid函数得到训练和验证的平均损失, 平均准确率
train_loss, train_acc = train(sub_train_)
valid_loss, valid_acc = valid(sub_valid_)
# 计算训练和验证的总耗时(秒)
secs = int(time.time() - start_time)
# 用分钟和秒表示
mins = secs / 60
secs = secs % 60
# 打印训练和验证耗时，平均损失，平均准确率
print('Epoch: %d' %(epoch + 1), " | time in %d minutes, %d seconds" %(mins, secs))
print(f'\tLoss: {train_loss:.4f}(train)\t|\tAcc: {train_acc * 100:.1f}%(train)')
print(f'\tLoss: {valid_loss:.4f}(valid)\t|\tAcc: {valid_acc * 100:.1f}%(valid)')
- 输出效果:
120000lines [00:06, 17834.17lines/s]
120000lines [00:11, 10071.77lines/s]
7600lines [00:00, 10432.95lines/s]
Epoch: 1 | time in 0 minutes, 36 seconds
Loss: 0.0592(train) | Acc: 63.9%(train)
Loss: 0.0005(valid) | Acc: 69.2%(valid)
Epoch: 2 | time in 0 minutes, 37 seconds
Loss: 0.0507(train) | Acc: 71.3%(train)
Loss: 0.0005(valid) | Acc: 70.7%(valid)
Epoch: 3 | time in 0 minutes, 36 seconds
Loss: 0.0484(train) | Acc: 72.8%(train)
Loss: 0.0005(valid) | Acc: 71.4%(valid)
Epoch: 4 | time in 0 minutes, 36 seconds
Loss: 0.0474(train) | Acc: 73.4%(train)
Loss: 0.0004(valid) | Acc: 72.0%(valid)
Epoch: 5 | time in 0 minutes, 36 seconds
Loss: 0.0455(train) | Acc: 74.8%(train)
Loss: 0.0004(valid) | Acc: 72.5%(valid)
Epoch: 6 | time in 0 minutes, 36 seconds
Loss: 0.0451(train) | Acc: 74.9%(train)
Loss: 0.0004(valid) | Acc: 72.3%(valid)
Epoch: 7 | time in 0 minutes, 36 seconds
Loss: 0.0446(train) | Acc: 75.3%(train)
Loss: 0.0004(valid) | Acc: 72.0%(valid)
Epoch: 8 | time in 0 minutes, 36 seconds
Loss: 0.0437(train) | Acc: 75.9%(train)
Loss: 0.0004(valid) | Acc: 71.4%(valid)
Epoch: 9 | time in 0 minutes, 36 seconds
Loss: 0.0431(train) | Acc: 76.2%(train)
Loss: 0.0004(valid) | Acc: 72.7%(valid)
Epoch: 10 | time in 0 minutes, 36 seconds
Loss: 0.0426(train) | Acc: 76.6%(train)
Loss: 0.0004(valid) | Acc: 72.6%(valid)
- 第五步: 查看embedding层嵌入的词向量.
# 打印从模型的状态字典中获得的Embedding矩阵
print(model.state_dict()['embedding.weight'])
- 输出效果:
tensor([[ 0.4401, -0.4177, -0.4161, ..., 0.2497, -0.4657, -0.1861],
[-0.2574, -0.1952, 0.1443, ..., -0.4687, -0.0742, 0.2606],
[-0.1926, -0.1153, -0.0167, ..., -0.0954, 0.0134, -0.0632],
...,
[-0.0780, -0.2331, -0.3656, ..., -0.1899, 0.4083, 0.3002],
[-0.0696, 0.4396, -0.1350, ..., 0.1019, 0.2792, -0.4749],
[-0.2978, 0.1872, -0.1994, ..., 0.3435, 0.4729, -0.2608]])
-
小节总结:
- 学习了关于新闻主题分类任务:
- 以一段新闻报道中的文本描述内容为输入, 使用模型帮助我们判断它最有可能属于哪一种类型的新闻, 这是典型的文本分类问题, 我们这里假定每种类型是互斥的, 即文本描述有且只有一种类型.
- 学习了新闻主题分类数据的获取和样式.
- 学习了整个案例的实现的五个步骤:
- 第一步: 构建带有Embedding层的文本分类模型.
- 第二步: 对数据进行batch处理.
- 第三步: 构建训练与验证函数.
- 第四步: 进行模型训练和验证.
- 第五步: 查看embedding层嵌入的词向量.
- 学习了关于新闻主题分类任务:
新闻分类.py
""" pip install torchtext """
# 导入torchtext.datasets中的文本分类任务
# from torchtext.datasets import text_classification
from day04 import My_text_classification
# 导入相关的torch工具包
import torch
import os
# 导入必备的torch模型构建工具
import torch.nn as nn
import torch.nn.functional as F
# 导入torch中的数据加载器方法
from torch.utils.data import DataLoader
# 导入时间工具包
import time
# 导入数据随机划分方法工具
from torch.utils.data.dataset import random_split
import numpy as np
#================================================================================================
# 指定BATCH_SIZE的大小
BATCH_SIZE = 16
# 进行可用设备检测, 有GPU的话将优先使用GPU
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
# 定义数据下载路径, 当前路径的data文件夹
load_data_path = "./data"
# 如果不存在该路径, 则创建这个路径
if not os.path.isdir(load_data_path):
os.mkdir(load_data_path)
# 选取torchtext中的文本分类数据集'AG_NEWS'即新闻主题分类数据, 保存在指定目录下
# 并将数值映射后的训练和验证数据加载到内存中
train_dataset, test_dataset = My_text_classification.DATASETS['AG_NEWS'](root=load_data_path)
"""
第一步: 构建带有Embedding层的文本分类模型
"""
class TextSentiment(nn.Module):
"""文本分类模型"""
def __init__(self, vocab_size, embed_dim, num_class):
"""
description: 类的初始化函数
:param vocab_size: 整个语料包含的不同词汇总数
:param embed_dim: 指定词嵌入的维度
:param num_class: 文本分类的类别总数
"""
super().__init__()
# 实例化embedding层, sparse=True代表每次对该层求解梯度时, 只更新部分权重.
#nn.Embedding(vocab_size 词汇总数, embed_dim 单词嵌入维度)
self.embedding = nn.Embedding(vocab_size, embed_dim, sparse=True) #vocab_size 95812，embed_dim 32
# 实例化线性层, 参数分别是embed_dim和num_class
self.fc = nn.Linear(embed_dim, num_class) #embed_dim 32，num_class 4
# 为各层初始化权重
self.init_weights()
def init_weights(self):
"""初始化权重函数"""
# 指定初始权重的取值范围数
initrange = 0.5
# 各层的权重参数都是初始化为均匀分布
self.embedding.weight.data.uniform_(-initrange, initrange) #初始化 -0.5 到 0.5之间
self.fc.weight.data.uniform_(-initrange, initrange) #初始化 -0.5 到 0.5之间
# 偏置初始化为0
self.fc.bias.data.zero_() ##初始化 0
"""
output = model(text)
每次训练传入的text为tensor类型的一维数组，数组中的值均为单词对应在词汇列表中的索引值。
text由16个句子的批量大小组成的一维数组，因此每个句子的长度都不一致的关系，
因此每个一维数组text的长度都不一致。
embedded = self.embedding(text)
embedded.shape为(m, 32)，m为批量大小16个句子的单词总数，并且每个批量的embedded(同text原理)的m都是不相同的，
32为单词的嵌入维度(权重维度)。
c = embedded.size(0) // BATCH_SIZE
embedded = embedded[:BATCH_SIZE*c]
已知m的值远大于BATCH_SIZE=16，为了在模型中以便通过fc层后能计算相应的损失，
实际即为了(m, 32)中的m维度值可以整除BATCH_SIZE，
因此还需要将(m, 32) 转化成 (m//BATCH_SIZE*BATCH_SIZE, 32)，c为批量个数，
即先用m整除BATCH_SIZE, 获得m*包含c个BATCH_SIZE，之后再从embedded中取c*BATCH_SIZE个向量得到新的embedded。
这个新的embedded中的向量个数可以整除BATCH_SIZE。
embedded = embedded.transpose(1, 0).unsqueeze(0)
embedded = F.avg_pool1d(embedded, kernel_size=c)
因为我们想利用平均池化的方法求embedded中指定行数的列的平均数，但平均池化方法是作用在行上的,
并且需要3维输入因此我们对新的embedded进行转置后并拓展维度。
首先transpose(1, 0)把embedded的(m, 32)转换为(32, m)，然后unsqueeze(0)拓展维度变成(1, 32, m)，
然后就是调用平均池化avg_pool1d方法, 并且核的大小kernel_size为c(批量个数)，
即取每c个的元素计算一次均值作为结果，即最终有 m/c(批量个数)个平均值，
即是对(m, 32)的embedded 中的m 即按行进行求平均值，
实际即 一共有m个单词，每个单词的嵌入维度为32，那么如果对每个单词的嵌入维度求平均值是毫无意义的，
应该是求一段单词的平均值，因此kernel_size=c 实际即卷积核的大小为c，那么就是按照c个单词数求一个平均值。
embedded.shape: torch.Size([625, 32])
c: 39 # 625 / batch_size(16) = 39.0625
embedded.shape: torch.Size([624, 32]) # 39 * batch_size(16) = 624
embedded.shape: torch.Size([1, 32, 624])
embedded.shape: torch.Size([1, 32, 16]) # 624 / 39 = 16
"""
def forward(self, text):
"""
:param text: 文本数值映射后的结果
:return: 与类别数尺寸相同的张量, 用以判断文本类别
"""
# 获得embedding的结果embedded
# >>> embedded.shape
# (m, 32) 其中m是BATCH_SIZE大小的数据中词汇总数
embedded = self.embedding(text)
print("embedded.shape:",embedded.shape)
# 接下来我们需要将(m, 32)转化成(BATCH_SIZE, 32)
# 以便通过fc层后能计算相应的损失
# 首先, 我们已知m的值远大于BATCH_SIZE=16,
# 用m整除BATCH_SIZE, 获得m*包含c个BATCH_SIZE
c = embedded.size(0) // BATCH_SIZE
print("c:",c)
# 之后再从embedded中取c*BATCH_SIZE个向量得到新的embedded
# 这个新的embedded中的向量个数可以整除BATCH_SIZE
embedded = embedded[:BATCH_SIZE*c]
print("embedded.shape:",embedded.shape)
# 因为我们想利用平均池化的方法求embedded中指定行数的列的平均数,
# 但平均池化方法是作用在行上的, 并且需要3维输入
# 因此我们对新的embedded进行转置并拓展维度
embedded = embedded.transpose(1, 0).unsqueeze(0)
print("embedded.shape:",embedded.shape)
# 然后就是调用平均池化的方法, 并且核的大小为c
# 即取每c的元素计算一次均值作为结果
embedded = F.avg_pool1d(embedded, kernel_size=c)
print("embedded.shape:",embedded.shape)
# 最后，还需要减去新增的维度, 然后转置回去输送给fc层
return self.fc(embedded[0].transpose(1, 0))
"""
实例化模型
"""
# 获得整个语料包含的不同词汇总数
VOCAB_SIZE = len(train_dataset.get_vocab())
# 指定词嵌入维度
EMBED_DIM = 32
# 获得类别总数
NUN_CLASS = len(train_dataset.get_labels())
# print("VOCAB_SIZE词数:",VOCAB_SIZE) # 95812
# print("NUN_CLASS类别数:",NUN_CLASS) # 4
# 实例化模型
model = TextSentiment(VOCAB_SIZE, EMBED_DIM, NUN_CLASS).to(device)
"""
第二步: 对数据进行batch处理
"""
def generate_batch(batch):
"""
description: 生成batch数据函数
:param batch: 由样本张量和对应标签的元组组成的batch_size大小的列表
形如:
[(sample1, label1), (sample2, lable2), ..., (sampleN, labelN)]
:return: 样本张量和标签各自的列表形式(张量)
形如:
text = tensor([sample1, sample2, ..., sampleN])
label = tensor([label1, label2, ..., labelN])
"""
"""
batch：16个数据，格式为(标签值,样本数据)，一共有16个这样的元祖构成一个列表
"""
# print("batch size批量样本数:",len(batch)) # 列表中有 16 个元祖
# print("batch[0][0]:",batch[0][0]) # 元祖中的 int标签值
# print("batch[0][1]:",batch[0][1]) # 元祖中的 样本数据，数据值为单词在词汇列表中的索引值
# 从batch中获得样本张量
# text = torch.tensor()
text = [entry[1] for entry in batch]
# 从batch中获得标签张量
label = [entry[0] for entry in batch]
# print("batch_size批量大小的样本特征:", len(text)) #16
# print("batch_size批量大小的标签:", len(label)) #16
# text中包含16个句子，每个句子的长度都不一样，最终通过cat函数把16个句子的单词都封装到一个列表中
#text中的值 实际是16句子中的单词对应的索引值，因此cat函数封装成的一个列表中的元素值都是单词对应的索引值
#每个批量的16个句子都是长度不一致，因此每个批量中的text.shape都可能是不相同
text = torch.cat(text)
label = torch.tensor(label) #使用 torch.tensor 把 list 转换为 tensor类型
# print("cat(text):", text.shape) #每个批量中的text的长度(16个句子一共的单词数)都可能是不相同
# print("label:", label.shape) #torch.Size([16]) 16个句子对应的标签值
# 返回结果
return text.to(device), label.to(device)
"""
第三步: 构建训练与验证函数
"""
# 指定训练轮数
N_EPOCHS = 10
# 定义初始的验证损失
# min_valid_loss = float('inf')
# 选择损失函数, 这里选择预定义的交叉熵损失函数
criterion = torch.nn.CrossEntropyLoss().to(device)
# 选择随机梯度下降优化器
optimizer = torch.optim.SGD(model.parameters(), lr=4.0)
# 选择优化器步长调节方法StepLR, 用来衰减学习率
scheduler = torch.optim.lr_scheduler.StepLR(optimizer, 1, gamma=0.9)
def train(train_data):
"""模型训练函数"""
# 初始化训练损失和准确率为0
train_loss = 0
train_acc = 0
# 使用数据加载器生成BATCH_SIZE大小的数据进行批次训练
# data就是N多个generate_batch函数处理后的BATCH_SIZE大小的数据生成器
data = DataLoader(train_data, batch_size=BATCH_SIZE, shuffle=True, collate_fn=generate_batch)
# 对data进行循环遍历, 使用每个batch的数据进行参数更新
for i, (text, cls) in enumerate(data):
# 设置优化器初始梯度为0
optimizer.zero_grad()
# 模型输入一个批次数据, 获得输出
output = model(text)
"""
模型最后一个输出层输出的维度是类别数4，那么因为每个批量大小为16，
那么一个批量中的真实标签值一共有16个int值(16个句子对应16个真实标签值)。
因此模型的预测输出的维度是torch.Size([16, 4])，和真实标签值列表[16] 进行比较是否相同。
"""
# print("output:",output.shape) #torch.Size([16, 4])
# 根据真实标签与模型输出计算损失
loss = criterion(output, cls)
# 将该批次的损失加到总损失中
train_loss += loss.item()
# print("loss:",loss.item()) # loss.item()为一个 float小数值
# 误差反向传播
loss.backward()
# 参数进行更新
optimizer.step()
"""
output维度为 [16, 4]。
output.argmax(1) 取的是每行中最大元素值的索引值，一共有16行，那么最终得出16个索引值。
output.argmax(0) 在此处使用的话为错误用法，只能取出4个值，取出的是 每列中最大元素值的索引值。
"""
# 将该批次的准确率加到总准确率中
train_acc += (output.argmax(1) == cls).sum().item()
# 调整优化器学习率
scheduler.step()
"""
len(train_data) 获取的是训练样本数量 114000
使用 train_loss/训练样本数量，train_acc/训练样本数量
"""
# 返回本轮训练的平均损失和平均准确率
return train_loss / len(train_data), train_acc / len(train_data)
def valid(valid_data):
"""模型验证函数"""
# 初始化验证损失和准确率为0
loss = 0
acc = 0
# 和训练相同, 使用DataLoader获得训练数据生成器
data = DataLoader(valid_data, batch_size=BATCH_SIZE, collate_fn=generate_batch)
# 按批次取出数据验证
for text, cls in data:
# 验证阶段, 不再求解梯度
with torch.no_grad():
# 使用模型获得输出
output = model(text)
# 计算损失
loss = criterion(output, cls)
# 将损失和准确率加到总损失和准确率中
loss += loss.item()
acc += (output.argmax(1) == cls).sum().item()
# 返回本轮验证的平均损失和平均准确率
return loss / len(valid_data), acc / len(valid_data)
"""
第四步: 进行模型训练和验证
"""
def train_model():
train_dataset_size = len(train_dataset)
# print("train_dataset 总样本数",train_dataset_size) #120000
# 从train_dataset取出0.95作为训练集, 先取其长度
train_len = int(train_dataset_size * 0.95)
valid_len = train_dataset_size - train_len
# print("train_len 训练样本数",train_len) #114000
# print("valid_len 验证本数",valid_len) #6000
# 然后使用random_split进行乱序划分, 得到对应的训练集和验证集
sub_train_, sub_valid_ = random_split(train_dataset, [train_len, valid_len])
"""
RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index'
分析：data数据或者model没有调用.to(device)
解决：
# 进行可用设备检测, 有GPU的话将优先使用GPU
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = model.to(device) 或 model = model.cuda(device)
output = model(input.to(device)) 或 output = model(input.cuda(device))
# 选择损失函数, 这里选择预定义的交叉熵损失函数
criterion = torch.nn.CrossEntropyLoss().to(device) 或 criterion = torch.nn.CrossEntropyLoss().cuda(device)
loss = criterion(output, lable.to(device)) 或 loss = criterion(output, lable.cuda(device))
"""
# 开始每一轮训练
for epoch in range(N_EPOCHS):
# 记录概论训练的开始时间
start_time = time.time()
# 调用train和valid函数得到训练和验证的平均损失, 平均准确率
train_loss, train_acc = train(sub_train_)
valid_loss, valid_acc = valid(sub_valid_)
# 计算训练和验证的总耗时(秒)
secs = int(time.time() - start_time)
# 用分钟和秒表示
mins = secs / 60
secs = secs % 60
# 打印训练和验证耗时，平均损失，平均准确率
print('Epoch: %d' % (epoch + 1), " | time in %d minutes, %d seconds" % (mins, secs))
print(f'\tLoss: {train_loss:.4f}(train)\t|\tAcc: {train_acc * 100:.1f}%(train)')
print(f'\tLoss: {valid_loss:.4f}(valid)\t|\tAcc: {valid_acc * 100:.1f}%(valid)')
if __name__ == '__main__':
train_model()
# passMy_text_classification.py
import logging
import torch
import io
from torchtext.utils import download_from_url, extract_archive, unicode_csv_reader
from torchtext.data.utils import ngrams_iterator
from torchtext.data.utils import get_tokenizer
from torchtext.vocab import build_vocab_from_iterator
from torchtext.vocab import Vocab
from tqdm import tqdm
URLS = {
'AG_NEWS':
'https://drive.google.com/uc?export=download&id=0Bz8a_Dbh9QhbUDNpeUdjb0wxRms',
'SogouNews':
'https://drive.google.com/uc?export=download&id=0Bz8a_Dbh9QhbUkVqNEszd0pHaFE',
'DBpedia':
'https://drive.google.com/uc?export=download&id=0Bz8a_Dbh9QhbQ2Vic1kxMmZZQ1k',
'YelpReviewPolarity':
'https://drive.google.com/uc?export=download&id=0Bz8a_Dbh9QhbNUpYQ2N3SGlFaDg',
'YelpReviewFull':
'https://drive.google.com/uc?export=download&id=0Bz8a_Dbh9QhbZlU4dXhHTFhZQU0',
'YahooAnswers':
'https://drive.google.com/uc?export=download&id=0Bz8a_Dbh9Qhbd2JNdDBsQUdocVU',
'AmazonReviewPolarity':
'https://drive.google.com/uc?export=download&id=0Bz8a_Dbh9QhbaW12WVVZS2drcnM',
'AmazonReviewFull':
'https://drive.google.com/uc?export=download&id=0Bz8a_Dbh9QhbZVhsUnRWRDhETzA'
}
def _csv_iterator(data_path, ngrams, yield_cls=False):
tokenizer = get_tokenizer("basic_english")
with io.open(data_path, encoding="utf8") as f:
reader = unicode_csv_reader(f)
for row in reader:
tokens = ' '.join(row[1:])
tokens = tokenizer(tokens)
if yield_cls:
yield int(row[0]) - 1, ngrams_iterator(tokens, ngrams)
else:
yield ngrams_iterator(tokens, ngrams)
def _create_data_from_iterator(vocab, iterator, include_unk):
data = []
labels = []
with tqdm(unit_scale=0, unit='lines') as t:
for cls, tokens in iterator:
if include_unk:
tokens = torch.tensor([vocab[token] for token in tokens])
else:
token_ids = list(filter(lambda x: x is not Vocab.UNK, [vocab[token]
for token in tokens]))
tokens = torch.tensor(token_ids)
if len(tokens) == 0:
logging.info('Row contains no tokens.')
data.append((cls, tokens))
labels.append(cls)
t.update(1)
return data, set(labels)
class TextClassificationDataset(torch.utils.data.Dataset):
"""Defines an abstract text classification datasets.
Currently, we only support the following datasets:
- AG_NEWS
- SogouNews
- DBpedia
- YelpReviewPolarity
- YelpReviewFull
- YahooAnswers
- AmazonReviewPolarity
- AmazonReviewFull
"""
def __init__(self, vocab, data, labels):
"""Initiate text-classification dataset.
Arguments:
vocab: Vocabulary object used for dataset.
data: a list of label/tokens tuple. tokens are a tensor after
numericalizing the string tokens. label is an integer.
[(label1, tokens1), (label2, tokens2), (label2, tokens3)]
label: a set of the labels.
{label1, label2}
Examples:
See the examples in examples/text_classification/
"""
super(TextClassificationDataset, self).__init__()
self._data = data
self._labels = labels
self._vocab = vocab
def __getitem__(self, i):
return self._data[i]
def __len__(self):
return len(self._data)
def __iter__(self):
for x in self._data:
yield x
def get_labels(self):
return self._labels
def get_vocab(self):
return self._vocab
def _setup_datasets(dataset_name, root='.data', ngrams=1, vocab=None, include_unk=False):
# dataset_tar = download_from_url(URLS[dataset_name], root=root)
# dataset_tar = "./data/ag_news_csv.tar.gz"
# extracted_files = extract_archive(dataset_tar)
extracted_files = ['./data/ag_news_csv/train.csv',
'./data/ag_news_csv/test.csv',
'./data/ag_news_csv/classes.txt',
'./data/ag_news_csv/readme.txt']
# print(extracted_files)
for fname in extracted_files:
if fname.endswith('train.csv'):
train_csv_path = fname
if fname.endswith('test.csv'):
test_csv_path = fname
if vocab is None:
logging.info('Building Vocab based on {}'.format(train_csv_path))
vocab = build_vocab_from_iterator(_csv_iterator(train_csv_path, ngrams))
else:
if not isinstance(vocab, Vocab):
raise TypeError("Passed vocabulary is not of type Vocab")
logging.info('Vocab has {} entries'.format(len(vocab)))
logging.info('Creating training data')
train_data, train_labels = _create_data_from_iterator(
vocab, _csv_iterator(train_csv_path, ngrams, yield_cls=True), include_unk)
logging.info('Creating testing data')
test_data, test_labels = _create_data_from_iterator(
vocab, _csv_iterator(test_csv_path, ngrams, yield_cls=True), include_unk)
if len(train_labels ^ test_labels) > 0:
raise ValueError("Training and test labels don't match")
return (TextClassificationDataset(vocab, train_data, train_labels),
TextClassificationDataset(vocab, test_data, test_labels))
def AG_NEWS(*args, **kwargs):
""" Defines AG_NEWS datasets.
The labels includes:
- 1 : World
- 2 : Sports
- 3 : Business
- 4 : Sci/Tech
Create supervised learning dataset: AG_NEWS
Separately returns the training and test dataset
Arguments:
root: Directory where the datasets are saved. Default: ".data"
ngrams: a contiguous sequence of n items from s string text.
Default: 1
vocab: Vocabulary used for dataset. If None, it will generate a new
vocabulary based on the train data set.
include_unk: include unknown token in the data (Default: False)
Examples:
>>> train_dataset, test_dataset = torchtext.datasets.AG_NEWS(ngrams=3)
"""
return _setup_datasets(*(("AG_NEWS",) + args), **kwargs)
def SogouNews(*args, **kwargs):
""" Defines SogouNews datasets.
The labels includes:
- 1 : Sports
- 2 : Finance
- 3 : Entertainment
- 4 : Automobile
- 5 : Technology
Create supervised learning dataset: SogouNews
Separately returns the training and test dataset
Arguments:
root: Directory where the datasets are saved. Default: ".data"
ngrams: a contiguous sequence of n items from s string text.
Default: 1
vocab: Vocabulary used for dataset. If None, it will generate a new
vocabulary based on the train data set.
include_unk: include unknown token in the data (Default: False)
Examples:
>>> train_dataset, test_dataset = torchtext.datasets.SogouNews(ngrams=3)
"""
return _setup_datasets(*(("SogouNews",) + args), **kwargs)
def DBpedia(*args, **kwargs):
""" Defines DBpedia datasets.
The labels includes:
- 1 : Company
- 2 : EducationalInstitution
- 3 : Artist
- 4 : Athlete
- 5 : OfficeHolder
- 6 : MeanOfTransportation
- 7 : Building
- 8 : NaturalPlace
- 9 : Village
- 10 : Animal
- 11 : Plant
- 12 : Album
- 13 : Film
- 14 : WrittenWork
Create supervised learning dataset: DBpedia
Separately returns the training and test dataset
Arguments:
root: Directory where the datasets are saved. Default: ".data"
ngrams: a contiguous sequence of n items from s string text.
Default: 1
vocab: Vocabulary used for dataset. If None, it will generate a new
vocabulary based on the train data set.
include_unk: include unknown token in the data (Default: False)
Examples:
>>> train_dataset, test_dataset = torchtext.datasets.DBpedia(ngrams=3)
"""
return _setup_datasets(*(("DBpedia",) + args), **kwargs)
def YelpReviewPolarity(*args, **kwargs):
""" Defines YelpReviewPolarity datasets.
The labels includes:
- 1 : Negative polarity.
- 2 : Positive polarity.
Create supervised learning dataset: YelpReviewPolarity
Separately returns the training and test dataset
Arguments:
root: Directory where the datasets are saved. Default: ".data"
ngrams: a contiguous sequence of n items from s string text.
Default: 1
vocab: Vocabulary used for dataset. If None, it will generate a new
vocabulary based on the train data set.
include_unk: include unknown token in the data (Default: False)
Examples:
>>> train_dataset, test_dataset = torchtext.datasets.YelpReviewPolarity(ngrams=3)
"""
return _setup_datasets(*(("YelpReviewPolarity",) + args), **kwargs)
def YelpReviewFull(*args, **kwargs):
""" Defines YelpReviewFull datasets.
The labels includes:
1 - 5 : rating classes (5 is highly recommended).
Create supervised learning dataset: YelpReviewFull
Separately returns the training and test dataset
Arguments:
root: Directory where the datasets are saved. Default: ".data"
ngrams: a contiguous sequence of n items from s string text.
Default: 1
vocab: Vocabulary used for dataset. If None, it will generate a new
vocabulary based on the train data set.
include_unk: include unknown token in the data (Default: False)
Examples:
>>> train_dataset, test_dataset = torchtext.datasets.YelpReviewFull(ngrams=3)
"""
return _setup_datasets(*(("YelpReviewFull",) + args), **kwargs)
def YahooAnswers(*args, **kwargs):
""" Defines YahooAnswers datasets.
The labels includes:
- 1 : Society & Culture
- 2 : Science & Mathematics
- 3 : Health
- 4 : Education & Reference
- 5 : Computers & Internet
- 6 : Sports
- 7 : Business & Finance
- 8 : Entertainment & Music
- 9 : Family & Relationships
- 10 : Politics & Government
Create supervised learning dataset: YahooAnswers
Separately returns the training and test dataset
Arguments:
root: Directory where the datasets are saved. Default: ".data"
ngrams: a contiguous sequence of n items from s string text.
Default: 1
vocab: Vocabulary used for dataset. If None, it will generate a new
vocabulary based on the train data set.
include_unk: include unknown token in the data (Default: False)
Examples:
>>> train_dataset, test_dataset = torchtext.datasets.YahooAnswers(ngrams=3)
"""
return _setup_datasets(*(("YahooAnswers",) + args), **kwargs)
def AmazonReviewPolarity(*args, **kwargs):
""" Defines AmazonReviewPolarity datasets.
The labels includes:
- 1 : Negative polarity
- 2 : Positive polarity
Create supervised learning dataset: AmazonReviewPolarity
Separately returns the training and test dataset
Arguments:
root: Directory where the datasets are saved. Default: ".data"
ngrams: a contiguous sequence of n items from s string text.
Default: 1
vocab: Vocabulary used for dataset. If None, it will generate a new
vocabulary based on the train data set.
include_unk: include unknown token in the data (Default: False)
Examples:
>>> train_dataset, test_dataset = torchtext.datasets.AmazonReviewPolarity(ngrams=3)
"""
return _setup_datasets(*(("AmazonReviewPolarity",) + args), **kwargs)
def AmazonReviewFull(*args, **kwargs):
""" Defines AmazonReviewFull datasets.
The labels includes:
1 - 5 : rating classes (5 is highly recommended)
Create supervised learning dataset: AmazonReviewFull
Separately returns the training and test dataset
Arguments:
root: Directory where the dataset are saved. Default: ".data"
ngrams: a contiguous sequence of n items from s string text.
Default: 1
vocab: Vocabulary used for dataset. If None, it will generate a new
vocabulary based on the train data set.
include_unk: include unknown token in the data (Default: False)
Examples:
>>> train_dataset, test_dataset = torchtext.datasets.AmazonReviewFull(ngrams=3)
"""
return _setup_datasets(*(("AmazonReviewFull",) + args), **kwargs)
DATASETS = {
'AG_NEWS': AG_NEWS,
'SogouNews': SogouNews,
'DBpedia': DBpedia,
'YelpReviewPolarity': YelpReviewPolarity,
'YelpReviewFull': YelpReviewFull,
'YahooAnswers': YahooAnswers,
'AmazonReviewPolarity': AmazonReviewPolarity,
'AmazonReviewFull': AmazonReviewFull
}
LABELS = {
'AG_NEWS': {1: 'World',
2: 'Sports',
3: 'Business',
4: 'Sci/Tech'},
'SogouNews': {1: 'Sports',
2: 'Finance',
3: 'Entertainment',
4: 'Automobile',
5: 'Technology'},
'DBpedia': {1: 'Company',
2: 'EducationalInstitution',
3: 'Artist',
4: 'Athlete',
5: 'OfficeHolder',
6: 'MeanOfTransportation',
7: 'Building',
8: 'NaturalPlace',
9: 'Village',
10: 'Animal',
11: 'Plant',
12: 'Album',
13: 'Film',
14: 'WrittenWork'},
'YelpReviewPolarity': {1: 'Negative polarity',
2: 'Positive polarity'},
'YelpReviewFull': {1: 'score 1',
2: 'score 2',
3: 'score 3',
4: 'score 4',
5: 'score 5'},
'YahooAnswers': {1: 'Society & Culture',
2: 'Science & Mathematics',
3: 'Health',
4: 'Education & Reference',
5: 'Computers & Internet',
6: 'Sports',
7: 'Business & Finance',
8: 'Entertainment & Music',
9: 'Family & Relationships',
10: 'Politics & Government'},
'AmazonReviewPolarity': {1: 'Negative polarity',
2: 'Positive polarity'},
'AmazonReviewFull': {1: 'score 1',
2: 'score 2',
3: 'score 3',
4: 'score 4',
5: 'score 5'}
}
上一篇： 决策树算法(二)