2019/08/16 学习整理
函数进阶(模块)
建议学习的时候去看官方文档学习
numpy模块
numpy官方文档：https://docs.scipy.org/doc/numpy/reference/?v=20190307135750
numpy是Python的一种开源的数值计算扩展库。这种库可用来存储和处理大型矩阵，比Python自身的嵌套列表结构要高效的多（该结构也可以用来表示矩阵）。
numpy库有两个作用：
- 区别于list列表，提供了数组操作、数组运算、以及统计分布和简单的数学模型
- 计算速度快，甚至要由于python内置的简单运算，使得其成为pandas、sklearn等模块的依赖包。高级的框架如TensorFlow、PyTorch等，其数组操作也和numpy非常相似。
创建矩阵(掌握)
矩阵即numpy的ndarray对象，创建矩阵就是把一个列表传入np.array()方法。
import numpy as np
# 创建一维的ndarray对象
arr = np.array([1, 2, 3])
print(arr, type(arr))
[1 2 3] <class 'numpy.ndarray'>
# 创建二维的ndarray对象
print(np.array([[1, 2, 3], [4, 5, 6]]))
[[1 2 3]
[4 5 6]]
# 创建三维的ndarray对象
print(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]))
[[1 2 3]
[4 5 6]
[7 8 9]]
获取矩阵的行列数(掌握)
arr = np.array([[1, 2, 3], [4, 5, 6]])
print(arr)
[[1 2 3]
[4 5 6]]
# 获取矩阵的行和列构成的数组
print(arr.shape)
(2, 3)
# 获取矩阵的行
print(arr.shape[0])
2
# 获取矩阵的列
print(arr.shape[1])
3
切割矩阵(掌握)
切分矩阵类似于列表的切割，但是与列表的切割不同的是，矩阵的切割涉及到行和列的切割，但是两者切割的方式都是从索引0开始，并且取头不取尾。
arr = np.array([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]])
print(arr)
[[ 1 2 3 4]
[ 5 6 7 8]
[ 9 10 11 12]]
# 取所有元素
print(arr[:, :])
[[ 1 2 3 4]
[ 5 6 7 8]
[ 9 10 11 12]]
# 取第一行的所有元素
print(arr[:1, :])
[[1 2 3 4]]
# 取第一行的所有元素
print(arr[0, [0, 1, 2, 3]])
[1 2 3 4]
# 取第一列的所有元素
print(arr[:, :1])
[[1]
[5]
[9]]
# 取第一列的所有元素
print(arr[(0, 1, 2), 0])
[1 5 9]
# 取第一行第一列的元素
print(arr[(0, 1, 2), 0])
[1 5 9]
# 取第一行第一列的元素
print(arr[0, 0])
1
# 取大于5的元素，返回一个数组
print(arr[arr > 5])
[ 6 7 8 9 10 11 12]
# 矩阵按运算符取元素的原理，即通过arr > 5生成一个布尔矩阵
print(arr > 5)
[[False False False False]
[False True True True]
[ True True True True]]
矩阵元素替换(掌握)
矩阵元素的替换，类似于列表元素的替换，并且矩阵也是一个可变类型的数据，即如果对矩阵进行替换操作，会修改原矩阵的元素，所以下面我们用.copy()方法举例矩阵元素的替换。
arr = np.array([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]])
print(arr)
[[ 1 2 3 4]
[ 5 6 7 8]
[ 9 10 11 12]]
# 取第一行的所有元素，并且让第一行的元素都为0
arr1 = arr.copy()
arr1[:1, :] = 0
print(arr1)
[[ 0 0 0 0]
[ 5 6 7 8]
[ 9 10 11 12]]
# 取所有大于5的元素，并且让大于5的元素为0
arr2 = arr.copy()
arr2[arr > 5] = 0
print(arr2)
[[1 2 3 4]
[5 0 0 0]
[0 0 0 0]]
# 对矩阵清零
arr3 = arr.copy()
arr3[:, :] = 0
print(arr3)
[[0 0 0 0]
[0 0 0 0]
[0 0 0 0]]
矩阵的合并(熟悉)
arr1 = np.array([[1, 2], [3, 4], [5, 6]])
print(arr1)
[[1 2]
[3 4]
[5 6]]
arr2 = np.array([[7, 8], [9, 10], [11, 12]])
print(arr2)
[[ 7 8]
[ 9 10]
[11 12]]
# 合并两个矩阵的行，注意使用hstack()方法合并矩阵，矩阵应该有相同的行，其中hstack的h表示horizontal水平的
print(np.hstack((arr1, arr2)))
[[ 1 2 7 8]
[ 3 4 9 10]
[ 5 6 11 12]]
# 合并两个矩阵，其中axis=1表示合并两个矩阵的行
print(np.concatenate((arr1, arr2), axis=1))
[[ 1 2 7 8]
[ 3 4 9 10]
[ 5 6 11 12]]
# 合并两个矩阵的列，注意使用vstack()方法合并矩阵，矩阵应该有相同的列，其中vstack的v表示vertical垂直的
print(np.vstack((arr1, arr2)))
[[ 1 2]
[ 3 4]
[ 5 6]
[ 7 8]
[ 9 10]
[11 12]]
# 合并两个矩阵，其中axis=0表示合并两个矩阵的列
print(np.concatenate((arr1, arr2), axis=0))
[[ 1 2]
[ 3 4]
[ 5 6]
[ 7 8]
[ 9 10]
[11 12]]
通过函数创建矩阵(掌握)
arange
# 构造0-9的ndarray数组
print(np.arange(10))
[0 1 2 3 4 5 6 7 8 9]
# 构造1-4的ndarray数组
print(np.arange(1, 5))
[1 2 3 4]
# 构造1-19且步长为2的ndarray数组
print(np.arange(1, 20, 2))
[ 1 3 5 7 9 11 13 15 17 19]
linspace/logspace
# 构造一个等差数列，取头也取尾，从0取到20，取5个数
print(np.linspace(0, 20, 5))
[ 0. 5. 10. 15. 20.]
# 构造一个等比数列，从10**0取到10**20，取5个数
print(np.logspace(0, 20, 5))
[ 1.00000000e+00 1.00000000e+05 1.00000000e+10 1.00000000e+15
1.00000000e+20]
zeros/ones/eye/empty
# 构造3*4的全0矩阵
print(np.zeros((3, 4)))
[[ 0. 0. 0. 0.]
[ 0. 0. 0. 0.]
[ 0. 0. 0. 0.]]
# 构造3*4的全1矩阵
print(np.ones((3, 4)))
[[ 1. 1. 1. 1.]
[ 1. 1. 1. 1.]
[ 1. 1. 1. 1.]]
# 构造3个主元的单位矩阵
print(np.eye(3))
[[ 1. 0. 0.]
[ 0. 1. 0.]
[ 0. 0. 1.]]
# 构造一个4*4的随机矩阵，里面的元素是随机生成的
print(np.empty((4, 4)))
[[ 1.72723371e-077 -2.68678116e+154 3.95252517e-323 0.00000000e+000]
[ 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000]
[ 0.00000000e+000 0.00000000e+000 0.00000000e+000 0.00000000e+000]
[ 0.00000000e+000 0.00000000e+000 0.00000000e+000 1.17248833e-308]]
fromstring/fromfunction(了解)
# fromstring通过对字符串的字符编码所对应ASCII编码的位置，生成一个ndarray对象
s = 'abcdef'
# np.int8表示一个字符的字节数为8
print(np.fromstring(s, dtype=np.int8))
[ 97 98 99 100 101 102]
def func(i, j):
"""其中i为矩阵的行，j为矩阵的列"""
return i*j
# 使用函数对矩阵元素的行和列的索引做处理，得到当前元素的值，索引从0开始，并构造一个3*4的矩阵
print(np.fromfunction(func, (3, 4)))
[[ 0. 0. 0. 0.]
[ 0. 1. 2. 3.]
[ 0. 2. 4. 6.]]
矩阵的运算(掌握)
普通矩阵运算
| 运算符 |
说明 |
| + |
两个矩阵对应元素相加 |
| - |
两个矩阵对应元素相减 |
| * |
两个矩阵对应元素相乘 |
| / |
两个矩阵对应元素相除，如果都是整数则取商 |
| % |
两个矩阵对应元素相除后取余数 |
| **n |
单个矩阵每个元素都取n次方，如**2：每个元素都取平方 |
arrarr1 = np.array([[1, 2], [3, 4], [5, 6]])
print(arr1)
[[1 2]
[3 4]
[5 6]]
arr2 = np.array([[7, 8], [9, 10], [11, 12]])
print(arr2)
[[ 7 8]
[ 9 10]
[11 12]]
print(arr1+arr2)
[[ 8 10]
[12 14]
[16 18]]
print(arr1**2)
[[ 1 4]
[ 9 16]
[25 36]]
常用矩阵运算函数(了解)
| 矩阵函数 |
详解 |
| np.sin(arr) |
对矩阵arr中每个元素取正弦，$sin(x)$ |
| np.cos(arr) |
对矩阵arr中每个元素取余弦，$cos(x)$ |
| np.tan(arr) |
对矩阵arr中每个元素取正切，$tan(x)$ |
| np.arcsin(arr) |
对矩阵arr中每个元素取反正弦，$arcsin(x)$ |
| np.arccos(arr) |
对矩阵arr中每个元素取反余弦，$arccos(x)$ |
| np.arctan(arr) |
对矩阵arr中每个元素取反正切，$arctan(x)$ |
| np.exp(arr) |
对矩阵arr中每个元素取指数函数，$e^x$ |
| np.sqrt(arr) |
对矩阵arr中每个元素开根号$\sqrt{x}$ |
arr = np.array([[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12]])
print(arr)
[[ 1 2 3 4]
[ 5 6 7 8]
[ 9 10 11 12]]
# 对矩阵的所有元素取正弦
print(np.sin(arr))
[[ 0.84147098 0.90929743 0.14112001 -0.7568025 ]
[-0.95892427 -0.2794155 0.6569866 0.98935825]
[ 0.41211849 -0.54402111 -0.99999021 -0.53657292]]
# 对矩阵的所有元素开根号
print(np.sqrt(arr))
[[ 1. 1.41421356 1.73205081 2. ]
[ 2.23606798 2.44948974 2.64575131 2.82842712]
[ 3. 3.16227766 3.31662479 3.46410162]]
# 对矩阵的所有元素取反正弦，如果元素不在定义域内，则会取nan值
print(np.arcsin(arr))
[[ 1.57079633 nan nan nan]
[ nan nan nan nan]
[ nan nan nan nan]]
/Applications/anaconda3/lib/python3.6/site-packages/ipykernel_launcher.py:2: RuntimeWarning: invalid value encountered in arcsin
矩阵的点乘(掌握)
矩阵的点乘必须满足第一个矩阵的列数等于第二个矩阵的行数，即$mn·{nm}=m*m$。
arr1 = np.array([[1, 2, 3], [4, 5, 6]])
print(arr1.shape)
(2, 3)
arr2 = np.array([[7, 8], [9, 10], [11, 12]])
print(arr2.shape)
(3, 2)
assert arr1.shape[0] == arr2.shape[1]
# 2*3·3*2 = 2*2
print(arr2.shape)
(3, 2)
矩阵的转置(掌握)
矩阵的转置，相当于矩阵的行和列互换。
arr = np.array([[1, 2, 3], [4, 5, 6]])
print(arr)
[[1 2 3]
[4 5 6]]
print(arr.transpose())
[[1 4]
[2 5]
[3 6]]
print(arr.T)
[[1 4]
[2 5]
[3 6]]
矩阵的逆(掌握)
矩阵行和列相同时，矩阵才可逆。
arr = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
print(arr)
[[1 2 3]
[4 5 6]
[7 8 9]]
print(np.linalg.inv(arr))
[[ 3.15251974e+15 -6.30503948e+15 3.15251974e+15]
[ -6.30503948e+15 1.26100790e+16 -6.30503948e+15]
[ 3.15251974e+15 -6.30503948e+15 3.15251974e+15]]
# 单位矩阵的逆是单位矩阵本身
arr = np.eye(3)
print(arr)
[[ 1. 0. 0.]
[ 0. 1. 0.]
[ 0. 0. 1.]]
print(np.linalg.inv(arr))
[[ 1. 0. 0.]
[ 0. 1. 0.]
[ 0. 0. 1.]]
矩阵其他操作(熟悉)
最大最小值
arr = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
print(arr)
[[1 2 3]
[4 5 6]
[7 8 9]]
# 获取矩阵所有元素中的最大值
print(arr.max())
9
# 获取矩阵所有元素中的最小值
print(arr.min())
1
# 获取举着每一行的最大值
print(arr.max(axis=0))
[7 8 9]
# 获取矩阵每一列的最大值
print(arr.max(axis=1))
[3 6 9]
# 获取矩阵最大元素的索引位置
print(arr.argmax(axis=1))
[2 2 2]
平均值
arr = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
print(arr)
[[1 2 3]
[4 5 6]
[7 8 9]]
# 获取矩阵所有元素的平均值
print(arr.mean())
5.0
# 获取矩阵每一列的平均值
print(arr.mean(axis=0))
[ 4. 5. 6.]
# 获取矩阵每一行的平均值
print(arr.mean(axis=1))
[ 2. 5. 8.]
方差
方差公式为
$$
mean(|x-x.mean()|^2)
$$
其中x为矩阵。
arr = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
print(arr)
[[1 2 3]
[4 5 6]
[7 8 9]]
# 获取矩阵所有元素的方差
print(arr.var())
6.66666666667
# 获取矩阵每一列的元素的方差
print(arr.var(axis=0))
[ 6. 6. 6.]
# 获取矩阵每一行的元素的方差
print(arr.var(axis=1))
[ 0.66666667 0.66666667 0.66666667]
标准差
标准差公式为
$$
\sqrt{mean|x-x.mean()|^2} = \sqrt{x.var()}
$$
arr = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
print(arr)
[[1 2 3]
[4 5 6]
[7 8 9]]
# 获取矩阵所有元素的标准差
print(arr.std())
2.58198889747
# 获取矩阵每一列的标准差
print(arr.std(axis=0))
[ 2.44948974 2.44948974 2.44948974]
# 获取矩阵每一行的标准差
print(arr.std(axis=1))
[ 0.81649658 0.81649658 0.81649658]
中位数
arr = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
print(arr)
[[1 2 3]
[4 5 6]
[7 8 9]]
# 获取矩阵所有元素的中位数
print(np.median(arr))
5.0
# 获取矩阵每一列的中位数
print(np.median(arr, axis=0))
[ 4. 5. 6.]
# 获取矩阵每一行的中位数
print(np.median(arr, axis=1))
[ 2. 5. 8.]
矩阵求和
arr = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
print(arr)
[[1 2 3]
[4 5 6]
[7 8 9]]
# 对矩阵的每一个元素求和
print(arr.sum())
45
# 对矩阵的每一列求和
print(arr.sum(axis=0))
[12 15 18]
# 对矩阵的每一行求和
print(arr.sum(axis=1))
[ 6 15 24]
累加和
arr = np.array([1, 2, 3, 4, 5])
print(arr)
[1 2 3 4 5]
# 第n个元素为前n-1个元素累加和
print(arr.cumsum())
[ 1 3 6 10 15]
numpy.random生成随机数(熟悉)
| 函数名称 |
函数功能 |
参数说明 |
| rand($d_0, d_1, \cdots , d_n$) |
产生均匀分布的随机数 |
$d_n$为第n维数据的维度 |
| randn($d_0, d_1, \cdots , d_n$) |
产生标准正态分布随机数 |
$d_n$为第n维数据的维度 |
| randint(low[, high, size, dtype]) |
产生随机整数 |
low:最小值；high:最大值；size:数据个数 |
| random_sample([size]) |
在$[0,1)$内产生随机数 |
size为随机数的shape，可以为元祖或者列表 |
| choice(a[, size]) |
从arr中随机选择指定数据 |
arr为1维数组；size为数据形状 |
# RandomState()方法会让数据值随机一次，之后都是相同的数据
rs = np.random.RandomState(1)
print(rs.rand(10))
[ 4.17022005e-01 7.20324493e-01 1.14374817e-04 3.02332573e-01
1.46755891e-01 9.23385948e-02 1.86260211e-01 3.45560727e-01
3.96767474e-01 5.38816734e-01]
# 构造3*4的均匀分布的矩阵
# seed()方法会让数据值随机一次，之后都是相同的数据
np.random.seed(1)
print(np.random.rand(3, 4))
[[ 4.17022005e-01 7.20324493e-01 1.14374817e-04 3.02332573e-01]
[ 1.46755891e-01 9.23385948e-02 1.86260211e-01 3.45560727e-01]
[ 3.96767474e-01 5.38816734e-01 4.19194514e-01 6.85219500e-01]]
# 构造3*4*5的均匀分布的矩阵
print(np.random.rand(3, 4, 5))
[[[ 0.20445225 0.87811744 0.02738759 0.67046751 0.4173048 ]
[ 0.55868983 0.14038694 0.19810149 0.80074457 0.96826158]
[ 0.31342418 0.69232262 0.87638915 0.89460666 0.08504421]
[ 0.03905478 0.16983042 0.8781425 0.09834683 0.42110763]]
[[ 0.95788953 0.53316528 0.69187711 0.31551563 0.68650093]
[ 0.83462567 0.01828828 0.75014431 0.98886109 0.74816565]
[ 0.28044399 0.78927933 0.10322601 0.44789353 0.9085955 ]
[ 0.29361415 0.28777534 0.13002857 0.01936696 0.67883553]]
[[ 0.21162812 0.26554666 0.49157316 0.05336255 0.57411761]
[ 0.14672857 0.58930554 0.69975836 0.10233443 0.41405599]
[ 0.69440016 0.41417927 0.04995346 0.53589641 0.66379465]
[ 0.51488911 0.94459476 0.58655504 0.90340192 0.1374747 ]]]
# 构造3*4的正态分布的矩阵
print(np.random.randn(3, 4))
[[ 0.30017032 -0.35224985 -1.1425182 -0.34934272]
[-0.20889423 0.58662319 0.83898341 0.93110208]
[ 0.28558733 0.88514116 -0.75439794 1.25286816]]
# 构造取值为1-5内的10个元素的ndarray数组
print(np.random.randint(1, 5, 10))
[1 1 1 2 3 1 2 1 3 4]
# 构造取值为0-1内的3*4的矩阵
print(np.random.random_sample((3, 4)))
[[ 0.62169572 0.11474597 0.94948926 0.44991213]
[ 0.57838961 0.4081368 0.23702698 0.90337952]
[ 0.57367949 0.00287033 0.61714491 0.3266449 ]]
arr = np.array([1, 2, 3])
# 随机选取arr中的两个元素
print(np.random.choice(arr, size=2))
[1 3]
pandas模块
pandas官方文档：https://pandas.pydata.org/pandas-docs/stable/?v=20190307135750
pandas基于Numpy，可以看成是处理文本或者表格数据。pandas中有两个主要的数据结构，其中Series数据结构类似于Numpy中的一维数组，DataFrame类似于多维表格数据结构。
pandas是python数据分析的核心模块。它主要提供了五大功能:
- 支持文件存取操作，支持数据库(sql)、html、json、pickle、csv(txt、excel)、sas、stata、hdf等。
- 支持增删改查、切片、高阶函数、分组聚合等单表操作，以及和dict、list的互相转换。
- 支持多表拼接合并操作。
- 支持简单的绘图操作。
- 支持简单的统计分析操作。
Series(熟悉)
import numpy as np
import pandas as pd
arr = np.array([1, 2, 3, 4, np.nan, ])
print(arr)
[ 1. 2. 3. 4. nan]
s = pd.Series(arr)
print(s)
0 1.0
1 2.0
2 3.0
3 4.0
4 NaN
dtype: float64
import random
random.randint(1,10)
1
import numpy as np
np.random.randn(6,4)
array([[-0.42660201, 2.61346133, 0.01214827, -1.43370137],
[-0.28285711, 0.14871693, 0.22235496, -2.63142648],
[ 0.78324411, -0.72633723, -0.23258796, 0.03855565],
[-0.30033472, -1.19873979, -1.72660722, 0.75214317],
[ 1.48194193, 0.11089792, 0.8845003 , -1.26433672],
[ 1.29958399, -1.75092753, 0.06823543, -0.64219199]])
DataFrame(掌握)
dates = pd.date_range('20190101', periods=6)
print(dates)
DatetimeIndex(['2019-01-01', '2019-01-02', '2019-01-03', '2019-01-04',
'2019-01-05', '2019-01-06'],
dtype='datetime64[ns]', freq='D')
np.random.seed(1)
arr = 10*np.random.randn(6, 4)
print(arr)
[[ 16.24345364 -6.11756414 -5.28171752 -10.72968622]
[ 8.65407629 -23.01538697 17.44811764 -7.61206901]
[ 3.19039096 -2.49370375 14.62107937 -20.60140709]
[ -3.22417204 -3.84054355 11.33769442 -10.99891267]
[ -1.72428208 -8.77858418 0.42213747 5.82815214]
[-11.00619177 11.4472371 9.01590721 5.02494339]]
df = pd.DataFrame(arr, index=dates, columns=['c1', 'c2', 'c3', 'c4'])
df
|
c1 |
c2 |
c3 |
c4 |
| 2019-01-01 |
16.243454 |
-6.117564 |
-5.281718 |
-10.729686 |
| 2019-01-02 |
8.654076 |
-23.015387 |
17.448118 |
-7.612069 |
| 2019-01-03 |
3.190391 |
-2.493704 |
14.621079 |
-20.601407 |
| 2019-01-04 |
-3.224172 |
-3.840544 |
11.337694 |
-10.998913 |
| 2019-01-05 |
-1.724282 |
-8.778584 |
0.422137 |
5.828152 |
| 2019-01-06 |
-11.006192 |
11.447237 |
9.015907 |
5.024943 |
# 使用pandas读取字典形式的数据
df2 = pd.DataFrame({'a': 1, 'b': [2, 3], 'c': np.arange(2), 'd': 'hello'})
df2
|
a |
b |
c |
d |
| 0 |
1 |
2 |
0 |
hello |
| 1 |
1 |
3 |
1 |
hello |
DataFrame属性(掌握)
| 属性 |
详解 |
| dtype |
查看数据类型 |
| index |
查看行序列或者索引 |
| columns |
查看各列的标签 |
| values |
查看数据框内的数据，也即不含表头索引的数据 |
| describe |
查看数据每一列的极值，均值，中位数，只可用于数值型数据 |
| transpose |
转置，也可用Ｔ来操作 |
| sort_index |
排序，可按行或列index排序输出 |
| sort_values |
按数据值来排序 |
# 查看数据类型
print(df2.dtypes)
a int64
b int64
c int64
d object
dtype: object
df
|
c1 |
c2 |
c3 |
c4 |
| 2019-01-01 |
16.243454 |
-6.117564 |
-5.281718 |
-10.729686 |
| 2019-01-02 |
8.654076 |
-23.015387 |
17.448118 |
-7.612069 |
| 2019-01-03 |
3.190391 |
-2.493704 |
14.621079 |
-20.601407 |
| 2019-01-04 |
-3.224172 |
-3.840544 |
11.337694 |
-10.998913 |
| 2019-01-05 |
-1.724282 |
-8.778584 |
0.422137 |
5.828152 |
| 2019-01-06 |
-11.006192 |
11.447237 |
9.015907 |
5.024943 |
print(df.index)
DatetimeIndex(['2019-01-01', '2019-01-02', '2019-01-03', '2019-01-04',
'2019-01-05', '2019-01-06'],
dtype='datetime64[ns]', freq='D')
print(df.columns)
Index(['c1', 'c2', 'c3', 'c4'], dtype='object')
print(df.values)
[[ 16.24345364 -6.11756414 -5.28171752 -10.72968622]
[ 8.65407629 -23.01538697 17.44811764 -7.61206901]
[ 3.19039096 -2.49370375 14.62107937 -20.60140709]
[ -3.22417204 -3.84054355 11.33769442 -10.99891267]
[ -1.72428208 -8.77858418 0.42213747 5.82815214]
[-11.00619177 11.4472371 9.01590721 5.02494339]]
df.describe()
|
c1 |
c2 |
c3 |
c4 |
| count |
6.000000 |
6.000000 |
6.000000 |
6.000000 |
| mean |
2.022213 |
-5.466424 |
7.927203 |
-6.514830 |
| std |
9.580084 |
11.107772 |
8.707171 |
10.227641 |
| min |
-11.006192 |
-23.015387 |
-5.281718 |
-20.601407 |
| 25% |
-2.849200 |
-8.113329 |
2.570580 |
-10.931606 |
| 50% |
0.733054 |
-4.979054 |
10.176801 |
-9.170878 |
| 75% |
7.288155 |
-2.830414 |
13.800233 |
1.865690 |
| max |
16.243454 |
11.447237 |
17.448118 |
5.828152 |
df.T
|
2019-01-01 00:00:00 |
2019-01-02 00:00:00 |
2019-01-03 00:00:00 |
2019-01-04 00:00:00 |
2019-01-05 00:00:00 |
2019-01-06 00:00:00 |
| c1 |
16.243454 |
8.654076 |
3.190391 |
-3.224172 |
-1.724282 |
-11.006192 |
| c2 |
-6.117564 |
-23.015387 |
-2.493704 |
-3.840544 |
-8.778584 |
11.447237 |
| c3 |
-5.281718 |
17.448118 |
14.621079 |
11.337694 |
0.422137 |
9.015907 |
| c4 |
-10.729686 |
-7.612069 |
-20.601407 |
-10.998913 |
5.828152 |
5.024943 |
# 按行标签从大到小排序
df.sort_index(axis=0)
|
c1 |
c2 |
c3 |
c4 |
| 2019-01-01 |
16.243454 |
-6.117564 |
-5.281718 |
-10.729686 |
| 2019-01-02 |
8.654076 |
-23.015387 |
17.448118 |
-7.612069 |
| 2019-01-03 |
3.190391 |
-2.493704 |
14.621079 |
-20.601407 |
| 2019-01-04 |
-3.224172 |
-3.840544 |
11.337694 |
-10.998913 |
| 2019-01-05 |
-1.724282 |
-8.778584 |
0.422137 |
5.828152 |
| 2019-01-06 |
-11.006192 |
11.447237 |
9.015907 |
5.024943 |
# 按列标签从大到小排序
df2.sort_index(axis=1)
|
a |
b |
c |
d |
| 0 |
1 |
2 |
0 |
hello |
| 1 |
1 |
3 |
1 |
hello |
# 按a列的值从大到小排序
df2.sort_values(by='a')
|
a |
b |
c |
d |
| 0 |
1 |
2 |
0 |
hello |
| 1 |
1 |
3 |
1 |
hello |
DataFrame取值(掌握)
df
|
c1 |
c2 |
c3 |
c4 |
| 2019-01-01 |
16.243454 |
-6.117564 |
-5.281718 |
-10.729686 |
| 2019-01-02 |
8.654076 |
-23.015387 |
17.448118 |
-7.612069 |
| 2019-01-03 |
3.190391 |
-2.493704 |
14.621079 |
-20.601407 |
| 2019-01-04 |
-3.224172 |
-3.840544 |
11.337694 |
-10.998913 |
| 2019-01-05 |
-1.724282 |
-8.778584 |
0.422137 |
5.828152 |
| 2019-01-06 |
-11.006192 |
11.447237 |
9.015907 |
5.024943 |
df['c2']
2019-01-01 -6.117564
2019-01-02 -23.015387
2019-01-03 -2.493704
2019-01-04 -3.840544
2019-01-05 -8.778584
2019-01-06 11.447237
Freq: D, Name: c2, dtype: float64
df[0:3]
|
c1 |
c2 |
c3 |
c4 |
| 2019-01-01 |
16.243454 |
-6.117564 |
-5.281718 |
-10.729686 |
| 2019-01-02 |
8.654076 |
-23.015387 |
17.448118 |
-7.612069 |
| 2019-01-03 |
3.190391 |
-2.493704 |
14.621079 |
-20.601407 |
loc/iloc
# 通过自定义的行标签选择数据
df.loc['2019-01-01':'2019-01-05']
|
c1 |
c2 |
c3 |
c4 |
| 2019-01-01 |
16.243454 |
-6.117564 |
-5.281718 |
-10.729686 |
| 2019-01-02 |
8.654076 |
-23.015387 |
17.448118 |
-7.612069 |
| 2019-01-03 |
3.190391 |
-2.493704 |
14.621079 |
-20.601407 |
| 2019-01-04 |
-3.224172 |
-3.840544 |
11.337694 |
-10.998913 |
| 2019-01-05 |
-1.724282 |
-8.778584 |
0.422137 |
5.828152 |
df
|
c1 |
c2 |
c3 |
c4 |
| 2019-01-01 |
16.243454 |
-6.117564 |
-5.281718 |
-10.729686 |
| 2019-01-02 |
8.654076 |
-23.015387 |
17.448118 |
-7.612069 |
| 2019-01-03 |
3.190391 |
-2.493704 |
14.621079 |
-20.601407 |
| 2019-01-04 |
-3.224172 |
-3.840544 |
11.337694 |
-10.998913 |
| 2019-01-05 |
-1.724282 |
-8.778584 |
0.422137 |
5.828152 |
| 2019-01-06 |
-11.006192 |
11.447237 |
9.015907 |
5.024943 |
df.values
array([[ 16.24345364, -6.11756414, -5.28171752, -10.72968622],
[ 8.65407629, -23.01538697, 17.44811764, -7.61206901],
[ 3.19039096, -2.49370375, 14.62107937, -20.60140709],
[ -3.22417204, -3.84054355, 11.33769442, -10.99891267],
[ -1.72428208, -8.77858418, 0.42213747, 5.82815214],
[-11.00619177, 11.4472371 , 9.01590721, 5.02494339]])
print(df.iloc[2, 1])
-2.49370375477
# 通过行索引选择数据
print(df.iloc[2, 1])
-2.49370375477
df.iloc[1:4, 1:4]
|
c2 |
c3 |
c4 |
| 2019-01-02 |
-23.015387 |
17.448118 |
-7.612069 |
| 2019-01-03 |
-2.493704 |
14.621079 |
-20.601407 |
| 2019-01-04 |
-3.840544 |
11.337694 |
-10.998913 |
df
|
c1 |
c2 |
c3 |
c4 |
| 2019-01-01 |
16.243454 |
-6.117564 |
-5.281718 |
-10.729686 |
| 2019-01-02 |
8.654076 |
-23.015387 |
17.448118 |
-7.612069 |
| 2019-01-03 |
3.190391 |
-2.493704 |
14.621079 |
-20.601407 |
| 2019-01-04 |
-3.224172 |
-3.840544 |
11.337694 |
-10.998913 |
| 2019-01-05 |
-1.724282 |
-8.778584 |
0.422137 |
5.828152 |
| 2019-01-06 |
-11.006192 |
11.447237 |
9.015907 |
5.024943 |
使用逻辑判断取值
df[df['c1'] > 0]
|
c1 |
c2 |
c3 |
c4 |
| 2019-01-01 |
16.243454 |
-6.117564 |
-5.281718 |
-10.729686 |
| 2019-01-02 |
8.654076 |
-23.015387 |
17.448118 |
-7.612069 |
| 2019-01-03 |
3.190391 |
-2.493704 |
14.621079 |
-20.601407 |
DataFrame值替换(掌握)
df
|
c1 |
c2 |
c3 |
c4 |
| 2019-01-01 |
16.243454 |
-6.117564 |
-5.281718 |
-10.729686 |
| 2019-01-02 |
8.654076 |
-23.015387 |
17.448118 |
-7.612069 |
| 2019-01-03 |
3.190391 |
-2.493704 |
14.621079 |
-20.601407 |
| 2019-01-04 |
-3.224172 |
-3.840544 |
11.337694 |
-10.998913 |
| 2019-01-05 |
-1.724282 |
-8.778584 |
0.422137 |
5.828152 |
| 2019-01-06 |
-11.006192 |
11.447237 |
9.015907 |
5.024943 |
df.iloc[0:3, 0:2] = 0
df
|
c1 |
c2 |
c3 |
c4 |
| 2019-01-01 |
0.000000 |
0.000000 |
-5.281718 |
-10.729686 |
| 2019-01-02 |
0.000000 |
0.000000 |
17.448118 |
-7.612069 |
| 2019-01-03 |
0.000000 |
0.000000 |
14.621079 |
-20.601407 |
| 2019-01-04 |
-3.224172 |
-3.840544 |
11.337694 |
-10.998913 |
| 2019-01-05 |
-1.724282 |
-8.778584 |
0.422137 |
5.828152 |
| 2019-01-06 |
-11.006192 |
11.447237 |
9.015907 |
5.024943 |
df
|
c1 |
c2 |
c3 |
c4 |
| 2019-01-01 |
16.243454 |
-6.117564 |
-5.281718 |
-10.729686 |
| 2019-01-02 |
8.654076 |
-23.015387 |
17.448118 |
-7.612069 |
| 2019-01-03 |
3.190391 |
-2.493704 |
14.621079 |
-20.601407 |
| 2019-01-04 |
-3.224172 |
-3.840544 |
11.337694 |
-10.998913 |
| 2019-01-05 |
-1.724282 |
-8.778584 |
0.422137 |
5.828152 |
| 2019-01-06 |
-11.006192 |
11.447237 |
9.015907 |
5.024943 |
df[df['c1'] > 0] = 100
df
|
c1 |
c2 |
c3 |
c4 |
| 2019-01-01 |
100.000000 |
100.000000 |
100.000000 |
100.000000 |
| 2019-01-02 |
100.000000 |
100.000000 |
100.000000 |
100.000000 |
| 2019-01-03 |
100.000000 |
100.000000 |
100.000000 |
100.000000 |
| 2019-01-04 |
-3.224172 |
-3.840544 |
11.337694 |
-10.998913 |
| 2019-01-05 |
-1.724282 |
-8.778584 |
0.422137 |
5.828152 |
| 2019-01-06 |
-11.006192 |
11.447237 |
9.015907 |
5.024943 |
读取CSV文件(掌握)
from io import StringIO
test_data = '''
5.1,,1.4,0.2
4.9,3.0,1.4,0.2
4.7,3.2,,0.2
7.0,3.2,4.7,1.4
6.4,3.2,4.5,1.5
6.9,3.1,4.9,
,,,
'''
# df = pd.read_csv('C:/Users/test_data.csv')
test_data = StringIO(test_data)
df = pd.read_csv(test_data)
df = pd.read_excel(test_data)
df.columns = ['c1', 'c2', 'c3', 'c4']
df
|
c1 |
c2 |
c3 |
c4 |
| 0 |
4.9 |
3.0 |
1.4 |
0.2 |
| 1 |
4.7 |
3.2 |
NaN |
0.2 |
| 2 |
7.0 |
3.2 |
4.7 |
1.4 |
| 3 |
6.4 |
3.2 |
4.5 |
1.5 |
| 4 |
6.9 |
3.1 |
4.9 |
NaN |
| 5 |
NaN |
NaN |
NaN |
NaN |
处理丢失数据(掌握)
df.isnull()
|
c1 |
c2 |
c3 |
c4 |
| 0 |
False |
False |
False |
False |
| 1 |
False |
False |
True |
False |
| 2 |
False |
False |
False |
False |
| 3 |
False |
False |
False |
False |
| 4 |
False |
False |
False |
True |
| 5 |
True |
True |
True |
True |
# 通过在isnull()方法后使用sum()方法即可获得该数据集某个特征含有多少个缺失值
print(df.isnull().sum())
c1 1
c2 1
c3 2
c4 2
dtype: int64
df
|
c1 |
c2 |
c3 |
c4 |
| 0 |
4.9 |
3.0 |
1.4 |
0.2 |
| 1 |
4.7 |
3.2 |
NaN |
0.2 |
| 2 |
7.0 |
3.2 |
4.7 |
1.4 |
| 3 |
6.4 |
3.2 |
4.5 |
1.5 |
| 4 |
6.9 |
3.1 |
4.9 |
NaN |
| 5 |
NaN |
NaN |
NaN |
NaN |
# axis=0删除有NaN值的行
df.dropna(axis=0)
|
c1 |
c2 |
c3 |
c4 |
| 0 |
4.9 |
3.0 |
1.4 |
0.2 |
| 2 |
7.0 |
3.2 |
4.7 |
1.4 |
| 3 |
6.4 |
3.2 |
4.5 |
1.5 |
# axis=1删除有NaN值的列
df.dropna(axis=1)
# 删除全为NaN值得行或列
df.dropna(how='all')
|
c1 |
c2 |
c3 |
c4 |
| 0 |
4.9 |
3.0 |
1.4 |
0.2 |
| 1 |
4.7 |
3.2 |
NaN |
0.2 |
| 2 |
7.0 |
3.2 |
4.7 |
1.4 |
| 3 |
6.4 |
3.2 |
4.5 |
1.5 |
| 4 |
6.9 |
3.1 |
4.9 |
NaN |
# 删除行不为4个值的
df.dropna(thresh=4)
|
c1 |
c2 |
c3 |
c4 |
| 0 |
4.9 |
3.0 |
1.4 |
0.2 |
| 2 |
7.0 |
3.2 |
4.7 |
1.4 |
| 3 |
6.4 |
3.2 |
4.5 |
1.5 |
# 删除c2中有NaN值的数据
df.dropna(subset=['c2'])
|
c1 |
c2 |
c3 |
c4 |
| 0 |
4.9 |
3.0 |
1.4 |
0.2 |
| 1 |
4.7 |
3.2 |
NaN |
0.2 |
| 2 |
7.0 |
3.2 |
4.7 |
1.4 |
| 3 |
6.4 |
3.2 |
4.5 |
1.5 |
| 4 |
6.9 |
3.1 |
4.9 |
NaN |
df
|
c1 |
c2 |
c3 |
c4 |
| 0 |
4.9 |
3.0 |
1.4 |
0.2 |
| 1 |
4.7 |
3.2 |
NaN |
0.2 |
| 2 |
7.0 |
3.2 |
4.7 |
1.4 |
| 3 |
6.4 |
3.2 |
4.5 |
1.5 |
| 4 |
6.9 |
3.1 |
4.9 |
NaN |
| 5 |
NaN |
NaN |
NaN |
NaN |
# 填充nan值
df.fillna(value=10)
|
c1 |
c2 |
c3 |
c4 |
| 0 |
4.9 |
3.0 |
1.4 |
0.2 |
| 1 |
4.7 |
3.2 |
10.0 |
0.2 |
| 2 |
7.0 |
3.2 |
4.7 |
1.4 |
| 3 |
6.4 |
3.2 |
4.5 |
1.5 |
| 4 |
6.9 |
3.1 |
4.9 |
10.0 |
| 5 |
10.0 |
10.0 |
10.0 |
10.0 |
导入导出数据(掌握)
使用df = pd.read_csv(filename)读取文件，使用df.to_csv(filename)保存文件。
# df = pd.read_csv("filename")
# 进行一堆处理后
# df.to_csv("filename", header=True, index=False)
合并数据(掌握)
df1 = pd.DataFrame(np.zeros((3, 4)))
df1
|
0 |
1 |
2 |
3 |
| 0 |
0.0 |
0.0 |
0.0 |
0.0 |
| 1 |
0.0 |
0.0 |
0.0 |
0.0 |
| 2 |
0.0 |
0.0 |
0.0 |
0.0 |
df2 = pd.DataFrame(np.ones((3, 4)))
df2
|
0 |
1 |
2 |
3 |
| 0 |
1.0 |
1.0 |
1.0 |
1.0 |
| 1 |
1.0 |
1.0 |
1.0 |
1.0 |
| 2 |
1.0 |
1.0 |
1.0 |
1.0 |
# axis=0合并列
pd.concat((df1, df2), axis=0)
|
0 |
1 |
2 |
3 |
| 0 |
0.0 |
0.0 |
0.0 |
0.0 |
| 1 |
0.0 |
0.0 |
0.0 |
0.0 |
| 2 |
0.0 |
0.0 |
0.0 |
0.0 |
| 0 |
1.0 |
1.0 |
1.0 |
1.0 |
| 1 |
1.0 |
1.0 |
1.0 |
1.0 |
| 2 |
1.0 |
1.0 |
1.0 |
1.0 |
# axis=1合并行
pd.concat((df1, df2), axis=1)
|
0 |
1 |
2 |
3 |
0 |
1 |
2 |
3 |
| 0 |
0.0 |
0.0 |
0.0 |
0.0 |
1.0 |
1.0 |
1.0 |
1.0 |
| 1 |
0.0 |
0.0 |
0.0 |
0.0 |
1.0 |
1.0 |
1.0 |
1.0 |
| 2 |
0.0 |
0.0 |
0.0 |
0.0 |
1.0 |
1.0 |
1.0 |
1.0 |
# append只能合并列
df1.append(df2)
|
0 |
1 |
2 |
3 |
| 0 |
0.0 |
0.0 |
0.0 |
0.0 |
| 1 |
0.0 |
0.0 |
0.0 |
0.0 |
| 2 |
0.0 |
0.0 |
0.0 |
0.0 |
| 0 |
1.0 |
1.0 |
1.0 |
1.0 |
| 1 |
1.0 |
1.0 |
1.0 |
1.0 |
| 2 |
1.0 |
1.0 |
1.0 |
1.0 |
读取sql语句(熟悉)
import numpy as np
import pandas as pd
import pymysql
def conn(sql):
# 连接到mysql数据库
conn = pymysql.connect(
host="localhost",
port=3306,
user="root",
passwd="123",
db="db1",
)
try:
data = pd.read_sql(sql, con=conn)
return data
except Exception as e:
print("SQL is not correct!")
finally:
conn.close()
sql = "select * from test1 limit 0, 10" # sql语句
data = conn(sql)
print(data.columns.tolist()) # 查看字段
print(data) # 查看数据
matplotlib模块
matplotlib官方文档：https://matplotlib.org/contents.html?v=20190307135750
matplotlib是一个绘图库，它可以创建常用的统计图，包括条形图、箱型图、折线图、散点图和直方图。
条形图(掌握)
import matplotlib.pyplot as plt
from matplotlib.font_manager import FontProperties
%matplotlib inline
font = FontProperties(fname='/Library/Fonts/Heiti.ttc')
# 修改背景为条纹
plt.style.use('ggplot')
classes = ['3班', '4班', '5班', '6班']
classes_index = range(len(classes))
print(list(classes_index))
[0, 1, 2, 3]
student_amounts = [66, 55, 45, 70]
# 画布设置
fig = plt.figure()
# 1,1,1表示一张画布切割成1行1列共一张图的第1个；2,2,1表示一张画布切割成2行2列共4张图的第一个（左上角）
ax1 = fig.add_subplot(1, 1, 1)
ax1.bar(classes_index, student_amounts, align='center', color='darkblue')
ax1.xaxis.set_ticks_position('bottom')
ax1.yaxis.set_ticks_position('left')
plt.xticks(classes_index, classes, rotation=0,
fontsize=13, fontproperties=font)
plt.xlabel('班级', fontproperties=font, fontsize=15)
plt.ylabel('学生人数', fontproperties=font, fontsize=15)
plt.title('班级-学生人数', fontproperties=font, fontsize=20)
# 保存图片，bbox_inches='tight'去掉图形四周的空白
# plt.savefig('classes_students.png', dpi=400, bbox_inches='tight')
plt.show()
直方图(掌握)
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.font_manager import FontProperties
%matplotlib inline
font = FontProperties(fname='/Library/Fonts/Heiti.ttc')
# 修改背景为条纹
plt.style.use('ggplot')
mu1, mu2, sigma = 50, 100, 10
# 构造均值为50的符合正态分布的数据
x1 = mu1+sigma*np.random.randn(10000)
print(x1)
[59.00855949 43.16272141 48.77109774 ... 57.94645859 54.70312714
58.94125528]
# 构造均值为100的符合正态分布的数据
x2 = mu2+sigma*np.random.randn(10000)
print(x2)
[115.19915511 82.09208214 110.88092454 ... 95.0872103 104.21549068
133.36025251]
fig = plt.figure()
ax1 = fig.add_subplot(121)
# bins=50表示每个变量的值分成50份，即会有50根柱子
ax1.hist(x1, bins=50, color='darkgreen')
ax2 = fig.add_subplot(122)
ax2.hist(x2, bins=50, color='orange')
fig.suptitle('两个正态分布', fontproperties=font, fontweight='bold', fontsize=15)
ax1.set_title('绿色的正态分布', fontproperties=font)
ax2.set_title('橙色的正态分布', fontproperties=font)
plt.show()
折线图(掌握)
import numpy as np
from numpy.random import randn
import matplotlib.pyplot as plt
from matplotlib.font_manager import FontProperties
%matplotlib inline
font = FontProperties(fname='/Library/Fonts/Heiti.ttc')
# 修改背景为条纹
plt.style.use('ggplot')
np.random.seed(1)
# 使用numpy的累加和，保证数据取值范围不会在（0，1）内波动
plot_data1 = randn(40).cumsum()
print(plot_data1)
[ 1.62434536 1.01258895 0.4844172 -0.58855142 0.2768562 -2.02468249
-0.27987073 -1.04107763 -0.72203853 -0.97140891 0.49069903 -1.56944168
-1.89185888 -2.27591324 -1.1421438 -2.24203506 -2.41446327 -3.29232169
-3.25010794 -2.66729273 -3.76791191 -2.6231882 -1.72159748 -1.21910314
-0.31824719 -1.00197505 -1.12486527 -2.06063471 -2.32852279 -1.79816732
-2.48982807 -2.8865816 -3.5737543 -4.41895994 -5.09020607 -5.10287067
-6.22018102 -5.98576532 -4.32596314 -3.58391898]
plot_data2 = randn(40).cumsum()
plot_data3 = randn(40).cumsum()
plot_data4 = randn(40).cumsum()
plt.plot(plot_data1, marker='o', color='red', linestyle='-', label='红实线')
plt.plot(plot_data2, marker='x', color='orange', linestyle='--', label='橙虚线')
plt.plot(plot_data3, marker='*', color='yellow', linestyle='-.', label='黄点线')
plt.plot(plot_data4, marker='s', color='green', linestyle=':', label='绿点图')
# loc='best'给label自动选择最好的位置
plt.legend(loc='best', prop=font)
plt.show()
散点图+直线图(掌握)
import numpy as np
from numpy.random import randn
import matplotlib.pyplot as plt
from matplotlib.font_manager import FontProperties
%matplotlib inline
font = FontProperties(fname='/Library/Fonts/Heiti.ttc')
# 修改背景为条纹
plt.style.use('ggplot')
x = np.arange(1, 20, 1)
print(x)
[ 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19]
# 拟合一条水平散点线
np.random.seed(1)
y_linear = x+10*np.random.randn(19)
print(y_linear)
[ 17.24345364 -4.11756414 -2.28171752 -6.72968622 13.65407629
-17.01538697 24.44811764 0.38793099 12.19039096 7.50629625
25.62107937 -8.60140709 9.77582796 10.15945645 26.33769442
5.00108733 15.27571792 9.22141582 19.42213747]
# 拟合一条x²的散点线
y_quad = x**2+10*np.random.randn(19)
print(y_quad)
[ 6.82815214 -7.00619177 20.4472371 25.01590721 30.02494339
45.00855949 42.16272141 62.77109774 71.64230566 97.3211192
126.30355467 137.08339248 165.03246473 189.128273 216.54794359
249.28753869 288.87335401 312.82689651 363.34415698]
# s是散点大小
fig = plt.figure()
ax1 = fig.add_subplot(121)
plt.scatter(x, y_linear, s=30, color='r', label='蓝点')
plt.scatter(x, y_quad, s=100, color='b', label='红点')
ax2 = fig.add_subplot(122)
plt.plot(x, y_linear, color='r')
plt.plot(x, y_quad, color='b')
# 限制x轴和y轴的范围取值
plt.xlim(min(x)-1, max(x)+1)
plt.ylim(min(y_quad)-10, max(y_quad)+10)
fig.suptitle('散点图+直线图', fontproperties=font, fontsize=20)
ax1.set_title('散点图', fontproperties=font)
ax1.legend(prop=font)
ax2.set_title('直线图', fontproperties=font)
plt.show()
