Python Pandas高级教程之时间处理
简介
时间应该是在数据处理中经常会用到的一种数据类型,除了numpy中datetime64 和 timedelta64 这两种数据类型之外,pandas 还整合了其他python库比如 scikits.timeseries 中的功能。
时间分类
pandas中有四种时间类型:
- date times : 日期和时间,可以带时区。和标准库中的 datetime.datetime 类似。
- time deltas: 绝对持续时间,和 标准库中的 datetime.timedelta 类似。
- time spans: 由时间点及其关联的频率定义的时间跨度。
- date offsets:基于日历计算的时间 和 dateutil.relativedelta.relativedelta 类似。
我们用一张表来表示:
类型 | 标量class | 数组class | pandas数据类型 | 主要创建方法 |
---|---|---|---|---|
date times | timestamp | datetimeindex | datetime64[ns] or datetime64[ns, tz] | to_datetime or date_range |
time deltas | timedelta | timedeltaindex | timedelta64[ns] | to_timedelta or timedelta_range |
time spans | period | periodindex | period[freq] | period or period_range |
date offsets | dateoffset | none | none | dateoffset |
看一个使用的例子:
in [19]: pd.series(range(3), index=pd.date_range("2000", freq="d", periods=3)) out[19]: 2000-01-01 0 2000-01-02 1 2000-01-03 2 freq: d, dtype: int64
看一下上面数据类型的空值:
in [24]: pd.timestamp(pd.nat) out[24]: nat in [25]: pd.timedelta(pd.nat) out[25]: nat in [26]: pd.period(pd.nat) out[26]: nat # equality acts as np.nan would in [27]: pd.nat == pd.nat out[27]: false
timestamp
timestamp 是最基础的时间类型,我们可以这样创建:
in [28]: pd.timestamp(datetime.datetime(2012, 5, 1)) out[28]: timestamp('2012-05-01 00:00:00') in [29]: pd.timestamp("2012-05-01") out[29]: timestamp('2012-05-01 00:00:00') in [30]: pd.timestamp(2012, 5, 1) out[30]: timestamp('2012-05-01 00:00:00')
datetimeindex
timestamp 作为index会自动被转换为datetimeindex:
in [33]: dates = [ ....: pd.timestamp("2012-05-01"), ....: pd.timestamp("2012-05-02"), ....: pd.timestamp("2012-05-03"), ....: ] ....: in [34]: ts = pd.series(np.random.randn(3), dates) in [35]: type(ts.index) out[35]: pandas.core.indexes.datetimes.datetimeindex in [36]: ts.index out[36]: datetimeindex(['2012-05-01', '2012-05-02', '2012-05-03'], dtype='datetime64[ns]', freq=none) in [37]: ts out[37]: 2012-05-01 0.469112 2012-05-02 -0.282863 2012-05-03 -1.509059 dtype: float64
date_range 和 bdate_range
还可以使用 date_range 来创建datetimeindex:
in [74]: start = datetime.datetime(2011, 1, 1) in [75]: end = datetime.datetime(2012, 1, 1) in [76]: index = pd.date_range(start, end) in [77]: index out[77]: datetimeindex(['2011-01-01', '2011-01-02', '2011-01-03', '2011-01-04', '2011-01-05', '2011-01-06', '2011-01-07', '2011-01-08', '2011-01-09', '2011-01-10', ... '2011-12-23', '2011-12-24', '2011-12-25', '2011-12-26', '2011-12-27', '2011-12-28', '2011-12-29', '2011-12-30', '2011-12-31', '2012-01-01'], dtype='datetime64[ns]', length=366, freq='d')
date_range 是日历范围,bdate_range 是工作日范围:
in [78]: index = pd.bdate_range(start, end) in [79]: index out[79]: datetimeindex(['2011-01-03', '2011-01-04', '2011-01-05', '2011-01-06', '2011-01-07', '2011-01-10', '2011-01-11', '2011-01-12', '2011-01-13', '2011-01-14', ... '2011-12-19', '2011-12-20', '2011-12-21', '2011-12-22', '2011-12-23', '2011-12-26', '2011-12-27', '2011-12-28', '2011-12-29', '2011-12-30'], dtype='datetime64[ns]', length=260, freq='b')
两个方法都可以带上 start, end, 和 periods 参数。
in [84]: pd.bdate_range(end=end, periods=20) in [83]: pd.date_range(start, end, freq="w") in [86]: pd.date_range("2018-01-01", "2018-01-05", periods=5)
origin
使用 origin参数,可以修改 datetimeindex 的起点:
in [67]: pd.to_datetime([1, 2, 3], unit="d", origin=pd.timestamp("1960-01-01")) out[67]: datetimeindex(['1960-01-02', '1960-01-03', '1960-01-04'], dtype='datetime64[ns]', freq=none)
默认情况下 origin='unix', 也就是起点是 1970-01-01 00:00:00.
in [68]: pd.to_datetime([1, 2, 3], unit="d") out[68]: datetimeindex(['1970-01-02', '1970-01-03', '1970-01-04'], dtype='datetime64[ns]', freq=none)
格式化
使用format参数可以对时间进行格式化:
in [51]: pd.to_datetime("2010/11/12", format="%y/%m/%d") out[51]: timestamp('2010-11-12 00:00:00') in [52]: pd.to_datetime("12-11-2010 00:00", format="%d-%m-%y %h:%m") out[52]: timestamp('2010-11-12 00:00:00')
period
period 表示的是一个时间跨度,通常和freq一起使用:
in [31]: pd.period("2011-01") out[31]: period('2011-01', 'm') in [32]: pd.period("2012-05", freq="d") out[32]: period('2012-05-01', 'd')
period可以直接进行运算:
in [345]: p = pd.period("2012", freq="a-dec") in [346]: p + 1 out[346]: period('2013', 'a-dec') in [347]: p - 3 out[347]: period('2009', 'a-dec') in [348]: p = pd.period("2012-01", freq="2m") in [349]: p + 2 out[349]: period('2012-05', '2m') in [350]: p - 1 out[350]: period('2011-11', '2m')
注意,period只有具有相同的freq才能进行算数运算。包括 offsets 和 timedelta
in [352]: p = pd.period("2014-07-01 09:00", freq="h") in [353]: p + pd.offsets.hour(2) out[353]: period('2014-07-01 11:00', 'h') in [354]: p + datetime.timedelta(minutes=120) out[354]: period('2014-07-01 11:00', 'h') in [355]: p + np.timedelta64(7200, "s") out[355]: period('2014-07-01 11:00', 'h')
period作为index可以自动被转换为periodindex:
in [38]: periods = [pd.period("2012-01"), pd.period("2012-02"), pd.period("2012-03")] in [39]: ts = pd.series(np.random.randn(3), periods) in [40]: type(ts.index) out[40]: pandas.core.indexes.period.periodindex in [41]: ts.index out[41]: periodindex(['2012-01', '2012-02', '2012-03'], dtype='period[m]', freq='m') in [42]: ts out[42]: 2012-01 -1.135632 2012-02 1.212112 2012-03 -0.173215 freq: m, dtype: float64
可以通过 pd.period_range 方法来创建 periodindex:
in [359]: prng = pd.period_range("1/1/2011", "1/1/2012", freq="m") in [360]: prng out[360]: periodindex(['2011-01', '2011-02', '2011-03', '2011-04', '2011-05', '2011-06', '2011-07', '2011-08', '2011-09', '2011-10', '2011-11', '2011-12', '2012-01'], dtype='period[m]', freq='m')
还可以通过periodindex直接创建:
in [361]: pd.periodindex(["2011-1", "2011-2", "2011-3"], freq="m") out[361]: periodindex(['2011-01', '2011-02', '2011-03'], dtype='period[m]', freq='m')
dateoffset
dateoffset表示的是频率对象。它和timedelta很类似,表示的是一个持续时间,但是有特殊的日历规则。比如timedelta一天肯定是24小时,而在 dateoffset中根据夏令时的不同,一天可能会有23,24或者25小时。
# this particular day contains a day light savings time transition in [144]: ts = pd.timestamp("2016-10-30 00:00:00", tz="europe/helsinki") # respects absolute time in [145]: ts + pd.timedelta(days=1) out[145]: timestamp('2016-10-30 23:00:00+0200', tz='europe/helsinki') # respects calendar time in [146]: ts + pd.dateoffset(days=1) out[146]: timestamp('2016-10-31 00:00:00+0200', tz='europe/helsinki') in [147]: friday = pd.timestamp("2018-01-05") in [148]: friday.day_name() out[148]: 'friday' # add 2 business days (friday --> tuesday) in [149]: two_business_days = 2 * pd.offsets.bday() in [150]: two_business_days.apply(friday) out[150]: timestamp('2018-01-09 00:00:00') in [151]: friday + two_business_days out[151]: timestamp('2018-01-09 00:00:00') in [152]: (friday + two_business_days).day_name() out[152]: 'tuesday'
dateoffsets 和frequency 运算是先关的,看一下可用的date offset 和它相关联的 frequency:
date offset | frequency string | 描述 |
---|---|---|
dateoffset | none | 通用的offset 类 |
bday or businessday | 'b' | 工作日 |
cday or custombusinessday | 'c' | 自定义的工作日 |
week | 'w' | 一周 |
weekofmonth | 'wom' | 每个月的第几周的第几天 |
lastweekofmonth | 'lwom' | 每个月最后一周的第几天 |
monthend | 'm' | 日历月末 |
monthbegin | 'ms' | 日历月初 |
bmonthend or businessmonthend | 'bm' | 营业月底 |
bmonthbegin or businessmonthbegin | 'bms' | 营业月初 |
cbmonthend or custombusinessmonthend | 'cbm' | 自定义营业月底 |
cbmonthbegin or custombusinessmonthbegin | 'cbms' | 自定义营业月初 |
semimonthend | 'sm' | 日历月末的第15天 |
semimonthbegin | 'sms' | 日历月初的第15天 |
quarterend | 'q' | 日历季末 |
quarterbegin | 'qs' | 日历季初 |
bquarterend | 'bq | 工作季末 |
bquarterbegin | 'bqs' | 工作季初 |
fy5253quarter | 'req' | 零售季( 52-53 week) |
yearend | 'a' | 日历年末 |
yearbegin | 'as' or 'bys' | 日历年初 |
byearend | 'ba' | 营业年末 |
byearbegin | 'bas' | 营业年初 |
fy5253 | 're' | 零售年 (aka 52-53 week) |
easter | none | 复活节假期 |
businesshour | 'bh' | business hour |
custombusinesshour | 'cbh' | custom business hour |
day | 'd' | 一天的绝对时间 |
hour | 'h' | 一小时 |
minute | 't' or 'min' | 一分钟 |
second | 's' | 一秒钟 |
milli | 'l' or 'ms' | 一微妙 |
micro | 'u' or 'us' | 一毫秒 |
nano | 'n' | 一纳秒 |
dateoffset还有两个方法 rollforward() 和 rollback() 可以将时间进行移动:
in [153]: ts = pd.timestamp("2018-01-06 00:00:00") in [154]: ts.day_name() out[154]: 'saturday' # businesshour's valid offset dates are monday through friday in [155]: offset = pd.offsets.businesshour(start="09:00") # bring the date to the closest offset date (monday) in [156]: offset.rollforward(ts) out[156]: timestamp('2018-01-08 09:00:00') # date is brought to the closest offset date first and then the hour is added in [157]: ts + offset out[157]: timestamp('2018-01-08 10:00:00')
上面的操作会自动保存小时,分钟等信息,如果想要设置为 00:00:00 , 可以调用normalize() 方法:
in [158]: ts = pd.timestamp("2014-01-01 09:00") in [159]: day = pd.offsets.day() in [160]: day.apply(ts) out[160]: timestamp('2014-01-02 09:00:00') in [161]: day.apply(ts).normalize() out[161]: timestamp('2014-01-02 00:00:00') in [162]: ts = pd.timestamp("2014-01-01 22:00") in [163]: hour = pd.offsets.hour() in [164]: hour.apply(ts) out[164]: timestamp('2014-01-01 23:00:00') in [165]: hour.apply(ts).normalize() out[165]: timestamp('2014-01-01 00:00:00') in [166]: hour.apply(pd.timestamp("2014-01-01 23:30")).normalize() out[166]: timestamp('2014-01-02 00:00:00')
作为index
时间可以作为index,并且作为index的时候会有一些很方便的特性。
可以直接使用时间来获取相应的数据:
in [99]: ts["1/31/2011"] out[99]: 0.11920871129693428 in [100]: ts[datetime.datetime(2011, 12, 25):] out[100]: 2011-12-30 0.56702 freq: bm, dtype: float64 in [101]: ts["10/31/2011":"12/31/2011"] out[101]: 2011-10-31 0.271860 2011-11-30 -0.424972 2011-12-30 0.567020 freq: bm, dtype: float64
获取全年的数据:
in [102]: ts["2011"] out[102]: 2011-01-31 0.119209 2011-02-28 -1.044236 2011-03-31 -0.861849 2011-04-29 -2.104569 2011-05-31 -0.494929 2011-06-30 1.071804 2011-07-29 0.721555 2011-08-31 -0.706771 2011-09-30 -1.039575 2011-10-31 0.271860 2011-11-30 -0.424972 2011-12-30 0.567020 freq: bm, dtype: float64
获取某个月的数据:
in [103]: ts["2011-6"] out[103]: 2011-06-30 1.071804 freq: bm, dtype: float64
df可以接受时间作为loc的参数:
in [105]: dft out[105]: a 2013-01-01 00:00:00 0.276232 2013-01-01 00:01:00 -1.087401 2013-01-01 00:02:00 -0.673690 2013-01-01 00:03:00 0.113648 2013-01-01 00:04:00 -1.478427 ... ... 2013-03-11 10:35:00 -0.747967 2013-03-11 10:36:00 -0.034523 2013-03-11 10:37:00 -0.201754 2013-03-11 10:38:00 -1.509067 2013-03-11 10:39:00 -1.693043 [100000 rows x 1 columns] in [106]: dft.loc["2013"] out[106]: a 2013-01-01 00:00:00 0.276232 2013-01-01 00:01:00 -1.087401 2013-01-01 00:02:00 -0.673690 2013-01-01 00:03:00 0.113648 2013-01-01 00:04:00 -1.478427 ... ... 2013-03-11 10:35:00 -0.747967 2013-03-11 10:36:00 -0.034523 2013-03-11 10:37:00 -0.201754 2013-03-11 10:38:00 -1.509067 2013-03-11 10:39:00 -1.693043 [100000 rows x 1 columns]
时间切片:
in [107]: dft["2013-1":"2013-2"] out[107]: a 2013-01-01 00:00:00 0.276232 2013-01-01 00:01:00 -1.087401 2013-01-01 00:02:00 -0.673690 2013-01-01 00:03:00 0.113648 2013-01-01 00:04:00 -1.478427 ... ... 2013-02-28 23:55:00 0.850929 2013-02-28 23:56:00 0.976712 2013-02-28 23:57:00 -2.693884 2013-02-28 23:58:00 -1.575535 2013-02-28 23:59:00 -1.573517 [84960 rows x 1 columns]
切片和完全匹配
考虑下面的一个精度为分的series对象:
in [120]: series_minute = pd.series( .....: [1, 2, 3], .....: pd.datetimeindex( .....: ["2011-12-31 23:59:00", "2012-01-01 00:00:00", "2012-01-01 00:02:00"] .....: ), .....: ) .....: in [121]: series_minute.index.resolution out[121]: 'minute'
时间精度小于分的话,返回的是一个series对象:
in [122]: series_minute["2011-12-31 23"] out[122]: 2011-12-31 23:59:00 1 dtype: int64
时间精度大于分的话,返回的是一个常量:
in [123]: series_minute["2011-12-31 23:59"] out[123]: 1 in [124]: series_minute["2011-12-31 23:59:00"] out[124]: 1
同样的,如果精度为秒的话,小于秒会返回一个对象,等于秒会返回常量值。
时间序列的操作
shifting
使用shift方法可以让 time series 进行相应的移动:
in [275]: ts = pd.series(range(len(rng)), index=rng) in [276]: ts = ts[:5] in [277]: ts.shift(1) out[277]: 2012-01-01 nan 2012-01-02 0.0 2012-01-03 1.0 freq: d, dtype: float64
通过指定 freq , 可以设置shift的方式:
in [278]: ts.shift(5, freq="d") out[278]: 2012-01-06 0 2012-01-07 1 2012-01-08 2 freq: d, dtype: int64 in [279]: ts.shift(5, freq=pd.offsets.bday()) out[279]: 2012-01-06 0 2012-01-09 1 2012-01-10 2 dtype: int64 in [280]: ts.shift(5, freq="bm") out[280]: 2012-05-31 0 2012-05-31 1 2012-05-31 2 dtype: int64
频率转换
时间序列可以通过调用 asfreq 的方法转换其频率:
in [281]: dr = pd.date_range("1/1/2010", periods=3, freq=3 * pd.offsets.bday()) in [282]: ts = pd.series(np.random.randn(3), index=dr) in [283]: ts out[283]: 2010-01-01 1.494522 2010-01-06 -0.778425 2010-01-11 -0.253355 freq: 3b, dtype: float64 in [284]: ts.asfreq(pd.offsets.bday()) out[284]: 2010-01-01 1.494522 2010-01-04 nan 2010-01-05 nan 2010-01-06 -0.778425 2010-01-07 nan 2010-01-08 nan 2010-01-11 -0.253355 freq: b, dtype: float64
asfreq还可以指定修改频率过后的填充方法:
in [285]: ts.asfreq(pd.offsets.bday(), method="pad") out[285]: 2010-01-01 1.494522 2010-01-04 1.494522 2010-01-05 1.494522 2010-01-06 -0.778425 2010-01-07 -0.778425 2010-01-08 -0.778425 2010-01-11 -0.253355 freq: b, dtype: float64
resampling 重新取样
给定的时间序列可以通过调用resample方法来重新取样:
in [286]: rng = pd.date_range("1/1/2012", periods=100, freq="s") in [287]: ts = pd.series(np.random.randint(0, 500, len(rng)), index=rng) in [288]: ts.resample("5min").sum() out[288]: 2012-01-01 25103 freq: 5t, dtype: int64
resample 可以接受各类统计方法,比如: sum, mean, std, sem, max, min, median, first, last, ohlc。
in [289]: ts.resample("5min").mean() out[289]: 2012-01-01 251.03 freq: 5t, dtype: float64 in [290]: ts.resample("5min").ohlc() out[290]: open high low close 2012-01-01 308 460 9 205 in [291]: ts.resample("5min").max() out[291]: 2012-01-01 460 freq: 5t, dtype: int64
总结
到此这篇关于python pandas高级教程之时间处理的文章就介绍到这了,更多相关pandas时间处理内容请搜索以前的文章或继续浏览下面的相关文章希望大家以后多多支持!
上一篇: Python深度学习pytorch卷积神经网络LeNet
下一篇: C语言文件操作详解