欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识！
您现在的位置是： 首页

M5 Forecasting - Accuracy：Multi-time series prediction with prophet

程序员文章站 2022-07-14 14:18:44
...

文章目录

总览

【目标】：预测最细粒度的商品id30490个时间序列数据，属于多时间序列预测问题。
【难点】：最细粒度的销量趋势不明显，大量0值存在。
【分析】：大粒度的ID聚合销量序列呈现明显的年、季度、趋势等周期特整。如何将聚合的稳定趋势预测问题转化为细粒度趋势预测？
【转化】：

  • 计算聚合id下销量，并用Prophet预测聚合ID下28天的销量。
  • 计算真实销量，针对某一聚合品类，分别计算其对应的细粒度商品id的真实销量占该品类真实销量的比例R。
  • 转化未来1天的销量预测值 = R * 未来1天的该品类的聚合销量。

细节：

【假设】：聚合id为dept_id（部门id），则不同取值有7个品类（FOODS_1、FOODS_2、FOODS_3、HOBBIES_1、HOBBIES_2、HOUSEHOLD_1、HOUSEHOLD_2）。
【Prophet预测】：得到7个时间序列（1-1914天）未来28天（1914-1941）的聚合销量数据。
【问题】：如何计算未来1天的某个商品id的销量？
【详情】：

1、目标：预测品类FOODS_1下某一个商品id=id_x的1941th的销量。
2、过程：
1）将品类=FOODS_1的原始销量数据全部拿出来，分别计算该品类下每个商品最近28天（1914th以前的28天）的销量和，并将商品id=id_x对应的销量和记为SUM1_idx1_FOODS1。计算该类目下最近28天所有商品的销量和为SUM2_FOODS1。则商品id_x1在该品类下的占比R_idx1_FOODS1 = SUM1_idx1_FOODS1 / SUM2_FOODS1。

2）Prophet预测品类=FOODS_1的一个聚合时间序列在1914的销量和Pre_FOODS1。
3）【id_x1在1914th的预测销量】= R_idx1_FOODS1 * Pre_FOODS1 = （SUM1_idx1_FOODS1 / SUM2_FOODS1）* Pre_FOODS1。

【疑问1】：为什么1）中计算最近的28天销量和？用最近一天计算的id销量占比转移到未来一天（这里假设未来一天id销量占比与最近的占比一致）不可以吗？
【答案】：案例来说可以。但是不稳定。最近一天商品id的销量计算带有不稳定性。因此计算的时间更长一些，也可以换成7、14等。

【疑问2】：为什么前面这么大的篇幅都是处理数据形态，变成那么复杂的 `多级索引`？
【答案】：数据量比较大，耗内存，其实主要是各种ID，将其转为多级索引，减少内存，且增加查询效率。

【疑问3】：不同的聚合ID选取的标准是什么？
【答案】：源于官方的评价标准的说明，评价标准考察了这些聚合维度销量 + 最细粒度向量。在这里选择了9个level的聚合方向，也就是说我们可以预测9种提交结果。

【疑问4】：这9种提交结果怎么处理？
【答案】：分别提交查看结果好坏。或者将所有结果求平均提交。

# Models
# --------
# FBProphet
# Loader/fast aggregrations from: https://www.kaggle.com/christoffer/pandas-multi-indices-for-hts-fast-loading-etc

# FPBTD_v1 Level 9, LB=0.603
# yearly_seasonality=2 weekly_seasonality=1, n_changepoints = 50, LRMSE=12.128

# FPBTD_v2 - Levels 1 to 9
# yearly_seasonality=10 weekly_seasonality=3, n_changepoints = 50, US holidays added, LRMSE=11.824
# lag365, lag28_roll28 min and max

# FPBTD_v3 - Levels 1 to 5
# Same as previous with quarter_fourier_order=4, L6-L6=0.614

# FPBTD_v4 - Levels 1 to 9
# quarter_fourier_order=8 + cumulated price, L6-LB=0.616

# FPBTD_v5 - Levels 1 to 9
# quarter_fourier_order=6, outliers removed, L6-LB=0.637

# FPBTD_v6 - Levels 1 to 9
# quarter_fourier_order=6, outliers removed + christmas, L6-LB=0.642

# FPBTD_v7 - Levels 1 to 9
# 2013, quarter_fourier_order=6, outliers removed + christmas, L6-LB=0.61x

# FPBTD_v8 - Levels 1 to 9
# 2014, quarter_fourier_order=6, outliers removed + christmas, L6-LB=0.63x

# FPBTD_v9 - Levels 1 to 9
# 2012, quarter_fourier_order=6, outliers removed + christmas, L6-LB=0.618

# FPBTD_v10 - Levels 1 to 9
# 2012, quarter_fourier_order=6, additional lags, L6-LB=

# !conda install -c conda-forge fbprophet

import os, sys, random, gc, math, glob, time
import numpy as np
import pandas as pd
import io, timeit, os, gc, pickle, psutil
import joblib
import time
from tqdm import tqdm
from matplotlib import cm
from datetime import datetime, timedelta
import warnings
from multiprocessing import Pool, cpu_count
from joblib import Parallel, delayed
from functools import partial
from collections import OrderedDict
# from tqdm.contrib.concurrent import process_map

# warnings.filterwarnings('ignore')
pd.set_option('display.max_colwidth', None)
pd.set_option('display.max_columns', 500)
pd.set_option('display.max_rows', 4000)

# !pip install psutil

seed = 2020
random.seed(seed)
np.random.seed(seed)

import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
DEFAULT_FIG_WIDTH = 20
sns.set_context("paper", font_scale=1.2) 

from sklearn.model_selection import train_test_split
from sklearn.model_selection import GridSearchCV, StratifiedKFold, TimeSeriesSplit, KFold, GroupKFold, ShuffleSplit
from sklearn.metrics import accuracy_score, precision_score, cohen_kappa_score, confusion_matrix
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error, mean_squared_log_error, mean_absolute_error
from sklearn.preprocessing import LabelEncoder
from sklearn.metrics import mean_squared_error
import csv
from collections import defaultdict

#import lightgbm as lgb

print('Python    : ' + sys.version.split('\n')[0])
print('Numpy     : ' + np.__version__)
print('Pandas    : ' + pd.__version__)
#print('LightGBM  : ' + lgb.__version__)

Python    : 3.6.5 | packaged by conda-forge | (default, Apr  6 2018, 13:39:56) 
Numpy     : 1.18.5
Pandas    : 1.0.5

# !pip install fbprophet --upgrade

import fbprophet
from fbprophet import Prophet
print('Prophet  : ' + fbprophet.__version__)

Importing plotly failed. Interactive plots will not work.


Prophet  : 0.6

数据说明

The data for the competition consists primarily of 30490 time series of sales data for 3049 items sold in 10 different stores in 3 states. The items are classified as being in one of 3 categories that are further subdivided into a total of 7 departments.

The representation we’ll look at in this notebook is representing each individual time series as a column in a data frame indexed by the day (d).

For the individual (level 12 series), we’ll index the series in the columns by (state_id, store_id, cat_id, dept_id, item_id).

  • 竞争的数据主要包括30490个时间序列的销售数据，涉及3个州10个不同商店销售的3049种商品。这些商品被划分为3个类别中的一个，再进一步细分为共7个部门。

  • 我们在这个笔记本中要看的表示方法是将每个单独的时间序列表示为一个数据框架中的一列，以日（d）为索引。

  • 对于个体（12级系列），我们将在列中按（state_id，store_id，cat_id，dept_id，item_id）来索引系列。

总结：

  • 行索引为日期
  • 列索引为各种id

1. 创建数据

Using Pandas directly to read the data and reshape it appears to be a bit slow and uses a significant amount of memory. Instead we’ll read the data line by line and store it in NumPy arrays (but we’ll try and keep the rest of the code in the notebook nicely vectorized and high-level =).

直接使用Pandas来读取数据并重塑数据似乎有点慢，而且会使用大量的内存。取而代之的是，我们将逐行读取数据，并将其存储在NumPy数组中（但我们会尽量让笔记本中的其他代码保持很好的矢量化和高级化=）。

HOME =  "./"
DATA_HOME = "../data/"
TRAIN_DATA_HOME = DATA_HOME

CALENDAR = DATA_HOME + "calendar.csv"
SALES = DATA_HOME + "sales_train_validation.csv"
PRICES = DATA_HOME + "sell_prices.csv"

MODELS_DIR = "models"
if not os.path.exists(MODELS_DIR):
    os.makedirs(MODELS_DIR)
            
NUM_SERIES = 30490
NUM_TRAINING = 1913
NUM_TEST = NUM_TRAINING + 2 * 28   # 初赛与复赛均是预测28天
MAX_LEVEL = 9 #9

# Load data
series_ids = np.empty(NUM_SERIES, dtype=object)
item_ids = np.empty(NUM_SERIES, dtype=object)
dept_ids = np.empty(NUM_SERIES, dtype=object)
cat_ids = np.empty(NUM_SERIES, dtype=object)
store_ids = np.empty(NUM_SERIES, dtype=object)
state_ids = np.empty(NUM_SERIES, dtype=object)

# 行为日期、列为各个商品
qties = np.zeros((NUM_TRAINING, NUM_SERIES), dtype=float) 
sell_prices = np.zeros((NUM_TEST, NUM_SERIES), dtype=float)

1.1 导入和重塑销售数据

  • 逐行读取数据，同时将id与销量分裂开，放到不同的容器内。
  • 销售数据中的每一行都由六个列组成，为一个系列的id，连同项目、部门、类别、商店、和、州五个层次。
# Sales
id_idx = {}
with open(SALES, "r", newline='') as f:
    is_header = True
    i = 0
    for row in tqdm(csv.reader(f)):
        if is_header:
            is_header = False
            continue
        series_id, item_id, dept_id, cat_id, store_id, state_id = row[0:6]
        # Remove '_validation/_evaluation' at end by regenerating series_id
        series_id = f"{item_id}_{store_id}"

        qty = np.array(row[6:], dtype=float)

        series_ids[i] = series_id

        item_ids[i] = item_id
        dept_ids[i] = dept_id
        cat_ids[i] = cat_id
        store_ids[i] = store_id
        state_ids[i] = state_id

        qties[:, i] = qty # 将一个商品的时间序列竖着放

        id_idx[series_id] = i # 哈希表————用***来编码30490个商品id

        i += 1
        
print('销量数据qties的shape为：', qties.shape)
print('销量数据商品HOBBIES_1_001_CA_1对应的编码id为：', id_idx['HOBBIES_1_001_CA_1'])

30491it [00:12, 2532.59it/s]

销量数据qties的shape为： (1913, 30490)
销量数据商品HOBBIES_1_001_CA_1对应的编码id为： 0

1.2 导入日历数据

日历数据有关于某一天是一周中的哪一天，是否有任何特殊事件的信息，最重要的是对于这个笔记本来说，这一天是在哪一周（wm_yr_wk）。我们需要这些来获得物品的价格，而这又是计算我们估算分数所需的权重所必需的。

  • wm_yr_wk_idx：盛放周序号对应的第几天（每个周序号都有七天）
# Calendar
wm_yr_wk_idx = defaultdict(list)  # map wmyrwk to d:s
with open(CALENDAR, "r", newline='') as f:
    for row in tqdm(csv.DictReader(f)):
        d = int(row['d'][2:]) # 将d列中"d_1"的1剥离出来，重新设置为d
        wm_yr_wk_idx[row['wm_yr_wk']].append(d)
        
print('周序号="11101"，对应的七天为：')
print(wm_yr_wk_idx['11101'])

1969it [00:00, 108381.80it/s]

周序号="11101"，对应的七天为：
[1, 2, 3, 4, 5, 6, 7]

1.3 导入价格数据

The price data describes the weekly prices for each item in every store.

# Price
with open(PRICES, "r", newline='') as f:
    is_header = True
    for row in tqdm(csv.reader(f)):
        if is_header:
            is_header = False
            continue
        store_id, item_id, wm_yr_wk, sell_price = row
        series_id = f"{item_id}_{store_id}"
        series_idx = id_idx[series_id] # 从id_idx中召唤商品id对应的数字编码
        for d in wm_yr_wk_idx[wm_yr_wk]: # 从哈希表wm_yr_wk_idx中召唤出周序号=wm_yr_wk对应的哪些天
            # sell_prices每一列代表一个商品id对应的所有日期的价格
            sell_prices[d - 1, series_idx] = float(sell_price) 

print('价格数据sell_prices的shape为：', sell_prices.shape)
print('最后一个商品{series_id}对应的周{wm_yr_wk}平均价格 = {sell_price}'.format(series_id=series_id, 
                                                                  wm_yr_wk=wm_yr_wk, 
                                                                  sell_price=sell_price))

6841122it [00:42, 161524.64it/s]

价格数据sell_prices的shape为： (1969, 30490)
最后一个商品FOODS_3_827_WI_3对应的周11621平均价格 = 1.0

2. Building DataFrame

We’ll store the dataset in two dataframes:

  • qty_ts: sales data.
  • price_ts: prices.
  • 在这里构建数据框时，就体现了多层列索引的东西
  • 设计的表为达宽表。行为日期，列为各种id的多级索引列。
# Aggregations - Levels
qty_ts = pd.DataFrame(qties,
                      index=range(1, NUM_TRAINING + 1),
                      columns=[state_ids, store_ids,
                               cat_ids, dept_ids, item_ids])

qty_ts.index.names = ['d']
qty_ts.columns.names = ['state_id', 'store_id',
                        'cat_id', 'dept_id', 'item_id']

print('===================================== 销量数据 =====================================')
print('输出qty_ts列的shape: ', qty_ts.shape)
print('输出qty_ts前两列的列名: \n', qty_ts.columns[:2])
price_ts = pd.DataFrame(sell_prices,
                        index=range(1, NUM_TEST + 1),
                        columns=[state_ids, store_ids,
                                 cat_ids, dept_ids, item_ids])
price_ts.index.names = ['d']
price_ts.columns.names = ['state_id', 'store_id',
                          'cat_id', 'dept_id', 'item_id']

print('===================================== 价格数据 =====================================')
print('输出price_ts列的shape: ', price_ts.shape)
print('输出price_ts前两列的列名: \n', price_ts.columns[:2])

===================================== 销量数据 =====================================
输出qty_ts列的shape:  (1913, 30490)
输出qty_ts前两列的列名: 
 MultiIndex([('CA', 'CA_1', 'HOBBIES', 'HOBBIES_1', 'HOBBIES_1_001'),
            ('CA', 'CA_1', 'HOBBIES', 'HOBBIES_1', 'HOBBIES_1_002')],
           names=['state_id', 'store_id', 'cat_id', 'dept_id', 'item_id'])
===================================== 价格数据 =====================================
输出price_ts列的shape:  (1969, 30490)
输出price_ts前两列的列名: 
 MultiIndex([('CA', 'CA_1', 'HOBBIES', 'HOBBIES_1', 'HOBBIES_1_001'),
            ('CA', 'CA_1', 'HOBBIES', 'HOBBIES_1', 'HOBBIES_1_002')],
           names=['state_id', 'store_id', 'cat_id', 'dept_id', 'item_id'])

3. 聚合

LEVELS

  • 在这次比赛中，我们的模型在12个不同的层次上进行评估，这些层次是由系列的分组组合定义的。
  • 重要的是，我们可以汇总我们的时间序列，例如，计算每个州的总销售额，这样我们就可以评估模型在每个级别上的每店商品销售数据预测。

COARSER

  • Pandas将所有的列级都看作是独立的，但在这里并不是，例如，所有具有相同dept_id的系列都属于同一个cat_id。当我们对列进行分组时，我们也会保留任何较粗的分组。
LEVELS = {
    1: [],
    2: ['state_id'],
    3: ['store_id'],
    4: ['cat_id'],
    5: ['dept_id'],
    6: ['state_id', 'cat_id'],
    7: ['state_id', 'dept_id'],
    8: ['store_id', 'cat_id'],
    9: ['store_id', 'dept_id'],
    10: ['item_id'],
    11: ['state_id', 'item_id'],
    12: ['item_id', 'store_id']
}

COARSER = {
    'state_id': [],
    'store_id': ['state_id'],
    'cat_id': [],
    'dept_id': ['cat_id'],
    'item_id': ['cat_id', 'dept_id']
}

COARSER

{'state_id': [],
 'store_id': ['state_id'],
 'cat_id': [],
 'dept_id': ['cat_id'],
 'item_id': ['cat_id', 'dept_id']}

def aggregate_all_levels(df):
    levels = []
    for i in range(1, max(LEVELS.keys()) + 1):
        level = aggregate_groupings(df, i, *LEVELS[i])
        levels.append(level)
    return pd.concat(levels, axis=1)

def aggregate_groupings(df, level_id, grouping_a=None, grouping_b=None):
    """Aggregate time series by summing over optional levels

    New columns are named according to the m5 competition.

    :param df: Time series as columns
    :param level_id: Numeric ID of level
    :param grouping_a: Grouping to aggregate over, if any
    :param grouping_b: Additional grouping to aggregate over, if any
    :return: Aggregated DataFrame with columns as series id:s
    """
    if grouping_a is None and grouping_b is None:
        new_df = df.sum(axis=1).to_frame()
    elif grouping_b is None:
        new_df = df.groupby(COARSER[grouping_a] + [grouping_a], axis=1).sum()
    else:
        assert grouping_a is not None
        new_df = df.groupby(COARSER[grouping_a] + COARSER[grouping_b] +
                            [grouping_a, grouping_b], axis=1).sum()

    new_df.columns = _restore_columns(df.columns, new_df.columns, level_id,
                                      grouping_a, grouping_b)
    return new_df

一个小的复杂情况是，Pandas在列的连通过程中并不对齐，即如果两个数据框有一些不同的列级别，pd.concat并不匹配帧之间相同的级别。

最简单的解决方法就是暂时把我们分组后丢失的级别加回来。

def _restore_columns(original_index, new_index, level_id, grouping_a, grouping_b):
    original_df = original_index.to_frame()
    new_df = new_index.to_frame()
    for column in original_df.columns:
        if column not in new_df.columns:
            new_df[column] = None

    # Set up `level` column
    new_df['level'] = level_id

    # Set up `id` column
    if grouping_a is None and grouping_b is None:
        new_df['id'] = 'Total_X'
    elif grouping_b is None:
        new_df['id'] = new_df[grouping_a] + '_X'
    else:
        assert grouping_a is not None
        new_df['id'] = new_df[grouping_a] + '_' + new_df[grouping_b]

    new_index = pd.MultiIndex.from_frame(new_df)
    # Remove "unnamed" level if no grouping
    if grouping_a is None and grouping_b is None:
        new_index = new_index.droplevel(0)
    new_levels = ['level'] + original_index.names + ['id']
    return new_index.reorder_levels(new_levels)

agg_pd = aggregate_all_levels(qty_ts)
agg_pd.head()
level 1 2 3 4 5 6 7 8 9 10 ... 12
state_id NaN CA TX WI CA TX WI NaN NaN CA TX WI CA TX WI CA TX WI CA TX WI CA TX WI CA TX WI NaN ... WI
store_id NaN NaN NaN NaN CA_1 CA_2 CA_3 CA_4 TX_1 TX_2 TX_3 WI_1 WI_2 WI_3 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN CA_1 CA_2 CA_3 CA_4 TX_1 TX_2 TX_3 WI_1 WI_2 WI_3 CA_1 CA_2 CA_3 CA_4 CA_1 CA_2 CA_3 CA_4 CA_1 CA_2 CA_3 CA_4 TX_1 TX_2 TX_3 TX_1 TX_2 TX_3 TX_1 TX_2 TX_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 NaN ... WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3 WI_1 WI_2 WI_3
cat_id NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN FOODS HOBBIES HOUSEHOLD FOODS HOBBIES HOUSEHOLD FOODS HOBBIES HOUSEHOLD FOODS HOBBIES HOUSEHOLD FOODS HOBBIES HOUSEHOLD FOODS FOODS FOODS HOBBIES HOBBIES HOBBIES HOUSEHOLD HOUSEHOLD HOUSEHOLD FOODS HOBBIES HOUSEHOLD FOODS HOBBIES HOUSEHOLD FOODS HOBBIES HOUSEHOLD FOODS HOBBIES HOUSEHOLD FOODS HOBBIES HOUSEHOLD FOODS HOBBIES HOUSEHOLD FOODS HOBBIES HOUSEHOLD FOODS HOBBIES HOUSEHOLD FOODS HOBBIES HOUSEHOLD FOODS HOBBIES HOUSEHOLD FOODS FOODS FOODS FOODS HOBBIES HOBBIES HOBBIES HOBBIES HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD FOODS FOODS FOODS HOBBIES HOBBIES HOBBIES HOUSEHOLD HOUSEHOLD HOUSEHOLD FOODS FOODS FOODS HOBBIES HOBBIES HOBBIES HOUSEHOLD HOUSEHOLD HOUSEHOLD FOODS ... HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD HOUSEHOLD
dept_id NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN FOODS_1 FOODS_2 FOODS_3 HOBBIES_1 HOBBIES_2 HOUSEHOLD_1 HOUSEHOLD_2 NaN NaN NaN NaN NaN NaN NaN NaN NaN FOODS_1 FOODS_2 FOODS_3 FOODS_1 FOODS_2 FOODS_3 FOODS_1 FOODS_2 FOODS_3 HOBBIES_1 HOBBIES_2 HOBBIES_1 HOBBIES_2 HOBBIES_1 HOBBIES_2 HOUSEHOLD_1 HOUSEHOLD_2 HOUSEHOLD_1 HOUSEHOLD_2 HOUSEHOLD_1 HOUSEHOLD_2 NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN FOODS_1 FOODS_2 FOODS_3 FOODS_1 FOODS_2 FOODS_3 FOODS_1 FOODS_2 FOODS_3 FOODS_1 FOODS_2 FOODS_3 HOBBIES_1 HOBBIES_2 HOBBIES_1 HOBBIES_2 HOBBIES_1 HOBBIES_2 HOBBIES_1 HOBBIES_2 HOUSEHOLD_1 HOUSEHOLD_2 HOUSEHOLD_1 HOUSEHOLD_2 HOUSEHOLD_1 HOUSEHOLD_2 HOUSEHOLD_1 HOUSEHOLD_2 FOODS_1 FOODS_2 FOODS_3 FOODS_1 FOODS_2 FOODS_3 FOODS_1 FOODS_2 FOODS_3 HOBBIES_1 HOBBIES_2 HOBBIES_1 HOBBIES_2 HOBBIES_1 HOBBIES_2 HOUSEHOLD_1 HOUSEHOLD_2 HOUSEHOLD_1 HOUSEHOLD_2 HOUSEHOLD_1 HOUSEHOLD_2 FOODS_1 FOODS_2 FOODS_3 FOODS_1 FOODS_2 FOODS_3 FOODS_1 FOODS_2 FOODS_3 HOBBIES_1 HOBBIES_2 HOBBIES_1 HOBBIES_2 HOBBIES_1 HOBBIES_2 HOUSEHOLD_1 HOUSEHOLD_2 HOUSEHOLD_1 HOUSEHOLD_2 HOUSEHOLD_1 HOUSEHOLD_2 FOODS_1 ... HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2 HOUSEHOLD_2
item_id NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN NaN FOODS_1_001 FOODS_1_002 FOODS_1_003 FOODS_1_004 FOODS_1_005 FOODS_1_006 FOODS_1_008 FOODS_1_009 FOODS_1_010 FOODS_1_011 FOODS_1_012 FOODS_1_013 FOODS_1_014 FOODS_1_015 FOODS_1_016 FOODS_1_017 FOODS_1_018 FOODS_1_019 FOODS_1_020 FOODS_1_021 FOODS_1_022 FOODS_1_023 FOODS_1_024 FOODS_1_025 FOODS_1_026 FOODS_1_027 FOODS_1_028 FOODS_1_029 FOODS_1_030 FOODS_1_031 FOODS_1_032 FOODS_1_033 FOODS_1_034 FOODS_1_035 FOODS_1_036 FOODS_1_037 FOODS_1_038 FOODS_1_039 FOODS_1_040 FOODS_1_041 FOODS_1_042 FOODS_1_043 FOODS_1_044 FOODS_1_045 FOODS_1_046 FOODS_1_047 FOODS_1_048 FOODS_1_049 FOODS_1_050 FOODS_1_051 FOODS_1_052 FOODS_1_053 FOODS_1_054 FOODS_1_055 FOODS_1_056 FOODS_1_057 FOODS_1_058 FOODS_1_059 FOODS_1_060 FOODS_1_061 FOODS_1_062 FOODS_1_063 FOODS_1_064 FOODS_1_065 FOODS_1_066 FOODS_1_067 FOODS_1_068 FOODS_1_069 FOODS_1_070 FOODS_1_071 FOODS_1_072 FOODS_1_073 FOODS_1_074 FOODS_1_075 FOODS_1_076 FOODS_1_077 FOODS_1_078 FOODS_1_079 FOODS_1_080 FOODS_1_081 FOODS_1_082 FOODS_1_083 FOODS_1_084 FOODS_1_085 FOODS_1_086 FOODS_1_087 FOODS_1_088 FOODS_1_089 FOODS_1_090 FOODS_1_091 FOODS_1_092 FOODS_1_093 FOODS_1_094 FOODS_1_095 FOODS_1_096 FOODS_1_097 ... HOUSEHOLD_2_433 HOUSEHOLD_2_434 HOUSEHOLD_2_434 HOUSEHOLD_2_434 HOUSEHOLD_2_435 HOUSEHOLD_2_435 HOUSEHOLD_2_435 HOUSEHOLD_2_436 HOUSEHOLD_2_436 HOUSEHOLD_2_436 HOUSEHOLD_2_437 HOUSEHOLD_2_437 HOUSEHOLD_2_437 HOUSEHOLD_2_438 HOUSEHOLD_2_438 HOUSEHOLD_2_438 HOUSEHOLD_2_439 HOUSEHOLD_2_439 HOUSEHOLD_2_439 HOUSEHOLD_2_440 HOUSEHOLD_2_440 HOUSEHOLD_2_440 HOUSEHOLD_2_441 HOUSEHOLD_2_441 HOUSEHOLD_2_441 HOUSEHOLD_2_442 HOUSEHOLD_2_442 HOUSEHOLD_2_442 HOUSEHOLD_2_443 HOUSEHOLD_2_443 HOUSEHOLD_2_443 HOUSEHOLD_2_444 HOUSEHOLD_2_444 HOUSEHOLD_2_444 HOUSEHOLD_2_445 HOUSEHOLD_2_445 HOUSEHOLD_2_445 HOUSEHOLD_2_446 HOUSEHOLD_2_446 HOUSEHOLD_2_446 HOUSEHOLD_2_447 HOUSEHOLD_2_447 HOUSEHOLD_2_447 HOUSEHOLD_2_448 HOUSEHOLD_2_448 HOUSEHOLD_2_448 HOUSEHOLD_2_449 HOUSEHOLD_2_449 HOUSEHOLD_2_449 HOUSEHOLD_2_450 HOUSEHOLD_2_450 HOUSEHOLD_2_450 HOUSEHOLD_2_451 HOUSEHOLD_2_451 HOUSEHOLD_2_451 HOUSEHOLD_2_452 HOUSEHOLD_2_452 HOUSEHOLD_2_452 HOUSEHOLD_2_453 HOUSEHOLD_2_453 HOUSEHOLD_2_453 HOUSEHOLD_2_454 HOUSEHOLD_2_454 HOUSEHOLD_2_454 HOUSEHOLD_2_455 HOUSEHOLD_2_455 HOUSEHOLD_2_455 HOUSEHOLD_2_456 HOUSEHOLD_2_456 HOUSEHOLD_2_456 HOUSEHOLD_2_457 HOUSEHOLD_2_457 HOUSEHOLD_2_457 HOUSEHOLD_2_458 HOUSEHOLD_2_458 HOUSEHOLD_2_458 HOUSEHOLD_2_459 HOUSEHOLD_2_459 HOUSEHOLD_2_459 HOUSEHOLD_2_460 HOUSEHOLD_2_460 HOUSEHOLD_2_460 HOUSEHOLD_2_461 HOUSEHOLD_2_461 HOUSEHOLD_2_461 HOUSEHOLD_2_462 HOUSEHOLD_2_462 HOUSEHOLD_2_462 HOUSEHOLD_2_463 HOUSEHOLD_2_463 HOUSEHOLD_2_463 HOUSEHOLD_2_464 HOUSEHOLD_2_464 HOUSEHOLD_2_464 HOUSEHOLD_2_465 HOUSEHOLD_2_465 HOUSEHOLD_2_465 HOUSEHOLD_2_466 HOUSEHOLD_2_466 HOUSEHOLD_2_466 HOUSEHOLD_2_467 HOUSEHOLD_2_467 HOUSEHOLD_2_467 HOUSEHOLD_2_468 HOUSEHOLD_2_468 HOUSEHOLD_2_468 HOUSEHOLD_2_469 HOUSEHOLD_2_469 HOUSEHOLD_2_469 HOUSEHOLD_2_470 HOUSEHOLD_2_470 HOUSEHOLD_2_470 HOUSEHOLD_2_471 HOUSEHOLD_2_471 HOUSEHOLD_2_471 HOUSEHOLD_2_472 HOUSEHOLD_2_472 HOUSEHOLD_2_472 HOUSEHOLD_2_473 HOUSEHOLD_2_473 HOUSEHOLD_2_473 HOUSEHOLD_2_474 HOUSEHOLD_2_474 HOUSEHOLD_2_474 HOUSEHOLD_2_475 HOUSEHOLD_2_475 HOUSEHOLD_2_475 HOUSEHOLD_2_476 HOUSEHOLD_2_476 HOUSEHOLD_2_476 HOUSEHOLD_2_477 HOUSEHOLD_2_477 HOUSEHOLD_2_477 HOUSEHOLD_2_478 HOUSEHOLD_2_478 HOUSEHOLD_2_478 HOUSEHOLD_2_479 HOUSEHOLD_2_479 HOUSEHOLD_2_479 HOUSEHOLD_2_480 HOUSEHOLD_2_480 HOUSEHOLD_2_480 HOUSEHOLD_2_481 HOUSEHOLD_2_481 HOUSEHOLD_2_481 HOUSEHOLD_2_482 HOUSEHOLD_2_482 HOUSEHOLD_2_482 HOUSEHOLD_2_483 HOUSEHOLD_2_483 HOUSEHOLD_2_483 HOUSEHOLD_2_484 HOUSEHOLD_2_484 HOUSEHOLD_2_484 HOUSEHOLD_2_485 HOUSEHOLD_2_485 HOUSEHOLD_2_485 HOUSEHOLD_2_486 HOUSEHOLD_2_486 HOUSEHOLD_2_486 HOUSEHOLD_2_487 HOUSEHOLD_2_487 HOUSEHOLD_2_487 HOUSEHOLD_2_488 HOUSEHOLD_2_488 HOUSEHOLD_2_488 HOUSEHOLD_2_489 HOUSEHOLD_2_489 HOUSEHOLD_2_489 HOUSEHOLD_2_490 HOUSEHOLD_2_490 HOUSEHOLD_2_490 HOUSEHOLD_2_491 HOUSEHOLD_2_491 HOUSEHOLD_2_491 HOUSEHOLD_2_492 HOUSEHOLD_2_492 HOUSEHOLD_2_492 HOUSEHOLD_2_493 HOUSEHOLD_2_493 HOUSEHOLD_2_493 HOUSEHOLD_2_494 HOUSEHOLD_2_494 HOUSEHOLD_2_494 HOUSEHOLD_2_495 HOUSEHOLD_2_495 HOUSEHOLD_2_495 HOUSEHOLD_2_496 HOUSEHOLD_2_496 HOUSEHOLD_2_496 HOUSEHOLD_2_497 HOUSEHOLD_2_497 HOUSEHOLD_2_497 HOUSEHOLD_2_498 HOUSEHOLD_2_498 HOUSEHOLD_2_498 HOUSEHOLD_2_499 HOUSEHOLD_2_499 HOUSEHOLD_2_499 HOUSEHOLD_2_500 HOUSEHOLD_2_500 HOUSEHOLD_2_500 HOUSEHOLD_2_501 HOUSEHOLD_2_501 HOUSEHOLD_2_501 HOUSEHOLD_2_502 HOUSEHOLD_2_502 HOUSEHOLD_2_502 HOUSEHOLD_2_503 HOUSEHOLD_2_503 HOUSEHOLD_2_503 HOUSEHOLD_2_504 HOUSEHOLD_2_504 HOUSEHOLD_2_504 HOUSEHOLD_2_505 HOUSEHOLD_2_505 HOUSEHOLD_2_505 HOUSEHOLD_2_506 HOUSEHOLD_2_506 HOUSEHOLD_2_506 HOUSEHOLD_2_507 HOUSEHOLD_2_507 HOUSEHOLD_2_507 HOUSEHOLD_2_508 HOUSEHOLD_2_508 HOUSEHOLD_2_508 HOUSEHOLD_2_509 HOUSEHOLD_2_509 HOUSEHOLD_2_509 HOUSEHOLD_2_510 HOUSEHOLD_2_510 HOUSEHOLD_2_510 HOUSEHOLD_2_511 HOUSEHOLD_2_511 HOUSEHOLD_2_511 HOUSEHOLD_2_512 HOUSEHOLD_2_512 HOUSEHOLD_2_512 HOUSEHOLD_2_513 HOUSEHOLD_2_513 HOUSEHOLD_2_513 HOUSEHOLD_2_514 HOUSEHOLD_2_514 HOUSEHOLD_2_514 HOUSEHOLD_2_515 HOUSEHOLD_2_515 HOUSEHOLD_2_515 HOUSEHOLD_2_516 HOUSEHOLD_2_516 HOUSEHOLD_2_516
id Total_X CA_X TX_X WI_X CA_1_X CA_2_X CA_3_X CA_4_X TX_1_X TX_2_X TX_3_X WI_1_X WI_2_X WI_3_X FOODS_X HOBBIES_X HOUSEHOLD_X FOODS_1_X FOODS_2_X FOODS_3_X HOBBIES_1_X HOBBIES_2_X HOUSEHOLD_1_X HOUSEHOLD_2_X CA_FOODS CA_HOBBIES CA_HOUSEHOLD TX_FOODS TX_HOBBIES TX_HOUSEHOLD WI_FOODS WI_HOBBIES WI_HOUSEHOLD CA_FOODS_1 CA_FOODS_2 CA_FOODS_3 TX_FOODS_1 TX_FOODS_2 TX_FOODS_3 WI_FOODS_1 WI_FOODS_2 WI_FOODS_3 CA_HOBBIES_1 CA_HOBBIES_2 TX_HOBBIES_1 TX_HOBBIES_2 WI_HOBBIES_1 WI_HOBBIES_2 CA_HOUSEHOLD_1 CA_HOUSEHOLD_2 TX_HOUSEHOLD_1 TX_HOUSEHOLD_2 WI_HOUSEHOLD_1 WI_HOUSEHOLD_2 CA_1_FOODS CA_1_HOBBIES CA_1_HOUSEHOLD CA_2_FOODS CA_2_HOBBIES CA_2_HOUSEHOLD CA_3_FOODS CA_3_HOBBIES CA_3_HOUSEHOLD CA_4_FOODS CA_4_HOBBIES CA_4_HOUSEHOLD TX_1_FOODS TX_1_HOBBIES TX_1_HOUSEHOLD TX_2_FOODS TX_2_HOBBIES TX_2_HOUSEHOLD TX_3_FOODS TX_3_HOBBIES TX_3_HOUSEHOLD WI_1_FOODS WI_1_HOBBIES WI_1_HOUSEHOLD WI_2_FOODS WI_2_HOBBIES WI_2_HOUSEHOLD WI_3_FOODS WI_3_HOBBIES WI_3_HOUSEHOLD CA_1_FOODS_1 CA_1_FOODS_2 CA_1_FOODS_3 CA_2_FOODS_1 CA_2_FOODS_2 CA_2_FOODS_3 CA_3_FOODS_1 CA_3_FOODS_2 CA_3_FOODS_3 CA_4_FOODS_1 CA_4_FOODS_2 CA_4_FOODS_3 CA_1_HOBBIES_1 CA_1_HOBBIES_2 CA_2_HOBBIES_1 CA_2_HOBBIES_2 CA_3_HOBBIES_1 CA_3_HOBBIES_2 CA_4_HOBBIES_1 CA_4_HOBBIES_2 CA_1_HOUSEHOLD_1 CA_1_HOUSEHOLD_2 CA_2_HOUSEHOLD_1 CA_2_HOUSEHOLD_2 CA_3_HOUSEHOLD_1 CA_3_HOUSEHOLD_2 CA_4_HOUSEHOLD_1 CA_4_HOUSEHOLD_2 TX_1_FOODS_1 TX_1_FOODS_2 TX_1_FOODS_3 TX_2_FOODS_1 TX_2_FOODS_2 TX_2_FOODS_3 TX_3_FOODS_1 TX_3_FOODS_2 TX_3_FOODS_3 TX_1_HOBBIES_1 TX_1_HOBBIES_2 TX_2_HOBBIES_1 TX_2_HOBBIES_2 TX_3_HOBBIES_1 TX_3_HOBBIES_2 TX_1_HOUSEHOLD_1 TX_1_HOUSEHOLD_2 TX_2_HOUSEHOLD_1 TX_2_HOUSEHOLD_2 TX_3_HOUSEHOLD_1 TX_3_HOUSEHOLD_2 WI_1_FOODS_1 WI_1_FOODS_2 WI_1_FOODS_3 WI_2_FOODS_1 WI_2_FOODS_2 WI_2_FOODS_3 WI_3_FOODS_1 WI_3_FOODS_2 WI_3_FOODS_3 WI_1_HOBBIES_1 WI_1_HOBBIES_2 WI_2_HOBBIES_1 WI_2_HOBBIES_2 WI_3_HOBBIES_1 WI_3_HOBBIES_2 WI_1_HOUSEHOLD_1 WI_1_HOUSEHOLD_2 WI_2_HOUSEHOLD_1 WI_2_HOUSEHOLD_2 WI_3_HOUSEHOLD_1 WI_3_HOUSEHOLD_2 FOODS_1_001_X FOODS_1_002_X FOODS_1_003_X FOODS_1_004_X FOODS_1_005_X FOODS_1_006_X FOODS_1_008_X FOODS_1_009_X FOODS_1_010_X FOODS_1_011_X FOODS_1_012_X FOODS_1_013_X FOODS_1_014_X FOODS_1_015_X FOODS_1_016_X FOODS_1_017_X FOODS_1_018_X FOODS_1_019_X FOODS_1_020_X FOODS_1_021_X FOODS_1_022_X FOODS_1_023_X FOODS_1_024_X FOODS_1_025_X FOODS_1_026_X FOODS_1_027_X FOODS_1_028_X FOODS_1_029_X FOODS_1_030_X FOODS_1_031_X FOODS_1_032_X FOODS_1_033_X FOODS_1_034_X FOODS_1_035_X FOODS_1_036_X FOODS_1_037_X FOODS_1_038_X FOODS_1_039_X FOODS_1_040_X FOODS_1_041_X FOODS_1_042_X FOODS_1_043_X FOODS_1_044_X FOODS_1_045_X FOODS_1_046_X FOODS_1_047_X FOODS_1_048_X FOODS_1_049_X FOODS_1_050_X FOODS_1_051_X FOODS_1_052_X FOODS_1_053_X FOODS_1_054_X FOODS_1_055_X FOODS_1_056_X FOODS_1_057_X FOODS_1_058_X FOODS_1_059_X FOODS_1_060_X FOODS_1_061_X FOODS_1_062_X FOODS_1_063_X FOODS_1_064_X FOODS_1_065_X FOODS_1_066_X FOODS_1_067_X FOODS_1_068_X FOODS_1_069_X FOODS_1_070_X FOODS_1_071_X FOODS_1_072_X FOODS_1_073_X FOODS_1_074_X FOODS_1_075_X FOODS_1_076_X FOODS_1_077_X FOODS_1_078_X FOODS_1_079_X FOODS_1_080_X FOODS_1_081_X FOODS_1_082_X FOODS_1_083_X FOODS_1_084_X FOODS_1_085_X FOODS_1_086_X FOODS_1_087_X FOODS_1_088_X FOODS_1_089_X FOODS_1_090_X FOODS_1_091_X FOODS_1_092_X FOODS_1_093_X FOODS_1_094_X FOODS_1_095_X FOODS_1_096_X FOODS_1_097_X ... HOUSEHOLD_2_433_WI_3 HOUSEHOLD_2_434_WI_1 HOUSEHOLD_2_434_WI_2 HOUSEHOLD_2_434_WI_3 HOUSEHOLD_2_435_WI_1 HOUSEHOLD_2_435_WI_2 HOUSEHOLD_2_435_WI_3 HOUSEHOLD_2_436_WI_1 HOUSEHOLD_2_436_WI_2 HOUSEHOLD_2_436_WI_3 HOUSEHOLD_2_437_WI_1 HOUSEHOLD_2_437_WI_2 HOUSEHOLD_2_437_WI_3 HOUSEHOLD_2_438_WI_1 HOUSEHOLD_2_438_WI_2 HOUSEHOLD_2_438_WI_3 HOUSEHOLD_2_439_WI_1 HOUSEHOLD_2_439_WI_2 HOUSEHOLD_2_439_WI_3 HOUSEHOLD_2_440_WI_1 HOUSEHOLD_2_440_WI_2 HOUSEHOLD_2_440_WI_3 HOUSEHOLD_2_441_WI_1 HOUSEHOLD_2_441_WI_2 HOUSEHOLD_2_441_WI_3 HOUSEHOLD_2_442_WI_1 HOUSEHOLD_2_442_WI_2 HOUSEHOLD_2_442_WI_3 HOUSEHOLD_2_443_WI_1 HOUSEHOLD_2_443_WI_2 HOUSEHOLD_2_443_WI_3 HOUSEHOLD_2_444_WI_1 HOUSEHOLD_2_444_WI_2 HOUSEHOLD_2_444_WI_3 HOUSEHOLD_2_445_WI_1 HOUSEHOLD_2_445_WI_2 HOUSEHOLD_2_445_WI_3 HOUSEHOLD_2_446_WI_1 HOUSEHOLD_2_446_WI_2 HOUSEHOLD_2_446_WI_3 HOUSEHOLD_2_447_WI_1 HOUSEHOLD_2_447_WI_2 HOUSEHOLD_2_447_WI_3 HOUSEHOLD_2_448_WI_1 HOUSEHOLD_2_448_WI_2 HOUSEHOLD_2_448_WI_3 HOUSEHOLD_2_449_WI_1 HOUSEHOLD_2_449_WI_2 HOUSEHOLD_2_449_WI_3 HOUSEHOLD_2_450_WI_1 HOUSEHOLD_2_450_WI_2 HOUSEHOLD_2_450_WI_3 HOUSEHOLD_2_451_WI_1 HOUSEHOLD_2_451_WI_2 HOUSEHOLD_2_451_WI_3 HOUSEHOLD_2_452_WI_1 HOUSEHOLD_2_452_WI_2 HOUSEHOLD_2_452_WI_3 HOUSEHOLD_2_453_WI_1 HOUSEHOLD_2_453_WI_2 HOUSEHOLD_2_453_WI_3 HOUSEHOLD_2_454_WI_1 HOUSEHOLD_2_454_WI_2 HOUSEHOLD_2_454_WI_3 HOUSEHOLD_2_455_WI_1 HOUSEHOLD_2_455_WI_2 HOUSEHOLD_2_455_WI_3 HOUSEHOLD_2_456_WI_1 HOUSEHOLD_2_456_WI_2 HOUSEHOLD_2_456_WI_3 HOUSEHOLD_2_457_WI_1 HOUSEHOLD_2_457_WI_2 HOUSEHOLD_2_457_WI_3 HOUSEHOLD_2_458_WI_1 HOUSEHOLD_2_458_WI_2 HOUSEHOLD_2_458_WI_3 HOUSEHOLD_2_459_WI_1 HOUSEHOLD_2_459_WI_2 HOUSEHOLD_2_459_WI_3 HOUSEHOLD_2_460_WI_1 HOUSEHOLD_2_460_WI_2 HOUSEHOLD_2_460_WI_3 HOUSEHOLD_2_461_WI_1 HOUSEHOLD_2_461_WI_2 HOUSEHOLD_2_461_WI_3 HOUSEHOLD_2_462_WI_1 HOUSEHOLD_2_462_WI_2 HOUSEHOLD_2_462_WI_3 HOUSEHOLD_2_463_WI_1 HOUSEHOLD_2_463_WI_2 HOUSEHOLD_2_463_WI_3 HOUSEHOLD_2_464_WI_1 HOUSEHOLD_2_464_WI_2 HOUSEHOLD_2_464_WI_3 HOUSEHOLD_2_465_WI_1 HOUSEHOLD_2_465_WI_2 HOUSEHOLD_2_465_WI_3 HOUSEHOLD_2_466_WI_1 HOUSEHOLD_2_466_WI_2 HOUSEHOLD_2_466_WI_3 HOUSEHOLD_2_467_WI_1 HOUSEHOLD_2_467_WI_2 HOUSEHOLD_2_467_WI_3 HOUSEHOLD_2_468_WI_1 HOUSEHOLD_2_468_WI_2 HOUSEHOLD_2_468_WI_3 HOUSEHOLD_2_469_WI_1 HOUSEHOLD_2_469_WI_2 HOUSEHOLD_2_469_WI_3 HOUSEHOLD_2_470_WI_1 HOUSEHOLD_2_470_WI_2 HOUSEHOLD_2_470_WI_3 HOUSEHOLD_2_471_WI_1 HOUSEHOLD_2_471_WI_2 HOUSEHOLD_2_471_WI_3 HOUSEHOLD_2_472_WI_1 HOUSEHOLD_2_472_WI_2 HOUSEHOLD_2_472_WI_3 HOUSEHOLD_2_473_WI_1 HOUSEHOLD_2_473_WI_2 HOUSEHOLD_2_473_WI_3 HOUSEHOLD_2_474_WI_1 HOUSEHOLD_2_474_WI_2 HOUSEHOLD_2_474_WI_3 HOUSEHOLD_2_475_WI_1 HOUSEHOLD_2_475_WI_2 HOUSEHOLD_2_475_WI_3 HOUSEHOLD_2_476_WI_1 HOUSEHOLD_2_476_WI_2 HOUSEHOLD_2_476_WI_3 HOUSEHOLD_2_477_WI_1 HOUSEHOLD_2_477_WI_2 HOUSEHOLD_2_477_WI_3 HOUSEHOLD_2_478_WI_1 HOUSEHOLD_2_478_WI_2 HOUSEHOLD_2_478_WI_3 HOUSEHOLD_2_479_WI_1 HOUSEHOLD_2_479_WI_2 HOUSEHOLD_2_479_WI_3 HOUSEHOLD_2_480_WI_1 HOUSEHOLD_2_480_WI_2 HOUSEHOLD_2_480_WI_3 HOUSEHOLD_2_481_WI_1 HOUSEHOLD_2_481_WI_2 HOUSEHOLD_2_481_WI_3 HOUSEHOLD_2_482_WI_1 HOUSEHOLD_2_482_WI_2 HOUSEHOLD_2_482_WI_3 HOUSEHOLD_2_483_WI_1 HOUSEHOLD_2_483_WI_2 HOUSEHOLD_2_483_WI_3 HOUSEHOLD_2_484_WI_1 HOUSEHOLD_2_484_WI_2 HOUSEHOLD_2_484_WI_3 HOUSEHOLD_2_485_WI_1 HOUSEHOLD_2_485_WI_2 HOUSEHOLD_2_485_WI_3 HOUSEHOLD_2_486_WI_1 HOUSEHOLD_2_486_WI_2 HOUSEHOLD_2_486_WI_3 HOUSEHOLD_2_487_WI_1 HOUSEHOLD_2_487_WI_2 HOUSEHOLD_2_487_WI_3 HOUSEHOLD_2_488_WI_1 HOUSEHOLD_2_488_WI_2 HOUSEHOLD_2_488_WI_3 HOUSEHOLD_2_489_WI_1 HOUSEHOLD_2_489_WI_2 HOUSEHOLD_2_489_WI_3 HOUSEHOLD_2_490_WI_1 HOUSEHOLD_2_490_WI_2 HOUSEHOLD_2_490_WI_3 HOUSEHOLD_2_491_WI_1 HOUSEHOLD_2_491_WI_2 HOUSEHOLD_2_491_WI_3 HOUSEHOLD_2_492_WI_1 HOUSEHOLD_2_492_WI_2 HOUSEHOLD_2_492_WI_3 HOUSEHOLD_2_493_WI_1 HOUSEHOLD_2_493_WI_2 HOUSEHOLD_2_493_WI_3 HOUSEHOLD_2_494_WI_1 HOUSEHOLD_2_494_WI_2 HOUSEHOLD_2_494_WI_3 HOUSEHOLD_2_495_WI_1 HOUSEHOLD_2_495_WI_2 HOUSEHOLD_2_495_WI_3 HOUSEHOLD_2_496_WI_1 HOUSEHOLD_2_496_WI_2 HOUSEHOLD_2_496_WI_3 HOUSEHOLD_2_497_WI_1 HOUSEHOLD_2_497_WI_2 HOUSEHOLD_2_497_WI_3 HOUSEHOLD_2_498_WI_1 HOUSEHOLD_2_498_WI_2 HOUSEHOLD_2_498_WI_3 HOUSEHOLD_2_499_WI_1 HOUSEHOLD_2_499_WI_2 HOUSEHOLD_2_499_WI_3 HOUSEHOLD_2_500_WI_1 HOUSEHOLD_2_500_WI_2 HOUSEHOLD_2_500_WI_3 HOUSEHOLD_2_501_WI_1 HOUSEHOLD_2_501_WI_2 HOUSEHOLD_2_501_WI_3 HOUSEHOLD_2_502_WI_1 HOUSEHOLD_2_502_WI_2 HOUSEHOLD_2_502_WI_3 HOUSEHOLD_2_503_WI_1 HOUSEHOLD_2_503_WI_2 HOUSEHOLD_2_503_WI_3 HOUSEHOLD_2_504_WI_1 HOUSEHOLD_2_504_WI_2 HOUSEHOLD_2_504_WI_3 HOUSEHOLD_2_505_WI_1 HOUSEHOLD_2_505_WI_2 HOUSEHOLD_2_505_WI_3 HOUSEHOLD_2_506_WI_1 HOUSEHOLD_2_506_WI_2 HOUSEHOLD_2_506_WI_3 HOUSEHOLD_2_507_WI_1 HOUSEHOLD_2_507_WI_2 HOUSEHOLD_2_507_WI_3 HOUSEHOLD_2_508_WI_1 HOUSEHOLD_2_508_WI_2 HOUSEHOLD_2_508_WI_3 HOUSEHOLD_2_509_WI_1 HOUSEHOLD_2_509_WI_2 HOUSEHOLD_2_509_WI_3 HOUSEHOLD_2_510_WI_1 HOUSEHOLD_2_510_WI_2 HOUSEHOLD_2_510_WI_3 HOUSEHOLD_2_511_WI_1 HOUSEHOLD_2_511_WI_2 HOUSEHOLD_2_511_WI_3 HOUSEHOLD_2_512_WI_1 HOUSEHOLD_2_512_WI_2 HOUSEHOLD_2_512_WI_3 HOUSEHOLD_2_513_WI_1 HOUSEHOLD_2_513_WI_2 HOUSEHOLD_2_513_WI_3 HOUSEHOLD_2_514_WI_1 HOUSEHOLD_2_514_WI_2 HOUSEHOLD_2_514_WI_3 HOUSEHOLD_2_515_WI_1 HOUSEHOLD_2_515_WI_2 HOUSEHOLD_2_515_WI_3 HOUSEHOLD_2_516_WI_1 HOUSEHOLD_2_516_WI_2 HOUSEHOLD_2_516_WI_3
d
1 32631.0 14195.0 9438.0 8998.0 4337.0 3494.0 4739.0 1625.0 2556.0 3852.0 3030.0 2704.0 2256.0 4038.0 23178.0 3764.0 5689.0 2343.0 4094.0 16741.0 3610.0 154.0 4105.0 1584.0 10101.0 1802.0 2292.0 6853.0 879.0 1706.0 6224.0 1083.0 1691.0 1157.0 1864.0 7080.0 500.0 1384.0 4969.0 686.0 846.0 4692.0 1739.0 63.0 837.0 42.0 1034.0 49.0 1527.0 765.0 1281.0 425.0 1297.0 394.0 3239.0 556.0 542.0 2193.0 538.0 763.0 3446.0 550.0 743.0 1223.0 158.0 244.0 1781.0 251.0 524.0 2788.0 385.0 679.0 2284.0 243.0 503.0 1581.0 615.0 508.0 1615.0 190.0 451.0 3028.0 278.0 732.0 297.0 674.0 2268.0 406.0 212.0 1575.0 341.0 627.0 2478.0 113.0 351.0 759.0 528.0 28.0 522.0 16.0 531.0 19.0 158.0 0.0 361.0 181.0 529.0 234.0 457.0 286.0 180.0 64.0 152.0 390.0 1239.0 210.0 534.0 2044.0 138.0 460.0 1686.0 241.0 10.0 364.0 21.0 232.0 11.0 418.0 106.0 480.0 199.0 383.0 120.0 343.0 108.0 1130.0 191.0 155.0 1269.0 152.0 583.0 2293.0 598.0 17.0 180.0 10.0 256.0 22.0 360.0 148.0 353.0 98.0 584.0 148.0 6.0 4.0 14.0 0.0 34.0 16.0 0.0 0.0 0.0 7.0 0.0 8.0 0.0 0.0 13.0 0.0 47.0 106.0 14.0 0.0 0.0 0.0 30.0 0.0 6.0 0.0 0.0 0.0 2.0 22.0 62.0 0.0 0.0 7.0 33.0 5.0 0.0 0.0 23.0 16.0 0.0 0.0 109.0 24.0 52.0 36.0 0.0 17.0 0.0 0.0 0.0 4.0 15.0 45.0 0.0 0.0 0.0 0.0 0.0 29.0 14.0 17.0 9.0 10.0 10.0 35.0 0.0 7.0 0.0 0.0 0.0 16.0 25.0 6.0 8.0 0.0 0.0 0.0 9.0 0.0 0.0 18.0 1.0 52.0 37.0 0.0 35.0 0.0 22.0 14.0 0.0 0.0 25.0 8.0 24.0 0.0 ... 0.0 0.0 0.0 0.0 1.0 1.0 0.0 0.0 0.0 0.0 1.0 3.0 1.0 0.0 0.0 0.0 1.0 2.0 1.0 0.0 0.0 2.0 0.0 0.0 0.0 2.0 0.0 1.0 1.0 0.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 1.0 0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 0.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 2.0 0.0 0.0 0.0 2.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 3.0 0.0 0.0 2.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0
2 31749.0 13805.0 9630.0 8314.0 4155.0 3046.0 4827.0 1777.0 2687.0 3937.0 3006.0 2194.0 1922.0 4198.0 22758.0 3357.0 5634.0 2216.0 4209.0 16333.0 3172.0 185.0 3858.0 1776.0 9862.0 1561.0 2382.0 7030.0 870.0 1730.0 5866.0 926.0 1522.0 1142.0 1987.0 6733.0 446.0 1426.0 5158.0 628.0 796.0 4442.0 1502.0 59.0 775.0 95.0 895.0 31.0 1515.0 867.0 1261.0 469.0 1082.0 440.0 3137.0 498.0 520.0 1921.0 397.0 728.0 3535.0 430.0 862.0 1269.0 236.0 272.0 1903.0 264.0 520.0 2870.0 359.0 708.0 2257.0 247.0 502.0 1327.0 443.0 424.0 1433.0 127.0 362.0 3106.0 356.0 736.0 284.0 655.0 2198.0 408.0 227.0 1286.0 326.0 758.0 2451.0 124.0 347.0 798.0 489.0 9.0 381.0 16.0 401.0 29.0 231.0 5.0 350.0 170.0 461.0 267.0 521.0 341.0 183.0 89.0 154.0 369.0 1380.0 169.0 584.0 2117.0 123.0 473.0 1661.0 246.0 18.0 320.0 39.0 209.0 38.0 372.0 148.0 525.0 183.0 364.0 138.0 306.0 83.0 938.0 184.0 128.0 1121.0 138.0 585.0 2383.0 433.0 10.0 120.0 7.0 342.0 14.0 273.0 151.0 268.0 94.0 541.0 195.0 6.0 5.0 8.0 0.0 32.0 9.0 0.0 0.0 0.0 1.0 0.0 9.0 0.0 0.0 5.0 0.0 26.0 104.0 22.0 0.0 0.0 0.0 28.0 0.0 7.0 0.0 0.0 0.0 0.0 21.0 63.0 0.0 0.0 12.0 23.0 4.0 0.0 0.0 38.0 17.0 0.0 0.0 72.0 32.0 45.0 28.0 0.0 19.0 0.0 0.0 0.0 7.0 19.0 52.0 0.0 0.0 0.0 0.0 0.0 32.0 12.0 7.0 8.0 7.0 6.0 32.0 0.0 27.0 0.0 0.0 0.0 13.0 37.0 10.0 3.0 0.0 0.0 0.0 1.0 0.0 0.0 35.0 1.0 64.0 39.0 0.0 20.0 0.0 5.0 12.0 0.0 0.0 21.0 3.0 28.0 0.0 ... 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 0.0 2.0 0.0 3.0 2.0 3.0 1.0 0.0 2.0 0.0 0.0 0.0 3.0 0.0 0.0 0.0 0.0 3.0 1.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 3.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 0.0 4.0 0.0 0.0 0.0 2.0 0.0 0.0 4.0 0.0 2.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 1.0 0.0 0.0 0.0 1.0 1.0 0.0 0.0 2.0 1.0 0.0 0.0 0.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0.0 1.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 2.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 5.0 0.0 1.0 2.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 1.0
3 23783.0 10108.0 6778.0 6897.0 2816.0 2121.0 3785.0 1386.0 1822.0 2731.0 2225.0 1562.0 2018.0 3317.0 17174.0 2682.0 3927.0 1657.0 3174.0 12343.0 2497.0 185.0 2827.0 1100.0 6944.0 1472.0 1692.0 5124.0 526.0 1128.0 5106.0 684.0 1107.0 787.0 1315.0 4842.0 381.0 1008.0 3735.0 489.0 851.0 3766.0 1422.0 50.0 435.0 91.0 640.0 44.0 1116.0 576.0 845.0 283.0 866.0 241.0 2008.0 415.0 393.0 1289.0 368.0 464.0 2701.0 438.0 646.0 946.0 251.0 189.0 1377.0 107.0 338.0 2082.0 229.0 420.0 1665.0 190.0 370.0 977.0 323.0 262.0 1586.0 113.0 319.0 2543.0 248.0 526.0 214.0 396.0 1398.0 238.0 138.0 913.0 260.0 544.0 1897.0 75.0 237.0 634.0 409.0 6.0 352.0 16.0 414.0 24.0 247.0 4.0 279.0 114.0 306.0 158.0 415.0 231.0 116.0 73.0 109.0 255.0 1013.0 159.0 390.0 1533.0 113.0 363.0 1189.0 96.0 11.0 194.0 35.0 145.0 45.0 269.0 69.0 303.0 117.0 273.0 97.0 208.0 77.0 692.0 154.0 199.0 1233.0 127.0 575.0 1841.0 317.0 6.0 95.0 18.0 228.0 20.0 196.0 66.0 250.0 69.0 420.0 106.0 4.0 7.0 3.0 0.0 13.0 20.0 0.0 0.0 0.0 5.0 0.0 20.0 0.0 0.0 0.0 0.0 30.0 47.0 21.0 0.0 0.0 0.0 22.0 0.0 7.0 0.0 0.0 0.0 1.0 14.0 34.0 0.0 0.0 8.0 13.0 1.0 0.0 0.0 17.0 13.0 0.0 0.0 79.0 16.0 42.0 15.0 0.0 4.0 0.0 0.0 0.0 1.0 20.0 24.0 0.0 0.0 0.0 0.0 0.0 31.0 8.0 11.0 7.0 7.0 8.0 42.0 0.0 6.0 0.0 0.0 0.0 8.0 12.0 5.0 4.0 0.0 0.0 0.0 0.0 0.0 0.0 16.0 0.0 45.0 39.0 0.0 11.0 0.0 4.0 7.0 0.0 0.0 24.0 14.0 40.0 0.0 ... 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 1.0 2.0 0.0 0.0 0.0 0.0 1.0 0.0 2.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 1.0 2.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 2.0 2.0 0.0 0.0 0.0 0.0 3.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 0.0 2.0 0.0 0.0 0.0 0.0 3.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0
4 25412.0 11047.0 7381.0 6984.0 3051.0 2324.0 4232.0 1440.0 2258.0 2954.0 2169.0 1251.0 2522.0 3211.0 18878.0 2669.0 3865.0 1508.0 3606.0 13764.0 2531.0 138.0 2732.0 1133.0 7864.0 1405.0 1778.0 5470.0 809.0 1102.0 5544.0 455.0 985.0 743.0 1452.0 5669.0 317.0 1316.0 3837.0 448.0 838.0 4258.0 1352.0 53.0 756.0 53.0 423.0 32.0 1148.0 630.0 820.0 282.0 764.0 221.0 2258.0 392.0 401.0 1540.0 350.0 434.0 3064.0 424.0 744.0 1002.0 239.0 199.0 1643.0 248.0 367.0 2180.0 359.0 415.0 1647.0 202.0 320.0 935.0 137.0 179.0 2013.0 124.0 385.0 2596.0 194.0 421.0 175.0 476.0 1607.0 240.0 174.0 1126.0 231.0 549.0 2284.0 97.0 253.0 652.0 383.0 9.0 344.0 6.0 395.0 29.0 230.0 9.0 278.0 123.0 270.0 164.0 480.0 264.0 120.0 79.0 126.0 418.0 1099.0 107.0 472.0 1601.0 84.0 426.0 1137.0 229.0 19.0 332.0 27.0 195.0 7.0 276.0 91.0 307.0 108.0 237.0 83.0 145.0 61.0 729.0 205.0 244.0 1564.0 98.0 533.0 1965.0 131.0 6.0 109.0 15.0 183.0 11.0 132.0 47.0 305.0 80.0 327.0 94.0 6.0 4.0 6.0 0.0 20.0 16.0 0.0 0.0 0.0 1.0 0.0 8.0 0.0 0.0 1.0 0.0 24.0 69.0 12.0 0.0 0.0 0.0 26.0 0.0 14.0 0.0 0.0 0.0 1.0 6.0 54.0 0.0 0.0 0.0 27.0 2.0 0.0 0.0 9.0 14.0 0.0 0.0 75.0 32.0 31.0 7.0 0.0 7.0 0.0 0.0 0.0 5.0 27.0 28.0 0.0 0.0 0.0 0.0 0.0 14.0 12.0 12.0 10.0 4.0 1.0 13.0 0.0 2.0 0.0 0.0 0.0 17.0 26.0 8.0 6.0 0.0 0.0 0.0 2.0 0.0 0.0 8.0 2.0 45.0 48.0 0.0 13.0 0.0 7.0 6.0 0.0 0.0 14.0 9.0 39.0 0.0 ... 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 2.0 0.0 0.0 0.0 0.0 1.0 2.0 0.0 0.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 4.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 2.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 1.0
5 19146.0 9925.0 5912.0 3309.0 2630.0 1942.0 3817.0 1536.0 1694.0 2492.0 1726.0 2.0 1175.0 2132.0 14603.0 1814.0 2729.0 1209.0 2869.0 10525.0 1714.0 100.0 1802.0 927.0 7178.0 1181.0 1566.0 4602.0 501.0 809.0 2823.0 132.0 354.0 729.0 1239.0 5210.0 297.0 1143.0 3162.0 183.0 487.0 2153.0 1139.0 42.0 448.0 53.0 127.0 5.0 935.0 631.0 598.0 211.0 269.0 85.0 2032.0 268.0 330.0 1278.0 296.0 368.0 2761.0 364.0 692.0 1107.0 253.0 176.0 1329.0 102.0 263.0 1890.0 257.0 345.0 1383.0 142.0 201.0 2.0 0.0 0.0 967.0 58.0 150.0 1854.0 74.0 204.0 182.0 354.0 1496.0 220.0 102.0 956.0 220.0 543.0 1998.0 107.0 240.0 760.0 263.0 5.0 283.0 13.0 344.0 20.0 249.0 4.0 195.0 135.0 217.0 151.0 420.0 272.0 103.0 73.0 108.0 356.0 865.0 128.0 419.0 1343.0 61.0 368.0 954.0 91.0 11.0 231.0 26.0 126.0 16.0 196.0 67.0 247.0 98.0 155.0 46.0 0.0 0.0 2.0 96.0 147.0 724.0 87.0 340.0 1427.0 0.0 0.0 57.0 1.0 70.0 4.0 0.0 0.0 118.0 32.0 151.0 53.0 7.0 3.0 3.0 0.0 10.0 14.0 0.0 0.0 0.0 0.0 0.0 14.0 0.0 0.0 7.0 0.0 13.0 51.0 6.0 0.0 0.0 0.0 15.0 0.0 5.0 0.0 0.0 0.0 1.0 13.0 48.0 0.0 0.0 1.0 26.0 1.0 0.0 0.0 8.0 6.0 0.0 0.0 65.0 18.0 27.0 13.0 0.0 7.0 0.0 0.0 0.0 6.0 11.0 20.0 0.0 0.0 0.0 0.0 0.0 21.0 11.0 4.0 7.0 13.0 3.0 18.0 0.0 4.0 0.0 0.0 0.0 6.0 11.0 6.0 4.0 0.0 0.0 0.0 0.0 0.0 0.0 8.0 0.0 25.0 13.0 0.0 17.0 0.0 3.0 9.0 0.0 0.0 7.0 9.0 28.0 0.0 ... 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 1.0 0.0 0.0 2.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

5 rows × 42840 columns

# All levels
df_train = agg_pd.T.reset_index()
df_train = df_train.set_index("id") # id as index
rename_dict = {}
for c in df_train.columns:
    if c not in ['level', 'state_id', 'store_id',   'cat_id',  'dept_id',  'item_id']:
        rename_dict[c] = "d_%s" % c
df_train.rename(columns=rename_dict, inplace=True)
day_cols = pd.Series([c for c in df_train.columns if c not in ['level', 'state_id', 'store_id',   'cat_id',  'dept_id',  'item_id']]) # d_1 to d_1913
print(df_train.shape)
df_train.head()

(42840, 1919)
d level state_id store_id cat_id dept_id item_id d_1 d_2 d_3 d_4 d_5 d_6 d_7 d_8 d_9 d_10 d_11 d_12 d_13 d_14 d_15 d_16 d_17 d_18 d_19 d_20 d_21 d_22 d_23 d_24 d_25 d_26 d_27 d_28 d_29 d_30 d_31 d_32 d_33 d_34 d_35 d_36 d_37 d_38 d_39 d_40 d_41 d_42 d_43 d_44 d_45 d_46 d_47 d_48 d_49 d_50 d_51 d_52 d_53 d_54 d_55 d_56 d_57 d_58 d_59 d_60 d_61 d_62 d_63 d_64 d_65 d_66 d_67 d_68 d_69 d_70 d_71 d_72 d_73 d_74 d_75 d_76 d_77 d_78 d_79 d_80 d_81 d_82 d_83 d_84 d_85 d_86 d_87 d_88 d_89 d_90 d_91 d_92 d_93 d_94 d_95 d_96 d_97 d_98 d_99 d_100 d_101 d_102 d_103 d_104 d_105 d_106 d_107 d_108 d_109 d_110 d_111 d_112 d_113 d_114 d_115 d_116 d_117 d_118 d_119 d_120 d_121 d_122 d_123 d_124 d_125 d_126 d_127 d_128 d_129 d_130 d_131 d_132 d_133 d_134 d_135 d_136 d_137 d_138 d_139 d_140 d_141 d_142 d_143 d_144 d_145 d_146 d_147 d_148 d_149 d_150 d_151 d_152 d_153 d_154 d_155 d_156 d_157 d_158 d_159 d_160 d_161 d_162 d_163 d_164 d_165 d_166 d_167 d_168 d_169 d_170 d_171 d_172 d_173 d_174 d_175 d_176 d_177 d_178 d_179 d_180 d_181 d_182 d_183 d_184 d_185 d_186 d_187 d_188 d_189 d_190 d_191 d_192 d_193 d_194 d_195 d_196 d_197 d_198 d_199 d_200 d_201 d_202 d_203 d_204 d_205 d_206 d_207 d_208 d_209 d_210 d_211 d_212 d_213 d_214 d_215 d_216 d_217 d_218 d_219 d_220 d_221 d_222 d_223 d_224 d_225 d_226 d_227 d_228 d_229 d_230 d_231 d_232 d_233 d_234 d_235 d_236 d_237 d_238 d_239 d_240 d_241 d_242 d_243 d_244 ... d_1664 d_1665 d_1666 d_1667 d_1668 d_1669 d_1670 d_1671 d_1672 d_1673 d_1674 d_1675 d_1676 d_1677 d_1678 d_1679 d_1680 d_1681 d_1682 d_1683 d_1684 d_1685 d_1686 d_1687 d_1688 d_1689 d_1690 d_1691 d_1692 d_1693 d_1694 d_1695 d_1696 d_1697 d_1698 d_1699 d_1700 d_1701 d_1702 d_1703 d_1704 d_1705 d_1706 d_1707 d_1708 d_1709 d_1710 d_1711 d_1712 d_1713 d_1714 d_1715 d_1716 d_1717 d_1718 d_1719 d_1720 d_1721 d_1722 d_1723 d_1724 d_1725 d_1726 d_1727 d_1728 d_1729 d_1730 d_1731 d_1732 d_1733 d_1734 d_1735 d_1736 d_1737 d_1738 d_1739 d_1740 d_1741 d_1742 d_1743 d_1744 d_1745 d_1746 d_1747 d_1748 d_1749 d_1750 d_1751 d_1752 d_1753 d_1754 d_1755 d_1756 d_1757 d_1758 d_1759 d_1760 d_1761 d_1762 d_1763 d_1764 d_1765 d_1766 d_1767 d_1768 d_1769 d_1770 d_1771 d_1772 d_1773 d_1774 d_1775 d_1776 d_1777 d_1778 d_1779 d_1780 d_1781 d_1782 d_1783 d_1784 d_1785 d_1786 d_1787 d_1788 d_1789 d_1790 d_1791 d_1792 d_1793 d_1794 d_1795 d_1796 d_1797 d_1798 d_1799 d_1800 d_1801 d_1802 d_1803 d_1804 d_1805 d_1806 d_1807 d_1808 d_1809 d_1810 d_1811 d_1812 d_1813 d_1814 d_1815 d_1816 d_1817 d_1818 d_1819 d_1820 d_1821 d_1822 d_1823 d_1824 d_1825 d_1826 d_1827 d_1828 d_1829 d_1830 d_1831 d_1832 d_1833 d_1834 d_1835 d_1836 d_1837 d_1838 d_1839 d_1840 d_1841 d_1842 d_1843 d_1844 d_1845 d_1846 d_1847 d_1848 d_1849 d_1850 d_1851 d_1852 d_1853 d_1854 d_1855 d_1856 d_1857 d_1858 d_1859 d_1860 d_1861 d_1862 d_1863 d_1864 d_1865 d_1866 d_1867 d_1868 d_1869 d_1870 d_1871 d_1872 d_1873 d_1874 d_1875 d_1876 d_1877 d_1878 d_1879 d_1880 d_1881 d_1882 d_1883 d_1884 d_1885 d_1886 d_1887 d_1888 d_1889 d_1890 d_1891 d_1892 d_1893 d_1894 d_1895 d_1896 d_1897 d_1898 d_1899 d_1900 d_1901 d_1902 d_1903 d_1904 d_1905 d_1906 d_1907 d_1908 d_1909 d_1910 d_1911 d_1912 d_1913
id
Total_X 1 NaN NaN NaN NaN NaN 32631.0 31749.0 23783.0 25412.0 19146.0 29211.0 28010.0 37932.0 32736.0 25572.0 23071.0 23658.0 23756.0 26445.0 34833.0 36380.0 21804.0 24070.0 21443.0 20318.0 23721.0 31689.0 29283.0 23966.0 20501.0 20757.0 20277.0 22529.0 29908.0 28707.0 21240.0 22872.0 22046.0 23475.0 23572.0 31202.0 34876.0 24562.0 22752.0 22560.0 22626.0 25572.0 32225.0 31417.0 24935.0 24021.0 22765.0 21779.0 23855.0 30599.0 29643.0 23101.0 20492.0 19230.0 18854.0 23350.0 29620.0 29866.0 21449.0 19581.0 18928.0 21742.0 28309.0 33478.0 33058.0 24852.0 23581.0 22656.0 22901.0 24754.0 31745.0 31536.0 24829.0 23218.0 21611.0 21964.0 25482.0 29234.0 29787.0 22389.0 19474.0 20188.0 19768.0 24279.0 29575.0 24018.0 20004.0 18113.0 17929.0 18145.0 22735.0 28101.0 29045.0 22837.0 24080.0 20761.0 21184.0 24486.0 28412.0 23688.0 25024.0 22231.0 22165.0 22413.0 25648.0 28736.0 31320.0 22213.0 19613.0 18770.0 18684.0 21649.0 27112.0 26246.0 20954.0 19203.0 19058.0 20480.0 22481.0 25527.0 23449.0 22538.0 19555.0 23165.0 22681.0 26781.0 30310.0 31604.0 26685.0 25089.0 24453.0 24782.0 26419.0 29940.0 31094.0 25752.0 24155.0 25258.0 22706.0 24712.0 28825.0 26106.0 24261.0 21810.0 21401.0 22780.0 24638.0 26482.0 27668.0 22642.0 19800.0 19930.0 21451.0 31033.0 33001.0 29173.0 23033.0 26596.0 26602.0 26727.0 27679.0 32486.0 31539.0 26592.0 24632.0 23445.0 23447.0 27349.0 30869.0 30334.0 24639.0 21931.0 22487.0 22778.0 26356.0 30284.0 28799.0 22895.0 21195.0 20504.0 21243.0 24542.0 28953.0 28677.0 26607.0 25475.0 27237.0 23713.0 26594.0 31959.0 32006.0 26354.0 26213.0 24578.0 24708.0 28544.0 33003.0 33909.0 28511.0 25266.0 23681.0 23376.0 27004.0 32258.0 33973.0 25324.0 22617.0 21935.0 21414.0 24268.0 30383.0 30626.0 22085.0 20350.0 21723.0 26299.0 29208.0 34032.0 26925.0 30158.0 26134.0 24445.0 24048.0 28246.0 33030.0 34632.0 26138.0 24645.0 23505.0 26343.0 26933.0 32195.0 32034.0 24367.0 22675.0 21796.0 21859.0 26706.0 32466.0 33228.0 24090.0 22788.0 22193.0 22383.0 ... 33707.0 32516.0 36963.0 45080.0 47596.0 35093.0 31295.0 31967.0 32076.0 36251.0 45236.0 48517.0 35885.0 37955.0 36590.0 39619.0 39270.0 47487.0 45248.0 45762.0 36169.0 38102.0 35693.0 40286.0 50103.0 51340.0 38071.0 37425.0 33645.0 33998.0 38689.0 48279.0 49979.0 36145.0 33518.0 33497.0 32967.0 38680.0 46636.0 47613.0 34262.0 30615.0 31270.0 37677.0 44376.0 54687.0 52895.0 42474.0 38855.0 35887.0 37819.0 43595.0 48787.0 49704.0 42240.0 35681.0 34248.0 36353.0 38422.0 47813.0 48964.0 35302.0 33104.0 32411.0 32770.0 38021.0 45006.0 46179.0 35092.0 32319.0 31267.0 33232.0 40514.0 42743.0 46994.0 36457.0 37997.0 32102.0 34675.0 39621.0 47341.0 48411.0 36915.0 33374.0 33338.0 33242.0 36195.0 47342.0 49755.0 35735.0 29174.0 32160.0 32365.0 38357.0 42793.0 44925.0 35140.0 34222.0 40124.0 23631.0 27581.0 34212.0 37782.0 32016.0 34818.0 33732.0 38029.0 38187.0 47025.0 48415.0 35408.0 32913.0 33690.0 32159.0 36343.0 45975.0 45133.0 37522.0 36130.0 34123.0 35454.0 34756.0 43488.0 43011.0 35522.0 36612.0 38752.0 36227.0 14.0 38377.0 35378.0 29964.0 31893.0 34127.0 43059.0 32651.0 44737.0 44340.0 36693.0 35306.0 32685.0 35944.0 38936.0 52880.0 47369.0 37986.0 35122.0 32231.0 35856.0 41790.0 51146.0 45533.0 38340.0 31988.0 32684.0 31779.0 36307.0 48927.0 47832.0 34389.0 30446.0 31349.0 32466.0 37639.0 49104.0 50831.0 41714.0 34928.0 39340.0 35424.0 40193.0 55040.0 46893.0 36804.0 38513.0 39353.0 44584.0 44222.0 53715.0 48337.0 46152.0 37702.0 36545.0 36436.0 40392.0 50857.0 52081.0 37328.0 33981.0 34068.0 34615.0 37649.0 47887.0 50915.0 37259.0 36536.0 38458.0 41459.0 39325.0 52322.0 57218.0 40562.0 37727.0 37032.0 38267.0 40887.0 52711.0 51421.0 42035.0 40117.0 36606.0 35009.0 39652.0 46181.0 47825.0 37360.0 35475.0 34786.0 34003.0 45611.0 53863.0 46360.0 36041.0 33857.0 32359.0 34681.0 45536.0 52672.0 56425.0 40418.0 39683.0 39134.0 38116.0 43220.0 56340.0 53856.0 42427.0 39069.0 35193.0 37529.0 41789.0 48362.0 51640.0 38059.0 37570.0 35343.0 35033.0 40517.0 48962.0 49795.0
CA_X 2 CA NaN NaN NaN NaN 14195.0 13805.0 10108.0 11047.0 9925.0 11322.0 12251.0 16610.0 14696.0 11822.0 10933.0 10350.0 11204.0 11354.0 14610.0 15630.0 9044.0 9496.0 8844.0 7835.0 10025.0 13870.0 13465.0 11997.0 9130.0 8941.0 8514.0 9847.0 13652.0 12491.0 9196.0 9782.0 9537.0 9710.0 10444.0 13784.0 16057.0 10953.0 10299.0 9951.0 10012.0 11189.0 13710.0 14340.0 10722.0 9293.0 9752.0 9504.0 10030.0 14509.0 14112.0 10255.0 9383.0 8490.0 7889.0 11011.0 13816.0 13661.0 9997.0 8516.0 8911.0 10059.0 12827.0 14791.0 15341.0 12356.0 10497.0 10101.0 10575.0 10971.0 14064.0 15291.0 11098.0 9181.0 8853.0 9430.0 10922.0 12809.0 13894.0 9955.0 8930.0 8582.0 8986.0 10967.0 13577.0 11721.0 8865.0 8176.0 8135.0 7951.0 9730.0 12428.0 12997.0 10657.0 10669.0 9596.0 9281.0 10617.0 12386.0 11231.0 11264.0 10430.0 9570.0 9312.0 10978.0 12496.0 13549.0 9542.0 8492.0 8258.0 8410.0 9809.0 12007.0 12301.0 9346.0 8541.0 8835.0 9176.0 10231.0 10959.0 11115.0 10768.0 8743.0 9662.0 9889.0 11788.0 13730.0 13997.0 11171.0 10489.0 10606.0 10266.0 12188.0 12534.0 13616.0 11284.0 10400.0 10380.0 9936.0 10661.0 12745.0 11822.0 11173.0 10049.0 10125.0 10442.0 11207.0 12037.0 13032.0 10458.0 8483.0 8956.0 9822.0 13455.0 14988.0 13502.0 10583.0 11916.0 12365.0 12070.0 12505.0 14046.0 14520.0 10926.0 9697.0 9553.0 9800.0 10931.0 13438.0 13484.0 10834.0 9406.0 9930.0 10209.0 11621.0 13827.0 13467.0 10083.0 9377.0 9379.0 9548.0 11085.0 12797.0 12841.0 11381.0 11236.0 11280.0 10499.0 11494.0 13371.0 14111.0 11813.0 11441.0 11556.0 10370.0 11918.0 14227.0 15256.0 11723.0 11037.0 10163.0 10282.0 11451.0 14025.0 15415.0 11166.0 10026.0 9921.0 9527.0 10979.0 13649.0 14060.0 9890.0 8931.0 9238.0 10865.0 12472.0 14568.0 13019.0 13807.0 12063.0 10804.0 10879.0 12294.0 14664.0 15086.0 10704.0 10089.0 9776.0 10061.0 11169.0 13587.0 14843.0 11168.0 10500.0 10185.0 9984.0 12504.0 14876.0 15644.0 11231.0 10813.0 10479.0 10203.0 ... 15929.0 15362.0 16338.0 21062.0 22447.0 15643.0 14187.0 14070.0 14387.0 16343.0 21273.0 23319.0 16223.0 16305.0 15825.0 16222.0 18059.0 20994.0 19898.0 21191.0 16508.0 16348.0 16385.0 17203.0 21581.0 23552.0 15918.0 15805.0 14209.0 14737.0 17192.0 22322.0 23160.0 17236.0 15753.0 15098.0 14610.0 17432.0 21638.0 22462.0 16259.0 13677.0 13952.0 15726.0 18782.0 24286.0 24729.0 18511.0 16163.0 15797.0 15862.0 18486.0 22084.0 21329.0 18473.0 15336.0 14786.0 14546.0 16249.0 21764.0 23087.0 16005.0 14988.0 14342.0 14111.0 16117.0 20049.0 22632.0 15588.0 14858.0 14253.0 14598.0 17112.0 20376.0 21945.0 15531.0 16052.0 14266.0 14421.0 16243.0 20676.0 21942.0 15020.0 14992.0 14410.0 13137.0 14592.0 19806.0 21637.0 15127.0 13774.0 13316.0 13261.0 15181.0 19932.0 20838.0 16056.0 14937.0 17095.0 9570.0 11537.0 15088.0 17216.0 14449.0 14823.0 14174.0 15020.0 16666.0 20202.0 21101.0 15260.0 13856.0 13767.0 13802.0 14612.0 19364.0 19713.0 15479.0 14188.0 14306.0 14753.0 14594.0 19940.0 18996.0 15417.0 16026.0 16522.0 13841.0 6.0 16365.0 16025.0 15043.0 14669.0 14779.0 16248.0 13364.0 19054.0 19624.0 16455.0 14421.0 13572.0 15418.0 16242.0 22922.0 22216.0 16206.0 14058.0 13001.0 13371.0 15800.0 21380.0 20852.0 18004.0 13383.0 14241.0 13937.0 15102.0 22843.0 21845.0 15839.0 13872.0 13768.0 13798.0 15904.0 21394.0 23506.0 16618.0 15146.0 15372.0 15456.0 16905.0 23210.0 18660.0 15566.0 15574.0 16325.0 16963.0 16856.0 21848.0 19238.0 19231.0 15249.0 14555.0 15306.0 17114.0 21348.0 22538.0 16269.0 14616.0 13943.0 14517.0 16346.0 21546.0 23157.0 16232.0 16297.0 15415.0 15621.0 16095.0 22022.0 25224.0 17148.0 16083.0 15544.0 15835.0 16275.0 22515.0 21937.0 16858.0 15051.0 14807.0 14196.0 16664.0 18900.0 19421.0 16541.0 15581.0 15141.0 15228.0 18694.0 23787.0 21296.0 15482.0 15319.0 14352.0 15601.0 18922.0 22171.0 24577.0 18193.0 16396.0 15807.0 15459.0 17714.0 23311.0 24063.0 17490.0 15192.0 14614.0 15069.0 16255.0 20564.0 23032.0 17052.0 15784.0 15148.0 14488.0 17095.0 21834.0 23187.0
TX_X 2 TX NaN NaN NaN NaN 9438.0 9630.0 6778.0 7381.0 5912.0 9006.0 6226.0 9440.0 9376.0 7319.0 6224.0 6740.0 6109.0 7202.0 10081.0 11725.0 6719.0 7460.0 6606.0 6412.0 6711.0 9150.0 9789.0 7297.0 6628.0 5814.0 6175.0 6286.0 8586.0 9086.0 6555.0 7082.0 6145.0 6639.0 6584.0 9238.0 10423.0 7623.0 6335.0 6941.0 6339.0 7203.0 9271.0 9542.0 7745.0 7834.0 6768.0 6759.0 7131.0 8203.0 8992.0 7075.0 6086.0 5914.0 5898.0 6249.0 8072.0 9434.0 6429.0 5728.0 5384.0 6096.0 7992.0 9134.0 9863.0 6648.0 6743.0 6689.0 7201.0 6797.0 9025.0 9648.0 7555.0 7302.0 7202.0 6338.0 7749.0 8573.0 9248.0 7218.0 6133.0 6387.0 5677.0 7542.0 8833.0 7897.0 6339.0 5535.0 5369.0 5372.0 6432.0 8240.0 9644.0 6125.0 6937.0 5771.0 6372.0 7924.0 8863.0 7440.0 8001.0 6523.0 6755.0 6708.0 8231.0 8663.0 10237.0 7141.0 6179.0 5787.0 5680.0 6337.0 8288.0 8518.0 6545.0 5813.0 5958.0 6283.0 6343.0 8286.0 7251.0 7152.0 5865.0 7780.0 6662.0 8130.0 8753.0 9712.0 8655.0 7955.0 7409.0 7772.0 7823.0 9628.0 10570.0 8731.0 7320.0 8142.0 6941.0 7876.0 9080.0 8521.0 7684.0 6636.0 5859.0 7007.0 7327.0 8300.0 9188.0 7157.0 6520.0 6162.0 6353.0 9592.0 9559.0 9194.0 7983.0 8431.0 7508.0 8617.0 7928.0 10024.0 9895.0 8868.0 7968.0 8087.0 7225.0 8809.0 9541.0 9758.0 7794.0 6577.0 6950.0 6885.0 7789.0 9133.0 9310.0 7536.0 6582.0 6447.0 6597.0 7294.0 8868.0 9379.0 9139.0 7367.0 8535.0 7618.0 8380.0 10505.0 11146.0 8302.0 8178.0 7266.0 7950.0 9338.0 11454.0 11302.0 9825.0 8152.0 7827.0 7770.0 8829.0 11170.0 12340.0 8584.0 7483.0 6530.0 6464.0 7184.0 9774.0 10547.0 7054.0 6258.0 6378.0 8827.0 8211.0 10987.0 7538.0 9371.0 7545.0 8306.0 6992.0 8722.0 10427.0 11559.0 8492.0 8996.0 7365.0 8914.0 8672.0 10510.0 10658.0 8065.0 7056.0 6403.0 6554.0 7942.0 10062.0 10890.0 7635.0 6924.0 6453.0 6762.0 ... 8427.0 8422.0 10480.0 12577.0 13820.0 10580.0 8651.0 8832.0 8540.0 9151.0 11377.0 12995.0 9144.0 10186.0 9203.0 10409.0 9859.0 12589.0 12134.0 12774.0 9278.0 9603.0 8895.0 10077.0 13142.0 14305.0 10455.0 9943.0 9327.0 9459.0 10495.0 12856.0 14179.0 9632.0 9238.0 9629.0 9187.0 10013.0 12286.0 12610.0 9332.0 8387.0 8352.0 10941.0 11330.0 13622.0 13932.0 11484.0 10763.0 10148.0 9959.0 11322.0 12528.0 13998.0 11422.0 10176.0 8157.0 10253.0 10286.0 12670.0 12973.0 10116.0 9083.0 8946.0 9326.0 11153.0 11592.0 10916.0 10108.0 8704.0 8751.0 9189.0 10320.0 9895.0 12814.0 9746.0 9959.0 8918.0 9225.0 10509.0 12764.0 12438.0 10320.0 9092.0 8878.0 9034.0 9988.0 11793.0 13557.0 10366.0 7636.0 9433.0 9213.0 9561.0 11792.0 12072.0 9741.0 9596.0 11632.0 7855.0 7928.0 9194.0 9885.0 8775.0 9739.0 8501.0 9694.0 9141.0 11449.0 12379.0 10135.0 8471.0 8780.0 8838.0 9747.0 12133.0 12663.0 10823.0 10523.0 10049.0 10397.0 9375.0 10986.0 11666.0 9754.0 9970.0 10625.0 10755.0 4.0 11906.0 9527.0 9165.0 8884.0 8740.0 11162.0 8845.0 10408.0 11081.0 9856.0 9332.0 7928.0 10507.0 10043.0 13158.0 12633.0 10526.0 10190.0 9539.0 9309.0 10858.0 12764.0 13106.0 10685.0 9010.0 8769.0 8657.0 10132.0 11836.0 12719.0 9645.0 8158.0 8512.0 8896.0 9741.0 11933.0 13014.0 11700.0 8943.0 10894.0 9311.0 10350.0 13611.0 14149.0 10266.0 10540.0 10835.0 12668.0 11836.0 14363.0 13392.0 12115.0 10341.0 9904.0 10102.0 10826.0 13282.0 14663.0 10415.0 9459.0 9842.0 9492.0 9897.0 11575.0 13376.0 9963.0 10842.0 9902.0 11396.0 10158.0 12495.0 14155.0 11603.0 9786.0 9091.0 10648.0 11066.0 13113.0 13466.0 11882.0 11659.0 10478.0 10170.0 10756.0 12210.0 13428.0 9973.0 9826.0 10341.0 9305.0 12235.0 13595.0 13336.0 10303.0 8322.0 8828.0 9306.0 11852.0 13095.0 14051.0 10497.0 10466.0 10338.0 10600.0 10766.0 13360.0 13818.0 11174.0 10418.0 9694.0 10292.0 10800.0 12460.0 13709.0 9868.0 10922.0 9600.0 9602.0 10615.0 12266.0 12282.0
WI_X 2 WI NaN NaN NaN NaN 8998.0 8314.0 6897.0 6984.0 3309.0 8883.0 9533.0 11882.0 8664.0 6431.0 5914.0 6568.0 6443.0 7889.0 10142.0 9025.0 6041.0 7114.0 5993.0 6071.0 6985.0 8669.0 6029.0 4672.0 4743.0 6002.0 5588.0 6396.0 7670.0 7130.0 5489.0 6008.0 6364.0 7126.0 6544.0 8180.0 8396.0 5986.0 6118.0 5668.0 6275.0 7180.0 9244.0 7535.0 6468.0 6894.0 6245.0 5516.0 6694.0 7887.0 6539.0 5771.0 5023.0 4826.0 5067.0 6090.0 7732.0 6771.0 5023.0 5337.0 4633.0 5587.0 7490.0 9553.0 7854.0 5848.0 6341.0 5866.0 5125.0 6986.0 8656.0 6597.0 6176.0 6735.0 5556.0 6196.0 6811.0 7852.0 6645.0 5216.0 4411.0 5219.0 5105.0 5770.0 7165.0 4400.0 4800.0 4402.0 4425.0 4822.0 6573.0 7433.0 6404.0 6055.0 6474.0 5394.0 5531.0 5945.0 7163.0 5017.0 5759.0 5278.0 5840.0 6393.0 6439.0 7577.0 7534.0 5530.0 4942.0 4725.0 4594.0 5503.0 6817.0 5427.0 5063.0 4849.0 4265.0 5021.0 5907.0 6282.0 5083.0 4618.0 4947.0 5723.0 6130.0 6863.0 7827.0 7895.0 6859.0 6645.0 6438.0 6744.0 6408.0 7778.0 6908.0 5737.0 6435.0 6736.0 5829.0 6175.0 7000.0 5763.0 5404.0 5125.0 5417.0 5331.0 6104.0 6145.0 5448.0 5027.0 4797.0 4812.0 5276.0 7986.0 8454.0 6477.0 4467.0 6249.0 6729.0 6040.0 7246.0 8416.0 7124.0 6798.0 6967.0 5805.0 6422.0 7609.0 7890.0 7092.0 6011.0 5948.0 5607.0 5684.0 6946.0 7324.0 6022.0 5276.0 5236.0 4678.0 5098.0 6163.0 7288.0 6457.0 6087.0 6872.0 7422.0 5596.0 6720.0 8083.0 6749.0 6239.0 6594.0 5756.0 6388.0 7288.0 7322.0 7351.0 6963.0 6077.0 5691.0 5324.0 6724.0 7063.0 6218.0 5574.0 5108.0 5484.0 5423.0 6105.0 6960.0 6019.0 5141.0 5161.0 6107.0 6607.0 8525.0 8477.0 6368.0 6980.0 6526.0 5335.0 6177.0 7230.0 7939.0 7987.0 6942.0 5560.0 6364.0 7368.0 7092.0 8098.0 6533.0 5134.0 5119.0 5208.0 5321.0 6260.0 7528.0 6694.0 5224.0 5051.0 5261.0 5418.0 ... 9351.0 8732.0 10145.0 11441.0 11329.0 8870.0 8457.0 9065.0 9149.0 10757.0 12586.0 12203.0 10518.0 11464.0 11562.0 12988.0 11352.0 13904.0 13216.0 11797.0 10383.0 12151.0 10413.0 13006.0 15380.0 13483.0 11698.0 11677.0 10109.0 9802.0 11002.0 13101.0 12640.0 9277.0 8527.0 8770.0 9170.0 11235.0 12712.0 12541.0 8671.0 8551.0 8966.0 11010.0 14264.0 16779.0 14234.0 12479.0 11929.0 9942.0 11998.0 13787.0 14175.0 14377.0 12345.0 10169.0 11305.0 11554.0 11887.0 13379.0 12904.0 9181.0 9033.0 9123.0 9333.0 10751.0 13365.0 12631.0 9396.0 8757.0 8263.0 9445.0 13082.0 12472.0 12235.0 11180.0 11986.0 8918.0 11029.0 12869.0 13901.0 14031.0 11575.0 9290.0 10050.0 11071.0 11615.0 15743.0 14561.0 10242.0 7764.0 9411.0 9891.0 13615.0 11069.0 12015.0 9343.0 9689.0 11397.0 6206.0 8116.0 9930.0 10681.0 8792.0 10256.0 11057.0 13315.0 12380.0 15374.0 14935.0 10013.0 10586.0 11143.0 9519.0 11984.0 14478.0 12757.0 11220.0 11419.0 9768.0 10304.0 10787.0 12562.0 12349.0 10351.0 10616.0 11605.0 11631.0 4.0 10106.0 9826.0 5756.0 8340.0 10608.0 15649.0 10442.0 15275.0 13635.0 10382.0 11553.0 11185.0 10019.0 12651.0 16800.0 12520.0 11254.0 10874.0 9691.0 13176.0 15132.0 17002.0 11575.0 9651.0 9595.0 9674.0 9185.0 11073.0 14248.0 13268.0 8905.0 8416.0 9069.0 9772.0 11994.0 15777.0 14311.0 13396.0 10839.0 13074.0 10657.0 12938.0 18219.0 14084.0 10972.0 12399.0 12193.0 14953.0 15530.0 17504.0 15707.0 14806.0 12112.0 12086.0 11028.0 12452.0 16227.0 14880.0 10644.0 9906.0 10283.0 10606.0 11406.0 14766.0 14382.0 11064.0 9397.0 13141.0 14442.0 13072.0 17805.0 17839.0 11811.0 11858.0 12397.0 11784.0 13546.0 17083.0 16018.0 13295.0 13407.0 11321.0 10643.0 12232.0 15071.0 14976.0 10846.0 10068.0 9304.0 9470.0 14682.0 16481.0 11728.0 10256.0 10216.0 9179.0 9774.0 14762.0 17406.0 17797.0 11728.0 12821.0 12989.0 12057.0 14740.0 19669.0 15975.0 13763.0 13459.0 10885.0 12168.0 14734.0 15338.0 14899.0 11139.0 10864.0 10595.0 10943.0 12807.0 14862.0 14326.0
CA_1_X 3 CA CA_1 NaN NaN NaN 4337.0 4155.0 2816.0 3051.0 2630.0 3276.0 3450.0 5437.0 4340.0 3157.0 2995.0 2710.0 2928.0 3078.0 4316.0 4354.0 2757.0 2430.0 2272.0 2009.0 3015.0 4553.0 3966.0 3264.0 2441.0 2377.0 2414.0 2701.0 3719.0 3548.0 2246.0 2549.0 2559.0 2568.0 2978.0 4096.0 4832.0 3138.0 2997.0 2772.0 2650.0 3249.0 4114.0 4094.0 2880.0 2384.0 2522.0 2435.0 2744.0 4142.0 3904.0 2635.0 2458.0 2079.0 2281.0 2740.0 3899.0 3999.0 2780.0 2167.0 2247.0 2576.0 3500.0 4325.0 4396.0 3642.0 2784.0 2602.0 2976.0 3294.0 4083.0 4194.0 3015.0 2512.0 2217.0 2520.0 2867.0 3571.0 4084.0 2760.0 2234.0 2261.0 2487.0 3155.0 4187.0 2946.0 2439.0 2085.0 2096.0 2086.0 2675.0 3560.0 3668.0 2911.0 2839.0 2610.0 2384.0 3125.0 3816.0 3033.0 3069.0 2985.0 2629.0 2277.0 3008.0 3754.0 3822.0 2582.0 2144.0 2174.0 2168.0 2664.0 3463.0 3597.0 2461.0 2160.0 2251.0 2296.0 2642.0 3214.0 3202.0 2884.0 2156.0 2529.0 2652.0 3200.0 3942.0 3871.0 3056.0 2763.0 2884.0 2787.0 3420.0 3588.0 3569.0 3149.0 2894.0 2679.0 2535.0 2971.0 3896.0 3109.0 2848.0 2936.0 2557.0 2624.0 3071.0 3354.0 3672.0 2912.0 2200.0 2166.0 2542.0 3681.0 4276.0 3885.0 2672.0 3177.0 3542.0 3362.0 3252.0 3923.0 4286.0 2785.0 2380.0 2501.0 2467.0 2922.0 3688.0 3587.0 2936.0 2373.0 2511.0 2585.0 3069.0 3851.0 3773.0 2762.0 2322.0 2337.0 2518.0 2915.0 2930.0 3493.0 2860.0 3018.0 2919.0 3003.0 3450.0 3818.0 3930.0 3246.0 2862.0 2699.0 2728.0 3292.0 4077.0 4202.0 2879.0 2718.0 2577.0 2574.0 3080.0 3702.0 4462.0 2722.0 2217.0 2440.0 2483.0 2873.0 3769.0 3701.0 2407.0 2114.0 2291.0 2668.0 3412.0 4127.0 3542.0 3485.0 3130.0 2674.0 2826.0 3238.0 3873.0 4046.0 2835.0 2383.0 2186.0 2388.0 2813.0 3515.0 3670.0 2606.0 2558.0 2453.0 2366.0 3475.0 4059.0 4086.0 2846.0 2438.0 2581.0 2437.0 ... 3868.0 3876.0 4198.0 5122.0 6222.0 3888.0 3364.0 3558.0 3544.0 4001.0 5368.0 5869.0 4164.0 3983.0 3785.0 4327.0 4728.0 5757.0 5476.0 5636.0 4106.0 3813.0 3845.0 4380.0 5467.0 6052.0 4133.0 3928.0 3405.0 3773.0 4675.0 5627.0 6123.0 4425.0 3912.0 3656.0 3430.0 4515.0 5421.0 5838.0 4053.0 3293.0 3377.0 3941.0 5145.0 6433.0 6668.0 4847.0 4138.0 3956.0 3992.0 4748.0 5923.0 6025.0 4880.0 3977.0 3665.0 3611.0 4122.0 5582.0 6214.0 4094.0 3717.0 3717.0 3211.0 4076.0 5164.0 6013.0 3953.0 3658.0 3400.0 3380.0 4523.0 5214.0 5260.0 4173.0 4103.0 3622.0 3541.0 4284.0 5414.0 5599.0 3828.0 3928.0 3702.0 2984.0 3652.0 4954.0 5817.0 3592.0 3503.0 3463.0 3359.0 4037.0 5115.0 5614.0 4607.0 3942.0 4438.0 2160.0 2696.0 3812.0 4585.0 3806.0 3879.0 3673.0 3691.0 4084.0 5286.0 5352.0 4020.0 3560.0 3417.0 3347.0 3733.0 5130.0 5080.0 3989.0 3567.0 3545.0 3791.0 3843.0 5317.0 5069.0 4451.0 4224.0 4300.0 3803.0 0.0 4137.0 4010.0 3867.0 3884.0 3613.0 3994.0 3418.0 5085.0 4986.0 4237.0 3481.0 3225.0 3867.0 4404.0 6077.0 5660.0 4001.0 3377.0 3252.0 3405.0 4026.0 5745.0 5422.0 4484.0 3026.0 3544.0 3502.0 3612.0 5923.0 5247.0 3741.0 3246.0 3031.0 3173.0 3871.0 5498.0 6032.0 4075.0 3765.0 3983.0 3928.0 4327.0 6412.0 4986.0 3933.0 3936.0 3995.0 4007.0 4111.0 5483.0 4834.0 4588.0 3650.0 3777.0 3868.0 4413.0 5283.0 5607.0 3905.0 3613.0 3369.0 3396.0 4087.0 5329.0 6198.0 3797.0 3830.0 3754.0 3769.0 4163.0 5630.0 6829.0 4393.0 3930.0 3672.0 3877.0 4049.0 5933.0 5446.0 4190.0 3457.0 3587.0 3329.0 4285.0 5659.0 6197.0 4242.0 3776.0 3770.0 3970.0 4904.0 6139.0 4669.0 3905.0 3788.0 3339.0 4076.0 4983.0 5877.0 6496.0 4812.0 4063.0 3964.0 3655.0 4719.0 6112.0 5969.0 4484.0 3579.0 3478.0 3637.0 3982.0 5437.0 5954.0 4345.0 3793.0 3722.0 3709.0 4387.0 5577.0 6113.0

5 rows × 1919 columns

# agg_price_pd = aggregate_all_levels(price_ts)
# agg_price_pd.head()

# df_train_price = agg_price_pd.T.reset_index()
# df_train_price = df_train_price.set_index("id") # id as index
# rename_dict = {}
# for c in df_train_price.columns:
#     if c not in ['level', 'state_id', 'store_id',   'cat_id',  'dept_id',  'item_id']:
#         rename_dict[c] = "d_%s" % c
# df_train_price.rename(columns=rename_dict, inplace=True)
# day_prices_cols = pd.Series([c for c in df_train_price.columns if c not in ['level', 'state_id', 'store_id',   'cat_id',  'dept_id',  'item_id']]) # d_1 to d_1913+
# print(df_train_price.shape)
# df_train_price.head()

# Level 12 only
df_sale = df_train[df_train["level"] == 12]
df_sale.shape

(30490, 1919)

# Levels <= 9
df_train = df_train[df_train["level"] <= MAX_LEVEL]
print(df_train.shape)

(154, 1919)

# df_train_price = df_train_price[df_train_price["level"] <= MAX_LEVEL]
# print(df_train_price.shape)

Prepare data for Prophet

# Prepare calendar columns
df_calendar = pd.read_csv(CALENDAR)
df_calendar.index = df_calendar['d'].values # d_xxx as index
df_calendar['ds'] = pd.to_datetime(df_calendar['date']) # move date as datetime in "ds" column
df_calendar['quarter'] = df_calendar['ds'].dt.quarter # add quarter feature
df_calendar.head()
date wm_yr_wk weekday wday month year d event_name_1 event_type_1 event_name_2 event_type_2 snap_CA snap_TX snap_WI ds quarter
d_1 2011-01-29 11101 Saturday 1 1 2011 d_1 NaN NaN NaN NaN 0 0 0 2011-01-29 1
d_2 2011-01-30 11101 Sunday 2 1 2011 d_2 NaN NaN NaN NaN 0 0 0 2011-01-30 1
d_3 2011-01-31 11101 Monday 3 1 2011 d_3 NaN NaN NaN NaN 0 0 0 2011-01-31 1
d_4 2011-02-01 11101 Tuesday 4 2 2011 d_4 NaN NaN NaN NaN 1 1 0 2011-02-01 1
d_5 2011-02-02 11101 Wednesday 5 2 2011 d_5 NaN NaN NaN NaN 1 0 1 2011-02-02 1 
# Generate holidays ds
events1 = pd.Series(df_calendar['event_name_1'].values, index=df_calendar['ds'].values).dropna()
events2 = pd.Series(df_calendar['event_name_2'].values, index=df_calendar['ds'].values).dropna()
holidays = pd.DataFrame(pd.concat([events1, events2], axis=0))
holidays['ds'] = holidays.index.values
holidays.rename({0: 'holiday'}, axis=1, inplace=True)
holidays.reset_index(drop=True, inplace=True)
del events1, events2
holidays.head()
holiday ds
0 SuperBowl 2011-02-06
1 ValentinesDay 2011-02-14
2 PresidentsDay 2011-02-21
3 LentStart 2011-03-09
4 LentWeek2 2011-03-16 
# Clean data: remove leading zeros and outliers
def clean_data(df_train, day_cols, indx):
    t = df_train.loc[indx].copy()
    t.loc[day_cols[((t.loc[day_cols]>0).cumsum()==0).values]] = np.nan
    q1 = t.loc[day_cols].quantile(0.25)
    q3 = t.loc[day_cols].quantile(0.75)
    iqr = q3-q1
    qm = (q3+1.5*iqr)
    t.loc[day_cols][t.loc[day_cols]>qm] = qm
    return t

future_preds = 28
day_fit_cols = day_cols

# Remove noticeable dates that are not in evaluation
ignore_dates = ["2011-12-25", "2012-12-25", "2013-12-25", "2014-12-25", "2015-12-25",
                "2011-12-24", "2012-12-24", "2013-12-24", "2014-12-24", "2015-12-24",
                "2011-12-31", "2012-12-31", "2013-12-31", "2014-12-31", "2015-12-31",
                "2012-01-01", "2013-01-01", "2014-01-01", "2015-01-01", "2016-01-01"]
ignore_days = [] # df_calendar[df_calendar['ds'].isin(ignore_dates)]["d"].apply(lambda x: int(x[2:])).values

FIRST = 338 # 1069 # 704 # 1 # 704
day_fit_cols = ["d_%d"%c for c in range(FIRST, 1914) if c not in ignore_days]
df_train = df_train[['item_id', 'dept_id', 'cat_id', 'store_id', 'state_id', 'level'] + day_fit_cols]
df_calendar = df_calendar[df_calendar["d"].isin(["d_%d"%c for c in range(FIRST, 1942) if c not in ignore_days])]
#df_prices = df_prices[["d_%d"%c for c in range(FIRST, 1942) if c not in ignore_days]]

4. Prophet聚合数据预测

关于Prophet预测的数据形式和参数含义

  • 使用add_regressor方法将附加的回归量添加到模型的线性部分
  • 输出

========================= y列增加28天的预测为nan =========================
(1604, 5)
============================== y列滞后项 ==============================
(1604, 11)
============================== df.columns ==============================
Index([‘ds’, ‘month’, ‘wday’, ‘quarter’, ‘y’, ‘ft_lag28’, ‘ft_lag35’,
‘ft_lag42’, ‘ft_lag28_roll28_std’, ‘ft_lag28_roll28_max’,
‘ft_lag28_roll28_min’],
dtype=‘object’)
target_first_valid_index: d_338
训练开始 ------------------------------------------------------------------
预测开始 ------------------------------------------------------------------
============================== 预测结果 ==============================
(1604, 63)
Average LRMSE=0.000

4.1 预测模型定义

# df_train: "id" ["d_1", "d_2", ...]
# holidays: ["holiday", "ds"]
# df_calendar d ["date", "ds", "weekday", "wday", "month", "year", "quarter" ...]
# df_prices: "id" ["d_1", "d_2", ...]
def make_prediction(indx, model_columns = 'yhat', ret_columns = 'yhat', full_predict = True):
    global df_train, holidays, df_calendar # df_train_price, df_prices
    # full_predict = True
    # Return either series or dataframe
    # model_columns = 'yhat' # ["yhat", "yhat_lower", "yhat_upper"]  # 'yhat' ["yhat"]
    # ret_columns = model_columns # + ["ds", "y"]
    changepoints=list()
    uncertainty_samples=False # False (True to include yhat_upper ...)
    changepoint_prior_scale=0.1
    changepoint_range=0.9
    n_changepoints=50
    holidays_prior_scale=10
    yearly_seasonality=10 #2
    weekly_seasonality=3 #1
    daily_seasonality=False
    monthly_fourier_order=8
    quarter_fourier_order=6 # 6 #None
    seasonality_prior_scale=10
    seasonality_mode = 'multiplicative'  # 'additive' 两种模式，默认情况下是加法模式
    
    target = df_train.loc[indx, day_fit_cols] # sales for one time series
    # target_price = df_train_price.loc[indx, day_prices_cols] # day_fit_cols

    snap_state_id = str(df_train.loc[indx, 'state_id'])
    cols = ['ds', 'month', 'wday', 'quarter']
    if snap_state_id in ["CA", "TX", "WI"]:
        cols = cols + ['snap_'+snap_state_id]

    # Create temporary dataframe for prediction from 2011-01-29	to 2016-05-22 (d_1941) initialized with NaN for values to predict
    # ["ds", "y", "prices", "month", "wday", "quarter", "snap_xx"] (snap matching to state related to id)
    df = df_calendar.iloc[:target.shape[0]+future_preds][cols].copy()
    df['y'] = target
    # print('========================= y列增加28天的预测为nan =========================')
    # print(df.shape)
    # print(df.tail())
    
    # Clip outliers in aggregated time series
    #q1 = df['y'].quantile(0.25)
    #q3 = df['y'].quantile(0.75)
    #iqr = q3-q1
    #qm_up = (q3+1.5*iqr)
    #qm_dw = (q1-1.5*iqr)
    #df.loc[df["y"] > qm_up, "y"] = qm_up
    #df.loc[df["y"] < qm_dw, "y"] = qm_dw
    
    #df['ft_lag365'] = df['y'].shift(365)
    #df["ft_lag365"].fillna(method ='bfill', inplace = True)
    df['ft_lag28'] = df['y'].shift(28)
    df["ft_lag28"].fillna(method ='bfill', inplace = True)
    df['ft_lag35'] = df['y'].shift(35)
    df["ft_lag35"].fillna(method ='bfill', inplace = True)
    df['ft_lag42'] = df['y'].shift(42)
    df["ft_lag42"].fillna(method ='bfill', inplace = True)
    df['ft_lag28_roll28_std'] = df['y'].shift(28).rolling(28).std()
    df["ft_lag28_roll28_std"].fillna(method ='bfill', inplace = True)
    df['ft_lag28_roll28_max'] = df['y'].shift(28).rolling(28).max()
    df["ft_lag28_roll28_max"].fillna(method ='bfill', inplace = True)
    df['ft_lag28_roll28_min'] = df['y'].shift(28).rolling(28).min()
    df["ft_lag28_roll28_min"].fillna(method ='bfill', inplace = True)
    # Add prices
    #df['prices'] = target_price.astype(np.float32)
    #df["prices"].fillna(method ='bfill', inplace = True)
    #df['ft_prices_roll7_std'] = df['prices'].rolling(7).std()
    #df["ft_prices_roll7_std"].fillna(method ='bfill', inplace = True)    

    # print('============================== y列滞后项 ==============================')
    # print(df.shape)
    # print(df.tail())
    
    # 相关参数解释见：
    # https://blog.csdn.net/anshuai_aw1/article/details/83412058 比较全的讲解Prophet的中文文档
    # https://blog.csdn.net/Joely_112/article/details/100090176  一些重要的Prophet参数调节说明
    # https://www.pianshen.com/article/3525122198/ 讲解也是蛮细的
    # https://zhuanlan.zhihu.com/p/33189135
    # https://zr9558.com/2018/11/30/timeseriespredictionfbprophet/
    GROWTH = 'linear' #'logistic' #'linear'
    m = Prophet(growth=GROWTH, 
                # holidays是节假日信息输入，holiday是一个文件，里面包含有节假日的信息
                # 存在突增、突降时使用。降低整体预测偏差
                holidays=holidays, 
                
                # yearly_seasonality 是年规律拟合，Prophet模型会描绘出以一年为单位的数据规律；
                # 同理于参数 weekly_seasonality，daily_seasonality
                yearly_seasonality=yearly_seasonality,
                daily_seasonality=daily_seasonality, 
                weekly_seasonality=weekly_seasonality,
                
                # changepoint_range 是设定转折点可以存在的范围，0.1表示存在于历史数据的前十分之一，
                # 0.5表示在历史数据的前半部分，其余同理。
                # changepoints=[] 是指定转折点的具体位置，指定转折点的具体位置，没有模型自适应拟合得好。
                changepoint_range=changepoint_range,
                
                # 调整趋势的灵活性。
                changepoint_prior_scale=changepoint_prior_scale, 
                
                # 转折点的数量，需要调整
                n_changepoints = n_changepoints,
                
                # uncertainty_samples：用于估计未来时间的增长趋势间隔的仿真绘制数，默认值：1000。
                uncertainty_samples=uncertainty_samples, 
                
                # holidays_prior_scale（holidays模型中的）：调节节假日模型组件的强度。
                # 值越大，该节假日对模型的影响越大，值越小，节假日的影响越小，默认值：10.0。
                holidays_prior_scale=holidays_prior_scale, 

                # 使用seasonality_mode ='multiplicative'，假日效果将被建模为乘法。
                # 默认情况下，任何添加的季节性或额外的回归量都将使用设置的seasonality_mode，
                # 但可以通过在添加季节性或回归量时指定mode ='additive’或mode ='multiplicative’作为参数来覆盖。
                seasonality_mode=seasonality_mode, 
                seasonality_prior_scale=seasonality_prior_scale)
    
    m.add_country_holidays(country_name='US')

    if not monthly_fourier_order is None:
        m.add_seasonality(name='monthly', period=365.25/12, fourier_order=monthly_fourier_order)
    if not quarter_fourier_order is None:
        m.add_seasonality(name='quarterly', period=365.25/4, fourier_order=quarter_fourier_order)

    # Add regressor for month, wday, quarter (snap_XX, prices)
    # print('============================== df.columns ==============================')
    # print(df.columns)
    for reg in df.columns:
        if reg not in ['ds', 'y']:
            # 增加要回归数据的特征列，可以使用add_regressor方法将附加的回归量添加到模型的线性部分。
            # 重要地方！！！！！！！！！！！！！！！！！！
            m.add_regressor(reg)

    target_first_valid_index = target.first_valid_index()
    # print('target_first_valid_index: ', target_first_valid_index)

    if GROWTH == "logistic":
        df["cap"] = df["y"].max()
        df["floor"] = df["y"].min()
        
    # Fit on existing data (first_valid_index = Return index for first non-NA/null value.)
    
    # print('训练开始 ------------------------------------------------------------------')
    m.fit(df.loc[target.loc[target_first_valid_index:].index])

    # Remove target
    if 'y' not in ret_columns:
        df.drop(['y'], axis=1, inplace=True)

    res = None
    if full_predict == True:
        # print('预测开始 ------------------------------------------------------------------')
        forecast = m.predict(df.loc[target_first_valid_index:]) # For all days with valid data
        # print('============================== 预测结果 ==============================')
        # print(forecast.shape)
        # print(forecast.head())
        
        forecast["yhat"] = forecast["yhat"].astype(np.float32)
        res = forecast[model_columns]
        # Update prediction from first valid index from 2016-05-22
        res.index = df.loc[target_first_valid_index:].index.values
        res = df.merge(res, left_index=True, right_index=True, how="left")[ret_columns]

    else:
        forecast = m.predict(df.iloc[-future_preds:]) # for last 28 days (2016-04-25 to 2016-05-22)
        forecast["yhat"] = forecast["yhat"].astype(np.float32)
        res = forecast[model_columns]
        # Update prediction index from d_1914 to d_1941
        res.index = df.iloc[-future_preds:].index.values

    return (indx, res)

4.2 预测一部分按照ID聚合之后的时间序列看看情况

# Basic EDA and hyperparameters tuning
# model_columns = ['yhat']
# ret_columns = model_columns + ["ds", "y"]

# L1
train_indxs = ["Total_X"]
# L2
train_indxs = train_indxs + ["CA_X", "TX_X", "WI_X"] 
# L3
train_indxs = train_indxs + ['CA_1_X', 'CA_2_X', 'CA_3_X','CA_4_X', 'TX_1_X', 'TX_2_X', 
                             'TX_3_X', 'WI_1_X', 'WI_2_X', 'WI_3_X']
# L4
train_indxs = train_indxs + ['HOBBIES_X', 'FOODS_X', 'HOUSEHOLD_X']
# L5
train_indxs = train_indxs + ['HOBBIES_1_X', 'HOBBIES_2_X', 'FOODS_1_X', 'FOODS_2_X', 
                             'FOODS_3_X', 'HOUSEHOLD_1_X', 'HOUSEHOLD_2_X'] 

m_score = 0
for train_ind in train_indxs:
    (_, pred) = make_prediction(train_ind, model_columns = ['yhat'], ret_columns = ['yhat', 'ds', 'y']) # 'prices'
    fig, ax = plt.subplots(figsize=(DEFAULT_FIG_WIDTH, 4))
    d = pred.set_index("ds").plot(kind="line", y=["y", "yhat"], ax=ax, linestyle='-', linewidth=0.8) # 'prices'
    
    score = np.log1p(mean_squared_error(pred["y"][:-28], pred["yhat"][:-28]))
    m_score = m_score + (score / len(train_indxs))
    plt.title("%s LRMSE=%.3f" % (train_ind, score))
    plt.show()
    #break
print("Average LRMSE=%.3f" % m_score)

Average LRMSE=11.870

pred.head(2)
yhat ds y
d_338 1359.151855 2012-01-01 962
d_339 1466.394287 2012-01-02 1462

4.3 所有聚合时间序列预测

  • 在这里预测是全天候，可以只预测要预测的天
  • 多时间序列预测
  • 规范化结果
  • 保存模型
print('Predicting...', flush=True)
start_time = time.time()
#pool = Pool()

train_indxs = df_train.index #[0:10]

# Memory crash with multiple CPU on Kaggle
# res = pool.map(make_prediction, train_indxs)
# res = process_map(make_prediction, train_indxs, chunksize=1)
res = []
print('========================= train_indxs长度时间序列数量 =========================')
print(len(train_indxs))
print('========================= 根据不同的id聚合的日销量预测 =========================')
for train_indx in tqdm(train_indxs):
    r = make_prediction(train_indx)
    res.append(r)
    
#pool.close()
#pool.join()
end_time = time.time()
print('Exec speed=%.2f for every ts' %((end_time-start_time)/train_indxs.shape[0]))

Predicting...


  0%|          | 0/154 [00:00<?, ?it/s]

========================= train_indxs长度时间序列数量 =========================
154
========================= 根据不同的id聚合的日销量预测 =========================


100%|██████████| 154/154 [08:46<00:00,  3.42s/it]

M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet
M5 Forecasting - Accuracy：Multi-time series prediction with prophet

Exec speed=3.42 for every ts

5. 重新规划要预测的最细粒度的时间序列数

# Convert back to initial format
tmp = pd.DataFrame()
for result in tqdm(res):
    uid = result[0]
    ret = result[1].rename(uid)
    tmp = pd.concat([tmp, ret], axis=1)
fbp_pd = tmp.T
fbp_pd.index.name = "id"

100%|██████████| 154/154 [00:00<00:00, 499.40it/s]

6. 小部分序列细粒度扩展计算结果

level_coef_dict = {
    12: ["id"], # L12 x30490
    10: ["item_id"], # L10 x3049
    5: ["dept_id"], # L5 x7
    4: ["cat_id"], # L4 x3
    3: ["store_id"], # L3 x10
    2: ["state_id"],  # L2 x3
    1: ["all"],  # L1 x1
    11: ["state_id", "item_id"], # L11 x9167
    7: ["state_id", "dept_id"],  # L7 x21
    9: ["store_id","dept_id"],  # L9 x70
    6: ["state_id", "cat_id"], # L6 x 9
    8: ["store_id","cat_id"], # L8 x30
}

6.1 结果计算函数

# Top-Down prediction
def predict_topdown(df, df_level12, items):
    df_level_forecast = pd.DataFrame()
    for idx, row in df.iterrows():
        item1 = items[0]
        if item1 is not "all":
            item1_id = row[item1]
#             print('item1:', item1)
#             print('item1_id:', item1_id)
        if len(items) == 2:
            item2 = items[1]
            item2_id = row[items[1]]

        ## 找到聚合序列中的每一个对应的id
        if item1 is not "all":
            # Find all level 12 items for the dept_id, store_id pair
            if len(items) == 2:
                df_item = df_level12.loc[(df_level12[item1] == item1_id) & (df_level12[item2] == item2_id)][['id']]
            else:
                df_item = df_level12.loc[df_level12[item1] == item1_id][['id']]
                
        else:
            df_item = df_level12[['id']]
        #print(df_item.shape)
        #display(df_item.head())

        # Sum sales from last 28 days in level 12 training
        # df_item['val']为item1_id最近28天的销量和
        if item1 is not "all":
            if len(items) == 2:
                df_item['val'] = df_level12[(df_level12[item1] == item1_id) & (df_level12[item2] == item2_id)].iloc[:, np.r_[0,-28:0]].sum(axis = 1)
            else:
                df_item['val'] = df_level12[df_level12[item1] == item1_id].iloc[:, np.r_[0,-28:0]].sum(axis = 1)
        else:
            df_item['val'] = df_level12.iloc[:, np.r_[0,-28:0]].sum(axis = 1)
        #display(df_item.head())

        # Back to per id prediction
        for i in range(1,29):
            col = "d_%d" % (1913 + i)
            p_col = "F%d" % i
            df_item[p_col] = (df_item['val'] * float(row[col]) / df_item['val'].sum())
#             print(df_item[p_col].head())
#             print(df_item[p_col].shape)
        #display(df_item.head())

        df_level_forecast = pd.concat([df_level_forecast, df_item])
    return df_level_forecast.drop(columns=["val"])

level_coef_dict

{12: ['id'],
 10: ['item_id'],
 5: ['dept_id'],
 4: ['cat_id'],
 3: ['store_id'],
 2: ['state_id'],
 1: ['all'],
 11: ['state_id', 'item_id'],
 7: ['state_id', 'dept_id'],
 9: ['store_id', 'dept_id'],
 6: ['state_id', 'cat_id'],
 8: ['store_id', 'cat_id']}

6.2 预测一部分时间序列看看结果

说明

  • yhat_df：为1-9个聚合等级下，用prophet得到的预测值+已知聚合xiaol数据。等级level字段是快速定位的关键（shape为【154, 1611】）。
  • prediction：为level等于具体取值时的yhat_df数据。销量聚合数据的天数【d_338——d_1941】（后面28天是prophet根据历史预测得到）。
  • df_sale：为原始销量数据，也是最细粒度的非聚合数据。

预测的思路

  • 因为最细粒度id的销量周期性不明显，且0销量取值太多，预测起来误差太大。
  • 分解的思想是关键。
  • 又因为粗粒度的聚合销量数据周期性明显，预测准确性高。
  • 对于真实销量：计算某一类目下，各个分量最细粒度ID最近28天销量和占比R。
  • 对于预测销量：需要预测各个聚合销量天级别S。
  • 未来一天细粒度销量（某个类目下的一个商品id1） = R_id1 * S
print('========================================== predictions ======================================')
print('预测表的列名..............')
print(yhat_df.columns)
for key, value in level_coef_dict.items():
    if (key <= MAX_LEVEL) and (key >= 1):
        predictions = yhat_df[yhat_df["level"] == key]
        # 重要！！！
        df_levelx_forecast = predict_topdown(predictions, df_sale, value)
        print("Top-Down prediction for level =【%s】,detail id =【%s】, items =【%s】 into %s" % (key, 
                                                                          value, 
                                                                          predictions.shape[0], 
                                                                          df_levelx_forecast.shape[0]))
        df_levelx_forecast.to_pickle(MODELS_DIR + "/level_%d.pkl.gz" % key, 
                                     compression="gzip")
        # display(df_levelx_forecast.head())
        print(df_levelx_forecast.shape)
df_levelx_forecast.head()

========================================== predictions ======================================
预测表的列名..............
Index(['id', 'd_338', 'd_339', 'd_340', 'd_341', 'd_342', 'd_343', 'd_344',
       'd_345', 'd_346',
       ...
       'd_1938', 'd_1939', 'd_1940', 'd_1941', 'level', 'state_id', 'store_id',
       'cat_id', 'dept_id', 'item_id'],
      dtype='object', length=1611)
Top-Down prediction for level =【5】,detail id =【['dept_id']】, items =【7】 into 30490
(30490, 29)
Top-Down prediction for level =【4】,detail id =【['cat_id']】, items =【3】 into 30490
(30490, 29)
Top-Down prediction for level =【3】,detail id =【['store_id']】, items =【10】 into 30490
(30490, 29)
Top-Down prediction for level =【2】,detail id =【['state_id']】, items =【3】 into 30490
(30490, 29)


/opt/conda/envs/python3/lib/python3.6/site-packages/ipykernel/__main__.py:35: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
/opt/conda/envs/python3/lib/python3.6/site-packages/ipykernel/__main__.py:42: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy


Top-Down prediction for level =【1】,detail id =【['all']】, items =【1】 into 30490
(30490, 29)
Top-Down prediction for level =【7】,detail id =【['state_id', 'dept_id']】, items =【21】 into 30490
(30490, 29)
Top-Down prediction for level =【9】,detail id =【['store_id', 'dept_id']】, items =【70】 into 30490
(30490, 29)
Top-Down prediction for level =【6】,detail id =【['state_id', 'cat_id']】, items =【9】 into 30490
(30490, 29)
Top-Down prediction for level =【8】,detail id =【['store_id', 'cat_id']】, items =【30】 into 30490
(30490, 29)
d id F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 F13 F14 F15 F16 F17 F18 F19 F20 F21 F22 F23 F24 F25 F26 F27 F28
0 FOODS_1_001_CA_1 1.050698 0.940785 0.878987 0.925223 1.136064 1.466935 1.703499 1.276676 1.171124 1.129501 0.988901 1.293892 1.621422 1.403054 1.264438 1.151234 1.060480 1.059198 1.215895 1.509826 1.590587 1.143320 1.013105 0.995392 1.011581 1.184087 1.478189 1.569705
4 FOODS_1_002_CA_1 0.382072 0.342104 0.319632 0.336445 0.413114 0.533431 0.619454 0.464246 0.425863 0.410728 0.359600 0.470506 0.589608 0.510201 0.459796 0.418631 0.385629 0.385163 0.442144 0.549028 0.578395 0.415753 0.368402 0.361961 0.367848 0.430577 0.537523 0.570802
8 FOODS_1_003_CA_1 0.700465 0.627190 0.585991 0.616815 0.757376 0.977957 1.135666 0.851117 0.780749 0.753001 0.659267 0.862594 1.080948 0.935369 0.842959 0.767490 0.706987 0.706132 0.810597 1.006551 1.060391 0.762213 0.675403 0.663594 0.674387 0.789391 0.985460 1.046470
12 FOODS_1_004_CA_1 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000 0.000000
16 FOODS_1_005_CA_1 1.082537 0.969293 0.905623 0.953260 1.170491 1.511388 1.755120 1.315363 1.206612 1.163728 1.018868 1.333100 1.670556 1.445571 1.302755 1.186120 1.092616 1.091294 1.252741 1.555578 1.638786 1.177966 1.043805 1.025555 1.042235 1.219968 1.522983 1.617271

6.3 可视化结果查看

  • id = FOODS_1_001_CA_1
tmp_pd1 = df_levelx_forecast[df_levelx_forecast["id"].isin(["FOODS_1_001_CA_1"])]
tmp_pd1 = pd.melt(frame = tmp_pd1, 
                  id_vars = ['id'],
                  var_name = "F",
                  value_vars = [c for c in tmp_pd1.columns if "F" in c],
                  value_name = "yhat")
d = tmp_pd1.plot(kind="line", x="F", y="yhat")
plt.show()

M5 Forecasting - Accuracy：Multi-time series prediction with prophet

7. 全量计算并规范化输出并保存

SAMPLE_SUBMISSION = TRAIN_DATA_HOME + "sample_submission.csv"
sub_ids = pd.read_csv(SAMPLE_SUBMISSION)[['id']]
sub_ids = sub_ids[sub_ids.id.str.endswith("validation")]

for key, value in tqdm(level_coef_dict.items()):
    if (key <= MAX_LEVEL) and (key >= 1):
        predictions = yhat_df[yhat_df["level"] == key]
        df_levelx_forecast = predict_topdown(predictions, df_sale, value)
        print("Top-Down prediction for level %s, %s, items=%s into %s" % (key, value, predictions.shape[0], df_levelx_forecast.shape[0]))
        df_levelx_forecast["id"] = df_levelx_forecast["id"].astype(str) + "_validation"
        part1 = pd.merge(sub_ids, df_levelx_forecast, on="id", how="left")
        part2 = part1.copy()
        part2["id"] = part2["id"].str.replace("validation$", "evaluation")
        sub = pd.concat([part1, part2])
        MODE_LEVEL_PATH = MODELS_DIR + "/level_%d" % key
        if not os.path.exists(MODE_LEVEL_PATH):
            os.makedirs(MODE_LEVEL_PATH)
        sub.to_csv(MODE_LEVEL_PATH + "/submission.csv.gz", compression="gzip", index=False)

  0%|          | 0/12 [00:00<?, ?it/s]

Top-Down prediction for level 5, ['dept_id'], items=7 into 30490


 25%|██▌       | 3/12 [00:06<00:18,  2.02s/it]

Top-Down prediction for level 4, ['cat_id'], items=3 into 30490


 33%|███▎      | 4/12 [00:11<00:25,  3.18s/it]

Top-Down prediction for level 3, ['store_id'], items=10 into 30490


 42%|████▏     | 5/12 [00:18<00:29,  4.15s/it]

Top-Down prediction for level 2, ['state_id'], items=3 into 30490


 50%|█████     | 6/12 [00:24<00:27,  4.66s/it]/opt/conda/envs/python3/lib/python3.6/site-packages/ipykernel/__main__.py:35: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
/opt/conda/envs/python3/lib/python3.6/site-packages/ipykernel/__main__.py:42: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy


Top-Down prediction for level 1, ['all'], items=1 into 30490


 58%|█████▊    | 7/12 [00:29<00:24,  4.81s/it]

Top-Down prediction for level 7, ['state_id', 'dept_id'], items=21 into 30490


 75%|███████▌  | 9/12 [00:36<00:13,  4.37s/it]

Top-Down prediction for level 9, ['store_id', 'dept_id'], items=70 into 30490


 83%|████████▎ | 10/12 [00:45<00:11,  5.86s/it]

Top-Down prediction for level 6, ['state_id', 'cat_id'], items=9 into 30490


 92%|█████████▏| 11/12 [00:51<00:05,  5.93s/it]

Top-Down prediction for level 8, ['store_id', 'cat_id'], items=30 into 30490


100%|██████████| 12/12 [00:58<00:00,  4.90s/it]

print(sub.shape)

(60980, 29)

8. 加载所有聚合level下预测的取平均

  • 这里计算了9个level
  • 其实也可以不用取平局，直接每一个都提交作为成绩，看看哪个结果好，然后用其中最好的即可
for key in tqdm(range(1, 10)):
    MODE_LEVEL_PATH = MODELS_DIR + "/level_%d" % key
    if not os.path.exists(MODE_LEVEL_PATH):
        os.makedirs(MODE_LEVEL_PATH)
    if key==1:
        cc = pd.read_csv(MODE_LEVEL_PATH + "/submission.csv.gz")
    else:
        cc.iloc[:, 1:] += pd.read_csv(MODE_LEVEL_PATH + "/submission.csv.gz").iloc[:, 1:]
cc.iloc[:, 1:] = cc.iloc[:, 1:]/9

100%|██████████| 9/9 [00:04<00:00,  1.91it/s]

cc.head()
cc.to_csv("../result/submission_prophet.csv", index=False)
相关标签： 比赛 预测

上一篇： 图片分类-K近邻分类器

下一篇： 用ARIMA模型做需求预测

推荐阅读