flowcontainer: 基于python3的网络流信息提取库

库介绍

flowcontainer是本人编写的基于python3的网络流量基本信息提取库，方便做加密网络流量的分类任务。给定pcap文件，该库会提取pcap所有的流的相关信息，其中流信息包括：流的源端口、源IP、目的IP、目的端口、IP数据包的长度序列、IP数据集的到达时间序列、有效载荷序列以及相应有效载荷的到达时间序列、TLS的SNI。库会对IP数据包做过滤，那些tcp/udp载荷不为0的数据包会统计到有效载荷序列里面。工具简单易用，扩展性和复用性高。

库的安装

pip3 install git+https://github.com/jmhIcoding/flowcontainer.git

库的环境

• python3
• numpy>=18.1
• 系统安装好tshark的最新版本,并将tshark所在的目录添加到系统的环境目录

库的使用

示例代码：
直接导入extract函数，然后给定pcap的路径即可。

from flowcontainer.extractor import extract
result = extract(r"1592995802_clear.pcap")
for key in result:
    ### The return vlaue result is a dict, the key is a tuple (filename,procotol,stream_id)
    ### and the value is an Flow object, user can access Flow object as flowcontainer.flows.Flow's attributes refer.

    value = result[key]
    print('Flow {0} info:'.format(key))
    ## access ip src
    print('src ip:',value.src)
    ## access ip dst
    print('dst ip:',value.dst)
    ## access srcport
    print('sport:',value.sport)
    ## access_dstport
    print('dport:',value.dport)
    ## access payload packet lengths
    print('payload lengths :',value.payload_lengths)
    ## access payload packet timestamps sequence:
    print('payload timestamps:',value.payload_timestamps)
    ## access ip packet lengths, (including packets with zero payload, and ip header)
    print('ip packets lengths:',value.ip_lengths)
    ## access ip packet timestamp sequence, (including packets with zero payload)
    print('ip packets timestamps:',value.ip_timestamps)

    ## access default lengths sequence, the default length sequences is the payload lengths sequences
    print('default length sequence:',value.lengths)
    ## access default timestamp sequence, the default timestamp sequence is the payload timestamp sequences
    print('default timestamp sequence:',value.timestamps)

    ##access sni of the flow if any else empty str
    print('sni:',value.sni)
'''
Output:
----------------
"C:\Program Files\Python36\python.exe" C:/Users/dk/Documents/flowcontainer/example.py
Reading 1592995802_clear.pcap...
Flow ('1592995802_clear.pcap', 'tcp', '0') info:
src ip: 192.168.0.100
dst ip: 23.51.209.190
sport: 49924
dport: 80
payload lengths : [166, -1448, -600, -244, 166, -1440, -852]
payload timestamps: [1592995818.017334, 1592995818.01734, 1592995818.017346, 1592995818.511819, 1592995818.511939, 1592995818.511947, 1592995818.800813]
ip packets lengths: [60, -60, 52, 218, -52, -1500, 52, -652, 52, -296, 52, 218, -52, -1492, 52, -904, 52, 52, -52, -52, 52]
ip packets timestamps: [1592995818.017318, 1592995818.017328, 1592995818.017331, 1592995818.017334, 1592995818.017337, 1592995818.01734, 1592995818.017343, 1592995818.017346, 1592995818.017348, 1592995818.511819, 1592995818.511828, 1592995818.511939, 1592995818.511942, 1592995818.511947, 1592995818.511953, 1592995818.800813, 1592995818.800824, 1592995825.934351, 1592995826.020253, 1592995826.976613, 1592995826.976616]
default length sequence: [166, -1448, -600, -244, 166, -1440, -852]
default timestamp sequence: [1592995818.017334, 1592995818.01734, 1592995818.017346, 1592995818.511819, 1592995818.511939, 1592995818.511947, 1592995818.800813]
sni:
Flow ('1592995802_clear.pcap', 'tcp', '1') info:
src ip: 192.168.0.100
dst ip: 34.240.55.180
sport: 47544
dport: 443
payload lengths : [172, -1448, -1448, -600, -832, -1034]
payload timestamps: [1592995818.800828, 1592995819.267969, 1592995819.267975, 1592995819.26802, 1592995819.268026, 1592995819.268031]
ip packets lengths: [60, -60, 52, 224, -52, 52, -52, -1500, 40, -1500, 40, -652, 40, -884, 40, -1086, 40, 40]
ip packets timestamps: [1592995818.511945, 1592995818.51195, 1592995818.511956, 1592995818.800828, 1592995818.800963, 1592995818.80098, 1592995818.854556, 1592995819.267969, 1592995819.267972, 1592995819.267975, 1592995819.26801, 1592995819.26802, 1592995819.268023, 1592995819.268026, 1592995819.268028, 1592995819.268031, 1592995819.268063, 1592995819.321972]
default length sequence: [172, -1448, -1448, -600, -832, -1034]
default timestamp sequence: [1592995818.800828, 1592995819.267969, 1592995819.267975, 1592995819.26802, 1592995819.268026, 1592995819.268031]
sni: tom.itv.com
Flow ('1592995802_clear.pcap', 'tcp', '2') info:
src ip: 192.168.0.100
dst ip: 223.119.236.227
sport: 42630
dport: 443
payload lengths : [172, -1448, -600, -816, -1232, -969, 126, 514, -258, -321, 537, -321, 514, -321, 565, -321, 31, -31]
payload timestamps: [1592995818.800974, 1592995818.854565, 1592995818.854572, 1592995818.85471, 1592995818.942261, 1592995818.942267, 1592995818.942447, 1592995819.10146, 1592995819.101487, 1592995819.415174, 1592995822.789448, 1592995823.264541, 1592995826.020247, 1592995826.330403, 1592995827.398372, 1592995827.779183, 1592995831.396575, 1592995831.396608]
ip packets lengths: [60, -60, 52, 224, -52, -1500, 52, -652, -868, 52, 52, -1284, 52, -1021, 52, 178, -52, 566, -52, -310, 52, -373, 52, 589, -52, -373, 52, 566, -52, -373, 52, 617, -52, -373, 52, 83, 52, -52, -52, -83, 40, -52, 40]
ip packets timestamps: [1592995818.800966, 1592995818.800969, 1592995818.800971, 1592995818.800974, 1592995818.800977, 1592995818.854565, 1592995818.854569, 1592995818.854572, 1592995818.85471, 1592995818.942249, 1592995818.942258, 1592995818.942261, 1592995818.942264, 1592995818.942267, 1592995818.942445, 1592995818.942447, 1592995818.94245, 1592995819.10146, 1592995819.10147, 1592995819.101487, 1592995819.101489, 1592995819.415174, 1592995819.415177, 1592995822.789448, 1592995822.789457, 1592995823.264541, 1592995823.26455, 1592995826.020247, 1592995826.020256, 1592995826.330403, 1592995826.330406, 1592995827.398372, 1592995827.426276, 1592995827.779183, 1592995827.779187, 1592995831.396575, 1592995831.396581, 1592995831.396587, 1592995831.396592, 1592995831.396608, 1592995831.396614, 1592995831.396619, 1592995831.396625]
default length sequence: [172, -1448, -600, -816, -1232, -969, 126, 514, -258, -321, 537, -321, 514, -321, 565, -321, 31, -31]
default timestamp sequence: [1592995818.800974, 1592995818.854565, 1592995818.854572, 1592995818.85471, 1592995818.942261, 1592995818.942267, 1592995818.942447, 1592995819.10146, 1592995819.101487, 1592995819.415174, 1592995822.789448, 1592995823.264541, 1592995826.020247, 1592995826.330403, 1592995827.398372, 1592995827.779183, 1592995831.396575, 1592995831.396608]
sni: cpt.itv.com
Flow ('1592995802_clear.pcap', 'tcp', '3') info:
src ip: 192.168.0.100
dst ip: 34.240.55.180
sport: 47546
dport: 443
payload lengths : [172, -1448, -600, -816, -32, -1440, -1026, 126, 288, -38, -426, 38, 124, -983, 125, -435, 131, -420, 118, -427, 123, -945, -38, 125, -433, 131, -382, -38, 118, -427, 124, -944, -38, 125, -433, 131, 120, -382, -38, -428, 125, -988, 127, -436, 133, -420, 120, -428, 42, 125, 42, -987, 127, -436, 42, 42, 133, -420, 183, -612, 120, -428, 126, -988, 127, -436, 133, -420, 120, -429, 125, -988, 127, -435, 133, -420, 120, -429, 126, -988, 127, -436, 133, 120, -420, -428, 126, -988, 127, -435, 42, 133, 120, -420, -427, 126, -950, -38, 127, -435, 133, -420]
payload timestamps: [1592995818.800991, 1592995819.512145, 1592995819.512162, 1592995819.512168, 1592995819.512221, 1592995819.512232, 1592995819.512264, 1592995819.574809, 1592995819.826353, 1592995820.282689, 1592995820.282694, 1592995821.093755, 1592995821.093838, 1592995821.093844, 1592995821.093849, 1592995821.375563, 1592995821.375571, 1592995822.563011, 1592995822.865031, 1592995823.264553, 1592995823.264559, 1592995823.554085, 1592995823.554238, 1592995823.60004, 1592995823.956123, 1592995823.972505, 1592995824.311251, 1592995824.311254, 1592995826.020238, 1592995826.330379, 1592995826.330391, 1592995826.827078, 1592995826.827257, 1592995826.827263, 1592995827.107589, 1592995827.107604, 1592995827.39358, 1592995827.426279, 1592995827.426282, 1592995827.723343, 1592995827.75746, 1592995828.096227, 1592995828.217933, 1592995828.595053, 1592995828.610983, 1592995828.964437, 1592995831.396597, 1592995831.611614, 1592995831.611627, 1592995832.443988, 1592995832.444017, 1592995832.444041, 1592995832.444054, 1592995833.443225, 1592995833.443238, 1592995833.443249, 1592995833.443261, 1592995833.443272, 1592995835.873283, 1592995835.873295, 1592995837.913008, 1592995838.157615, 1592995838.163851, 1592995838.603329, 1592995838.603352, 1592995838.925834, 1592995838.957103, 1592995839.295437, 1592995840.231051, 1592995840.961888, 1592995840.961895, 1592995841.140672, 1592995841.140848, 1592995841.350632, 1592995842.342914, 1592995842.342935, 1592995846.494982, 1592995847.208562, 1592995847.208568, 1592995847.282154, 1592995847.282333, 1592995847.827889, 1592995847.827894, 1592995848.166325, 1592995848.166342, 1592995848.49875, 1592995848.498762, 1592995848.934556, 1592995848.934569, 1592995849.179916, 1592995849.180113, 1592995849.180119, 1592995849.180127, 1592995849.361546, 1592995849.539365, 1592995849.539371, 1592995849.865436, 1592995849.865665, 1592995849.881012, 1592995850.232922, 1592995850.232944, 1592995850.718411]
ip packets lengths: [60, -60, 52, 224, -52, -1500, 52, -652, 52, -868, 52, -84, 52, -1492, 52, -1078, 52, 178, -52, 340, -52, -90, -478, 64, 52, 90, -52, 176, -52, -1035, 52, 177, -52, -487, 52, 183, -52, -472, 52, 170, -52, -479, 52, 175, -52, -997, -90, 52, 177, -52, -485, 183, -52, -434, -90, 52, 170, -52, -479, 176, -52, -996, -90, 52, 177, -52, -485, 183, -52, 172, -52, -434, -90, 52, -480, 177, -52, -1040, 52, 179, -52, -488, 52, 185, -52, -472, 52, 172, -52, -480, 52, 94, -52, 177, -52, 94, -52, -1039, 52, 179, -52, -488, 52, 94, -52, 94, -52, 185, -52, -472, 52, 235, -52, -664, 52, 172, -52, -480, 52, 178, -52, -1040, 52, 179, -52, -488, 52, 185, -52, -472, 52, 172, -52, -481, 52, 177, -52, -1040, 52, 179, -52, -487, 52, 185, -52, -472, 52, 172, -52, -481, 52, 178, -52, -1040, 52, 179, -52, -488, 52, 185, -52, 172, -52, -472, 52, -480, 52, 178, -52, -1040, 52, 179, -52, -487, 52, 94, -52, 185, -52, 172, -52, -472, 52, -479, 52, 178, -52, -1002, -90, 52, 179, -52, -487, 185, -52, -472, 52, 52, -52, -52, 52]
ip packets timestamps: [1592995818.800982, 1592995818.800985, 1592995818.800988, 1592995818.800991, 1592995818.800993, 1592995819.512145, 1592995819.512148, 1592995819.512162, 1592995819.512165, 1592995819.512168, 1592995819.512201, 1592995819.512221, 1592995819.512223, 1592995819.512232, 1592995819.512262, 1592995819.512264, 1592995819.512267, 1592995819.574809, 1592995819.574817, 1592995819.826353, 1592995819.826376, 1592995820.282689, 1592995820.282694, 1592995820.282697, 1592995821.093746, 1592995821.093755, 1592995821.093835, 1592995821.093838, 1592995821.093841, 1592995821.093844, 1592995821.093847, 1592995821.093849, 1592995821.093852, 1592995821.375563, 1592995821.375566, 1592995821.375571, 1592995821.375574, 1592995822.563011, 1592995822.563021, 1592995822.865031, 1592995822.865035, 1592995823.264553, 1592995823.264556, 1592995823.264559, 1592995823.264562, 1592995823.554085, 1592995823.554238, 1592995823.554241, 1592995823.60004, 1592995823.600045, 1592995823.956123, 1592995823.972505, 1592995823.972508, 1592995824.311251, 1592995824.311254, 1592995824.324861, 1592995826.020238, 1592995826.02025, 1592995826.330379, 1592995826.330391, 1592995826.3304, 1592995826.827078, 1592995826.827257, 1592995826.82726, 1592995826.827263, 1592995826.827269, 1592995827.107589, 1592995827.107604, 1592995827.10776, 1592995827.39358, 1592995827.393584, 1592995827.426279, 1592995827.426282, 1592995827.426285, 1592995827.723343, 1592995827.75746, 1592995827.791067, 1592995828.096227, 1592995828.113777, 1592995828.217933, 1592995828.234674, 1592995828.595053, 1592995828.595057, 1592995828.610983, 1592995828.640008, 1592995828.964437, 1592995828.995039, 1592995831.396597, 1592995831.396603, 1592995831.611614, 1592995831.611621, 1592995831.611627, 1592995831.611633, 1592995832.443988, 1592995832.444011, 1592995832.444017, 1592995832.444023, 1592995832.444041, 1592995832.444048, 1592995832.444054, 1592995832.44406, 1592995833.443225, 1592995833.443232, 1592995833.443238, 1592995833.443244, 1592995833.443249, 1592995833.443255, 1592995833.443261, 1592995833.443267, 1592995833.443272, 1592995833.443278, 1592995835.873283, 1592995835.873289, 1592995835.873295, 1592995835.873301, 1592995837.913008, 1592995837.913012, 1592995838.157615, 1592995838.157844, 1592995838.163851, 1592995838.176152, 1592995838.603329, 1592995838.603339, 1592995838.603352, 1592995838.603356, 1592995838.925834, 1592995838.925847, 1592995838.957103, 1592995839.063726, 1592995839.295437, 1592995839.327109, 1592995840.231051, 1592995840.231054, 1592995840.961888, 1592995840.961892, 1592995840.961895, 1592995840.961898, 1592995841.140672, 1592995841.140844, 1592995841.140848, 1592995841.140852, 1592995841.350632, 1592995841.350635, 1592995842.342914, 1592995842.342925, 1592995842.342935, 1592995842.342938, 1592995846.494982, 1592995846.49517, 1592995847.208562, 1592995847.208565, 1592995847.208568, 1592995847.208571, 1592995847.282154, 1592995847.282318, 1592995847.282333, 1592995847.827699, 1592995847.827889, 1592995847.827892, 1592995847.827894, 1592995847.827897, 1592995848.166325, 1592995848.166338, 1592995848.166342, 1592995848.166345, 1592995848.49875, 1592995848.49876, 1592995848.498762, 1592995848.498766, 1592995848.934556, 1592995848.934566, 1592995848.934569, 1592995848.934575, 1592995849.179916, 1592995849.18011, 1592995849.180113, 1592995849.180116, 1592995849.180119, 1592995849.180121, 1592995849.180127, 1592995849.36134, 1592995849.361546, 1592995849.361554, 1592995849.539365, 1592995849.539368, 1592995849.539371, 1592995849.550906, 1592995849.865436, 1592995849.865665, 1592995849.865669, 1592995849.881012, 1592995849.896329, 1592995850.232922, 1592995850.232944, 1592995850.233163, 1592995850.718411, 1592995850.718416, 1592995852.488969, 1592995852.902292, 1592995853.893155, 1592995853.893161]
default length sequence: [172, -1448, -600, -816, -32, -1440, -1026, 126, 288, -38, -426, 38, 124, -983, 125, -435, 131, -420, 118, -427, 123, -945, -38, 125, -433, 131, -382, -38, 118, -427, 124, -944, -38, 125, -433, 131, 120, -382, -38, -428, 125, -988, 127, -436, 133, -420, 120, -428, 42, 125, 42, -987, 127, -436, 42, 42, 133, -420, 183, -612, 120, -428, 126, -988, 127, -436, 133, -420, 120, -429, 125, -988, 127, -435, 133, -420, 120, -429, 126, -988, 127, -436, 133, 120, -420, -428, 126, -988, 127, -435, 42, 133, 120, -420, -427, 126, -950, -38, 127, -435, 133, -420]
default timestamp sequence: [1592995818.800991, 1592995819.512145, 1592995819.512162, 1592995819.512168, 1592995819.512221, 1592995819.512232, 1592995819.512264, 1592995819.574809, 1592995819.826353, 1592995820.282689, 1592995820.282694, 1592995821.093755, 1592995821.093838, 1592995821.093844, 1592995821.093849, 1592995821.375563, 1592995821.375571, 1592995822.563011, 1592995822.865031, 1592995823.264553, 1592995823.264559, 1592995823.554085, 1592995823.554238, 1592995823.60004, 1592995823.956123, 1592995823.972505, 1592995824.311251, 1592995824.311254, 1592995826.020238, 1592995826.330379, 1592995826.330391, 1592995826.827078, 1592995826.827257, 1592995826.827263, 1592995827.107589, 1592995827.107604, 1592995827.39358, 1592995827.426279, 1592995827.426282, 1592995827.723343, 1592995827.75746, 1592995828.096227, 1592995828.217933, 1592995828.595053, 1592995828.610983, 1592995828.964437, 1592995831.396597, 1592995831.611614, 1592995831.611627, 1592995832.443988, 1592995832.444017, 1592995832.444041, 1592995832.444054, 1592995833.443225, 1592995833.443238, 1592995833.443249, 1592995833.443261, 1592995833.443272, 1592995835.873283, 1592995835.873295, 1592995837.913008, 1592995838.157615, 1592995838.163851, 1592995838.603329, 1592995838.603352, 1592995838.925834, 1592995838.957103, 1592995839.295437, 1592995840.231051, 1592995840.961888, 1592995840.961895, 1592995841.140672, 1592995841.140848, 1592995841.350632, 1592995842.342914, 1592995842.342935, 1592995846.494982, 1592995847.208562, 1592995847.208568, 1592995847.282154, 1592995847.282333, 1592995847.827889, 1592995847.827894, 1592995848.166325, 1592995848.166342, 1592995848.49875, 1592995848.498762, 1592995848.934556, 1592995848.934569, 1592995849.179916, 1592995849.180113, 1592995849.180119, 1592995849.180127, 1592995849.361546, 1592995849.539365, 1592995849.539371, 1592995849.865436, 1592995849.865665, 1592995849.881012, 1592995850.232922, 1592995850.232944, 1592995850.718411]
sni: tom.itv.com

解析速度

flowcontainer的解析的速度还是可以的，50G左右的流量2个小时左右即可完成所有流信息的提取。
5G左右的流量12分钟即可解析完毕。

本文地址：https://blog.csdn.net/jmh1996/article/details/107148871