欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页  >  IT编程

flowcontainer: 基于python3的网络流信息提取库

程序员文章站 2022-07-07 19:37:39
库介绍flowcontainer是本人编写的基于python3的网络流量基本信息提取库,方便做加密网络流量的分类任务。给定pcap文件,该库会提取pcap所有的流的相关信息,其中流信息包括:流的源端口、源IP、目的IP、目的端口、IP数据包的长度序列、IP数据集的到达时间序列、有效载荷序列以及相应有效载荷的到达时间序列、TLS的SNI。库会对IP数据包做过滤,那些tcp/udp载荷不为0的数据包会统计到有效载荷序列里面。工具简单易用,扩展性和复用性高。库的安装pip3 install git+http...

库介绍

flowcontainer是本人编写的基于python3的网络流量基本信息提取库,方便做加密网络流量的分类任务。给定pcap文件,该库会提取pcap所有的流的相关信息,其中流信息包括:流的源端口、源IP、目的IP、目的端口、IP数据包的长度序列、IP数据集的到达时间序列、有效载荷序列以及相应有效载荷的到达时间序列、TLS的SNI。库会对IP数据包做过滤,那些tcp/udp载荷不为0的数据包会统计到有效载荷序列里面。工具简单易用,扩展性和复用性高。

库的安装

pip3 install git+https://github.com/jmhIcoding/flowcontainer.git

库的环境

  • python3
  • numpy>=18.1
  • 系统安装好tshark的最新版本,并将tshark所在的目录添加到系统的环境目录

库的使用

示例代码:
直接导入extract函数,然后给定pcap的路径即可。

from flowcontainer.extractor import extract
result = extract(r"1592995802_clear.pcap")
for key in result:
    ### The return vlaue result is a dict, the key is a tuple (filename,procotol,stream_id)
    ### and the value is an Flow object, user can access Flow object as flowcontainer.flows.Flow's attributes refer.

    value = result[key]
    print('Flow {0} info:'.format(key))
    ## access ip src
    print('src ip:',value.src)
    ## access ip dst
    print('dst ip:',value.dst)
    ## access srcport
    print('sport:',value.sport)
    ## access_dstport
    print('dport:',value.dport)
    ## access payload packet lengths
    print('payload lengths :',value.payload_lengths)
    ## access payload packet timestamps sequence:
    print('payload timestamps:',value.payload_timestamps)
    ## access ip packet lengths, (including packets with zero payload, and ip header)
    print('ip packets lengths:',value.ip_lengths)
    ## access ip packet timestamp sequence, (including packets with zero payload)
    print('ip packets timestamps:',value.ip_timestamps)

    ## access default lengths sequence, the default length sequences is the payload lengths sequences
    print('default length sequence:',value.lengths)
    ## access default timestamp sequence, the default timestamp sequence is the payload timestamp sequences
    print('default timestamp sequence:',value.timestamps)

    ##access sni of the flow if any else empty str
    print('sni:',value.sni)
'''
Output:
----------------
"C:\Program Files\Python36\python.exe" C:/Users/dk/Documents/flowcontainer/example.py
Reading 1592995802_clear.pcap...
Flow ('1592995802_clear.pcap', 'tcp', '0') info:
src ip: 192.168.0.100
dst ip: 23.51.209.190
sport: 49924
dport: 80
payload lengths : [166, -1448, -600, -244, 166, -1440, -852]
payload timestamps: [1592995818.017334, 1592995818.01734, 1592995818.017346, 1592995818.511819, 1592995818.511939, 1592995818.511947, 1592995818.800813]
ip packets lengths: [60, -60, 52, 218, -52, -1500, 52, -652, 52, -296, 52, 218, -52, -1492, 52, -904, 52, 52, -52, -52, 52]
ip packets timestamps: [1592995818.017318, 1592995818.017328, 1592995818.017331, 1592995818.017334, 1592995818.017337, 1592995818.01734, 1592995818.017343, 1592995818.017346, 1592995818.017348, 1592995818.511819, 1592995818.511828, 1592995818.511939, 1592995818.511942, 1592995818.511947, 1592995818.511953, 1592995818.800813, 1592995818.800824, 1592995825.934351, 1592995826.020253, 1592995826.976613, 1592995826.976616]
default length sequence: [166, -1448, -600, -244, 166, -1440, -852]
default timestamp sequence: [1592995818.017334, 1592995818.01734, 1592995818.017346, 1592995818.511819, 1592995818.511939, 1592995818.511947, 1592995818.800813]
sni:
Flow ('1592995802_clear.pcap', 'tcp', '1') info:
src ip: 192.168.0.100
dst ip: 34.240.55.180
sport: 47544
dport: 443
payload lengths : [172, -1448, -1448, -600, -832, -1034]
payload timestamps: [1592995818.800828, 1592995819.267969, 1592995819.267975, 1592995819.26802, 1592995819.268026, 1592995819.268031]
ip packets lengths: [60, -60, 52, 224, -52, 52, -52, -1500, 40, -1500, 40, -652, 40, -884, 40, -1086, 40, 40]
ip packets timestamps: [1592995818.511945, 1592995818.51195, 1592995818.511956, 1592995818.800828, 1592995818.800963, 1592995818.80098, 1592995818.854556, 1592995819.267969, 1592995819.267972, 1592995819.267975, 1592995819.26801, 1592995819.26802, 1592995819.268023, 1592995819.268026, 1592995819.268028, 1592995819.268031, 1592995819.268063, 1592995819.321972]
default length sequence: [172, -1448, -1448, -600, -832, -1034]
default timestamp sequence: [1592995818.800828, 1592995819.267969, 1592995819.267975, 1592995819.26802, 1592995819.268026, 1592995819.268031]
sni: tom.itv.com
Flow ('1592995802_clear.pcap', 'tcp', '2') info:
src ip: 192.168.0.100
dst ip: 223.119.236.227
sport: 42630
dport: 443
payload lengths : [172, -1448, -600, -816, -1232, -969, 126, 514, -258, -321, 537, -321, 514, -321, 565, -321, 31, -31]
payload timestamps: [1592995818.800974, 1592995818.854565, 1592995818.854572, 1592995818.85471, 1592995818.942261, 1592995818.942267, 1592995818.942447, 1592995819.10146, 1592995819.101487, 1592995819.415174, 1592995822.789448, 1592995823.264541, 1592995826.020247, 1592995826.330403, 1592995827.398372, 1592995827.779183, 1592995831.396575, 1592995831.396608]
ip packets lengths: [60, -60, 52, 224, -52, -1500, 52, -652, -868, 52, 52, -1284, 52, -1021, 52, 178, -52, 566, -52, -310, 52, -373, 52, 589, -52, -373, 52, 566, -52, -373, 52, 617, -52, -373, 52, 83, 52, -52, -52, -83, 40, -52, 40]
ip packets timestamps: [1592995818.800966, 1592995818.800969, 1592995818.800971, 1592995818.800974, 1592995818.800977, 1592995818.854565, 1592995818.854569, 1592995818.854572, 1592995818.85471, 1592995818.942249, 1592995818.942258, 1592995818.942261, 1592995818.942264, 1592995818.942267, 1592995818.942445, 1592995818.942447, 1592995818.94245, 1592995819.10146, 1592995819.10147, 1592995819.101487, 1592995819.101489, 1592995819.415174, 1592995819.415177, 1592995822.789448, 1592995822.789457, 1592995823.264541, 1592995823.26455, 1592995826.020247, 1592995826.020256, 1592995826.330403, 1592995826.330406, 1592995827.398372, 1592995827.426276, 1592995827.779183, 1592995827.779187, 1592995831.396575, 1592995831.396581, 1592995831.396587, 1592995831.396592, 1592995831.396608, 1592995831.396614, 1592995831.396619, 1592995831.396625]
default length sequence: [172, -1448, -600, -816, -1232, -969, 126, 514, -258, -321, 537, -321, 514, -321, 565, -321, 31, -31]
default timestamp sequence: [1592995818.800974, 1592995818.854565, 1592995818.854572, 1592995818.85471, 1592995818.942261, 1592995818.942267, 1592995818.942447, 1592995819.10146, 1592995819.101487, 1592995819.415174, 1592995822.789448, 1592995823.264541, 1592995826.020247, 1592995826.330403, 1592995827.398372, 1592995827.779183, 1592995831.396575, 1592995831.396608]
sni: cpt.itv.com
Flow ('1592995802_clear.pcap', 'tcp', '3') info:
src ip: 192.168.0.100
dst ip: 34.240.55.180
sport: 47546
dport: 443
payload lengths : [172, -1448, -600, -816, -32, -1440, -1026, 126, 288, -38, -426, 38, 124, -983, 125, -435, 131, -420, 118, -427, 123, -945, -38, 125, -433, 131, -382, -38, 118, -427, 124, -944, -38, 125, -433, 131, 120, -382, -38, -428, 125, -988, 127, -436, 133, -420, 120, -428, 42, 125, 42, -987, 127, -436, 42, 42, 133, -420, 183, -612, 120, -428, 126, -988, 127, -436, 133, -420, 120, -429, 125, -988, 127, -435, 133, -420, 120, -429, 126, -988, 127, -436, 133, 120, -420, -428, 126, -988, 127, -435, 42, 133, 120, -420, -427, 126, -950, -38, 127, -435, 133, -420]
payload timestamps: [1592995818.800991, 1592995819.512145, 1592995819.512162, 1592995819.512168, 1592995819.512221, 1592995819.512232, 1592995819.512264, 1592995819.574809, 1592995819.826353, 1592995820.282689, 1592995820.282694, 1592995821.093755, 1592995821.093838, 1592995821.093844, 1592995821.093849, 1592995821.375563, 1592995821.375571, 1592995822.563011, 1592995822.865031, 1592995823.264553, 1592995823.264559, 1592995823.554085, 1592995823.554238, 1592995823.60004, 1592995823.956123, 1592995823.972505, 1592995824.311251, 1592995824.311254, 1592995826.020238, 1592995826.330379, 1592995826.330391, 1592995826.827078, 1592995826.827257, 1592995826.827263, 1592995827.107589, 1592995827.107604, 1592995827.39358, 1592995827.426279, 1592995827.426282, 1592995827.723343, 1592995827.75746, 1592995828.096227, 1592995828.217933, 1592995828.595053, 1592995828.610983, 1592995828.964437, 1592995831.396597, 1592995831.611614, 1592995831.611627, 1592995832.443988, 1592995832.444017, 1592995832.444041, 1592995832.444054, 1592995833.443225, 1592995833.443238, 1592995833.443249, 1592995833.443261, 1592995833.443272, 1592995835.873283, 1592995835.873295, 1592995837.913008, 1592995838.157615, 1592995838.163851, 1592995838.603329, 1592995838.603352, 1592995838.925834, 1592995838.957103, 1592995839.295437, 1592995840.231051, 1592995840.961888, 1592995840.961895, 1592995841.140672, 1592995841.140848, 1592995841.350632, 1592995842.342914, 1592995842.342935, 1592995846.494982, 1592995847.208562, 1592995847.208568, 1592995847.282154, 1592995847.282333, 1592995847.827889, 1592995847.827894, 1592995848.166325, 1592995848.166342, 1592995848.49875, 1592995848.498762, 1592995848.934556, 1592995848.934569, 1592995849.179916, 1592995849.180113, 1592995849.180119, 1592995849.180127, 1592995849.361546, 1592995849.539365, 1592995849.539371, 1592995849.865436, 1592995849.865665, 1592995849.881012, 1592995850.232922, 1592995850.232944, 1592995850.718411]
ip packets lengths: [60, -60, 52, 224, -52, -1500, 52, -652, 52, -868, 52, -84, 52, -1492, 52, -1078, 52, 178, -52, 340, -52, -90, -478, 64, 52, 90, -52, 176, -52, -1035, 52, 177, -52, -487, 52, 183, -52, -472, 52, 170, -52, -479, 52, 175, -52, -997, -90, 52, 177, -52, -485, 183, -52, -434, -90, 52, 170, -52, -479, 176, -52, -996, -90, 52, 177, -52, -485, 183, -52, 172, -52, -434, -90, 52, -480, 177, -52, -1040, 52, 179, -52, -488, 52, 185, -52, -472, 52, 172, -52, -480, 52, 94, -52, 177, -52, 94, -52, -1039, 52, 179, -52, -488, 52, 94, -52, 94, -52, 185, -52, -472, 52, 235, -52, -664, 52, 172, -52, -480, 52, 178, -52, -1040, 52, 179, -52, -488, 52, 185, -52, -472, 52, 172, -52, -481, 52, 177, -52, -1040, 52, 179, -52, -487, 52, 185, -52, -472, 52, 172, -52, -481, 52, 178, -52, -1040, 52, 179, -52, -488, 52, 185, -52, 172, -52, -472, 52, -480, 52, 178, -52, -1040, 52, 179, -52, -487, 52, 94, -52, 185, -52, 172, -52, -472, 52, -479, 52, 178, -52, -1002, -90, 52, 179, -52, -487, 185, -52, -472, 52, 52, -52, -52, 52]
ip packets timestamps: [1592995818.800982, 1592995818.800985, 1592995818.800988, 1592995818.800991, 1592995818.800993, 1592995819.512145, 1592995819.512148, 1592995819.512162, 1592995819.512165, 1592995819.512168, 1592995819.512201, 1592995819.512221, 1592995819.512223, 1592995819.512232, 1592995819.512262, 1592995819.512264, 1592995819.512267, 1592995819.574809, 1592995819.574817, 1592995819.826353, 1592995819.826376, 1592995820.282689, 1592995820.282694, 1592995820.282697, 1592995821.093746, 1592995821.093755, 1592995821.093835, 1592995821.093838, 1592995821.093841, 1592995821.093844, 1592995821.093847, 1592995821.093849, 1592995821.093852, 1592995821.375563, 1592995821.375566, 1592995821.375571, 1592995821.375574, 1592995822.563011, 1592995822.563021, 1592995822.865031, 1592995822.865035, 1592995823.264553, 1592995823.264556, 1592995823.264559, 1592995823.264562, 1592995823.554085, 1592995823.554238, 1592995823.554241, 1592995823.60004, 1592995823.600045, 1592995823.956123, 1592995823.972505, 1592995823.972508, 1592995824.311251, 1592995824.311254, 1592995824.324861, 1592995826.020238, 1592995826.02025, 1592995826.330379, 1592995826.330391, 1592995826.3304, 1592995826.827078, 1592995826.827257, 1592995826.82726, 1592995826.827263, 1592995826.827269, 1592995827.107589, 1592995827.107604, 1592995827.10776, 1592995827.39358, 1592995827.393584, 1592995827.426279, 1592995827.426282, 1592995827.426285, 1592995827.723343, 1592995827.75746, 1592995827.791067, 1592995828.096227, 1592995828.113777, 1592995828.217933, 1592995828.234674, 1592995828.595053, 1592995828.595057, 1592995828.610983, 1592995828.640008, 1592995828.964437, 1592995828.995039, 1592995831.396597, 1592995831.396603, 1592995831.611614, 1592995831.611621, 1592995831.611627, 1592995831.611633, 1592995832.443988, 1592995832.444011, 1592995832.444017, 1592995832.444023, 1592995832.444041, 1592995832.444048, 1592995832.444054, 1592995832.44406, 1592995833.443225, 1592995833.443232, 1592995833.443238, 1592995833.443244, 1592995833.443249, 1592995833.443255, 1592995833.443261, 1592995833.443267, 1592995833.443272, 1592995833.443278, 1592995835.873283, 1592995835.873289, 1592995835.873295, 1592995835.873301, 1592995837.913008, 1592995837.913012, 1592995838.157615, 1592995838.157844, 1592995838.163851, 1592995838.176152, 1592995838.603329, 1592995838.603339, 1592995838.603352, 1592995838.603356, 1592995838.925834, 1592995838.925847, 1592995838.957103, 1592995839.063726, 1592995839.295437, 1592995839.327109, 1592995840.231051, 1592995840.231054, 1592995840.961888, 1592995840.961892, 1592995840.961895, 1592995840.961898, 1592995841.140672, 1592995841.140844, 1592995841.140848, 1592995841.140852, 1592995841.350632, 1592995841.350635, 1592995842.342914, 1592995842.342925, 1592995842.342935, 1592995842.342938, 1592995846.494982, 1592995846.49517, 1592995847.208562, 1592995847.208565, 1592995847.208568, 1592995847.208571, 1592995847.282154, 1592995847.282318, 1592995847.282333, 1592995847.827699, 1592995847.827889, 1592995847.827892, 1592995847.827894, 1592995847.827897, 1592995848.166325, 1592995848.166338, 1592995848.166342, 1592995848.166345, 1592995848.49875, 1592995848.49876, 1592995848.498762, 1592995848.498766, 1592995848.934556, 1592995848.934566, 1592995848.934569, 1592995848.934575, 1592995849.179916, 1592995849.18011, 1592995849.180113, 1592995849.180116, 1592995849.180119, 1592995849.180121, 1592995849.180127, 1592995849.36134, 1592995849.361546, 1592995849.361554, 1592995849.539365, 1592995849.539368, 1592995849.539371, 1592995849.550906, 1592995849.865436, 1592995849.865665, 1592995849.865669, 1592995849.881012, 1592995849.896329, 1592995850.232922, 1592995850.232944, 1592995850.233163, 1592995850.718411, 1592995850.718416, 1592995852.488969, 1592995852.902292, 1592995853.893155, 1592995853.893161]
default length sequence: [172, -1448, -600, -816, -32, -1440, -1026, 126, 288, -38, -426, 38, 124, -983, 125, -435, 131, -420, 118, -427, 123, -945, -38, 125, -433, 131, -382, -38, 118, -427, 124, -944, -38, 125, -433, 131, 120, -382, -38, -428, 125, -988, 127, -436, 133, -420, 120, -428, 42, 125, 42, -987, 127, -436, 42, 42, 133, -420, 183, -612, 120, -428, 126, -988, 127, -436, 133, -420, 120, -429, 125, -988, 127, -435, 133, -420, 120, -429, 126, -988, 127, -436, 133, 120, -420, -428, 126, -988, 127, -435, 42, 133, 120, -420, -427, 126, -950, -38, 127, -435, 133, -420]
default timestamp sequence: [1592995818.800991, 1592995819.512145, 1592995819.512162, 1592995819.512168, 1592995819.512221, 1592995819.512232, 1592995819.512264, 1592995819.574809, 1592995819.826353, 1592995820.282689, 1592995820.282694, 1592995821.093755, 1592995821.093838, 1592995821.093844, 1592995821.093849, 1592995821.375563, 1592995821.375571, 1592995822.563011, 1592995822.865031, 1592995823.264553, 1592995823.264559, 1592995823.554085, 1592995823.554238, 1592995823.60004, 1592995823.956123, 1592995823.972505, 1592995824.311251, 1592995824.311254, 1592995826.020238, 1592995826.330379, 1592995826.330391, 1592995826.827078, 1592995826.827257, 1592995826.827263, 1592995827.107589, 1592995827.107604, 1592995827.39358, 1592995827.426279, 1592995827.426282, 1592995827.723343, 1592995827.75746, 1592995828.096227, 1592995828.217933, 1592995828.595053, 1592995828.610983, 1592995828.964437, 1592995831.396597, 1592995831.611614, 1592995831.611627, 1592995832.443988, 1592995832.444017, 1592995832.444041, 1592995832.444054, 1592995833.443225, 1592995833.443238, 1592995833.443249, 1592995833.443261, 1592995833.443272, 1592995835.873283, 1592995835.873295, 1592995837.913008, 1592995838.157615, 1592995838.163851, 1592995838.603329, 1592995838.603352, 1592995838.925834, 1592995838.957103, 1592995839.295437, 1592995840.231051, 1592995840.961888, 1592995840.961895, 1592995841.140672, 1592995841.140848, 1592995841.350632, 1592995842.342914, 1592995842.342935, 1592995846.494982, 1592995847.208562, 1592995847.208568, 1592995847.282154, 1592995847.282333, 1592995847.827889, 1592995847.827894, 1592995848.166325, 1592995848.166342, 1592995848.49875, 1592995848.498762, 1592995848.934556, 1592995848.934569, 1592995849.179916, 1592995849.180113, 1592995849.180119, 1592995849.180127, 1592995849.361546, 1592995849.539365, 1592995849.539371, 1592995849.865436, 1592995849.865665, 1592995849.881012, 1592995850.232922, 1592995850.232944, 1592995850.718411]
sni: tom.itv.com

解析速度

flowcontainer的解析的速度还是可以的,50G左右的流量2个小时左右即可完成所有流信息的提取。
5G左右的流量12分钟即可解析完毕。

本文地址:https://blog.csdn.net/jmh1996/article/details/107148871