Linux json文件(排序|去重)

如果有这样一些日志，里面是json内容的格式，我们需要对其某些字段进行排序，最方便的就是使用jq命令处理了，当然使用awk也可以处理，最方便还是jq，此文介绍jq在排序中的使用。

使用教程

文件test.logjson文件数据

{"time":1537111143,"ti":1537111143233,"data":{"v":"1.1.0","ext":{"ak":"3ea689f0-8054-11e8-a08e-ab12c01f89c9","ccode":"chaojitanqiu-wechatmp-01","type":"login_out","uid":"oLogu5RL3ArbLkVYEHtM5pJ1k63M","scene":1095},"device":{"screenWidth":360,"statusBarHeight":24,"pixelRatio":2,"system":"Android 6.0.1","benchmarkLevel":7,"windowWidth":360,"brand":"Xiaomi","screenHeight":640,"version":"6.7.2","fontSizeSetting":16,"language":"zh_CN","windowHeight":640,"model":"Redmi 4","platform":"android","SDKVersion":"2.3.0","devicePixelRatio":2}}}
{"time":1537111634,"ti":1537111634410,"data":{"v":"1.1.0","ext":{"ak":"3ea689f0-8054-11e8-a08e-ab12c01f89c9","ccode":"chaojitanqiu-wechatmp-01","type":"login_out","uid":"oLogu5Z5kftizEU3Wx6mGj3m5z7o","scene":1095},"device":{"screenWidth":360,"statusBarHeight":24,"pixelRatio":3,"system":"Android 6.0","benchmarkLevel":4,"windowWidth":360,"brand":"HUAWEI","screenHeight":640,"version":"6.7.2","fontSizeSetting":16,"language":"zh_CN","windowHeight":640,"model":"HUAWEI GRA-CL10","platform":"android","SDKVersion":"2.3.0","devicePixelRatio":3}}}
{"time":1537111471,"ti":1537111471439,"data":{"v":"1.1.0","ext":{"ak":"3ea689f0-8054-11e8-a08e-ab12c01f89c9","ccode":"chaojitanqiu-wechatmp-01","type":"login_out","uid":"oLogu5RL3ArbLkVYEHtM5pJ1k63M","scene":1095},"device":{"screenWidth":360,"statusBarHeight":24,"pixelRatio":2,"system":"Android 6.0.1","benchmarkLevel":7,"windowWidth":360,"brand":"Xiaomi","screenHeight":640,"version":"6.7.2","fontSizeSetting":16,"language":"zh_CN","windowHeight":640,"model":"Redmi 4","platform":"android","SDKVersion":"2.3.0","devicePixelRatio":2}}}

根据data.ext.uid字段排序

cat test.log | jq  -s "sort_by(.data.ext.uid) | .[]"

使用jq去重数据,并只显示data.ext.uid字段

cat test.log | jq ".data.ext.uid" | sort | uniq

多维度去重

cat test.log |  jq ".data.ext.appKey + .data.ext.uid" | sort | uniq | wc -l

PS 以下可用于验证JSON串

jq -R -r '. as $line | try fromjson catch $line' | awk -F '.log:' '{print $NF}'