centos下对文件某些特定字符串分组统计出现次数

假如现有数据：

 

{ "@timestamp": "2018-10-13t21:55:58+08:00", "remote_addr": "100.120.34.3", "referer": "-", "request": "get /api/gourd/activeupload?idfa=58237fa9-a1b3-4202-b5f3-9536983119e5&appid=1410137206&source=rehulu http/1.1", "status": 200, "request_time": 0.076, "cookie":"-","host":"cms.369wan.com","bytes": 48, "agent": "mozilla/5.0 (x11; ubuntu; linux x86_64; rv:18.0) gecko/20100101 firefox/18.0", "proxy_x_forwarded": "139.129.97.187, 100.120.34.3" "upstr_addr": "127.0.0.1:9000","upstr_host": "-","ups_resp_time": 0.076 }
{ "@timestamp": "2018-10-13t21:56:06+08:00", "remote_addr": "100.120.34.101", "referer": "-", "request": "get /api/gourd/activeupload?idfa=e9d7f87a-9042-46b4-82e8-e5f64b74466b&appid=1410137206&source=rehulu http/1.1", "status": 200, "request_time": 0.076, "cookie":"-","host":"cms.369wan.com","bytes": 48, "agent": "mozilla/5.0 (x11; ubuntu; linux x86_64; rv:18.0) gecko/20100101 firefox/18.0", "proxy_x_forwarded": "139.129.97.187, 100.120.34.101" "upstr_addr": "127.0.0.1:9000","upstr_host": "-","ups_resp_time": 0.076 }
{ "@timestamp": "2018-10-13t21:56:08+08:00", "remote_addr": "100.120.34.29", "referer": "-", "request": "get /api/gourd/activeupload?idfa=d5b924f3-7d25-4b52-bae9-3270b08ea32d&appid=1410137206&source=rehulu http/1.1", "status": 200, "request_time": 0.075, "cookie":"-","host":"cms.369wan.com","bytes": 48, "agent": "mozilla/5.0 (x11; ubuntu; linux x86_64; rv:18.0) gecko/20100101 firefox/18.0", "proxy_x_forwarded": "139.129.97.187, 100.120.34.29" "upstr_addr": "127.0.0.1:9000","upstr_host": "-","ups_resp_time": 0.075 }
{ "@timestamp": "2018-10-13t21:56:10+08:00", "remote_addr": "100.120.34.75", "referer": "-", "request": "get /api/gourd/activeupload?idfa=d166459d-e823-4847-9094-6f4bf90625b2&appid=1410137206&source=rehulu http/1.1", "status": 200, "request_time": 0.078, "cookie":"-","host":"cms.369wan.com","bytes": 48, "agent": "mozilla/5.0 (x11; ubuntu; linux x86_64; rv:18.0) gecko/20100101 firefox/18.0", "proxy_x_forwarded": "139.129.97.187, 100.120.34.75" "upstr_addr": "127.0.0.1:9000","upstr_host": "-","ups_resp_time": 0.078 }
{ "@timestamp": "2018-10-13t21:56:18+08:00", "remote_addr": "100.120.34.39", "referer": "-", "request": "get /api/gourd/activeupload?idfa=08c65c3b-eed2-4a65-b0c1-67fc7fb78e18&appid=1410137206&source=rehulu http/1.1", "status": 200, "request_time": 0.082, "cookie":"-","host":"cms.369wan.com","bytes": 48, "agent": "mozilla/5.0 (x11; ubuntu; linux x86_64; rv:18.0) gecko/20100101 firefox/18.0", "proxy_x_forwarded": "139.129.97.187, 100.120.34.39" "upstr_addr": "127.0.0.1:9000","upstr_host": "-","ups_resp_time": 0.082 }
{ "@timestamp": "2018-10-13t21:56:31+08:00", "remote_addr": "100.120.34.68", "referer": "-", "request": "get /api/gourd/activeupload?idfa=d5b924f3-7d25-4b52-bae9-3270b08ea32d&appid=1410137206&source=rehulu http/1.1", "status": 200, "request_time": 0.079, "cookie":"-","host":"cms.369wan.com","bytes": 48, "agent": "mozilla/5.0 (x11; ubuntu; linux x86_64; rv:18.0) gecko/20100101 firefox/18.0", "proxy_x_forwarded": "139.129.97.187, 100.120.34.68" "upstr_addr": "127.0.0.1:9000","upstr_host": "-","ups_resp_time": 0.079 }

临时保存为tmp.log

awk -f 'idfa=' '{print $2}' tmp.log 

 

出现如下结果：

58237fa9-a1b3-4202-b5f3-9536983119e5&appid=1410137206&source=rehulu http/1.1", "status": 200, "request_time": 0.076, "cookie":"-","host":"cms.369wan.com","bytes": 48, "agent": "mozilla/5.0 (x11; ubuntu; linux x86_64; rv:18.0) gecko/20100101 firefox/18.0", "proxy_x_forwarded": "139.129.97.187, 100.120.34.3" "upstr_addr": "127.0.0.1:9000","upstr_host": "-","ups_resp_time": 0.076 }
e9d7f87a-9042-46b4-82e8-e5f64b74466b&appid=1410137206&source=rehulu http/1.1", "status": 200, "request_time": 0.076, "cookie":"-","host":"cms.369wan.com","bytes": 48, "agent": "mozilla/5.0 (x11; ubuntu; linux x86_64; rv:18.0) gecko/20100101 firefox/18.0", "proxy_x_forwarded": "139.129.97.187, 100.120.34.101" "upstr_addr": "127.0.0.1:9000","upstr_host": "-","ups_resp_time": 0.076 }
d5b924f3-7d25-4b52-bae9-3270b08ea32d&appid=1410137206&source=rehulu http/1.1", "status": 200, "request_time": 0.075, "cookie":"-","host":"cms.369wan.com","bytes": 48, "agent": "mozilla/5.0 (x11; ubuntu; linux x86_64; rv:18.0) gecko/20100101 firefox/18.0", "proxy_x_forwarded": "139.129.97.187, 100.120.34.29" "upstr_addr": "127.0.0.1:9000","upstr_host": "-","ups_resp_time": 0.075 }

执行 

awk -f 'idfa=' '{print $2}' tmp.log | awk -f '&source=' '{print $1}' 

出现如下结果：
58237fa9-a1b3-4202-b5f3-9536983119e5&appid=1410137206
e9d7f87a-9042-46b4-82e8-e5f64b74466b&appid=1410137206
d5b924f3-7d25-4b52-bae9-3270b08ea32d&appid=1410137206

执行 

awk -f 'idfa=' '{print $2}' tmp.log | awk -f '&source=' '{print $1}'  | sort  (这一步将结果相同的放在一起)

出现如下结果：
58237fa9-a1b3-4202-b5f3-9536983119e5&appid=1410137206
e9d7f87a-9042-46b4-82e8-e5f64b74466b&appid=1410137206
d5b924f3-7d25-4b52-bae9-3270b08ea32d&appid=1410137206

执行

awk -f 'idfa=' '{print $2}' cms_\(2\).log | awk -f '&source=' '{print $1}' |sort| uniq -c
出现最终结果(次数 和 各字符串)：
1 58237fa9-a1b3-4202-b5f3-9536983119e5&appid=1410137206
1 e9d7f87a-9042-46b4-82e8-e5f64b74466b&appid=1410137206
1 d5b924f3-7d25-4b52-bae9-3270b08ea32d&appid=1410137206