欢迎您访问程序员文章站本站旨在为大家提供分享程序员计算机编程知识!
您现在的位置是: 首页

如何扩展python的logging组件支持json日志输出

程序员文章站 2022-05-03 22:30:32
...


这两天在优化公司一个python的项目,顺便研究了一下如何将python日志转成json格式,原来在Java的项目中搞过类似的事情,知道日志转成json之后有很多便利的之处,最常见的就是可以直接对接各种日志分析系统,如开源的ELK,将数据导入之后就能快速的进行查询和分析,方便做各种统计,监控或报警等。


python里面的logging组件,其实已经是组件化了,有Logger组件,Handler组件,Fomatter组件,如下图所示:

如何扩展python的logging组件支持json日志输出
            
    
    博客分类: Python jsonpythonlogging 

logger=>handler=>formatter分别是一对多的关系,日志的格式其实是由formatter决定的,所以想要扩展成你想要的各种格式,就重写定制formatter组件就可以了,它实际上和Java里面Log4j的LayOut组件类似。


核心代码如下:

```python
 REMOVE_ATTR = ["filename", "module", "exc_text", "stack_info", "created", "msecs", "relativeCreated", "exc_info", "msg"]


class JSONFormatter(logging.Formatter):
    host_name, host_ip = HostIp.get_host_ip()

    def format(self, record):
        extra = self.build_record(record)
        self.set_format_time(extra)  # set time
        self.set_host_ip(extra)  # set host name and host ip
        extra['message'] = record.msg  # set message
        if record.exc_info:
            extra['exc_info'] = self.formatException(record.exc_info)
        if self._fmt == 'pretty':
            return json.dumps(extra, indent=1, ensure_ascii=False)
        else:
            return json.dumps(extra, ensure_ascii=False)

    @classmethod
    def build_record(cls, record):
        return {
            attr_name: record.__dict__[attr_name]
            for attr_name in record.__dict__
            if attr_name not in REMOVE_ATTR
        }

    @classmethod
    def set_format_time(cls, extra):
        now = datetime.datetime.utcnow()
        format_time = now.strftime("%Y-%m-%dT%H:%M:%S" + ".%03d" % (now.microsecond / 1000) + "Z")
        extra['@timestamp'] = format_time
        return format_time

    @classmethod
    def set_host_ip(cls, extra):
        extra['host_name'] = JSONFormatter.host_name
        extra['host_ip'] = JSONFormatter.host_ip
 
```


使用的时候,可以通过简单配置即可:
```
 [loggers]
keys=root

[handlers]
keys=console,file

[formatters]
keys=json,json_pretty

[logger_root]
level=DEBUG
;handlers=console,file,rotate
handlers=console,file

[handler_console]
class=StreamHandler
level=INFO
formatter=json_pretty
args=(sys.stderr,)


[handler_file]
class=FileHandler
level=INFO
formatter=json
args=('log/base_conf.log','a','utf-8')

[handler_rotate]
class=handlers.TimedRotatingFileHandler
level=INFO
formatter=json
args=('log/rotate.log', 'D',1,0,'utf-8')

[formatter_json]
class=format.json_formatter.JSONFormatter

[formatter_json_pretty]
format=pretty
class=format.json_formatter.JSONFormatter
```


如下的一段异常代码:


```
    fileConfig('log_conf.ini')
    log = logging.getLogger(__name__)
    try:
        a = 1 / 0
    except Exception:
        log.exception(" occurred exception ")

```

输出结果如下:

```
{
 "name": "__main__",
 "args": [],
 "levelname": "ERROR",
 "levelno": 40,
 "pathname": "C:/texx.py",
 "lineno": 17,
 "funcName": "base_configuration",
 "thread": 10608,
 "threadName": "MainThread",
 "processName": "MainProcess",
 "process": 11916,
 "@timestamp": "2019-01-10T12:50:20.392Z",
 "host_name": "your-PC",
 "host_ip": "192.168.10.11",
 "message": " occurred exception ",
 "exc_info": "Traceback (most recent call last):\n  File \"C:/txxx.py\", line 14, in base_configuration\n    a = 1 / 0\nZeroDivisionError: division by zero"
}
```


可以看到内容非常详细,并且组件还支持自定义字段的加入,在收集到日志系统上,可以非常的方便检索统计。

详细的解释和代码,可以fork我的github连接:

https://github.com/qindongliang/python_log_json