Logging is an important and widely embraced industry practice that provides us with valuable insight into how our applications are performing in real-time. So in this article, I’ll delve into the process of setting up logging in a Python application.

Python’s logging module

While there are a few amazing third-party logging libraries, today I’ll cover the built-in logging module in Python, as it’s by far the most prevalent.

The logging module is part of the Python standard library and provides tracking for events that occur during the runtime of an application.

The simplest way to access a logger in Python is to call the logging methods directly, which will utilize the root logger.

import logging
logging.basicConfig(level=logging.DEBUG)
logging.debug('This is a debug message')
logging.info('This is an info message')
logging.warning('This is a warning message')
logging.error('This is an error message')
logging.critical('This is a critical message')

The logging module has five built-in levels of severity: DEBUG, INFO, WARNING, ERROR, and CRITICAL.

DEBUG: Detailed information, typically of interest only when diagnosing problems.

INFO: Confirmation that things are working as expected.

WARNING: An indication that something unexpected happened, or indicative of some problem in the near future (e.g. ‘disk space low’). The software is still working as expected.

ERROR: Due to a more serious problem, the software has not been able to perform some functions.

CRITICAL: A serious error, indicating that the program itself may be unable to continue running.

EXCEPTION: An exception was raised, signaling the death of a program.

As you can see, the logger has a good selection of levels to help us structure our logs and provide an at-a-glance idea of the state of our program.

Setting up a logger

While utilizing the root logger is the simplest way to log messages and is fine in small applications, it’s not recommended for larger applications.

We can create a new logger by calling the getLogger method and passing in the name of the logger. This will create a new logger with the name we passed in.

It’s common to use __name__ as the name of the logger, as this will create a logger with the name of the module that the logger is created in.

import logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

logger.debug('This is a debug message')
logger.info('This is an info message')
logger.warning('This is a warning message')
logger.error('This is an error message')
logger.critical('This is a critical message')

Writing to a file

By default, the logging module will log messages to the console. However, we can easily configure the logger to write to a file. We do so by using the FileHandler class.

import logging

logger = logging.getLogger(__name__)

file_handler = logging.FileHandler('app.log')
logger.addHandler(file_handler)

We can also create a handler that will rotate the log file when it reaches a certain size by using the RotatingFileHandler, and I would recommend its use when you expect your programs to be running for long periods of time.

import logging.handlers

logger = logging.getLogger(__name__)

file_handler = logging.handlers.RotatingFileHandler('app.log', maxBytes=1000, backupCount=3)
logger.addHandler(file_handler)

The logger allows us to define multiple handlers, so we can write to the console and to a file at the same time.

import logging

logger = logging.getLogger(__name__)

file_handler = logging.FileHandler('app.log')
console_handler = logging.StreamHandler()

logger.addHandler(file_handler)
logger.addHandler(console_handler)

By doing this, we can write to both the console and to a file at the same time.

Formatting the log messages

We can also format the log messages to include more information, such as the time the message was logged, the name of the logger, the log level, and the message itself.

import logging

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
formatter = logging.Formatter(
        '%(asctime)s - %(name)s - %(levelname)s - %(message)s',
        datefmt='%Y-%m-%d %H:%M:%S'
)

stream_handler = logging.StreamHandler()
stream_handler.setFormatter(formatter)

logger.addHandler(stream_handler)
logger.debug('This is a debug message')

Notice the syntax used in the format string? This is the syntax used by the logging module to format the log messages.

The logging module provides a number of attributes that can be used in the format string to include more information in the log messages.

• %(asctime)s: The time the message was logged.
• %(name)s: The name of the logger.
• %(levelname)s: The log level.
• %(message)s: The log message itself.
• %(filename)s: The filename of the module that the logger is created in.
• %(lineno)d: The line number of the module that the logger is created in.
• %(funcName)s: The name of the function that the logger is created in.

It also allows us to set a date format using the same syntax.

Filtering log messages

We can also apply various filters to the log messages to only log messages that meet certain criteria.

import logging

class ImportantFilter(logging.Filter):
    def filter(self, record):
	    return 'important' in record.getMessage()

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger('my_logger')

logger.addFilter(ImportantFilter())
logger.info('This is a regular message')
logger.info('This is an important message')

In this example, we have created a filter that will only log messages that are of the INFO level or above (WARNING, ERROR) and contain the word ‘important’ in the message.

But I just need a simple logger?

Sometimes you won’t really need all of this setup, especially for simple, one file scripts, and the logging module has a solution for this too.

import logging
logging.basicConfig(
    level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s',
    datefmt='%Y-%m-%d %H:%M:%S',
    filename='apply.log'
    )

This is perfect for when you just want a tiny little logger with no frills.

Final thoughts

Logging not only helps you track the behavior of your applications at runtime, it also allows us to detect patterns over periods of time, which can give us clearer insight into how and why certain bugs and errors are happening. This is especially useful for detecting those bugs that show up in very rare, specific edge cases.

And if you want to become a logging pro, check out my video here.