Video Transcript (English)

In this video, you’ll learn about log rotation and retention in MongoDB.

We’ll take a look at the retention period for logs in Atlas and how to use MongoDB's standard approach to log rotation. Then we’ll configure the Linux logrotate service to automate the rotation of MongoDB log files on self-managed deployments.

Logs are a valuable tool in MongoDB because they provide detailed information about the performance of your deployments and help diagnose issues. But log files are always growing, so to keep them usable it's important to have a good rotation and retention strategy.

Regularly rotating and archiving logs can prevent them from occupying excessive disk space and taking several minutes to open. Otherwise, we run the risk of low disk space, degraded performance, and an inhibited ability to troubleshoot issues.

Rotating a log refers to the process of maintaining a log’s size. This can be done automatically or manually, but the goal is the same, which is to prevent the log files from growing without bounds.

MongoDB Atlas retains the last 30 days of log messages and system event audit messages for each node in a cluster. For example, if you have a primary and two secondary nodes, Atlas would keep logs for all three nodes. This is a useful feature that is set up automatically and works out of the box.

As a reminder, to view and download logs in MongoDB Atlas, your user must have the "Project Data Access Read Only" role. The feature is available on M10-or-above Atlas clusters.

For self-managed deployments, log files are retained indefinitely unless we explicitly tell the deployment to rotate the logs. This can be done by sending a `SIGUSR1` signal to the MongoDB service, or by running the logRotate command.

Now let’s discuss MongoDB's approach to log rotation. There are two main types of log rotation behavior: `rename`, which is MongoDB’s default method, and `reopen`.

With rename, the MongoDB service will respond to a `logRotate` command or the `SIGUSR1` signal by renaming the current log file with a UTC timestamp, opening a new log, closing the old one, and sending all new events to the new log file.

For example, if we start the service manually with just the logpath parameter, then the default behavior will be “rename”. If we then send the `SIGUSR1` signal, or run the logRotate command, the resulting file names will look something like this

In the output, we see a list of files, each starting with the filename we provided to the launch parameter – `server1.log`. After the logRotate command was called, the older log files were renamed from `server1.log` to `server1.log.<timestamp>`.

The `reopen` option responds to the logRotate command by closing and reopening the file. The `reopen` option is required when using an external log rotation tool such as the Linux logrotate service.

When using `reopen`, you must also specify the `logappend` option to append log messages to the same file. Without logappend, the MongoDB service creates a new log file every time.

For example, if we start the service manually and set logRotate to reopen with logappend, and then rotate the logs by using SIGUSR1 or the logRotate command, we expect to see only one log file.

MongoDB recommends that you automate the process of rotating your logs based on a specific log file size, or at a specific time interval.

A sustainable file size will depend on your configuration, but generally the log files should be small enough to easily share.

Logging can impact performance when both the log files, and database files are located on the same disk. Additionally, when the log files grow abnormally large, they can consume disk space otherwise allocated for normal operation of the database.

Now, let’s look at how to hook into the Linux `logrotate` utility to automate log rotation.

On Linux, `logrotate` was designed to help system admins manage the large number of log files being generated from all sorts of utilities in a Linux environment.

The rotation process begins by renaming the active log file. Then, it creates a new log file with the same name. Finally, it invokes other logic such as compression to complete the rotation process.

`logrotate`can be configured to rotate log files on a daily, weekly, or monthly basis.

We can configure MongoDB to use the `logrotate` utility to automate the rotation of log files for your deployment.

To do this, we first need to make a few changes to the MongoDB configuration file. Let’s open the config file.

In this file, we locate the section titled `systemLog` and make sure that `destination` is set to `file` and `logAppend` is set to `true`. Just like the `logAppend` command line argument, adding this option to our configuration tells MongoDB to append data to the end of the file.

Finally, make sure `logRotate` is set to `reopen` so that MongoDB reopens the log file when instructed by the logrotate service.

Now that the MongoDB configuration file is all set, the last step is to create a configuration file for the logrotate service`.

Let’s create the MongoDB-specific configuration file in the default logrotate service location.

In this example, the Linux `logrotate` service will send a `SIGUSR1` signal to the MongoDB service once a day, or when it reaches a maximum size of 10 megabytes. The rest of the configuration looks for any empty files that match a pattern and removes them. The Linux logrotate service is responsible for renaming and compressing the file.

Once we have this script set up, we’ll restart the MongoDB service.

Finally, let’s test our log rotation by sending a signal to the MongoDB service. At the same time, we’ll use the tail command on the log file. The `tail -F` command allows us to see lines written to the file in real time. It will be interesting to see what happens in the event of a “reopen rotate”.

Now that we’re viewing the log file, we will issue the `SIGUSR1` in another tab to the MongoDB service. The command, pidof, is a Linux/Unix utility that will return the process ID when given the name of a process.

As you can see, we can still view the file because it was simply reopened by MongoDB. However, tail reports that the log has been truncated. In reality, the original file has been renamed by logrotate and a new one with the same name as the original was created.

Well done! In this video, you learned how to identify the log retention period in MongoDB Atlas. You also learned how to use MongoDB’s standard rename log rotation approach by providing just the –logpath parameter. We also briefly covered how to use the reopen or rename options with the logRotate and logappend parameters. Finally, you learned how to configure MongoDB to use the Linux logrotate service to automate log rotation.