banner
云野阁

云野阁

闲云野鹤，八方逍遥
HomeArchives关于

Docker container log configuration management

#运维#docker#知识梳理#Linux18
AI Translation
This post is translated from Chinese into English through AI.View Original
AI-generated summary
The document discusses managing Docker container logs and addresses an issue where a running container stopped due to insufficient disk space. The error message indicated that the disk was full, specifically in the Docker storage directory. By checking the disk usage with the `df -h` command, it was found that the `containers` directory was consuming 409G of space, primarily due to a large log file associated with the container. Deleting this log file allowed the container to restart and the business to resume normal operations. To prevent this issue from recurring, it is recommended to configure the Docker daemon's log settings in the `/etc/docker/daemon.json` file. The suggested configuration includes setting the log driver to `json-file`, with a maximum log size of 50MB and retaining a maximum of 2 log files. After making these changes, the Docker service should be restarted using `sudo systemctl daemon-reload` and `sudo systemctl restart docker`. Additional resources on Docker logging are provided in a linked article.

The running project container suddenly stopped, and upon checking the container logs, it was found that there was insufficient disk space, which caused the container to be unable to run.

Error response from daemon: Cannot restart container docker-p: mkdir /home/dockerData/overlay2/7bef1e2fa4788ab4d5db7e2e850d7ceeed09185b55c175f20c1500d28a0cc874d: no space left on device

First, use df -h to check the disk usage.

overlay 413G 413G 0 100% /home/dockerData/overlay2/fea24dfbbe9f7d6d8309....../merged
overlay 413G 413G 0 100% /home/dockerData/overlay2/7bef1e2fa4788ab4d5db....../merged

The container's storage directory /home/dockerData/ (its default storage directory is /var/lib/docker, but I have reconfigured it) has completely used up the disk.

Once the cause is found, it’s easy to handle. Directly go to the container's storage directory /home/dockerData/ and use the du -sh command to check the overlay2 and containers directories and their files. It was found that the container file in the containers directory 98bbe7f17fa335f1b6e17...... occupied 409G.

du -sh overlay2
3.7G overlay2

du -sh containers/98bbe7f17fa335f1b6e17......
409G containers/98bbe7f17fa335f1b6e17......

Further investigation into the container files revealed that the source was the container's log file 98bbe7f17fa335f1b6e17......-json.log.

After deleting this log file, restarting the container restored normal operations.

To prevent this situation from happening again, it is necessary to configure the logging driver in the Docker daemon /etc/docker/daemon.json, with the specific configuration as follows:

  "log-driver":"json-file",
  "log-opts":{"max-size" :"50m","max-file":"2"}

"log-driver":"json-file" The default logging driver for Docker. The log format is JSON.

"max-size" :"50m" The maximum size before log rotation is 50M.

"max-file":"2" The maximum number of log files to retain is 2.

After the configuration is complete, restart Docker.

sudo systemctl daemon-reload
sudo systemctl restart docker

For more related content about Docker logs, please refer to the content in the link below.

https://segmentfault.com/a/1190000023144155

Loading...
Ownership of this post data is guaranteed by blockchain and smart contracts to the creator alone.