Just the other day, one of my Percona Monitoring and Management (PMM) instances failed to update and ended up in a funny state:
No matter how many times I clicked the “check for updates” button in the lower right, PMM still showed “no updates available”. Basically, PMM forgot what version it is and because of that it thinks there are no updates available
If you run into this (or possibly other PMM update problems), you may find the following helpful to troubleshoot or resolve the problem.
I have deployed PMM through docker, so the first step is to get inside the docker container to be able to investigate and troubleshoot the installation.
1 | docker exec -it pmm-server bash |
Percona Monitoring and Management (as of version 2.26) learns its own version from the version of “pmm-update” package. In my case, this did not work due to a corrupted rpm database:
1 2 3 4 5 6 7 8 | [root@812fb43b3a96 opt]# yum info pmm-update error: rpmdb: BDB0113 Thread/process 13841/140435865503552 failed: BDB1507 Thread died in Berkeley DB library error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery error: cannot open Packages index using db5 - (-30973) error: cannot open Packages database in /var/lib/rpm CRITICAL:yum.main: Error: rpmdb open failed |
It is not clear to me why the database has become corrupted in my case, but unfortunately, it did and needed to be repaired:
1 2 3 | mv /var/lib/rpm/__db* /tmp/ rpm --rebuilddb rm -rf /var/cache/yum |
These commands backup the current database to /tmp/ (just in case) to rebuild the current database as well as remove already downloaded packages (it has nothing to do with database corruption but reduces reliance on the local state).
1 2 | # supervisorctl start pmm-update-perform pmm-update-perform: started |
This command starts the update process. You probably could also trigger an update through the Web GUI at this point but I want you to be aware of this option, too.
As this simply starts the background update process you do not get much information about what is going on. To find that out, you need to check the log:
1 | tail /srv/logs/pmm-update-perform.log |
Note while the update is going, the PMM instance may not be available at all. You could see this message:
Do not freak out – wait for the update to finish before trying to use PMM again.
The update log does not really have a very clear message which states what update was completed successfully. Here is, however, an example of how the end of a log from a successful update may look:
1 2 3 4 | PLAY RECAP ********************************************************************* localhost : ok=79 changed=31 unreachable=0 failed=0 skipped=16 rescued=0 ignored=0 time="2022-04-01T15:36:39Z" level=info msg="Waiting for Grafana dashboards update to finish..." |
With that, the update has been completed and my PMM instance has successfully updated to the recent version.
I hope this blog post will help you to troubleshoot your PMM update problems too if you happen to run into one! 🙂
