diff --git a/CHANGELOG.rst b/CHANGELOG.rst index 7cfa24c9..ef7e3942 100644 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -4,26 +4,41 @@ pgMonitor Release Notes .. contents:: Topics -v4.12.0 -======= +v5.2.0 +====== Release Summary --------------- -Crunchy Data is pleased to announce the availability of pgMonitor 4.12.0. This release primarily brings support for Grafana 10.4. See Changelog for additional information. +Crunchy Data is pleased to announce the availability of pgMonitor 5.2.0. This release brings support for the latest pgmonitor extension version 2.1.0 and many bug fixes. -Breaking Changes / Porting Guide --------------------------------- +Major Changes +------------- -- grafana - Update the dashboards to support Grafana 10.4 so that we're on an officially supported version of Grafana. This does potentially break backward compatibility with Grafana 9.x, so an update to Grafana 10.4 will be required with this version of pgMonitor. +- sql_exporter - Updated metric queries for pgmonitor-extension 2.1.0 + +Minor Changes +------------- + +- grafana - Improve Query Statistics dashboard to more accurately report lifetime vs time range statistics +- grafana - Removed the `Last Backup Size (Total)` panel from the pgBackRest Grafana dashboard since it was backed by the removed metric. +- postgres_exporter - Removed the metric `ccp_backrest_last_info_repo_total_size_bytes`. +- prometheus - Added new alerts to monitor the new metrics for the minimum required version of the pgmonitor extension +- sql_exporter - Add additional metrics for monitoring replication slot status. For PG16+ monitor for conflicts. For PG17+, monitor synced and failover status. +- sql_exporter - Add scrape_error_drop_interval setting to the configuration example +- sql_exporter - Added new metrics to monitor pgmonitor-extension version. - `ccp_pgmonitor_extension_global_version` - shows the currently installed version of the extension on the global database as an integer - `ccp_pgmonitor_extension_per_db_version` - shows the currently installed version of the extension on each monitored user database as an integer - `ccp_pgmonitor_extension_global_min_version_installed` - shows whether the currently installed version of the extension is the minimum required for this version of pgMonitor on the global database (0 true, 1 false) - `ccp_pgmonitor_extension_per_db_min_version_installed` - shows whether the currently installed version of the extension is the minimum required for this version of pgMonitor on each monitored user database (0 true, 1 false) +- sql_exporter - Removed the metric `ccp_backrest_last_info_repo_total_size_bytes`. When block incremental backups are enabled, this metric is no longer available from pgBackRest. Bugfixes -------- -- grafana - Fix etcd dashboard to use new metric names in etcd 3.5 -- postgres_exporter - Fix query for database table size to remove duplicate word -- postgres_exporter - Fix query for pgBackRest monitoring to handle 3 number versions - +- grafana - Fix Cache Hit Ratio panel on PG Details dashboard to always be lines. Depending on data returned was sometimes being shown as points. +- grafana - Fix pgBackRest recovery window panel showing multiple values after a PostgreSQL switchover +- postgres_exporter - Disable all collectors included with postgres_exporter by default in example configuration. The other options to disable default metrics are not applied to the new collections. +- postgres_exporter - Fix the `ccp_table_size_size_bytes` metric to remove the duplicate word and just be `ccp_table_size_bytes` +- sql_exporter - Fix the `ccp_table_size_size_bytes` metric to remove the duplicate word and just be `ccp_table_size_bytes` +- sql_exporter - Fix the names of ccp_pgbouncer_database_db_conn_perc_used, ccp_pgbouncer_database_paused, ccp_pgbouncer_database_disabled, and ccp_pgbouncer_list_item_count to be consistent with the old metric names from postgres exporter. These new names are the ones expected in the Grafana dashboard so this change fixes that to work properly again. +- sql_exporter - Removed extraneous double quote at the end of the pgbouncer fdw collector file v5.1.1 ====== @@ -46,7 +61,6 @@ Release Summary Crunchy Data is pleased to announce the availability of pgMonitor 5.1.0. This release brings support for PostgreSQL 17. It also brings more flexible Grafana dashboards, an HAProxy dashboard and better support for etcd and pgBackRest. - Major Changes ------------- @@ -56,7 +70,7 @@ Major Changes Minor Changes ------------- -- grafana - Add a new variable dropdown to all dashboards for the Datasource. Allows more flexiblity when importing the dashboard to different environments. +- grafana - Add a new variable dropdown to all dashboards for the Datasource. Allows more flexibility when importing the dashboard to different environments. - grafana - Add panel to PG Details dashboard to track autovac workers running vs max - grafana - add a dashboard for HAProxy - sql_exporter - Add metrics to track current autovacuum workers running and max autovacuum workers @@ -104,6 +118,26 @@ Bugfixes - sql_exporter - add new metric for n_tup_newpage_upd - sql_exporter - use the new views from pgmonitor-extension instead of full queries +v4.12.0 +======= + +Release Summary +--------------- + +Crunchy Data is pleased to announce the availability of pgMonitor 4.12.0. This release primarily brings support for Grafana 10.4. See Changelog for additional information. + +Breaking Changes / Porting Guide +-------------------------------- + +- grafana - Update the dashboards to support Grafana 10.4 so that we're on an officially supported version of Grafana. This does potentially break backward compatibility with Grafana 9.x, so an update to Grafana 10.4 will be required with this version of pgMonitor. + +Bugfixes +-------- + +- grafana - Fix etcd dashboard to use new metric names in etcd 3.5 +- postgres_exporter - Fix query for database table size to remove duplicate word +- postgres_exporter - Fix query for pgBackRest monitoring to handle 3 number versions + v4.11.0 ======= diff --git a/changelogs/changelog.yaml b/changelogs/changelog.yaml index 8d69d1d2..5dbc47be 100644 --- a/changelogs/changelog.yaml +++ b/changelogs/changelog.yaml @@ -1,6 +1,5 @@ ancestor: null releases: - 4.10.0: changes: major_changes: @@ -39,6 +38,27 @@ releases: - 378.yml - release.yml release_date: '2024-01-24' + 4.12.0: + changes: + breaking_changes: + - grafana - Update the dashboards to support Grafana 10.4 so that we're on an + officially supported version of Grafana. This does potentially break backward + compatibility with Grafana 9.x, so an update to Grafana 10.4 will be required + with this version of pgMonitor. + bugfixes: + - grafana - Fix etcd dashboard to use new metric names in etcd 3.5 + - postgres_exporter - Fix query for database table size to remove duplicate + word + - postgres_exporter - Fix query for pgBackRest monitoring to handle 3 number + versions + release_summary: Crunchy Data is pleased to announce the availability of pgMonitor + 4.12.0. This release primarily brings support for Grafana 10.4. See Changelog + for additional information. + fragments: + - 412_release.yml + - 425.yml + - 441.yml + release_date: '2024-11-04' 4.8.0: changes: bugfixes: @@ -167,7 +187,7 @@ releases: - sql_exporter - add support for PG17 minor_changes: - grafana - Add a new variable dropdown to all dashboards for the Datasource. - Allows more flexiblity when importing the dashboard to different environments. + Allows more flexibility when importing the dashboard to different environments. - grafana - Add panel to PG Details dashboard to track autovac workers running vs max - grafana - add a dashboard for HAProxy @@ -208,24 +228,66 @@ releases: - 435.yml - release_summary.yml release_date: '2024-10-04' - 4.12.0: + 5.2.0: changes: - breaking_changes: - - grafana - Update the dashboards to support Grafana 10.4 so that we're on an - officially supported version of Grafana. This does potentially break backward - compatibility with Grafana 9.x, so an update to Grafana 10.4 will be required - with this version of pgMonitor. bugfixes: - - grafana - Fix etcd dashboard to use new metric names in etcd 3.5 - - postgres_exporter - Fix query for database table size to remove duplicate - word - - postgres_exporter - Fix query for pgBackRest monitoring to handle 3 number - versions + - grafana - Fix Cache Hit Ratio panel on PG Details dashboard to always be lines. + Depending on data returned was sometimes being shown as points. + - grafana - Fix pgBackRest recovery window panel showing multiple values after + a PostgreSQL switchover + - postgres_exporter - Disable all collectors included with postgres_exporter + by default in example configuration. The other options to disable default + metrics are not applied to the new collections. + - postgres_exporter - Fix the `ccp_table_size_size_bytes` metric to remove the + duplicate word and just be `ccp_table_size_bytes` + - sql_exporter - Fix the `ccp_table_size_size_bytes` metric to remove the duplicate + word and just be `ccp_table_size_bytes` + - sql_exporter - Fix the names of ccp_pgbouncer_database_db_conn_perc_used, + ccp_pgbouncer_database_paused, ccp_pgbouncer_database_disabled, and ccp_pgbouncer_list_item_count + to be consistent with the old metric names from postgres exporter. These new + names are the ones expected in the Grafana dashboard so this change fixes + that to work properly again. + - sql_exporter - Removed extraneous double quote at the end of the pgbouncer + fdw collector file + major_changes: + - sql_exporter - Updated metric queries for pgmonitor-extension 2.1.0 + minor_changes: + - grafana - Improve Query Statistics dashboard to more accurately report lifetime + vs time range statistics + - grafana - Removed the `Last Backup Size (Total)` panel from the pgBackRest + Grafana dashboard since it was backed by the removed metric. + - postgres_exporter - Removed the metric `ccp_backrest_last_info_repo_total_size_bytes`. + - prometheus - Added new alerts to monitor the new metrics for the minimum required + version of the pgmonitor extension + - sql_exporter - Add additional metrics for monitoring replication slot status. + For PG16+ monitor for conflicts. For PG17+, monitor synced and failover status. + - sql_exporter - Add scrape_error_drop_interval setting to the configuration + example + - sql_exporter - Added new metrics to monitor pgmonitor-extension version. - + `ccp_pgmonitor_extension_global_version` - shows the currently installed version + of the extension on the global database as an integer - `ccp_pgmonitor_extension_per_db_version` + - shows the currently installed version of the extension on each monitored + user database as an integer - `ccp_pgmonitor_extension_global_min_version_installed` + - shows whether the currently installed version of the extension is the minimum + required for this version of pgMonitor on the global database (0 true, 1 false) + - `ccp_pgmonitor_extension_per_db_min_version_installed` - shows whether the + currently installed version of the extension is the minimum required for this + version of pgMonitor on each monitored user database (0 true, 1 false) + - sql_exporter - Removed the metric `ccp_backrest_last_info_repo_total_size_bytes`. + When block incremental backups are enabled, this metric is no longer available + from pgBackRest. release_summary: Crunchy Data is pleased to announce the availability of pgMonitor - 4.12.0. This release primarily brings support for Grafana 10.4. See Changelog - for additional information. + 5.2.0. This release brings support for the latest pgmonitor extension version + 2.1.0 and many bug fixes. fragments: - - 412_release.yml - - 425.yml - - 441.yml - release_date: '2024-11-04' + - 431.yml + - 434.yml + - 437.yml + - 439.yml + - 448.yml + - 449.yml + - 456.yml + - 459.yml + - package_docs.yml + - release.yml + release_date: '2025-03-25' diff --git a/changelogs/fragments/431.yml b/changelogs/fragments/431.yml deleted file mode 100644 index 1437b435..00000000 --- a/changelogs/fragments/431.yml +++ /dev/null @@ -1,3 +0,0 @@ ---- -minor_changes: - - sql_exporter - Add scrape_error_drop_interval setting to the configuration example diff --git a/changelogs/fragments/434.yml b/changelogs/fragments/434.yml deleted file mode 100644 index e17e014d..00000000 --- a/changelogs/fragments/434.yml +++ /dev/null @@ -1,3 +0,0 @@ ---- -bugfixes: - - grafana - Fix Cache Hit Ratio panel on PG Details dashboard to always be lines. Depending on data returned was sometimes being shown as points. diff --git a/changelogs/fragments/437.yml b/changelogs/fragments/437.yml deleted file mode 100644 index 92a823ef..00000000 --- a/changelogs/fragments/437.yml +++ /dev/null @@ -1,3 +0,0 @@ ---- -bugfixes: - - grafana - Fix pgBackRest recovery window panel showing multiple values after a PostgreSQL switchover diff --git a/changelogs/fragments/439.yml b/changelogs/fragments/439.yml deleted file mode 100644 index 5e377782..00000000 --- a/changelogs/fragments/439.yml +++ /dev/null @@ -1,4 +0,0 @@ ---- -bugfix: - - sql_exporter - Fix the `ccp_table_size_size_bytes` metric to remove the duplicate word and just be `ccp_table_size_bytes` - - postgres_exporter - Fix the `ccp_table_size_size_bytes` metric to remove the duplicate word and just be `ccp_table_size_bytes` diff --git a/changelogs/fragments/448.yml b/changelogs/fragments/448.yml deleted file mode 100644 index c719ae8d..00000000 --- a/changelogs/fragments/448.yml +++ /dev/null @@ -1,3 +0,0 @@ ---- -trivial: - - documentation - Update copyright year diff --git a/changelogs/fragments/449.yml b/changelogs/fragments/449.yml deleted file mode 100644 index e557733c..00000000 --- a/changelogs/fragments/449.yml +++ /dev/null @@ -1,3 +0,0 @@ ---- -minor_changes: - - grafana - Improve Query Statistics dashboard to more accurately report lifetime vs time range statistics diff --git a/changelogs/fragments/456.yml b/changelogs/fragments/456.yml deleted file mode 100644 index 8d0d6a0c..00000000 --- a/changelogs/fragments/456.yml +++ /dev/null @@ -1,15 +0,0 @@ ---- -major_changes: - - sql_exporter - Updated metric queries for pgmonitor-extension 2.1.0 -minor_changes: - - sql_exporter - Added new metrics to monitor pgmonitor-extension version. - - `ccp_pgmonitor_extension_global_version` - shows the currently installed version of the extension on the global database as an integer - - `ccp_pgmonitor_extension_per_db_version` - shows the currently installed version of the extension on each monitored user database as an integer - - `ccp_pgmonitor_extension_global_min_version_installed` - shows whether the currently installed version of the extension is the minimum required for this version of pgMonitor on the global database (0 true, 1 false) - - `ccp_pgmonitor_extension_per_db_min_version_installed` - shows whether the currently installed version of the extension is the minimum required for this version of pgMonitor on each monitored user database (0 true, 1 false) - - sql_exporter - Add additional metrics for monitoring replication slot status. For PG16+ monitor for conflicts. For PG17+, monitor synced and failover status. - - prometheus - Added new alerts to monitor the new metrics for the minimum required version of the pgmonitor extension -bugfixes: - - postgres_exporter - Disable all collectors included with postgres_exporter by default in example configuration. The other options to disable default metrics are not applied to the new collections. - - sql_exporter - Removed extraneous double quote at the end of the pgbouncer fdw collector file - - sql_exporter - Fix the names of ccp_pgbouncer_database_db_conn_perc_used, ccp_pgbouncer_database_paused, ccp_pgbouncer_database_disabled, and ccp_pgbouncer_list_item_count to be consistent with the old metric names from postgres exporter. These new names are the ones expected in the Grafana dashboard so this change fixes that to work properly again. diff --git a/changelogs/fragments/459.yml b/changelogs/fragments/459.yml deleted file mode 100644 index 67fcd376..00000000 --- a/changelogs/fragments/459.yml +++ /dev/null @@ -1,4 +0,0 @@ -minor_changes: - - sql_exporter - Removed the metric `ccp_backrest_last_info_repo_total_size_bytes`. When block incremental backups are enabled, this metric is no longer available from pgBackRest. - - postgres_exporter - Removed the metric `ccp_backrest_last_info_repo_total_size_bytes`. - - grafana - Removed the `Last Backup Size (Total)` panel from the pgBackRest Grafana dashboard since it was backed by the removed metric. diff --git a/changelogs/fragments/package_docs.yml b/changelogs/fragments/package_docs.yml deleted file mode 100644 index 4dd28106..00000000 --- a/changelogs/fragments/package_docs.yml +++ /dev/null @@ -1,3 +0,0 @@ ---- -trivial: - - documentation - Make documentation around package vs non-package installs more clear and consistent. Remove postgres_exporter file references from PG Metrics list. diff --git a/hugo/content/_index.md b/hugo/content/_index.md index 23e7fd40..a4461fb5 100644 --- a/hugo/content/_index.md +++ b/hugo/content/_index.md @@ -50,6 +50,9 @@ pgMonitor combines multiple open-source software services to create a robust Pos ## Supported Platforms +### pgMonitor Extension + - As of pgMonitor version 5.2, the pgMonitor extension version requirement is a minimum of 2.1.0 + ### Operating Systems - RHEL 8/9 (Build/Run Testing, Setup Instructions) diff --git a/hugo/content/exporter/_index.md b/hugo/content/exporter/_index.md index 420d5f4c..216266a1 100644 --- a/hugo/content/exporter/_index.md +++ b/hugo/content/exporter/_index.md @@ -35,13 +35,13 @@ After installing via these packages, continue reading at the [Setup](#setup) sec | Package Name | Description | |---------------------------------------|---------------------------------------------------------------------------| | blackbox-exporter | Package for the blackbox_exporter | -| pgmonitor-blackbox-exporter-extras | Crunchy-optimized configurations for blackbox_exporter | | node-exporter | Base package for node_exporter | | pg-bloat-check | Package for pg_bloat_check script | +| pgmonitor-blackbox-exporter-extras | Crunchy-optimized configurations for blackbox_exporter | | pgmonitor-node_exporter-extras | Crunchy-optimized configurations for node_exporter | | pgmonitor-pg##-extension | Crunchy monitoring PostgreSQL extension used by sql_exporter | -| sql-exporter | Base package for sql_exporter | | pgmonitor-sql-exporter-extras | Crunchy-optimized configurations for sql_exporter | +| sql-exporter | Base package for sql_exporter | @@ -94,7 +94,7 @@ The following pgMonitor configuration files should be placed according to the fo sql_exporter takes advantage of the Crunchy Data pgmonitor-extension (https://github.com/CrunchyData/pgmonitor-extension) to provide a much easier configuration and setup. The extension takes care of creating all the necessary objects inside the database. -The minimum required version of pgmonitor-extension is currently 1.0.0. +The minimum required version of pgmonitor-extension is currently 2.1.0. The following pgMonitor configuration files should be placed according to the following mapping: @@ -158,7 +158,7 @@ shared_preload_libraries = 'pg_stat_statements,auto_explain,pgmonitor_bgw' You will need to restart your PostgreSQL instance for the change to take effect. pgMonitor has optional metrics that can be collected via pg_stat_statements. auto_explain does not do anything to your database without further configuration. But even if neither of these extensions are initially used, they are very good to have enabled here by default for when they may be needed in the future. -The pgmonitor-extension uses its own background worker to refresh metric data. It also requires that you set which databases will have metrics monitoring enabled. You do this with the `pgmonitor_bgw.dbname` GUC in your postgresql.conf. At a minimum the "global" database needs to be mentioned, typically `postgres`, but if per-database metrics for other databases are also desired (tables statistics, bloat, etc), they must be listed here in CSV format. This value can be changed at any time with just a reload of PostgreSQL. +The pgmonitor-extension uses its own background worker to refresh certain metric data. It also requires that you set which databases will have metrics monitoring enabled. You do this with the `pgmonitor_bgw.dbname` GUC in your postgresql.conf. At a minimum the "global" database needs to be mentioned, typically `postgres`, but if per-database metrics for other databases are also desired (tables statistics, bloat, etc), they must be listed here in CSV format. This value can be changed at any time with just a reload of PostgreSQL. ``` pgmonitor_bgw.dbname = 'postgres,alphadb,betadb,etc' ``` @@ -179,9 +179,9 @@ CREATE EXTENSION pg_stat_statements; | crunchy_bloat_check_collector.yml | Collection file with pg_bloat_check queries and metrics | | crunchy_global_collector.yml | Collection file with global level queries and metrics | | crunchy_per_db_collector.yml | Collection file with general per-database level queries and metrics | -| crunchy_pgbouncer_collector_121.yml | Collection file with pgBouncer queries and metrics for a minimum version of 1.21 | | crunchy_pg_stat_statements_collector.yml | Collection file with pg_stat_statements queries and metrics | crunchy_pg_stat_statements_reset_collector.yml | Collection file with options to allow resetting of pg_stat_statements metrics | +| crunchy_pgbouncer_collector_121.yml | Collection file with pgBouncer queries and metrics for a minimum version of 1.21 | Run the `setup_db.sql` file on all databases that will be monitored by pgMonitor. At minimum this must be at least the global database so the necessary database objects are created. The `pgmonitor-extension` is expected to be available to be installed in the target database(s) when running this file. Note the `setup_db.sql` file is a convenience file and the steps contained within it can be done manually and customized as needed. Note that a default password is not set for the `ccp_monitoring` database role. @@ -194,7 +194,7 @@ For replica servers, the setup is the same except that the setup_db.sql file doe ##### Enabling/Disabling Extension Metrics -As mentioned above, metrics that sql_exporter actively tries to collect are maintained by the collection files that are added to the configuration. However the pgmonitor-extension also has some settings to enable/disable the refresh of the materialized views that it maintains. Most are enabled by default, but it's good to review the `active` column in both the `metric_views` and `metric_tables` configuration tables. For example, in `metric_tables`, the pgBackRest data is not enabled by default since not everyone will have that backup solution in place. +As mentioned above, metrics that sql_exporter actively tries to collect are maintained by the collection files that are added to the configuration. However the pgmonitor-extension also has some settings to enable/disable the refresh of the materialized views that it maintains. Not all are enabled by default, so it is good to review the `active` column in both the `metric_matviews` and `metric_tables` configuration tables. For example, in `metric_tables`, the pgBackRest data is not enabled by default since not everyone will have that backup solution in place. ``` postgres=# SELECT * FROM pgmonitor_ext.metric_tables ; -[ RECORD 1 ]-----+--------------------------------------- @@ -282,7 +282,9 @@ sudo systemctl start crunchy-sql-exporter@sql_exporter sudo systemctl status crunchy-sql-exporter@sql_exporter ``` -To allow the possible use of multiple sql_exporters running on a single system, and to avoid maintaining many similar service files, a systemd template service file is used. The name of the sysconfig EnvironmentFile to be used by the service is passed as the value after the "@" and before ".service" in the service name. The default exporter's sysconfig file is named "sql_exporter". If you need to run multiple sql_exporters on a single system, simply make a new copy of the sysconfig file and pass that to the service name. +To allow the possible use of multiple sql_exporters running on a single system, and to avoid maintaining many similar service files, a systemd template service file is used. Note that is is NOT necessary to run multiple sql_exporters to monitoring multiple databases in a single instance of PostgreSQL. Nor is it even necessary for multiple exporters when monitoring multiple PostgreSQL instances. A single sql_exporter can connect to as many target PostgreSQL instances as you need, so in general multiple will not be needed. + +The name of the sysconfig EnvironmentFile to be used by the service is passed as the value after the "@" and before ".service" in the service name. The default exporter's sysconfig file is named "sql_exporter". If you need to run multiple sql_exporters on a single system, simply make a new copy of the sysconfig file and pass that to the service name. ```bash sudo systemctl enable crunchy-sql-exporter@sql_exporter_cluster2 @@ -532,7 +534,7 @@ There are many other suggestions, projects, and exporters out there that can pro ## Legacy postgres_exporter Setup {#postgres-exporter} -If you had been using pgMonitor prior to version 5.0.0, postgres_exporter was the method used to collect PostgreSQL metrics. This exporter can still be used with 5.0.0, but there are some additional steps required and it will be deprecated in the near future. It is HIGHLY recommended to switch to using sql_exporter as soon as possible. Custom query support will be dropped from postgres_exporter at some point in the future and that will break pgMonitor since it relies solely on custom queries. No new features of pgMonitor are being developed around postgres_exporter. +If you had been using pgMonitor prior to version 5.0.0, postgres_exporter was the method used to collect PostgreSQL metrics. This exporter can still be used with 5.0.0, but there are some additional steps required. You MUST migrate to sql_exporter ASAP as postgres_exporter has been deprecated and will be removed in the near future. Custom query support will be dropped upstream from postgres_exporter at some point in the future which will break pgMonitor as it relies solely on custom queries. No new features of pgMonitor are being developed around postgres_exporter. Most of the installation steps are the same as above with the below differences for the relevant sections.