In this post, I am sharing an important demonstration on how to improve query performance by adding Extended Statistics in PostgreSQL.

PostgreSQL: Important Statistics Table, Used by the Query Planner

The database statistics are always important in any database system. Because query optimizer is creating Query Execution Plan basis on stored database statistics.

Understand below official note which copied from here.

It is common to see slow queries running bad execution plans because multiple columns used in the query clauses are correlated.

The planner normally assumes that multiple conditions are independent of each other, an assumption that does not hold when column values are correlated. Regular statistics, because of their per-individual-column nature, cannot capture any knowledge about cross-column correlation.

Because the number of possible column combinations is very large, it’s impractical to compute multivariate statistics automatically. Instead, extended statistics objects, more often called just statistics objects.

Let me explain, by my way,

For example, you give two filters in a where condition like zip code and city_name.
Now, zip code is unique for dataset, but one city has multiple zip codes.
Now, this both conditions and columns called correlated columns.

To inform the planner about functional dependencies, ANALYZE can collect measurements of cross-column dependency.
It is advisable to create dependencies statistics only for column groups that are strongly correlated, to avoid unnecessary overhead in both ANALYZE and later query planning.

Check the below demonstration:

Create a sample table:

CREATE TABLE tbl_TestStats
 (
     ID SERIAL
     ,CodeDate TIMESTAMPTZ
     ,CodeName TEXT
 );

Two sample insert for different series:
For example, 2016 year for code ‘abc’ and 2017 year for code ‘xyz’

INSERT INTO tbl_TestStats 
(CodeDate, CodeName)
SELECT x, 'abc' 
FROM generate_series('2016-01-01 00:00:00'::timestamptz, '2016-12-31 00:00:00'::timestamptz,'10 seconds'::interval) a(x);


INSERT INTO tbl_TestStats 
(CodeDate, CodeName)
SELECT x, 'xyz' 
FROM generate_series('2017-01-01 00:00:00'::timestamptz, '2017-12-31 00:00:00'::timestamptz,'10 seconds'::interval) a(x);

Total inserted records:

SELECT COUNT(1) FROM tbl_TestStats
-- '6298562'

Now, Execute ANALYZE:

ANALYZE tbl_TestStats;

Check the execution plan of the below query:

EXPLAIN ANALYZE
SELECT *FROM tbl_TestStats
WHERE CodeDate = '2016-12-03'
	AND CodeName = 'abc'
    
'Gather  (cost=1000.00..80484.88 rows=1 width=16) (actual time=347.355..347.496 rows=1 loops=1)'
'  Workers Planned: 2'
'  Workers Launched: 2'
'  ->  Parallel Seq Scan on tbl_teststats  (cost=0.00..79484.77 rows=1 width=16) (actual time=278.702..340.557 rows=0 loops=3)'
'        Filter: ((codedate = '2016-12-03 00:00:00+05:30'::timestamp with time zone) AND (codename = 'abc'::text))'
'        Rows Removed by Filter: 2099520'
'Planning time: 0.168 ms'
'Execution time: 353.298 ms'

Now, Create an Extended Statistics for correlated columns:

CREATE STATISTICS Ext_Stat_tbl_TestStats (dependencies) ON CodeDate, CodeName 
FROM tbl_TestStats;

Now, Execute ANALYZE:

ANALYZE tbl_TestStats;

Check execution plan for the same query:
Comparing to first result, you can find slightly performance improvement in the second result.

EXPLAIN ANALYZE
SELECT *FROM tbl_TestStats
WHERE CodeDate = '2016-12-03'
	AND CodeName = 'abc'
    
'Gather  (cost=1000.00..80484.86 rows=1 width=16) (actual time=150.732..311.567 rows=1 loops=1)'
'  Workers Planned: 2'
'  Workers Launched: 2'
'  ->  Parallel Seq Scan on tbl_teststats  (cost=0.00..79484.76 rows=1 width=16) (actual time=251.261..304.857 rows=0 loops=3)'
'        Filter: ((codedate = '2016-12-03 00:00:00+05:30'::timestamp with time zone) AND (codename = 'abc'::text))'
'        Rows Removed by Filter: 2099520'
'Planning time: 0.167 ms'
'Execution time: 316.685 ms'

In this post, I am sharing an important gp_toolkit view to finding a list of a table on which ANALYZE is required to be run in Greenplum Database Server.

An ANALYZE command is used to update the object or database statistics information. The query planner always uses the statistics for building the execution plans.

A gp_toolkit is an administrative schema of Greenplum Database server. It provides the list of various views to monitor the Greenplum server.

A gp_stats_missing is one of the views of a gp_toolkit schema which gives us a list of tables which may require an ANALYZE to be run.

Below is a query:

SELECT * FROM gp_toolkit.gp_stats_missing;

After executing the above query, you can get the list of five columns. A column “smisize” is an important column to check the status of table statistics.

If it contains value ‘f’, a table does not have row count and row sizing statistics recorded in the system catalog, which indicates that the table needs to be analyzed.

The meaning of other columns like “simcols”: number of columns in the table, “simrecs”: number of rows in the table.

SQL Server Database Statistics is one of the important topics in the Query optimization. I already wrote a couple of articles on SQL Server Database Statistics.

SQL Server: Enable Auto Create and Auto Update Statistics

In the Interview of SQL DBA, If you say like you are also good in Query Optimization, there is 99% chance that interviewer asks questions on SQL Database Statistics.

Why is Database Statistics important?

Simple answer – Query optimizer creates a query execution plan basis on Object Statistics information like record count, some real pages, page location. If statistics are not up to date, it generates the wrong query execution plan which will slow down the query performance.

How can we update Statistics automatically?

Statistics updates can be either synchronous (the default) or asynchronous.

AUTO_UPDATE_STATISTICS – For synchronous update:
This is a synchronous option, and if it is running, a query has to wait for updated statistics. Updating statistics synchronously is the default behaviour, and once the statistics are updated, then the query will be executed.

ALTER DATABASE database_name SET AUTO_UPDATE_STATISTICS ON
GO

AUTO_UPDATE_STATISTICS_ASYNC – For asynchronous update:
This is an asynchronous option, and if it is running, a query hasn’t required waiting for updated statistics because the query will be executed against the existing statistics.

In the background, at the same time database statistics update automatically using AUTO_UPDATE_STATISTICS_ASYNC option.
Once the background statistics update process completes, the new query can use the newly updated database statistics.

ALTER DATABASE database_name SET AUTO_UPDATE_STATISTICS ON
GO
ALTER DATABASE database_name SET AUTO_UPDATE_STATISTICS_ASYNC ON
GO

Please note: Before enabling the AUTO_UPDATE_STATISTICS_ASYNC option, Must have AUTO_UPDATE_STATISTICS enabled first