In this post, I am sharing an important demonstration on how to improve query performance by adding Extended Statistics in PostgreSQL.

PostgreSQL: Important Statistics Table, Used by the Query Planner

The database statistics are always important in any database system. Because query optimizer is creating Query Execution Plan basis on stored database statistics.

Understand below official note which copied from here.

It is common to see slow queries running bad execution plans because multiple columns used in the query clauses are correlated.

The planner normally assumes that multiple conditions are independent of each other, an assumption that does not hold when column values are correlated. Regular statistics, because of their per-individual-column nature, cannot capture any knowledge about cross-column correlation.

Because the number of possible column combinations is very large, it’s impractical to compute multivariate statistics automatically. Instead, extended statistics objects, more often called just statistics objects.

Let me explain, by my way,

For example, you give two filters in a where condition like zip code and city_name.
Now, zip code is unique for dataset, but one city has multiple zip codes.
Now, this both conditions and columns called correlated columns.

To inform the planner about functional dependencies, ANALYZE can collect measurements of cross-column dependency.
It is advisable to create dependencies statistics only for column groups that are strongly correlated, to avoid unnecessary overhead in both ANALYZE and later query planning.

Check the below demonstration:

Create a sample table:

CREATE TABLE tbl_TestStats
 (
     ID SERIAL
     ,CodeDate TIMESTAMPTZ
     ,CodeName TEXT
 );

Two sample insert for different series:
For example, 2016 year for code ‘abc’ and 2017 year for code ‘xyz’

INSERT INTO tbl_TestStats 
(CodeDate, CodeName)
SELECT x, 'abc' 
FROM generate_series('2016-01-01 00:00:00'::timestamptz, '2016-12-31 00:00:00'::timestamptz,'10 seconds'::interval) a(x);


INSERT INTO tbl_TestStats 
(CodeDate, CodeName)
SELECT x, 'xyz' 
FROM generate_series('2017-01-01 00:00:00'::timestamptz, '2017-12-31 00:00:00'::timestamptz,'10 seconds'::interval) a(x);

Total inserted records:

SELECT COUNT(1) FROM tbl_TestStats
-- '6298562'

Now, Execute ANALYZE:

ANALYZE tbl_TestStats;

Check the execution plan of the below query:

EXPLAIN ANALYZE
SELECT *FROM tbl_TestStats
WHERE CodeDate = '2016-12-03'
	AND CodeName = 'abc'
    
'Gather  (cost=1000.00..80484.88 rows=1 width=16) (actual time=347.355..347.496 rows=1 loops=1)'
'  Workers Planned: 2'
'  Workers Launched: 2'
'  ->  Parallel Seq Scan on tbl_teststats  (cost=0.00..79484.77 rows=1 width=16) (actual time=278.702..340.557 rows=0 loops=3)'
'        Filter: ((codedate = '2016-12-03 00:00:00+05:30'::timestamp with time zone) AND (codename = 'abc'::text))'
'        Rows Removed by Filter: 2099520'
'Planning time: 0.168 ms'
'Execution time: 353.298 ms'

Now, Create an Extended Statistics for correlated columns:

CREATE STATISTICS Ext_Stat_tbl_TestStats (dependencies) ON CodeDate, CodeName 
FROM tbl_TestStats;

Now, Execute ANALYZE:

ANALYZE tbl_TestStats;

Check execution plan for the same query:
Comparing to first result, you can find slightly performance improvement in the second result.

EXPLAIN ANALYZE
SELECT *FROM tbl_TestStats
WHERE CodeDate = '2016-12-03'
	AND CodeName = 'abc'
    
'Gather  (cost=1000.00..80484.86 rows=1 width=16) (actual time=150.732..311.567 rows=1 loops=1)'
'  Workers Planned: 2'
'  Workers Launched: 2'
'  ->  Parallel Seq Scan on tbl_teststats  (cost=0.00..79484.76 rows=1 width=16) (actual time=251.261..304.857 rows=0 loops=3)'
'        Filter: ((codedate = '2016-12-03 00:00:00+05:30'::timestamp with time zone) AND (codename = 'abc'::text))'
'        Rows Removed by Filter: 2099520'
'Planning time: 0.167 ms'
'Execution time: 316.685 ms'

Last month, we were testing huge data loading in PostgreSQL.
We tried different options like COPY command, ETL Tool, CSV Load, and manual INSERT.

While testing this load, got to know about one more option which is the UNLOGGED Table of PostgreSQL.

Important notice: The UNLOGGED Table is not a safe because it is not written to the write-ahead log, so it is not crash safe.

If your database crash or shutdown abnormally, the UNLOGGED table may lose or truncate automatically.

But in any database system, there are few tables which always for standby, backup and snapshot purpose.
Sometimes, we need faster data loading on this kind of tables.
When you create any indexes on the UNLOGGED table, those indexes also keep UNLOGGED.

Check the below sample table, and try it yourself:

CREATE UNLOGGED TABLE tbl_TestUnlogged (ID INTEGER, Name Character Varying)

SQL Server 2012 enhanced the performance of INDEX REBUILD by introducing the SORT_IN_TEMPDB option.
I observed that many SQL developers are not using this feature or maybe they don’t know about it.

The INDEX REBUILD operation is removing the fragmentation of indexes which improves the query performance. It scans the data pages of base table to retrieve the key values and builds an index leaf row for each row.

This process requires internal sorting which it stores in the buffer area.

Once the buffer filled with leaf index entries, it stores the sorted result into a disk as an intermediate sort run. Next, It again starts with the same process for further data pages.

As per the MSDN,

When you create or rebuild an index, by setting the SORT_IN_TEMPDB option to ON you can direct the SQL Server Database Engine to use tempdb to store the intermediate sort results that are used to build the index.

Although this option increases the amount of temporary disk space that is used to create an index, the option could reduce the time that is required to create or rebuild an index when tempdb is on a set of disks different from that of the user database.

If SORT_IN_TEMPDB is set to ON, there must be sufficient free space in tempdb to store the collection of sort runs for the largest index.

When you are rebuilding the large size of Indexes, SORT_IN_TEMPDB helps you lot by reducing the operational execution time.

Sample Script with option SORT_IN_TEMPDB:

ALTER INDEX ALL ON MyTableName
REBUILD WITH 
(FILLFACTOR = 80, SORT_IN_TEMPDB = ON,
 STATISTICS_NORECOMPUTE = ON);
GO

Since SQL Server 2005, we have a concept of Forced Parameterization of SQL Query.

It looks like a very old concept, but the use of Forced Parameterization improves the performance of Query Optimizer which responsible for the best execution plan.

SQL Server provides two types of query parameterization, one is Simple Parameterization and second is Forced Parameterization.

Why it is important and How it works?

When you are executing any query, SQL Server Query Optimizer analyzes your query and prepares the best execution plan for it.

If you are executing the same query again, previously generated execution plan may use for the same query. We can also say that It caches your query execution plan.

It is doing this using a Simple Parameterization which is default in SQL Server.

Let me add one more thing,
Your frequent SQL Query uses the cache query plan, but what If you change the value in WHERE clause.

If you are changing just column filter value in WHERE clause, It uses the cache query plan and If you are modifying your parameter list, It requires to generate a new query plan.

This is about the Simple Parameterization.

You can see in the below image:

I have executed two different queries with passing different values for TrackingEventID.

But You can see TrackingEventID = @1 in query execution plan, means It uses same query plan for both the query which increase your query performance.

Now check below image:

I have used ISNULL() in query and It skipped to put parameter in the query of execution plan.
You can see static date value instead of parameter values.

If you are executing this kind of queries frequently, It prepares and generates an execution plan every time which decreases the query performance.

Now, enable the Force Parameterization and check the same query.

You can check below image where you can see parameter values instead of static date value.

Once you enable Force Parameterization, It forcefully adds the parameter in the query of execution plan.

Which improves the query performance by reducing the cost of preparing and compiling an execution plan again and again.

But we should also take care about this because It might generate SQL Parameter Sniffing Issue so we should keep database statistics up to date.

Script to change the database in Forced Parameterization:

ALTER DATABASE Database_Name
SET PARAMETERIZATION FORCED
GO

Script to change the database in Simple Parameterization:

ALTER DATABASE Database_Name
SET PARAMETERIZATION Simple
GO

Other related articles:

SQL Server: The Importance of Statistics and Why It is important

SQL Server Parameter Sniffing

Please read this previous article, To Understand the Power of BRIN – Block Range Index.

Database Theory: What is BRIN (Block Range Index), How is faster than BTREE Index

PostgreSQL 9.5 introduced the powerful BRIN Index, which is performance much faster than the regular BTREE Index.
The most important two lines of the BRIN are: It stores only minimum and maximum value per block so it does not require more space. For extremely large table It runs faster than any other Indexes.

In this post, I am going to show the example of BRIN index with the full performance report (testing over the 6gb of Table Data).

Below are steps:

First create one sample table:

CREATE TABLE tbl_ItemTransactions 
 (
     TranID SERIAL
     ,TransactionDate TIMESTAMPTZ
     ,TransactionName TEXT
 );

Insert Millions of data to test the performance of BRIN Index:

INSERT INTO tbl_ItemTransactions 
(TransactionDate, TransactionName)
SELECT x, 'dbrnd' 
FROM generate_series('2008-01-01 00:00:00'::timestamptz, '2016-08-01 00:00:00'::timestamptz,'2 seconds'::interval) a(x);

Check the total size of table:

SELECT pg_size_pretty(pg_total_relation_size('tbl_ItemTransactions')) AS TableSize;

/*
    TableSize
------------------
    6741 MB
*/

Now Check the performance without any Index:

EXPLAIN ANALYSE 
SELECT COUNT(1) FROM tbl_ItemTransactions 
WHERE TransactionDate BETWEEN '2012-01-01 00:00:00' and '2014-08-08 08:08:08';
/*
--Result:
			QueryPlan
-------------------------------------------------------------------------
Aggregate  (cost=2997896.81..2997896.82 rows=1 width=0) 
	(actual time=40651.793..40651.793 rows=1 loops=1)
  ->  Seq Scan on tbl_itemtransactions  (cost=0.00..2894105.00 rows=41516724 width=0) 
	(actual time=0.009..38726.686 rows=41054645 loops=1)
        Filter: ((transactiondate >= '2012-01-01 00:00:00+05:30'::timestamp with time zone) 
		AND (transactiondate <= '2014-08-08 08:08:08+05:30'::timestamp with time zone))
        Rows Removed by Filter: 94377356
Planning time: 0.860 ms
Execution time: 80651.837 ms
*/

Create BRIN index on TransactionDate Column:

CREATE INDEX idx_tbl_ItemTransactions_TransactionDate
ON tbl_ItemTransactions
USING BRIN (TransactionDate);

Now Check the performance of the same query which has BRIN index:

EXPLAIN ANALYSE 
SELECT COUNT(1) FROM tbl_ItemTransactions 
WHERE TransactionDate BETWEEN '2012-01-01 00:00:00' and '2014-08-08 08:08:08';
/*
			QueryPlan
---------------------------------------------------------------------------
Aggregate  (cost=2014834.09..2014834.10 rows=1 width=0) 
	(actual time=7108.998..7108.998 rows=1 loops=1)
  ->  Bitmap Heap Scan on tbl_itemtransactions  (cost=425666.42..1911042.28 rows=41516724 width=0) 
	(actual time=16.995..5415.086 rows=41054645 loops=1)
        Recheck Cond: ((transactiondate >= '2012-01-01 00:00:00+05:30'::timestamp with time zone) 
		AND (transactiondate <= '2014-08-08 08:08:08+05:30'::timestamp with time zone))
        Rows Removed by Index Recheck: 21579
        Heap Blocks: lossy=261632
        ->  Bitmap Index Scan on idx_tbl_itemtransactions_transactiondate  
		(cost=0.00..415287.24 rows=41516724 width=0) (actual time=15.547..15.547 rows=2616320 loops=1)
              Index Cond: ((transactiondate >= '2012-01-01 00:00:00+05:30'::timestamp with time zone) 
              AND (transactiondate <= '2014-08-08 08:08:08+05:30'::timestamp with time zone))
Planning time: 0.059 ms
Execution time: 7109.060 ms
*/

Now, you can see the difference between the result of the above two queries.
With the BRIN index same query took only 7 seconds and without BRIN it took around 80 seconds.

Create Partial BRIN index on TransactionDate Column:
You can also create Partial BRIN index for your individual range of data. The Partial BRIN index is also faster than normal BRIN index, but we should apply proper filter based on created Partial BRIN index.

CREATE INDEX idx_tbl_ItemTransactions_TransactionDate_2012
ON tbl_ItemTransactions
USING BRIN (TransactionDate)
WHERE TransactionDate BETWEEN '2012-01-01' AND '2012-12-31';

CREATE INDEX idx_tbl_ItemTransactions_TransactionDate_2013
ON tbl_ItemTransactions
USING BRIN (TransactionDate)
WHERE TransactionDate BETWEEN '2013-01-01' AND '2013-12-31';