In this post, I am sharing a script for checking the status of AutoVacuum for all the PostgreSQL Tables.

If you don’t know about the MVCC architecture, you must visit the below article. Because of MVCC, we have to work on dead tuples which are generated by transactions in PostgreSQL.

What is Multi Version Concurrency Control (MVCC)

You can configure auto-vacuum on tables, but periodically you should check the status of it. Use below script for checking the status of Autovacuum.

PostgreSQL: Short note on VACUUM, VACUUM FULL and ANALYZE

PostgreSQL: Force Autovacuum for running Aggressive

SELECT 
    schemaname
    ,relname
    ,n_live_tup
    ,n_dead_tup
    ,last_autovacuum
FROM pg_stat_all_tables
ORDER BY n_dead_tup
    /(n_live_tup
      * current_setting('autovacuum_vacuum_scale_factor')::float8
      + current_setting('autovacuum_vacuum_threshold')::float8)
     DESC

In this post, I am sharing an important gp_toolkit view to finding a list of a table on which VACUUM is required to be run in Greenplum Database Server.

Greenplum is based on PostgreSQL 8.2 and MVCC architecture. It generates dead tuples at every UPDATE and DELETE which is a fragmentation in the form of dead tuples which we can remove by executing a VACUUM/VACUUM FULL command.

What is Multi Version Concurrency Control (MVCC)

A gp_toolkit is an administrative schema of Greenplum Database server. It provides the list of various views to monitor the Greenplum server.

A gp_bloat_diag is one of the views of a gp_toolkit schema which gives us a list of tables which may require a VACUUM to be run.

It returns the actual number of pages and the expected number of pages. If a count of an actual number of pages is greater than expected number of pages, that table requires VACUUM.

If you want to re-use the wasted space of table, execute VACUUM FULL on it which requires ACCESS EXCLUSIVE LOCK on a table because it re-writes all the data pages from the scratch.

If you execute only VACUUM, you cannot reclaims the free space of table.
You must execute ANALYZE after the execution of VACUUM/VACUUM FULL for updating database statistics.

Below is a query:

SELECT * FROM gp_toolkit.gp_bloat_diag;

After executing the above query, you can get the list of six columns. The important columns are “bdirelpages” and “bdiexppages” means real pages and expected pages.

If you find your table in the result set, your table requires VACUUM/VACUUM FULL.

In this post, I am sharing a system view which we can use to check the progress of running vacuum process of PostgreSQL.

PostgreSQL based on MVCC, and in this architecture VACUUM is a routine task of DBA for removing dead tuples.

What is Multi Version Concurrency Control (MVCC)

Now the question is, How to monitor the progress of VACUUM?

Simple, we can use pg_stat_progress_vacuum view for this purpose. In this view, we can check phase column which indicates the different stages of running the vacuum.

select * from pg_stat_progress_vacuum

Type of phases:

initializing: VACUUM is preparing to begin scanning the heap
scanning heap: VACUUM is currently scanning the heap
vacuuming indexes: VACUUM is currently vacuuming the indexes
vacuuming heap: VACUUM is currently vacuuming the heap
cleaning up indexes: VACUUM is currently cleaning up indexes
truncating heap: VACUUM is currently truncating the heap to return empty pages at the end of the relation to the operating system
performing final cleanup: VACUUM is performing final cleanup. During this phase, VACUUM will vacuum the free space map, update statistics in pg_class, and report statistics to the statistics collector

PostgreSQL: What is a Free Space Map (FSM)?

Reference taken from the official page:

In this post, I am sharing a script to find a list of tables with the information like the last action_date of VACUUM and ANALYZE in Greenplum.

Before executing VACUUM or ANALYZE on tables, we should check the last execution time of it. If it recently executed, we should skip those tables for VACUUM or ANALYZE.
In below script, I am finding this information from pg_stat_last_operation.

select
       pn.nspname
       ,pc.relname
       ,pslo.staactionname
       ,pslo.stasubtype
       ,pslo.statime as action_date
from pg_stat_last_operation pslo
right outer join pg_class pc
                on pc.oid = pslo.objid and pslo.staactionname in ('VACUUM','ANALYZE')
join pg_namespace pn
                on pn.oid = pc.relnamespace
where pc.relkind IN ('r','s')
AND pc.relstorage IN ('h', 'a', 'c');

In this post, sharing a small note on one of the internal architecture things of PostgreSQL.

Each Table and Index has an FSM which used to keep track of available space in the relation.

It stores all free space related information in an alongside primary relation and that relation starts with the file node number plus the suffix _fsm.

If you do not know about the MVCC (Multiversion Concurrency Control), please visit below an article.

What is Multi Version Concurrency Control (MVCC)

Because of MVCC architecture, PostgreSQL generates dead tuples on each update and delete action.

Why VACUUM requires in MVCC because VACUUM removes dead row versions in tables and marks the space available for future reuse.

The VACUUM process also updates the Free Space Map and using VACUUM FULL we can recover those free spaces.

The FSM pages are organized as a tree.
The bottom level FSM pages store the free space available on each page, using one byte to represent each such page.
The upper levels aggregate information from the lower levels.

We can also examine free space pages using pg_freespace module.

CREATE EXTENSION pg_freespace;
SELECT * FROM pg_freespace('table_name');

This is one of the very important post for all PostgreSQL Database Professionals.

In this post, I am sharing a small, but very powerful script to know about the Live Tuples (Rows) and Dead Tuples (Rows) of the PostgreSQL Object like: Tables and Indexes.

If you don’t know about the MVCC (Multi Version Concurrency Control), Please visit this article.

What is Multi Version Concurrency Control (MVCC)

PostgreSQL is based on MVCC Architecture.

In MVCC Architecture, When you update or delete any row, Internally It creates the new row and mark old row as unused.

This kind of data, we call as Dead Tuples or Dead Rows. (We can also say like, This is an internal fragmentation).

Periodically, We should find dead rows of the object and we should remove it using VACUUM techniques of PostgreSQL.
By this way, we can increase the overall performance of PostgreSQL Database Server.

PostgreSQL: Short note on VACUUM, VACUUM FULL and ANALYZE

Find out Live Tuples or Dead Tuples using two different scripts.

Using system function and pg_class:

SELECT 
	relname AS ObjectName
	,pg_stat_get_live_tuples(c.oid) AS LiveTuples
	,pg_stat_get_dead_tuples(c.oid) AS DeadTuples
FROM pg_class c;

Using pg_stat_user_tables:

SELECT 
	relname AS TableName
	,n_live_tup AS LiveTuples
	,n_dead_tup AS DeadTuples
FROM pg_stat_user_tables;

In any RDBMS system, Different type of Statistics is very important for Query Optimizer and Planner to choose and generate the best Query Execution Plan.

PostgreSQL Database Server also managing different types of Statistic which are used by the Query Optimizer to generate the best Query Execution Plan.

Statistical data should be updated, Why?
For Example,

We have one table with 100000 rows, but table statistic data are not updated and showing only 1000 tuples information.

Query optimizer always looks into Statistic tables, It never looks into direct table files.

So in this situation, It generates the wrong Query Execution Plan because It has only 1000 rows information, but actually there are 100000 rows.

PostgreSQL Database Server is mainly using two Statistical Table.

SELECT *FROM pg_class;
SELECT *FROM pg_stats;

You can also query these tables and can find information about Tables, Indexes, Pages and others.

When you look into table pg_class, you can find “reltuples” column. You can compare this column value with your actual table record count.

If you find any mismatch, for that table statistic information is not updated.

Execute Analyze command:

When you find that Statistical information is not updated, you can execute the ANALYZE command on the Database or Table.
After executing the ANALYZE command, It updates all require statistics.

PostgreSQL: Short note on VACUUM, VACUUM FULL and ANALYZE

How to perform VACUUM FULL without disk space?

The above question is common for all PostgreSQL DBA. This is really a very important topic that I am going to discuss in this post.

We require VACUUM and VACUUM FULL in PostgreSQL because of MVCC Architecture.
You can visit this article for MVCC:

Postgres VACUUM FULL reclaims all free space released by VACUUM by removing dead rows physically.

This method also requires extra disk space, since it writes a new copy of the table and doesn’t release the old copy until the operation is complete.

In one of our Postgres Database Server, we require VACUUM FULL on one table, and this table size is 13GB, and we don’t have enough free space on hard-disk.

I worked around this problem and created one of the best solutions to resolve this problem.

To solve this problem, we need additional storage like any other network drive or portable hard disk.
Please do not forget that VACUUM FULL requires an exclusive lock on the table so during this operation your table cannot be accessible.

Now, add new hard disk and make it as table-space.

Create a new tablespace:

CREATE TABLESPACE temptablespace LOCATION '/path/../';

Check your table current tablespace:

SELECT tablespace FROM pg_tables WHERE tablename = 'mybigtable';

If it is NULL, it has a default tablespace.

Move table to new tablespace:

ALTER TABLE mybigtable SET TABLESPACE temptablespace;

Perform VACUUM FULL:

VACUUM FULL mybigtable;

Move table to old tablespace:(moving to pg_default)

ALTER TABLE mybigtable SET TABLESPACE pg_default;

Drop that temp table space:

DROP TABLESPACE temptablespace;

VACUUM, VACUUM FULL and ANALYZE: These are the maintenance related commands of PostgreSQL which requires frequent execution because PostgreSQL based on MVCC architecture where every UPDATE and DELETE generates dead rows or dead tuples as an internal fragmentation.

If you don’t about the MVCC, you must visit the below article.
What is MVCC?

VACUUM:

VACUUM reclaims storage occupied by dead tuples.
In the MCVV architecture tuples that deleted which is not physically remove. It presents in a disk until a VACUUM is done.

If you are executing frequent UPDATE/DELETE on your table, you must find the fragmentation and must execute VACUUM on it.
VACUUM does not require an exclusive lock on the table.
Syntax:

VACUUM Table_Name;

VACUUM FULL:

Whatever space reclaims by VACUUM that space might require to use for next storage.
VACUUM FULL rewrites the entire table with data and releases all fragmented space of an old table.

We should avoid the VACUUM FULL because it shrinks the whole table and writes everything into the new disk block which requires more resources and disk space to complete this operation.

If you find a huge number of dead rows, you should execute VACUUM FULL on that table.
VACUUM FULL requires an exclusive lock on the table and also require a free disk space as same as your table size.
Syntax:

VACUUM FULL Table_Name;

ANALYZE:

After execution of VACUUM or VACUUM FULL, you must execute this command to update the statistics.
ANALYZE updates all require statistics and stores the results in the pg_statistic system catalog.

The Query Planner uses this database statistics information to prepare an efficient query execution plan.
You can also execute VACUUM ANALYZE, which executes the first VACUUM and then executes ANALYZE.
Syntax:

ANALYZE Table_Name;
VACUUM ANALYZE Table_Name;