If you don’t know about that data pages, you must visit below article.
What is MVCC ?

Because of MVCC architecture, PostgreSQL generates dead rows at every UPDATE and DELETE.

You must remove the fragmentation by executing VACUUM ANALYZE command which removes the dead rows and update the related database statistics.

You can also use VACUUM VERBOSE to troubleshoot total number of data pages, total number of removable dead row version while executing a VACUUM ANALYZE.
(Check your query message window, where you can find detailed information generated by VACUUM VERBOSE.)

This is really very helpful information for PostgreSQL DBA to make sure about the fragmentation.

I have prepared a detailed demonstration of this.

Create a table with sample data:

CREATE TABLE tbl_ItemTransactions 
 (
     TranID SERIAL
     ,TransactionDate TIMESTAMPTZ
     ,TransactionName TEXT
 ) DISTRIBUTED BY (TranID);

INSERT INTO tbl_ItemTransactions 
(TransactionDate, TransactionName)
SELECT x, 'ggkproduct' 
FROM generate_series('2015-01-01 00:00:00'::timestamptz, '2016-08-01 00:00:00'::timestamptz,'5 seconds'::interval) a(x);

Check the size of table:

SELECT pg_size_pretty(pg_total_relation_size('tbl_ItemTransactions')) AS TableSize;

The size is: "496 MB"

Execute VACUUM VERBOSE to get data pages and dead row versions related information:

VACUUM VERBOSE ANALYZE tbl_ItemTransactions;

"tbl_itemtransactions": found 0 removable, 2496960 nonremovable row versions in 3970 pages  (seg0 sdw1:40000 pid=17647)

Execute one DELETE to generate few dead tuples:

DELETE FROM tbl_ItemTransactions 
WHERE TransactionDate BETWEEN '2016-04-01' AND '2016-08-01';

Now, examine the result of VACUUM VERBOSE where you can find 527040 removable rows:
You can find total 3970 pages and 527040 removable rows.

VACUUM VERBOSE ANALYZE tbl_ItemTransactions;

INFO:  "tbl_itemtransactions": found 527040 removable, 1969920 nonremovable row versions in 3970 pages  (seg1 sdw1:40001 pid=17649)
839 pages contain useful free space.

Now, execute one more time and you can find 3132 pages and 0 removable rows:

VACUUM VERBOSE ANALYZE tbl_ItemTransactions;

"tbl_itemtransactions": found 0 removable, 1969920 nonremovable row versions in 3132 pages  (seg0 sdw1:40000 pid=17647)
1 pages contain useful free space.

Check the size of table:

SELECT pg_size_pretty(pg_total_relation_size('tbl_ItemTransactions')) AS TableSize;

The size is: "392 MB"

Before reading this article, I would suggest please read about MVCC because of PostgreSQL based on MVCC architecture.

What is Multi Version Concurrency Control (MVCC)

The day before yesterday, one of our developers asked a question like I want updated data on a table while a function is running in PostgreSQL.

If you are from Microsoft SQL Server background, of course, you will need this because you can get an updated data on a table while a function is running. Why? Because SQL Server is auto-committed by default.

When you create a function in PostgreSQL, you must put all your code between BEGIN and END. In PostgreSQL, it treats as transaction block, and you can get only updated table data once function execution completed or you can get the previous state of your table if execution of a function failed.

Why? because of MVCC architecture you cannot get partially updated table otherwise other users can update those value which breaks the rule of MVCC.

While function is running, a user can still access the old records of a table, and once function execution got completed, a user can access the newly updated records of a table.

CREATE FUNCTION fn_test()
  RETURNS as
$$
declare
	-- declare variables...
begin
	-- do operations...
end
$$ language plpgsql;

Then, they asked one more question like If the function code is default between BEGIN and END transaction block when we should define explicitly BEGIN TRANSACTION and COMMIT TRANSACTION in a PostgreSQL function?

My answer was, you can also set BEGIN TRANSACTION and COMMIT TRANSACTION explicitly when you want ROLLBACK or SAVEPOINT action.
But if you do not define this, by default everything is in transaction block and you will get all the output once your function completes its execution.

REATE FUNCTION fn_test()
  RETURNS as
$$
declare
	-- declare variables...
begin	
	-- do operations...
	begin transaction
		-- do transactions...
	commit transaction
end
$$ language plpgsql;

In this post, I am sharing a system view which we can use to check the progress of running vacuum process of PostgreSQL.

PostgreSQL based on MVCC, and in this architecture VACUUM is a routine task of DBA for removing dead tuples.

What is Multi Version Concurrency Control (MVCC)

Now the question is, How to monitor the progress of VACUUM?

Simple, we can use pg_stat_progress_vacuum view for this purpose. In this view, we can check phase column which indicates the different stages of running the vacuum.

select * from pg_stat_progress_vacuum

Type of phases:

initializing: VACUUM is preparing to begin scanning the heap
scanning heap: VACUUM is currently scanning the heap
vacuuming indexes: VACUUM is currently vacuuming the indexes
vacuuming heap: VACUUM is currently vacuuming the heap
cleaning up indexes: VACUUM is currently cleaning up indexes
truncating heap: VACUUM is currently truncating the heap to return empty pages at the end of the relation to the operating system
performing final cleanup: VACUUM is performing final cleanup. During this phase, VACUUM will vacuum the free space map, update statistics in pg_class, and report statistics to the statistics collector

PostgreSQL: What is a Free Space Map (FSM)?

Reference taken from the official page:

If you don’t know about the MVCC, please visit the below article.
Because without knowing the MVCC, you should not start the learning of PostgreSQL.

What is Multi Version Concurrency Control (MVCC)

Many times, I shared information on MVCC like ONLY INSERT concept where every update is a new version of data.

In today’s post, I am sharing a proof of MVCC.
Not in detail, but with the small and easy demonstration.

The XMIN is a pseudo column of PostgreSQL which used to track concurrent row access.
Using XMIN, you can get an exact number which internally uses by MVCC operations.

Please check the below demonstration:

Create a table with a sample record:

CREATE TABLE tbl_MVCCDemo (ID INT, CODE CHARACTER VARYING);

INSERT INTO tbl_MVCCDemo VALUES (1,'XVK98');

Check the value of XMIN pseudo column:

SELECT XMIN,ID,CODE
FROM tbl_MVCCDemo;

 xmin  | id  |  code
----------------------
  622  |  1  | XVK98

Now, update a record using XMIN:

UPDATE tbl_MVCCDemo SET code = 'T2738'
WHERE XMIN = 622;

Check the value of XMIN pseudo column:
We updated the value base on XMIN, then also the value of XMIN changed from 622 TO 623.
So this is proof that MVCC architecture inserts the new row for each update.

SELECT XMIN,ID,CODE
FROM tbl_MVCCDemo;

 xmin  | id  |  code
----------------------
  623  |  1  | XVK98

What is HOT UPDATE (Heap Only Tuple) and How It helps to increase the performance of UPDATE Query?
In this post, I am going to explain the answer of this question.

This is one of the internal of PostgreSQL, but I am sure that most of you guys don’t know about PostgreSQL HOT.

PostgreSQL requires HOT because of MVCC architecture.
Please read below articles, before start to knowing about the HOT.

What is MVCC?

What is Fillfactor in PostgreSQL?

What is Vacuum in PostgreSQL?

In the MVCC architecture, When we update any record, internally MVCC architecture inserts a new record with new values and marks old record as deleted or discarded.
When we vacuum, we can clean that deleted records and utilize that space.

What is HOT (Hot Only Tuple)?

Now imagine that, we have total 500 data pages and we are updating record of page number 3, but unfortunately new record insert in page number 500.

It looks like very big internal fragmentation because of MVCC architecture.
But actually this is not happening.

PostgreSQL has a concept of HOT, With a HOT dead tuple space can be automatically reclaimed at the time of INSERT or UPDATE if no changes are made to indexed columns. This allows for more consistent performance.

HOT means, creating a new update tuple if possible on the same page as the old tuple, and maintaining a chain of updated tuples linking a new version to the old tuple.

HOT updates can only reuse dead tuples from previous transactions directly, not from the same or concurrent ones.
If you have big updates, changing large portions of the table at once, you might want to split them up in a couple of chunks.

HOT updates only work when the changed columns are not involved in indexes in any way because the indexes pointing the the old tuples need to point to the new version of it as of transaction id.

You might be blocking HOT updates with indexes on updated columns. If those are expendable, you might get better overall performance without them.

Let’s understand PostgreSQL HOT practically.

Create a table with Sample data:

CREATE TABLE tbl_Employees
(
	EmpID INT
	,EmpName CHARACTER VARYING 	
	,EmpDepartment CHARACTER VARYING
	,EmpSalary INT
);
 
INSERT INTO tbl_Employees
VALUES 
(1,'Anvesh','Database',90000)
,(2,'Jenny','JAVA',65000)
,(3,'Martin','PHP',85000)
,(4,'Roy','PHP',94000)
,(5,'Eric','PHP',70000)
,(6,'Rajesh','Animation',50000)
,(7,'Mahi','Database',40000)
,(8,'Sofia','JAVA',60000);

Delete one record:

DELETE FROM tbl_Employees WHERE EmpID = 7;

Insert one New Record:

INSERT INTO tbl_Employees
VALUES 
(7,'Junik','Database',10000);

Please check If any HOT Update or not on table:

SELECT pg_stat_get_tuples_hot_updated('tbl_employees'::regclass::oid);
/*
Result:
pg_stat_get_tuples_hot_updated
------------------------------
0
*/

Please update the statestic of that table:

ANALYZE tbl_Employees;

Now execute one more update command:

UPDATE tbl_Employees SET EmpName = 'Robort' WHERE EmpID=7;

Please check If any HOT Update or not on table:

SELECT pg_stat_get_tuples_hot_updated('tbl_employees'::regclass::oid);
/*
Result:
pg_stat_get_tuples_hot_updated
------------------------------
1
*/

Again, do more update:

UPDATE tbl_Employees SET EmpName = 'Manish' WHERE EmpID=8;
UPDATE tbl_Employees SET EmpName = 'Cerlin' WHERE EmpID=4;

Please check If any change in number of HOT Update of a table:

SELECT pg_stat_get_tuples_hot_updated('tbl_employees'::regclass::oid);
/*
Result:
pg_stat_get_tuples_hot_updated
------------------------------
3
*/

Now Apply, Index on EmpName Column:

CREATE INDEX idx_tbl_employees_empname 
ON tbl_employees USING BTREE (EmpName);

After Index, execute few more update:

UPDATE tbl_Employees SET EmpName = 'Kulin' WHERE EmpID=3;
UPDATE tbl_Employees SET EmpName = 'Priya' WHERE EmpID=5;

After Index, Please check If any change in number of HOT Update of a table:
The result is same because If column has an Index, HOT Update does not work.

SELECT pg_stat_get_tuples_hot_updated('tbl_employees'::regclass::oid);
/*
Result:
pg_stat_get_tuples_hot_updated
------------------------------
3
*/

You can also check number of HOT update of current transaction:

SELECT pg_stat_get_xact_tuples_hot_updated('tbl_employees'::regclass::oid);

As a database professional, I work with both databases on a regular basis, and I typically use all the features of both databases for design and implementation.
I have got many replies to write about the core differences between PostgreSQL and MySQL.

As innovation on these databases has progressed, each development community has actively made changes to address their respective sets of perceived disadvantages.
The result that it has gotten more difficult to objectively determine which database is likely to be better suited for a given application.

MySQL is controlled by Oracle, whereas Postgres is available under an open-source license from the PostgreSQL Global Development Group.

PostgreSQL is closer to the ANSI SQL standard, MySQL is closer to the ODBC standard.

One of the most notable differences between MySQL and PostgreSQL is the fact that you can’t do nested subqueries of subselects in MySQL. MySQL doesn’t support CTE, window functions, full outer joins and arrays.

Compare to PostgreSQL, MySQL has its own large user community and various type of materials.

MySQL uses nonstandard ‘#’ to begin a comment line and PostgreSQL use ‘–‘ to begin a comment line.

MySQL provides different type of table storage engines like: InnoDB, MyIsam, Memory. PostgreSQL does not provide different type of table stroage engines.

PostgreSQL is case-sensitive for string comparisons. The field “DbRnd” is not the same as the field “dbrnd”. This is a big change for many users from MySQL and other small database systems.

PostgreSQL has some nice features like: generate_series, custom aggregate functions, arrays etc. other hand MySQL has also some nice feature like: session variables in queries, FORCE INDEX, etc.

In PostgreSQL, when you are inside a transaction almost any operation can be undone. There are some irreversible operations (like creating or destroying a database or tablespace), but normal table modifications can be backed out by issuing a ROLLBACK via its Write-Ahead Log design.

MySQL doesn’t support any sort of rollback when using MyISAM. With InnoDB, the server has an implicit commit that occurs even if the normal auto-commit behavior is turned off. This means that any single table alteration or similar change is immediately committed.

Postgres obliterates MySQL in all manner of query-level goodness, and we can use Postgres for data mining and batch analytics databases, where we run lots of very large, hand-coded queries that can take advantage of Postgres’ advantages in join methods (MySQL only supports Nested Loop and Nested Loop over Index, while Postgres supports numerous methods), generally richer querying capabilities, and other good stuff like partial indexes can be used extensively.

PostgreSQL uses a robust locking model called MVCC that limits situations where individual clients interfere with each other. A short summary of the main benefit of MVCC would be “readers are never blocked by writers”.

MySQL’s InnoDB implements MVCC using a rollback segment and InnoDB databases supports all four SQL standard transaction isolation levels.

PostgreSQL uses cost-based query optimization methods in order to get good performance for many different types of joins. Query costs are assessed based on planner statistics collected when tables are analyzed combined with adjustable planner costs and advanced features such as the Genetic Query Optimizer.

MySQL doesn’t have this level of sophistication in its planner, and the tunables for Controlling Query Optimizer Performance are crude. Developers instead do things like explicitly provide index hints to make sure joins are executed correctly.

PostgreSQL has different replication options than MySQL. MySQL has no point-in-time recovery.

One operation that PostgreSQL is known to be slow performing is doing a full count of rows in a table, typically using this SQL:

SELECT COUNT(*) FROM your_table_name;

The reason why this is slow is related to the MVCC implementation in PostgreSQL. The fact that multiple transactions can see different states of the data means that there can be no straightforward way for “COUNT(*)” to summarize data across the whole table; PostgreSQL must walk through all rows, in some sense.

In MySQL, MyISAM tables cache the row count information, making this type of count operation almost instant. But if you’re using InnoDB instead, this is no longer the case and on InnoDB can’t assume that a full row count will be fast.

PostgreSQL stores data into logical equivalent pages.

The size of each data page is 8KB.
The structure used to store the table is a heap file which also stores header information like: checksum, start of free space, end of free space.

It contains a pair of offset & length, which actually points to the actual items.

Postgres data called immutable row data. These immutable rows are called “tuples” in PostgreSQL.
Tuples are uniquely identified by ctid. A ctid conceptually represents the physical disk location for a tuple.

Because of MVCC architecture, multiple ctid can describe a single row which sometime degrade the write performance of PostgreSQL.

What is Multi Version Concurrency Control (MVCC)

For example,
When we are updating any data, internally MVCC inserts new version of record and marks old record as dead row.

This operation requires to create new ctid for new rows, new link for index, remove old link of record.
Yes, this is good to avoid locking issue, but write performance is not that much good.

A CTIDs are transparent to users, and any user can check table CTID by adding one additional ctid column in any of the table.
PostgreSQL internally creates CTID column for each and every table.

For example,

SELECT CTID,* FROM tbl_Employees;

Execute VACUUM or AUTO-VACUUM is required to increase the overall performance of PostgreSQL which removes the dead rows or old version rows, generated by MVCC architecture.

PostgreSQL: Short note on VACUUM, VACUUM FULL and ANALYZE

The Visibility Map is one of the important architectural notes of PostgreSQL.

The Visibility Map associates with each Table and Index.

It stores in separate relation alongside with main relation and It starts with the file node number plus a suffix _vm.

The Visibility Map uses to keep track of which pages contain only tuples that are known to be visible to all active transactions.

It stores one bit per page. If it set bit, that means no free space on the page and all tuples can be visible to running transaction.

If you do not know about the MVCC (Multiversion Concurrency Control), please visit below an article.

What is Multi Version Concurrency Control (MVCC)

Why VACUUM requires in MVCC because VACUUM removes dead row versions in tables and marks the space available for future reuse.

With the Visibility Map, VACUUM will be now able to perform partial scans of the table data, skipping pages which marks as fully visible.

Now Database Administrator happy because partial scans require fewer I/O operations for VACUUM.

The Visibility map bits are only set by vacuum but are cleared by any data-modifying operations on a page.

In this post, sharing a small note on one of the internal architecture things of PostgreSQL.

Each Table and Index has an FSM which used to keep track of available space in the relation.

It stores all free space related information in an alongside primary relation and that relation starts with the file node number plus the suffix _fsm.

If you do not know about the MVCC (Multiversion Concurrency Control), please visit below an article.

What is Multi Version Concurrency Control (MVCC)

Because of MVCC architecture, PostgreSQL generates dead tuples on each update and delete action.

Why VACUUM requires in MVCC because VACUUM removes dead row versions in tables and marks the space available for future reuse.

The VACUUM process also updates the Free Space Map and using VACUUM FULL we can recover those free spaces.

The FSM pages are organized as a tree.
The bottom level FSM pages store the free space available on each page, using one byte to represent each such page.
The upper levels aggregate information from the lower levels.

We can also examine free space pages using pg_freespace module.

CREATE EXTENSION pg_freespace;
SELECT * FROM pg_freespace('table_name');

This is one of the very important post for all PostgreSQL Database Professionals.

In this post, I am sharing a small, but very powerful script to know about the Live Tuples (Rows) and Dead Tuples (Rows) of the PostgreSQL Object like: Tables and Indexes.

If you don’t know about the MVCC (Multi Version Concurrency Control), Please visit this article.

What is Multi Version Concurrency Control (MVCC)

PostgreSQL is based on MVCC Architecture.

In MVCC Architecture, When you update or delete any row, Internally It creates the new row and mark old row as unused.

This kind of data, we call as Dead Tuples or Dead Rows. (We can also say like, This is an internal fragmentation).

Periodically, We should find dead rows of the object and we should remove it using VACUUM techniques of PostgreSQL.
By this way, we can increase the overall performance of PostgreSQL Database Server.

PostgreSQL: Short note on VACUUM, VACUUM FULL and ANALYZE

Find out Live Tuples or Dead Tuples using two different scripts.

Using system function and pg_class:

SELECT 
	relname AS ObjectName
	,pg_stat_get_live_tuples(c.oid) AS LiveTuples
	,pg_stat_get_dead_tuples(c.oid) AS DeadTuples
FROM pg_class c;

Using pg_stat_user_tables:

SELECT 
	relname AS TableName
	,n_live_tup AS LiveTuples
	,n_dead_tup AS DeadTuples
FROM pg_stat_user_tables;