The newly introduced BRIN (Block Range Index) Index is one my favourite index type of RDBMS system like PostgreSQL.

The BRIN index is stored the only summary of data pages instead of storing actual key data. It is 10 times faster than the normal BTREE Index.

I have already published articles on What is BRIN and How we can implement BRIN in PostgreSQL 9.5.?

Database Theory: What is BRIN (Block Range Index), How is faster than BTREE Index

PostgreSQL 9.5: Introduced BRIN – Block Range Index with Performance Report

In this post, I am sharing maintenance part of BRIN Index.

Database Administrators are doing maintenance of Indexes on a regular basis like Rebuild Index, Reorganize Index, Update Statistics of Indexes.

But these types of maintenance will not work for BRIN Indexes because BRIN Indexes are not storing any actual data and it stores only a summary of the data page.

PostgreSQL 9.5 also introduced brin_summarize_new_values() BRIN Index maintenance function.
This function is used to find a new range of pages which are not yet summarized by BRIN Index.

It performs the table scanning and creates a new summary index tuple for newly added data pages and adds into BRIN Index. It also returns a count of total newly added pages.

The syntax is:

SELECT brin_summarize_new_values('BRIN_Index_Name');

I already a shared article on ON CONFLICT clause of PostgreSQL 9.5

PostgreSQL 9.5: Insert IF not Exists, Update IF Exists (Insert ON CONFLICT option)

In above article, I used only one single key in ON CONFLICT clause.

A day before yesterday, I got an email like, does it require to add a unique index on those columns which we require to be in ON CONFLICT clause?
The answer is: Yes

You must need to define a unique index on those columns which you are planning to use in ON CONFLICT clause because it can only check the duplicates bases on unique indexes only.

Below is a demonstration of this:

Create a table with sample data with composite PRIMARY KEY:

CREATE TABLE tbl_Employee
(	
	EmpID INT 
	,EmpName CHARACTER VARYING 
	,CONSTRAINT pk_tbl_Employee_EmpID_EmpName PRIMARY KEY (EmpID,EmpName)
);
 
INSERT INTO tbl_Employee 
VALUES (1,'Anvesh'),(2,'Roy'),(3,'Lee')
,(4,'Nivu'),(5,'Rajesh'),(6,'Nupur');

Try to insert a duplicate EmpID record, using option INSERT ON DO NOTHING:
It will insert a new record because key EmpID + EmpName is not duplicate. But if you execute the same, insert the second time, it will do nothing.

INSERT INTO tbl_Employee 
VALUES (6,'Noor')
ON CONFLICT (EmpID,EmpName)
DO NOTHING;

Check the result:

SELECT *FROM tbl_Employee;

/*
EmpID   EmpName
-----------------
1       Anvesh
2       Roy
3       Lee
4       Nivu
5       Rajesh
6       Nupur
6       Noor
*/

Recently, PostgreSQL introduced JSON data type which is very powerful to manipulate JSON formatted data.

I have published a couple of articles on JSON Data Type.
You can start from here.

PostgreSQL 9.4: The JSON data type is Awesome (Part 1/3)

Recently, I have received a couple of comments on my FB page related use of PostgreSQL JSONB.

A JSONB data type does not preserve any white spaces. Whatever format we have specified, It removes and store without white spaces.

PostgreSQL 9.5 introduced jsonb_pretty() to get JSON formatted data.
When we are troubleshooting JSON related data at that time we require to get JSON in readable format.

Below is a small demonstration of jsonb_pretty():

Create table with sample JSON data:

CREATE TABLE tbl_TestJSON
(
	EmpID INTEGER PRIMARY KEY
	,DepartName VARCHAR(250)
	,EmployeeDetails JSONB
);

INSERT INTO tbl_TestJSON
VALUES 
(
	1
	,'Sales'
	,'
	{"employees":[
	    {"firstName":"Anvesh", "lastName":"Patel"}	  
	]}'
)
,(
	2
	,'Production'
	,'
	{"employees":[	    
	    {"firstName":"Nancy", "lastName":"Shah"}
	]}'
);

Using jsonb_pretty format function:

SELECT jsonb_pretty(EmployeeDetails) AS EmployeeDetails FROM tbl_TestJSON;

Result:

employeedetails
-----------------------------------
 {                                +
     "employees": [               +
         {                        +
             "lastName": "Patel", +
             "firstName": "Anvesh"+
         }                        +
     ]                            +
 }
 {                                +
     "employees": [               +
         {                        +
             "lastName": "Shah",  +
             "firstName": "Nancy" +
         }                        +
     ]                            +
 }
(2 rows)

If you are searching about, What is Expression Index, this is the one of the right article for you.

We know about the different types of PostgreSQL Index.
If you don’t about this, please visit below few links.

What is BRIN (Block Range Index )?

What is Partial Index?

What is BTree Index?

Most of the Database Administrators or Developers are doing a common mistake,
When you create any Index on Table Column, dose not mean that you can use the Indexed column in any expression and Index will work as per the expectation.

When you used Indexed column in any kind of expression, query planner simply skips the scanning of Indexes of that column.

For example,
We have created one index on Date column and in WHERE are extracting days from this Date column.
In this situation, the Index will not work on Date column and for such specific requirement, we have to create the Expression Index of PostgreSQL.

Let me demonstrate this.

First create one sample table:

CREATE TABLE tbl_ItemTransactions 
(
	TranID SERIAL
	,TransactionDate TIMESTAMPTZ
	,TransactionName TEXT
);

Generate sample data for testing the performance of Indexes:

INSERT INTO tbl_ItemTransactions 
(TransactionDate, TransactionName)
SELECT x, 'dbrnd' 
FROM generate_series('2015-01-01 00:00:00'::timestamptz, '2016-08-01 00:00:00'::timestamptz,'2 seconds'::interval) a(x);

Total inserted record count is 24969601:

SELECT COUNT(1) FROM tbl_ItemTransactions;

Now create index on TransactionDate column:

CREATE INDEX idx_tbl_ItemTransactions_TransactionDate
ON tbl_ItemTransactions (TransactionDate);

Lets see the plan of query with date filter:

EXPLAIN ANALYZE
SELECT *FROM 
tbl_ItemTransactions 
WHERE TransactionDate BETWEEN '20150808' AND '20160108';
/*
"Index Scan using idx_tbl_itemtransactions_transactiondate on tbl_itemtransactions  
(cost=0.44..249472.54 rows=6681405 width=18) 
(actual time=61.558..17765.979 rows=6609601 loops=1)"
"Index Cond: ((transactiondate >= '2015-08-08 00:00:00+05:30'::timestamp with time zone) AND 
(transactiondate <= '2016-01-08 00:00:00+05:30'::timestamp with time zone))"
"Planning time: 87.751 ms"
"Execution time: 29691.279 ms"
*/

Check Index usage by above query
(Result is one index scan):

SELECT *
FROM pg_stat_user_indexes
WHERE indexrelname='idx_tbl_itemtransactions_transactiondate';

Lets see the plan of query with date function filter:

EXPLAIN ANALYZE
SELECT *FROM 
tbl_ItemTransactions 
WHERE EXTRACT(day FROM TransactionDate) = 8;
/*
"Seq Scan on tbl_itemtransactions  
(cost=0.00..533587.00 rows=124848 width=18) 
(actual time=1246.093..37028.883 rows=820800 loops=1)"
"  Filter: (date_part('day'::text, transactiondate) = '8'::double precision)"
"  Rows Removed by Filter: 24148801"
"Planning time: 0.396 ms"
"Execution time: 58230.847 ms"
*/

Check Index usage by above query:
The Result is one index scan, means above query processed without scaning any index.
We created one index on TransactionDate, but when we use this column with any of default function, planner skips the index fot that column.

SELECT *
FROM pg_stat_user_indexes
WHERE indexrelname='idx_tbl_itemtransactions_transactiondate';

Now we should create one expression Index:
Please provide your require timezone otherwise It produces error like:

ERROR: functions in index expression must be marked IMMUTABLE

CREATE INDEX idx_tbl_ItemTransactions_TransactionDate_day
ON tbl_ItemTransactions ((EXTRACT(day FROM TransactionDate AT TIME ZONE 'UTC')));

Lets execute same query with date function filter:

EXPLAIN ANALYZE
SELECT *FROM 
tbl_ItemTransactions 
WHERE EXTRACT(day FROM TransactionDate AT TIME ZONE 'UTC') = 8;
/*
"Bitmap Heap Scan on tbl_itemtransactions  (cost=2340.01..160893.01 rows=124848 width=18) 
(actual time=367.653..2181.489 rows=820800 loops=1)"
"  Recheck Cond: (date_part('day'::text, 
(transactiondate)::timestamp without time zone) = '8'::double precision)"
"  Heap Blocks: exact=5248"
"  ->  Bitmap Index Scan on idx_tbl_itemtransactions_transactiondate_day  
(cost=0.00..2308.80 rows=124848 width=0) 
(actual time=365.386..365.386 rows=820800 loops=1)"
"        Index Cond: (date_part('day'::text, 
(transactiondate)::timestamp without time zone) = '8'::double precision)"
"Planning time: 0.297 ms"
"Execution time: 3608.713 ms"
*/

Check created new Expression Index usage by above query
(Result is one index scan):

SELECT *
FROM pg_stat_user_indexes
WHERE indexrelname='idx_tbl_itemtransactions_transactiondate_day';

Please read this previous article, To Understand the Power of BRIN – Block Range Index.

Database Theory: What is BRIN (Block Range Index), How is faster than BTREE Index

PostgreSQL 9.5 introduced the powerful BRIN Index, which is performance much faster than the regular BTREE Index.
The most important two lines of the BRIN are: It stores only minimum and maximum value per block so it does not require more space. For extremely large table It runs faster than any other Indexes.

In this post, I am going to show the example of BRIN index with the full performance report (testing over the 6gb of Table Data).

Below are steps:

First create one sample table:

CREATE TABLE tbl_ItemTransactions 
 (
     TranID SERIAL
     ,TransactionDate TIMESTAMPTZ
     ,TransactionName TEXT
 );

Insert Millions of data to test the performance of BRIN Index:

INSERT INTO tbl_ItemTransactions 
(TransactionDate, TransactionName)
SELECT x, 'dbrnd' 
FROM generate_series('2008-01-01 00:00:00'::timestamptz, '2016-08-01 00:00:00'::timestamptz,'2 seconds'::interval) a(x);

Check the total size of table:

SELECT pg_size_pretty(pg_total_relation_size('tbl_ItemTransactions')) AS TableSize;

/*
    TableSize
------------------
    6741 MB
*/

Now Check the performance without any Index:

EXPLAIN ANALYSE 
SELECT COUNT(1) FROM tbl_ItemTransactions 
WHERE TransactionDate BETWEEN '2012-01-01 00:00:00' and '2014-08-08 08:08:08';
/*
--Result:
			QueryPlan
-------------------------------------------------------------------------
Aggregate  (cost=2997896.81..2997896.82 rows=1 width=0) 
	(actual time=40651.793..40651.793 rows=1 loops=1)
  ->  Seq Scan on tbl_itemtransactions  (cost=0.00..2894105.00 rows=41516724 width=0) 
	(actual time=0.009..38726.686 rows=41054645 loops=1)
        Filter: ((transactiondate >= '2012-01-01 00:00:00+05:30'::timestamp with time zone) 
		AND (transactiondate <= '2014-08-08 08:08:08+05:30'::timestamp with time zone))
        Rows Removed by Filter: 94377356
Planning time: 0.860 ms
Execution time: 80651.837 ms
*/

Create BRIN index on TransactionDate Column:

CREATE INDEX idx_tbl_ItemTransactions_TransactionDate
ON tbl_ItemTransactions
USING BRIN (TransactionDate);

Now Check the performance of the same query which has BRIN index:

EXPLAIN ANALYSE 
SELECT COUNT(1) FROM tbl_ItemTransactions 
WHERE TransactionDate BETWEEN '2012-01-01 00:00:00' and '2014-08-08 08:08:08';
/*
			QueryPlan
---------------------------------------------------------------------------
Aggregate  (cost=2014834.09..2014834.10 rows=1 width=0) 
	(actual time=7108.998..7108.998 rows=1 loops=1)
  ->  Bitmap Heap Scan on tbl_itemtransactions  (cost=425666.42..1911042.28 rows=41516724 width=0) 
	(actual time=16.995..5415.086 rows=41054645 loops=1)
        Recheck Cond: ((transactiondate >= '2012-01-01 00:00:00+05:30'::timestamp with time zone) 
		AND (transactiondate <= '2014-08-08 08:08:08+05:30'::timestamp with time zone))
        Rows Removed by Index Recheck: 21579
        Heap Blocks: lossy=261632
        ->  Bitmap Index Scan on idx_tbl_itemtransactions_transactiondate  
		(cost=0.00..415287.24 rows=41516724 width=0) (actual time=15.547..15.547 rows=2616320 loops=1)
              Index Cond: ((transactiondate >= '2012-01-01 00:00:00+05:30'::timestamp with time zone) 
              AND (transactiondate <= '2014-08-08 08:08:08+05:30'::timestamp with time zone))
Planning time: 0.059 ms
Execution time: 7109.060 ms
*/

Now, you can see the difference between the result of the above two queries.
With the BRIN index same query took only 7 seconds and without BRIN it took around 80 seconds.

Create Partial BRIN index on TransactionDate Column:
You can also create Partial BRIN index for your individual range of data. The Partial BRIN index is also faster than normal BRIN index, but we should apply proper filter based on created Partial BRIN index.

CREATE INDEX idx_tbl_ItemTransactions_TransactionDate_2012
ON tbl_ItemTransactions
USING BRIN (TransactionDate)
WHERE TransactionDate BETWEEN '2012-01-01' AND '2012-12-31';

CREATE INDEX idx_tbl_ItemTransactions_TransactionDate_2013
ON tbl_ItemTransactions
USING BRIN (TransactionDate)
WHERE TransactionDate BETWEEN '2013-01-01' AND '2013-12-31';

After a long time of waiting, PostgreSQL 9.5 introduced INSERT ON CONFLICT [DO UPDATE] [DO NOTHING].

This option basically helps to perform DML actions like, Insert IF not Exists, Update IF Exists.

Previously, we have to use upsert or merge statement to do this kind of operation. I have also published an article on it.

PostgreSQL: Insert – Update or Upsert – Merge using writable CTE

This newly option has two varieties:

• INSERT ON CONFLICT DO UPDATE: If record matched, it is updated with the new data value.
• INSERT ON CONFLICT DO NOTHING: If record matched, it skips the record or error.

Below is a full demonstration of this:

Create a table with sample data:

CREATE TABLE tbl_Employee
(	
	EmpID INT PRIMARY KEY
	,EmpName CHARACTER VARYING 
);

INSERT INTO tbl_Employee 
VALUES (1,'Anvesh'),(2,'Roy'),(3,'Lee')
,(4,'Nivu'),(5,'Rajesh'),(6,'Nupur');

Insert one more row with option INSERT ON CONFLICT DO UPDATE:

Using this option, if a conflict occurs then it will update the mentioned data.
Here, I have use “Excluded” table which is special table and contains the row-to-be-inserted.

INSERT INTO tbl_Employee 
VALUES (7,'Ramu')
ON CONFLICT (EmpID)
DO UPDATE SET EmpName = Excluded.EmpName;

INSERT INTO tbl_Employet 
VALUES (7,'Mahi')
ON CONFLICT (EmpID)
DO UPDATE SET EmpName = Excluded.EmpName;

INSERT INTO tbl_Employee 
VALUES (8,'Noor')
ON CONFLICT (EmpID)
DO NOTHING;

INSERT INTO tbl_Employee 
VALUES (8,'Noor')
ON CONFLICT (EmpID)
DO NOTHING;

SELECT *FROM tbl_Employee;

/*
EmpID   EmpName
-----------------
1       Anvesh
2       Roy
3       Lee
4       Nivu
5       Rajesh
6       Nupur
7       Mahi
8       Noor
*/

PostgreSQL 9.5 introduced one of the powerful feature called SKIP LOCKED which is used for SKIP uncommitted data.

I am not going to make this more complex topic, explaining by one practical example.

In our project we have implemented thread mechanism. Different threads are running and taking a fixed set of records from the Database.
These threads basically implemented for performing Reverse Geo Coding on the stored Latitude and Longitude.
I am giving 10000 records to each thread in each batch.
Now the average execution time of each thread is 5 second to 7 second.

Now, one batch of 10000 records taken by thread_1 for 5 second to 7 second that same records should not taken by any other thread because that first 10000 records already under the process by thead_1.
Thread_2 has taken next 10000 records which are not taken by thread_1.

Now with the PostgreSQL 9.5, we do not require any additional locking mechanism or isolation level to achieve this scenario.
We can very easily achieve using FOR UPDATE SKIP LOCKED option. When we SELECT data using this option, it will skip all running transactions (uncommitted transaction) and return only required records to be processed.

Below is a full demonstration of this.

First create a table with sample data:

CREATE TABLE tbl_Employee 
(
	EmpID INT
	,EmpName CHARACTER VARYING
);

INSERT INTO tbl_Employee 
VALUES (1,'Anvesh'),(2,'Roy')
,(3,'Benny'),(4,'Nivu');

Now we are going to test this option with the help of two different sessions.

Open a new session and execute below queries in Session 1:
We have inserted four sample records and now will update one of record in OPEN TRANSACTION with out committing anything.

BEGIN TRANSACTION

UPDATE tbl_Employee SET EmpName = 'Martin' 
WHERE EmpID = 3;

Open another session and execute below queries in Session 2:
First SELECT with out any option:

If you do not know, please read about the MVCC.

What is Multi Version Concurrency Control (MVCC)

PostgreSQL has a default MVCC architecture, in which you can select last committed row version.
In the above query we have updated one record for EmpID=3, but when I execute simple SELECT statement then it returned the old value of EmpID=3.

This data is based on last committed row version.

SELECT *FROM tbl_Employee;

/* Result:
EmpID    EmpName
--------------------
  1      Anvesh
  2      Roy
  4      Nivu
  3      Benny
*/

Now SELECT with NOWAIT option:

When you are trying to SELECT running transaction, It will raise the error like : can not obtain lock on row. You can not SELECT EmpID=3, but you can SELECT EmpID IN (1,2,4).

Please find out below query and the result.

SELECT *FROM tbl_Employee WHERE EmpID=3 FOR UPDATE NOWAIT;
SELECT *FROM tbl_Employee FOR UPDATE NOWAIT;

/* Result:
********** Error **********

ERROR: could not obtain lock on row in relation "tbl_employee"
SQL state: 55P03
*/

Now finally, we test SKIP LOCKED option:

A SKIP LOCKED option simply sikped the records or rows which are the part of ROW Level Locking.
You can find out in the below result, EmpID=3 is not in result because it is not committed yet and running in different session.

/* Result:
SELECT *FROM tbl_Employee FOR UPDATE SKIP LOCKED;

EmpID    EmpName
--------------------
  1      Anvesh
  2      Roy
  4      Nivu
*/

Hello guys, Since last two months I have published about the SQL Server Interview Questions and Answers.
During this time, I got lots of messages and requests about the PostgreSQL.

In this August – 2016, I am going to post about only PostgreSQL and will try to complete all pending requests of readers.

In this post, I am going to publish about The Row Level Security which is now a common feature of all new version of RDBMSs like SQL Server 2016, PostgreSQL 9.5.

With the concept of Row Level Security, we can restrict the user for a particular set of records.
Previously, we are doing this kind of restriction using by creating filtered VIEWS and assign related VIEWS to related Users.

PostgreSQL 9.5 has this Row Level Security feature which extends the overall security policy of the PostgreSQL.

Let me demonstrate this.

Execute below all steps using Postgres Super user:

First create two sample users:

CREATE USER abc LOGIN PASSWORD '123';
CREATE USER xyz LOGIN PASSWORD '321';

CREATE TABLE tbl_Employees
(
	EmpID INT
	,EmpName CHARACTER VARYING 
);
INSERT INTO tbl_Employees VALUES 
(1,'Anvesh'),(2,'Martin'),(3,'Roy')
,(4,'Jeeny'),(5,'Kavita'),(6,'Neevan');

ALTER TABLE tbl_Employees ADD pgUser CHARACTER VARYING;

UPDATE tbl_Employees SET pgUser = 'abc' WHERE EmpID IN (1,2,3);
UPDATE tbl_Employees SET pgUser = 'xyz' WHERE EmpID IN (4,5,6);

GRANT SELECT,UPDATE,INSERT,DELETE ON tbl_Employees TO abc;
GRANT SELECT,UPDATE,INSERT,DELETE ON tbl_Employees TO xyz;

CREATE POLICY policy_employee_user ON tbl_Employees FOR ALL
TO PUBLIC USING (pgUser = current_user);

ALTER TABLE tbl_Employees ENABLE ROW LEVEL SECURITY;

SELECT *FROM tbl_Employees;

empid   empname   pguser
--------------------------
 1      Anvesh     abc
 2      Martin     abc
 3      Roy        abc

empid   empname   pguser
--------------------------
 4      Jeeny      xyz
 5      Kavita     xyz
 6      Neevan     xyz