This is one of the important post for all PostgreSQL Database Administrator who are dealing with Bulk Insertion in their PostgreSQL Server.

We should optimize Bulk Insertion operation because It is also utilizing more system resources.

Few steps to increase the performance of Bulk Insertion:

• If It is only one time data load, remove all Index and create after INSERT. For regular Bulk Insertion, look into Index part and try to optimize it.
• Take a backup of all the Triggers and drop all the Triggers.
• Use COPY command instead of INSERT.
• If you can not use COPY command because of of INSERT multiple value, you should not give more than 1000 multiple values in single INSERT. All INSERT multiple values stored in the memory area which may decrease the INSERT performance.
• Disable other database logging process.
• If BULK INSERTION is running with a single transaction, you should also disable all FOREIGN KEYs and later you can enable it.
• Change synchronous_commit=off to reduce the cost of committing.
• Change Checkpoint default configuration, Increase the checkpoint_segments value.
• Increase the size of shared_buffers, effective_cache_size, work_mem.
• Prepared all statements on server side.
• Disable Auto-Vacuum on Table.
• Disable Auto Database Backup process.
• Increase the system resource (Memory, Disk, CPU), If it is required.
• Use SSD type of Disk for faster storage.

Note: Whatever you changed for BULK INSERTION, please do not forget to rollback after completion of BULK INSERTION.

In this post, I am sharing few important parameters for tunning the PostgreSQL for better performance.

When we are writing SQL, we are doing our best and trying to write database optimized code.
We are also checking Indexes and best execution plan, and accordingly, we are making our SQL queries.

But sometimes, we are not getting the good performance so at that time we should look at PostgreSQL Server level configuration and parameters.

Below are important parameters:
You can set all parameters in postgresql.conf file.

max_connections:
We can configure the maximum number of client connections and this is very important because every connection requires memory and other resources.
PostgreSQL can handle some hundred of connections, but if we are planning for thousands of connections, we should use some connection pooling mechanism to reduce overhead of connections.

shared_buffers:
Instead of big shared memory pool, PostgreSQL has a separate process for each database connection. PostgreSQL uses this dedicated memory area for caching purpose, so its provide lots of performance gain in your system.

If we have more loads, we can set large values for shared_buffers.
Generally, we should set shared_buffers values up to 1/4 part of the main memory.

You can calculate using, below formula

(0.25 * Main Memory)

wal_buffers:
This parameter defines, how much space is required for caching to write-ahead log entries. This is really very small size value but it is required to change in heavily loaded servers.

Recommended value is: 8MB.

work_mem:
This is also one of the most important parameter which is used to set a private work space for each and every new connection.
We have shared_buffers which we are using as the dedicated buffer for all connections, but using work_mem we can set memory for each query.

If we have configured wrongly, it really creates a big problem for us because if we have a big, complex query and which is running for multiple times then it will create a big problem for us.

We should configure this parameter base on the number of connections and what type of queries we are running on the server.

We can calculate using this method,
Memory(MB) / Max_connections.
If the result is very near to Max_connections, perform the divide by 2 again.

The result should be at least 16MB otherwise we should buy more RAM.

maintenance_work_mem:
This is also same as like work_mem parameter, but use only for maintenance related queries like, VACUUM, CLUSTER.
Generally, we are doing maintenance when query load is not too high.
I suggest you set a big value for parameter so that we can quickly complete our maintenance task.

Default is 16MB and Recommended is (Memory / 8).

effective_cache_size:
This parameter is mostly dedicated to PostgreSQL query planner because query planner is responsible to execute a query by choosing a better optimized execution path.
The query planner also requires some space to perform their work so this parameter helps the query planner.

Default is 65536 and Recommended is (Memory (MB) * 0.75)

autovacuum:
This is also one of the important parameter because PostgreSQL support MVCC nature.
This is really very required and it solves lots of performance related issues by removing dead tuples.
Whenever table is free, Autovacuum performs vacuuming on that table so another individual Vacuum executes fast because it has very less to remove.

synchronous_commit:
You can set number of transactions to commit per second. e.g. set 100 transaction means per second, 100 transactions is going to commit.
This parameter should be ON or OFF is up to our choice because when we put OFF, our bulk insertion is faster, but there is the chance of failure and when we put ON, bulk insertion may slow down but the chance of data failure is also very less.