I am telling you personally that I hate the use of DISTINCT.

DISTINCT used by those people, who are not sure about their data set or SELECT statement or JOINS.

Whenever I get any query with DISTINCT, immediately I suggest to remove it.

I asked to many developers that, does it makes sense to use DISTINCT or do we have any alternative?
Second, I asked that Why are you required to use DISTINCT.

Guys, what are you thinking? What was that answer?

Very simple, they keep quiet for a minute and were telling like: we are using to avoid duplicate result set.

This is not the correct answer for the use of DISTINCT.

Actually, most of developers are using DISTINCT because they want to hide their mistakes. They are writing incorrect joins and hiding duplication of data.

This is not only for database developer, DBA is a same responsible for this. The DBA must stop the duplication of data.

The DBA should give the answer that how duplicates data inserted in the database and who is responsible for that duplication.

Do you think, the use of DISTINCT is good for query performance?

My answer is simply no. Why, because it fetches unnecessary duplicate data and returns unique from it.

You should know the logical sequence of SQL Query execution. In the logical sequence, DISTINCT execute after SELECT means It checks all the data first and after that It avoids the duplicate data.

As I have raised a question for DISTINCT, few developers asked me to use GROUP BY instead of DISTINCT.

My simple answer was, Please do not use anything to remove duplicates from your result set.

Explicitly, you should remove the duplicate data and then also If you require to count few unique master group data, you can use GROUP BY instead of DISTINCT.

SQL Server uses a Maximum Degree of Parallelism (MAXDOP) for achieving the parallel query processing to increase the query execution performance.

The default setting of MAXDOP is 0 means your query execution plan chooses the maximum degree of parallelism automatically.

Which is OK, but in big transaction system you should limit your query by specifying an explicit value of MAXDOP (Number of CPU Core for Parallel Query Execution). Which requires proper analysis of your system loads, CPU configuration and number of concurrent connections.

Below are some different ways to change the value MAXDOP.

Using SSMS:

In SSMS, right click on server go to Server Properties -> Advanced -> Parallelism -> Max Degree of Parallelism. Default is 0.

Using sp_configure:

sp_configure 'show advanced options', 1;
GO
RECONFIGURE WITH OVERRIDE;
GO
sp_configure 'max degree of parallelism', 8;
GO
RECONFIGURE WITH OVERRIDE;
GO

Using MAXDOP hint option at query level:

SELECT *
FROM [Person].[Address]
OPTION (MAXDOP 1)

Yesterday, I was solving a performance issue of a client, and I found that their in-house DBAs scheduled to restart a SQL Server at every Sunday.

I simply stopped my all tuning work and immediately set a meeting with DBA team.
To restart a SQL Server, again and again, is a very bad practice and it leads to the biggest performance issue.

You must be aware of the following points, before restarting a SQL Server.

A Plan Cache of SQL Server stores the precompiled execution plan for frequently executed of queries.
It improves the query performance by reducing a cost of creating and compiling execution plan again and again.

Apparently, most of our queries are frequently executing.

Once you restart the SQL Server, it clears the Plan Cache area, and all your queries require to create a new query plan which degrades the performance.

A Buffer Pool of SQL Server, very capable of storing the result of your frequently executed queries.

It reduces the disk/io and improves the query performance by sending a result directly from the memory area.
If a result is not available in the buffer pool, it fetches from the disk.

When you restart the SQL Server, it clears the Buffer Pool area, and it takes more time to fill it again.

The Dynamic Management Views and Functions of SQL Server, return server state information that can be used to monitor the health of a server instance, diagnose problems, and tune performance.

The DMVs and DMFs are the first tool for SQL DBA to monitor the health of SQL Server. The DMVs and DMFs store its data into the process space of sqlservr.exe.

Once you restart your SQL Server, it clears all DMVs and DMFs statistics by destroying the process of sqlservr.exe.

The rollback of uncommitted transactions of SQL Server, this is a part of SQL Server crash recovery which rollback all uncommitted transactions.

Now imagine that you are going to restart your SQL Server and few of queries are still running.

In this situation, SQL Server rollbacks all uncommitted transactions which degrade the performance and restart process might take more time to complete the action.

In this post, I am sharing the list of common practices for Database Designing which should follow by Database Developers and Application Developers.

• Foreign keys should have indexes on them, and it should often cover multiple columns base on the queries you need to execute.
• You should make sure about the referential integrity by enforcing the Foreign Keys.
  Choose right column for CLUSTER Index.
• You should apply proper indexes and do not create duplicate indexes and avoid creation of unnecessary indexes.
• You should not put auto-number data type as a PRIMARY KEY to join a FOREIGN KEY in a parent/child table relationship.
• If your system is running large number of INSERT/DELETE, you should avoid frequent update of database Statistics.
• You should choose the write data-type for your data columns.
• Please separate big size of columns from main table and store in a different table.
• You should apply proper column name.
• Use proper data types for Latitude, Longitude, IP-Address, Network Address, Mac Address.
• Use Natural Primary key rather than Surrogate Primary key. Natural Primary keys are keys base on an external meaningful data. Surrogate Primary key are keys base on an auto-increment number or values which have no meaning outside the system.
• You must use Stored Procedures or Stored Functions.
• Please avoid too much database normalization and try to design your database to achieve Database Normalization + Database Denormalization. You should design this combination to achieve the performance.
• Avoid inappropriate or excessive use of cursors.
• Avoid inappropriate use of Database Transactions (BEGIN…COMMIT).
• Try to avoid the use of OR condition because an AND condition restricts the data set, whereas an OR condition grows it.
• You should do detailed performance analysis on queries.
• Try to use the Prepared SQL Statement which caches the compiled query and the execution plan for better performance. It will also better protect you against SQL injection attacks.
• Please do not simplify your complex queries through the views.
• Please do not use DISTINCT frequently. Many developers are using DISTINCT to hide duplicate data, but you should stop and remove the duplicate data.
• Try to avoid Correlated Subqueries.
• Do not store comma delimited list in a table.
• Do not store JSON data in TEXT COLUMN, you should use JSON data type for it.
• Please do not use comma separated table names in SELECT query, use explicit JOINs conditions to avoid internal Cartesian products.

 
The list is not final yet, you guys can add/share a comment and make it final…

What is HOT UPDATE (Heap Only Tuple) and How It helps to increase the performance of UPDATE Query?
In this post, I am going to explain the answer of this question.

This is one of the internal of PostgreSQL, but I am sure that most of you guys don’t know about PostgreSQL HOT.

PostgreSQL requires HOT because of MVCC architecture.
Please read below articles, before start to knowing about the HOT.

What is MVCC?

What is Fillfactor in PostgreSQL?

What is Vacuum in PostgreSQL?

In the MVCC architecture, When we update any record, internally MVCC architecture inserts a new record with new values and marks old record as deleted or discarded.
When we vacuum, we can clean that deleted records and utilize that space.

What is HOT (Hot Only Tuple)?

Now imagine that, we have total 500 data pages and we are updating record of page number 3, but unfortunately new record insert in page number 500.

It looks like very big internal fragmentation because of MVCC architecture.
But actually this is not happening.

PostgreSQL has a concept of HOT, With a HOT dead tuple space can be automatically reclaimed at the time of INSERT or UPDATE if no changes are made to indexed columns. This allows for more consistent performance.

HOT means, creating a new update tuple if possible on the same page as the old tuple, and maintaining a chain of updated tuples linking a new version to the old tuple.

HOT updates can only reuse dead tuples from previous transactions directly, not from the same or concurrent ones.
If you have big updates, changing large portions of the table at once, you might want to split them up in a couple of chunks.

HOT updates only work when the changed columns are not involved in indexes in any way because the indexes pointing the the old tuples need to point to the new version of it as of transaction id.

You might be blocking HOT updates with indexes on updated columns. If those are expendable, you might get better overall performance without them.

Let’s understand PostgreSQL HOT practically.

Create a table with Sample data:

CREATE TABLE tbl_Employees
(
	EmpID INT
	,EmpName CHARACTER VARYING 	
	,EmpDepartment CHARACTER VARYING
	,EmpSalary INT
);
 
INSERT INTO tbl_Employees
VALUES 
(1,'Anvesh','Database',90000)
,(2,'Jenny','JAVA',65000)
,(3,'Martin','PHP',85000)
,(4,'Roy','PHP',94000)
,(5,'Eric','PHP',70000)
,(6,'Rajesh','Animation',50000)
,(7,'Mahi','Database',40000)
,(8,'Sofia','JAVA',60000);

Delete one record:

DELETE FROM tbl_Employees WHERE EmpID = 7;

Insert one New Record:

INSERT INTO tbl_Employees
VALUES 
(7,'Junik','Database',10000);

Please check If any HOT Update or not on table:

SELECT pg_stat_get_tuples_hot_updated('tbl_employees'::regclass::oid);
/*
Result:
pg_stat_get_tuples_hot_updated
------------------------------
0
*/

Please update the statestic of that table:

ANALYZE tbl_Employees;

Now execute one more update command:

UPDATE tbl_Employees SET EmpName = 'Robort' WHERE EmpID=7;

Please check If any HOT Update or not on table:

SELECT pg_stat_get_tuples_hot_updated('tbl_employees'::regclass::oid);
/*
Result:
pg_stat_get_tuples_hot_updated
------------------------------
1
*/

Again, do more update:

UPDATE tbl_Employees SET EmpName = 'Manish' WHERE EmpID=8;
UPDATE tbl_Employees SET EmpName = 'Cerlin' WHERE EmpID=4;

Please check If any change in number of HOT Update of a table:

SELECT pg_stat_get_tuples_hot_updated('tbl_employees'::regclass::oid);
/*
Result:
pg_stat_get_tuples_hot_updated
------------------------------
3
*/

Now Apply, Index on EmpName Column:

CREATE INDEX idx_tbl_employees_empname 
ON tbl_employees USING BTREE (EmpName);

After Index, execute few more update:

UPDATE tbl_Employees SET EmpName = 'Kulin' WHERE EmpID=3;
UPDATE tbl_Employees SET EmpName = 'Priya' WHERE EmpID=5;

After Index, Please check If any change in number of HOT Update of a table:
The result is same because If column has an Index, HOT Update does not work.

SELECT pg_stat_get_tuples_hot_updated('tbl_employees'::regclass::oid);
/*
Result:
pg_stat_get_tuples_hot_updated
------------------------------
3
*/

You can also check number of HOT update of current transaction:

SELECT pg_stat_get_xact_tuples_hot_updated('tbl_employees'::regclass::oid);

In this post, I am sharing one script to find Outdated Index Statistics of SQL Server.

Database Statistics are playing the main role to prepare a best query execution plan because Query Optimizer reads the Object Statistics and takes the decision accordingly.

SQL Server Statistics stores information like: object meta-data, location information, number of records.
If your Object Statistics are not updated, It impacts your query performance so You should find outdated statistics and periodically, you should update the statistics data.

SQL Server:Warning in Query Execution Plan, “Operator used tempdb to spill data during execution with spill level 2”

Script to find Outdated Index Statistics Information:

SELECT 
	OBJECT_NAME(id)
	,name
	,STATS_DATE(id, indid)
	,rowmodctr
FROM sys.sysindexes
WHERE STATS_DATE(id, indid)<=DATEADD(DAY,-1,GETDATE()) 
AND rowmodctr>0 
AND id IN (SELECT object_id FROM sys.tables)
GO

In online forums, I am observing one bad query practice. People are using multiple tables in the SELECT query by separating using commas.
Bad Query Like:

SELECT * FROM a, b, c WHERE a.id = b.id AND b.id = c.id;

Many of the Database Professionals are thinking that It is working like JOINs.

I am writing this article in the “Database Designing” category, means It applies to all database professionals for knowing the truth of this kind of bad SQL Query.

When you write a query like above sample, internally Query Planner creates a virtual join between these tables and this is not a strait forward joins creation. A Query Planner has to check all possible combinations like: first join A TO B and then join C, join A TO C and then join B, join B TO C and then join A.

When you write such types of bad queries, Query Planner has to do additional work for joining the tables correctly and return the correct result.
But actually It checks the data by creating a full Cartesian Product between these tables which is very costly and slow down the query performance.

We have taken example of three tables only, now let’s assume if we have 10 tables and people wrote a SELECT query using comma separated tables.
So please I am requesting, never use such a bad query and try to write explicit JOINs in the SELECT Query.

Another interesting point is:

When I discussed this point with our team, one of database developers said It is good that Query Planner is deciding the order of JOINS. If we give explicit joins, It may not be in the correct order all the times so let’s Query Planner to do.

My simple answer was:
I am 5% agree with you, but do not forget that we are enough professional and can define the JOINs order correctly. We should not rely on Query Planner because It requires more internal I/O to just prepare a joins of the table.

Correct Query should like:

SELECT a.* FROM a JOIN b ON a.id=b.id JOIN c ON b.id=c.id

The PostgreSQL Prepared Statement or Parameterized Statements are always good for specific purpose like: can create Prepared Statement for frequently executing query of a session, It also prevents from SQL Injections.

Prepared Statements are faster for a particular session because It does not require parsing and compiling for each execution. It is only session specific and once a session kill, Prepared Statements automatically destroy.

Here, You can access more on Prepared Statements.

Prepared or Parameterized Statements in Database System

Small demonstration of PostgreSQL Prepared Statement:

Create a table with sample records:

CREATE TABLE tbl_Students 
(
	StudID INTEGER PRIMARY KEY
	,StudName CHARACTER VARYING
	,StudClass CHAR(1)	
);

INSERT INTO tbl_Students 
VALUES (1,'Anvesh','A')
,(2,'Neevan','B'),(3,'Jenny','C')
,(4,'Roy','C'),(5,'Martin','C');

Create a Prepared Statement to SELECT Students data:

PREPARE pre_GetStudents (INT) AS 
SELECT *FROM tbl_Students WHERE StudID = $1;

Execute a Prepared Statement:

EXECUTE pre_GetStudents(2);

System view to check the Prepare statements of session:

SELECT *FROM pg_prepared_statements;

For reading this article title, Please do not surprise.
Yes, this is possible in PostgreSQL.

We can simply copy the PostgreSQL DATA Directory and can use as a full PostgreSQL Database Server backup.

The File System Level backup is not always recommended, but we can use this backup method in certain condition.

In our organization, I am managing PostgreSQL backup using two approaches.

The First approach is a regular method for taking individual Database Backup using pgDump.

The Second approach is to take “frozen snapshot” of the PostgreSQL DATA directory. Our System Admin department is taking File System Level Backup of PostgreSQL.

Whenever I required new duplicate PostgreSQL server, I used this kind of file system backup of PostgreSQL.

It is faster than pgDump because It does not require any additional CPU I/O as full DATA directory already available on DISK.
System Admin has to perform simple COPY and PASTE like any other file or folder.

Few Restrictions are:

The database server must be shut down in order to get a usable backup and disallowing all connections will not work.

It is impossible to restore only a table or few tables. A File system backups only work for complete backup and restoration of an entire database cluster.

If your data files and WAL log are on different disks, or if tablespaces are on different file systems, it might not be possible to use snapshot backup or file system level backup.

A file system backup will typically be larger than an SQL dump.

After restoring of file system backup, It requires to rebuild all your Database Indexes.(For improving the performance)

What is the use of Service Broker?

Service Broker helps developers build asynchronous, loosely coupled applications in which independent components work together to accomplish a task. These application components exchange messages that contain the information that is required to complete the task.

Service Broker can be useful for any application that needs to perform processing asynchronously, or that needs to distribute processing across a number of computers.

What are the main components of Service Broker?

• Conversation components: Conversation groups, conversations, and messages form the run-time structure of a Service Broker application. Applications exchange messages as part of a conversation.
• Service definition components: These are design-time components that specify the basic structure of the conversations that the application uses.
• Networking and security components: These components define the infrastructure used to exchange messages between instances of the Database Engine. To help database administrators manage
• Changing environments: Service Broker lets administrators configure these components independently of the application code.

What are the advantages of Service Broker?

Database integration enhances application performance and simplifies administration.

Message ordering and coordination for simplified application development

Loose application coupling provides workload flexibility.

Related message locking allows more than one instance of an application to process messages from the same queue without explicit synchronization.

Automatic activation allows applications to scale with the message volume.

I have a database which has several tables that have very heavy write operations. These table are very large and some are over a hundred gigabytes. I noticed performance of this database is getting slower and after some investigation we suspect that the Auto Update Statistics function is causing a performance degradation.

What should be your solution of this problem?

Statistics are objects which contain information about the distribution of values in one or more columns of a table or indexed view. The query optimizer uses this statistical data to estimate the number of rows in the query result. In the case where the statistics are invalid or outdated then the query optimizer is likely to choose a wrong execution plan and query performance will decrease. Therefore keeping statistics up to date is an essential part of SQL Server maintenance.

If we experience problems due to statistics automatically being updated then the solution might be to disable the Auto Update Statistics option. However, we cannot live with outdated statistics on large databases, so if you turn this option off you should create a scheduled job which explicitly updates the statistics during off-peak periods. To do this, we can use the sp_updatestats procedure.

What are the different type of Fragmentations?

• Logical/Phsical Disk Fragmentation
• Extent Fragmentation
• File Fragmentation
• Page Fragmentation
• Index Fragmentation

What are the index statistics?

Index statistics contain information about the distribution of index key values. By distribution, I mean the number of rows associated with each key value. SQL Server uses this information to determine what kind of execution plan to use when processing a query.

When are index statistics updated?

The AUTO_UPDATE_STATISTICS database setting controls when statistics are automatically updated.  Once statistics have been created, SQL Server then determines when to update those statistics based on how out-of-date the statistics might be.

SQL Server identifies out of date statistics based on the number of inserts, updates, and deletes that have occurred since the last time statistics were updated, and then recreates the statistics based on a threshold. The threshold is relative to the number of records in the table. (Enable the properties – “auto create statistics” and “auto update statistics” for OLTP)

Explain database options “Auto Update Statistics” and “Auto Update Statistics Asynchronous”?

Auto Update Statistics: If there is an incoming query but statistics are stale then sql server first update the statistics before building the execution plan.

Auto Update Statistics Asynchronous: If there is an incoming query but statistics are stale then sql servers uses the stale statistics, builds the execution plan and then update the statistics.

How to update statistics manually?

If you want to manually update statistics, you can use either sp_updatestats or UPDATE STATISTICS <statistics name>

What are the various types of statistics available?

There are three different types of statistics available.

• Statistics created due to index creation.
• Statistics created by optimizer.
• User defined statistics created from “CREATE STATISTICS”

What are the various options to be considered while designing a new execution plan?

There are a number of factors that influence the decision in choosing the proper execution plan. One of the most important ones are cardinality estimations, the process of calculating the number of qualifying rows that are likely after filtering operations are applied.

A query execution plan selected with inaccurate cardinality estimates can perform several orders of magnitude slower than one selected with accurate estimates. These cardinality estimations also influence plan design options, such as join-order and parallelism. Even memory allocation, the amount of memory that a query requests, is guided by cardinality estimations.

How to find out when statistics updated last time?

A simple logic is, run the query and observe the values for both “estimated rows” and “actual rows”, if they both are close to each other you need not worried about the statistics. If you find big difference between them then you need to think about updating statistics.

In general we can find out last statistics updated info from below query

select object_name(object_id) as table_name
,name as stats_name
,stats_date(object_id, stats_id) as last_update
from sys.stats
where objectproperty(object_id, ‘IsUserTable’) = 1
order by last_update

What are the statistics in SQL Server?

Statistics determine the selectivity of the indexes. If an indexed column has unique values then the selectivity of that index is more, as opposed to an index with non-unique values. Query optimizer uses these indexes in determining whether to choose an index or not while executing a query.

In which situation you should update statistics?

If there is significant change in the key values in the index

If a large amount of data in an indexed column has been added, changed, or removed (that is, if the distribution of key values has changed), or the table has been truncated using the TRUNCATE TABLE statement and then repopulated

Database is upgraded from a previous version.

What are the steps to improve the performance of a query?

• Number of joins and use of complex views/cursors have to be reduced.
• The use of the stored procedures and indexes have to be maximized.
• The optimized use of the complex conditional checks and computer columns have to be in place.
• Tracking of performance analysis for the query helps us in identifying the right aspects to optimize.

Could you please give some Optimization Tips in writing SQL Queries ?

• Always try to use views and stored procedures instead of doing work with heavy queries.
• Make a habit to use constraints instead of triggers whenever it is possible.
• When you need n number of row from database try to use top keyword or SET ROWCOUNT statement
• Always use table variables in place of temporary tables.
• Avoid Union and try to use UNION ALL statement.
• Always avoid using the DISTINCT clause, whenever possible.
• Always try to avoid using SQL Server cursors.
• Always try to avoid the HAVING clause.
• Do not use select count(*) to get number of rows
• Try to include SET NOCOUNT ON statement into your stored procedures to stop the message indicating the number of rows affected by a T-SQL statement.
• Always use file system to store large binary objects and use the file path in database.
• Sometimes we may have to apply more than one sub queries in our main query. Try to minimize the number of sub query block in your query