I am telling you personally that I hate the use of DISTINCT.

DISTINCT used by those people, who are not sure about their data set or SELECT statement or JOINS.

Whenever I get any query with DISTINCT, immediately I suggest to remove it.

I asked to many developers that, does it makes sense to use DISTINCT or do we have any alternative?
Second, I asked that Why are you required to use DISTINCT.

Guys, what are you thinking? What was that answer?

Very simple, they keep quiet for a minute and were telling like: we are using to avoid duplicate result set.

This is not the correct answer for the use of DISTINCT.

Actually, most of developers are using DISTINCT because they want to hide their mistakes. They are writing incorrect joins and hiding duplication of data.

This is not only for database developer, DBA is a same responsible for this. The DBA must stop the duplication of data.

The DBA should give the answer that how duplicates data inserted in the database and who is responsible for that duplication.

Do you think, the use of DISTINCT is good for query performance?

My answer is simply no. Why, because it fetches unnecessary duplicate data and returns unique from it.

You should know the logical sequence of SQL Query execution. In the logical sequence, DISTINCT execute after SELECT means It checks all the data first and after that It avoids the duplicate data.

As I have raised a question for DISTINCT, few developers asked me to use GROUP BY instead of DISTINCT.

My simple answer was, Please do not use anything to remove duplicates from your result set.

Explicitly, you should remove the duplicate data and then also If you require to count few unique master group data, you can use GROUP BY instead of DISTINCT.

In this post, I am sharing a full demonstration of the concept of Partitioned view in SQL Server.

Using Partitioned view, you can implement Table Partitioning in the Non-Enterprise Edition of SQL Server because Table Partition is the feature of SQL Server Enterprise Edition.

Many small level companies are searching for this kind of solution, and the concept of Partitioned view is also giving a good performance.

We can also say that this is a manual table partitioning where you must create separate physical tables basis on your require partition ranges and require to apply proper CHECK constraint for validating data ranges.

After this, using one view, you can access all table and accordingly query execution plan also access only require table which reduces the unnecessary load on all or whole table.

Please check the below demonstration and try it your self.

Create a SEQUENCE object:

CREATE SEQUENCE dbo.seq_ProductHistory
    START WITH 1  
    INCREMENT BY 1 ;  
GO

Create four physical tables:

CREATE TABLE dbo.tbl_ProductHistory_Q1
(
	ProductID INT
	,ProductName VARCHAR(20)
	,SoldDate DATE
	,ProductSoldQuarter TINYINT CHECK (ProductSoldQuarter = 1)
	,CONSTRAINT pk_tbl_ProductHistory_Q1_ProductID_ProductSoldQuarter PRIMARY KEY(ProductID, ProductSoldQuarter)
)
GO

CREATE TABLE dbo.tbl_ProductHistory_Q2
(
	ProductID INT 
	,ProductName VARCHAR(20)
	,SoldDate DATE
	,ProductSoldQuarter TINYINT CHECK (ProductSoldQuarter = 2)
	,CONSTRAINT pk_tbl_ProductHistory_Q2_ProductID_ProductSoldQuarter PRIMARY KEY(ProductID, ProductSoldQuarter)
)
GO

CREATE TABLE dbo.tbl_ProductHistory_Q3
(
	ProductID INT 
	,ProductName VARCHAR(20)
	,SoldDate DATE
	,ProductSoldQuarter TINYINT CHECK (ProductSoldQuarter = 3)
	,CONSTRAINT pk_tbl_ProductHistory_Q3_ProductID_ProductSoldQuarter PRIMARY KEY(ProductID, ProductSoldQuarter)
)
GO

CREATE TABLE dbo.tbl_ProductHistory_Q4
(
	ProductID INT 
	,ProductName VARCHAR(20)
	,SoldDate DATE
	,ProductSoldQuarter TINYINT CHECK (ProductSoldQuarter = 4)
	,CONSTRAINT pk_tbl_ProductHistory_Q4_ProductID_ProductSoldQuarter PRIMARY KEY(ProductID, ProductSoldQuarter)
)
GO

Create a Partitioned View:

CREATE VIEW dbo.vw_ProductHistory
WITH SCHEMABINDING
AS
SELECT ProductID, ProductName, SoldDate, ProductSoldQuarter FROM dbo.tbl_ProductHistory_Q1
UNION ALL
SELECT ProductID, ProductName, SoldDate, ProductSoldQuarter FROM dbo.tbl_ProductHistory_Q2
UNION ALL
SELECT ProductID, ProductName, SoldDate, ProductSoldQuarter FROM dbo.tbl_ProductHistory_Q3
UNION ALL
SELECT ProductID, ProductName, SoldDate, ProductSoldQuarter FROM dbo.tbl_ProductHistory_Q4
GO

Use view and Insert data into four tables:

INSERT INTO dbo.vw_ProductHistory 
VALUES (NEXT VALUE FOR dbo.seq_ProductHistory, 'Computer', '2017-01-08', 1)
GO

INSERT INTO dbo.vw_ProductHistory 
VALUES (NEXT VALUE FOR dbo.seq_ProductHistory, 'Laptop', '2017-04-14', 2)
GO

INSERT INTO dbo.vw_ProductHistory 
VALUES (NEXT VALUE FOR dbo.seq_ProductHistory, 'Pendrive', '2017-08-16', 3)
GO

INSERT INTO dbo.vw_ProductHistory 
VALUES (NEXT VALUE FOR dbo.seq_ProductHistory, 'Mobile', '2017-12-26', 4)
GO

SELECT all records from all four tables via view:
Check the execution plan, where you can see four different access on physical tables.

SELECT *FROM dbo.vw_ProductHistory

Result:

ProductID   ProductName          SoldDate   ProductSoldQuarter
----------- -------------------- ---------- ------------------
1           Computer             2017-01-08 1
2           Laptop               2017-04-14 2
3           Pendrive             2017-08-16 3
4           Mobile               2017-12-26 4

Check the execution plan: It is accessing all four physical tables

SELECT few records:
Check the execution plan, where you can find only access of Quarter-4 table because we are selecting data only for it.

SELECT *FROM dbo.vw_ProductHistory 
WHERE ProductSoldQuarter = 4

Result:

ProductID   ProductName          SoldDate   ProductSoldQuarter
----------- -------------------- ---------- ------------------
4           Mobile               2017-12-26 4

Check the execution plan: It is accessing only Quarter-4 table.

In this post, I am sharing performance test of the problem of mismatched column data type in SQL Server.

The Database Designer is also defining a wrong mismatched column data type by giving different data types to the same column for many tables.

For example, empid column -> assigned INT in tbl_emp table and -> assigned BIGINT in tbl_dept table. Because of this kind of problem, your query will execute slowly, and it requires implicit data conversion for each record.

Another example is, your column data type is VARCHAR, and you are checking N’DATA in WHERE clause which is NVARCHAR, so this is also required implicit data conversion.

Please check below full demonstration and, test it yourself:

Create a sample table, where [id] column type is varchar:

CREATE TABLE tbl_DiffDataTypes 
(	id varchar(25)
	,name VARCHAR(25)
	,constraint pk_tbl_DiffDataTypes_id primary key (id)
)
GO

Insert few dummy records:

DECLARE @i INT,@max INT
SET @i =1 
SET @max =99999
WHILE @i<=@max
BEGIN
   INSERT INTO tbl_DiffDataTypes VALUES(@i,'dbrnd.com')
   SET @i=@i+1
END

Please enable STATISTICS:

SET STATISTICS IO ON
SET STATISTICS TIME ON

Test WHERE filter with VARCHAR value: (Only 3 logical reads)

SELECT * FROM tbl_DiffDataTypes WHERE id = '1'

Result of Statistics:

Table 'tbl_DiffDataTypes'. Scan count 0, logical reads 3, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

Test WHERE filter with NVARCHAR value: (62 logical reads)

SELECT * FROM tbl_DiffDataTypes WHERE id = N'1'

Result of Statistics:

Table 'tbl_DiffDataTypes'. Scan count 1, logical reads 62, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

Test WHERE filter with INTEGER value: (621 logical reads)

SELECT * FROM tbl_DiffDataTypes WHERE id = 1

Result of Statistics:

Table 'tbl_DiffDataTypes'. Scan count 1, logical reads 621, physical reads 0, read-ahead reads 0, lob logical reads 0, lob physical reads 0, lob read-ahead reads 0.

I am also adding a Database Design category for this post, because what I am feeling that you guys must know about this before designing tables or indexes.

You can create Indexes on VARCHAR columns, but you should not create indexes on big size of VARCHAR Columns.

Indexes also require storing data as key for accessing associated records. There is a size limitation of index key values.

Before SQL Server 2016, Index key size limitation was 900 bytes for Clustered and Non-Clustered index column.
SQL Server 2016 increased this limitation for Non-Clustered Index and now it is up to 1700 bytes, but for Clustered Index column it is still 900 bytes.

Let me demonstrate this.

Create a sample table with Nonclustered Index:

CREATE TABLE tbl_TestVarcharNonClustered
(
	ID INT
	,Name VARCHAR(2000)
)
GO

CREATE NONCLUSTERED INDEX idx_tbl_TestVarcharNonClustered_Name ON tbl_TestVarcharNonClustered(Name)
GO

You will get a below warning:
Because we have set Name VARCHAR(2000).

Warning! The maximum key length for a nonclustered index is 1700 bytes. The index 'idx_tbl_TestVarcharNonClustered_Name' has maximum length of 2000 bytes. For some combination of large values, the insert/update operation will fail.

Try to insert 1701 bytes data:

INSERT INTO tbl_TestVarcharNonClustered 
VALUES (1,REPLICATE('a',1701))
GO

You will get a below error:

Msg 1946, Level 16, State 3, Line 1
Operation failed. The index entry of length 1701 bytes for the index 'idx_tbl_TestVarcharNonClustered_Name' exceeds the maximum length of 1700 bytes for nonclustered indexes.

You can successfully run below INSERT, because it inserts only 1700 bytes:

INSERT INTO tbl_TestVarcharNonClustered 
VALUES (1,REPLICATE('a',1700))
GO

Now, create a sample table with Clustered Index:

CREATE TABLE tbl_TestVarcharClustered
(
	ID INT
	,Name VARCHAR(2000)
)
GO

CREATE CLUSTERED INDEX idx_tbl_TestVarcharClustered_Name ON tbl_TestVarcharClustered(Name)
GO

You will get a below warning:
Because we have set Name VARCHAR(2000) on cluster index column.

Warning! The maximum key length for a clustered index is 900 bytes. The index 'idx_tbl_TestVarcharClustered_Name' has maximum length of 2000 bytes. For some combination of large values, the insert/update operation will fail.

Try to insert a over size data like more than 900 bytes:

INSERT INTO tbl_TestVarcharClustered 
VALUES (1,REPLICATE('a',1001))
GO

You will get a below error:

Msg 1946, Level 16, State 3, Line 1
Operation failed. The index entry of length 1001 bytes for the index 'idx_tbl_TestVarcharClustered_Name' exceeds the maximum length of 900 bytes for clustered indexes.

You can successfully run below INSERT, because it inserts only 900 bytes:

INSERT INTO tbl_TestVarcharClustered 
VALUES (1,REPLICATE('a',900))
GO

In this post, I am sharing the list of common practices for Database Designing which should follow by Database Developers and Application Developers.

• Foreign keys should have indexes on them, and it should often cover multiple columns base on the queries you need to execute.
• You should make sure about the referential integrity by enforcing the Foreign Keys.
  Choose right column for CLUSTER Index.
• You should apply proper indexes and do not create duplicate indexes and avoid creation of unnecessary indexes.
• You should not put auto-number data type as a PRIMARY KEY to join a FOREIGN KEY in a parent/child table relationship.
• If your system is running large number of INSERT/DELETE, you should avoid frequent update of database Statistics.
• You should choose the write data-type for your data columns.
• Please separate big size of columns from main table and store in a different table.
• You should apply proper column name.
• Use proper data types for Latitude, Longitude, IP-Address, Network Address, Mac Address.
• Use Natural Primary key rather than Surrogate Primary key. Natural Primary keys are keys base on an external meaningful data. Surrogate Primary key are keys base on an auto-increment number or values which have no meaning outside the system.
• You must use Stored Procedures or Stored Functions.
• Please avoid too much database normalization and try to design your database to achieve Database Normalization + Database Denormalization. You should design this combination to achieve the performance.
• Avoid inappropriate or excessive use of cursors.
• Avoid inappropriate use of Database Transactions (BEGIN…COMMIT).
• Try to avoid the use of OR condition because an AND condition restricts the data set, whereas an OR condition grows it.
• You should do detailed performance analysis on queries.
• Try to use the Prepared SQL Statement which caches the compiled query and the execution plan for better performance. It will also better protect you against SQL injection attacks.
• Please do not simplify your complex queries through the views.
• Please do not use DISTINCT frequently. Many developers are using DISTINCT to hide duplicate data, but you should stop and remove the duplicate data.
• Try to avoid Correlated Subqueries.
• Do not store comma delimited list in a table.
• Do not store JSON data in TEXT COLUMN, you should use JSON data type for it.
• Please do not use comma separated table names in SELECT query, use explicit JOINs conditions to avoid internal Cartesian products.

 
The list is not final yet, you guys can add/share a comment and make it final…

In online forums, I am observing one bad query practice. People are using multiple tables in the SELECT query by separating using commas.
Bad Query Like:

SELECT * FROM a, b, c WHERE a.id = b.id AND b.id = c.id;

Many of the Database Professionals are thinking that It is working like JOINs.

I am writing this article in the “Database Designing” category, means It applies to all database professionals for knowing the truth of this kind of bad SQL Query.

When you write a query like above sample, internally Query Planner creates a virtual join between these tables and this is not a strait forward joins creation. A Query Planner has to check all possible combinations like: first join A TO B and then join C, join A TO C and then join B, join B TO C and then join A.

When you write such types of bad queries, Query Planner has to do additional work for joining the tables correctly and return the correct result.
But actually It checks the data by creating a full Cartesian Product between these tables which is very costly and slow down the query performance.

We have taken example of three tables only, now let’s assume if we have 10 tables and people wrote a SELECT query using comma separated tables.
So please I am requesting, never use such a bad query and try to write explicit JOINs in the SELECT Query.

Another interesting point is:

When I discussed this point with our team, one of database developers said It is good that Query Planner is deciding the order of JOINS. If we give explicit joins, It may not be in the correct order all the times so let’s Query Planner to do.

My simple answer was:
I am 5% agree with you, but do not forget that we are enough professional and can define the JOINs order correctly. We should not rely on Query Planner because It requires more internal I/O to just prepare a joins of the table.

Correct Query should like:

SELECT a.* FROM a JOIN b ON a.id=b.id JOIN c ON b.id=c.id

In the previous article, I discussed the 8KB data page of the SQL Server, but what happened when the row is larger and cannot fit into a single 8KB page?

Each row is strictly prohibited to use only 8KB size per data page, but the data type like char, varchar, nvarchar, and varbinary require more space to store data into data page.

Sometimes, the individual column may fit in the limit of 8KB, but if we have multiple columns and the combination of all these types of columns exceeds the limit of 8KB, SQL Server moves data in different allocation unit or raise an error.

e.g. we have three separate columns like VARCHAR(MAX), CHAR(6000) and TEXT so combination of these all columns cannot fit in the 8KB size.

The SQL Server Database Engine moves the record column with the largest width to another page in the ROW_OVERFLOW_DATA allocation unit, and it is maintaining a 24-byte pointer on the original page.

To moving of the large records to another page is a dynamic process because records can become a larger at any point of time while executing UPDATE operations.

If the UPDATE operation makes a small size of the record, it backs to the original page in the IN_ROW_DATA allocation unit.

Whenever we are combining VARCHAR, NVARCHAR, BLOB, VARBINARY kind of data column in a table, internally SQL Server separates the data pages because it may exceed the 8KB row size.

My suggestion is: if we have large size of data columns, we should create a separate table for those columns. Because asynchronously, we can perform JOIN operations, which is performing better than selecting data from internal fragmented pages.

What should be our precautions and strategies to avoid a Deadlock in a Database System.

We should use clustered index maximum for all types of transaction. The clustered key stored in the same data page so whenever we are updating or deleting data, it saves the data searching time.
You should always write UPDATE and DELETE statement with the filter of a clustered key.

The RDBMS systems provide different types of isolation level, you should write your transactions under the proper isolation level.
If your system is ok with dirty read, you can SELECT uncommitted data which never block another transaction. You can also use MVCC architecture for this.

You should avoid high row count SQL statements that can cause a table lock. For example, instead of inserting all rows from one table to another at a time, do the batchwise insertion.

You should break the long transaction into small parts and arrange the proper sequence for that sorter transaction and also write properly COMMIT and ROLLBACK statement.

You should try to avoid unnecessarily hidden triggers which are manipulating transaction for another table. Many times, I found that Deadlock occurs because of trigger hold a lock on another table.

You should avoid SELECT * and specify the only require column list. You can visit this article for more detail. “SELECT all columns to be good or bad in database system”

Should we store comma – separated list in a database?
My suggestion and answer are: Never store comma – separated or delimited list in a Database.

I found this discussion in many forums and blogs so, in this post, I would like to share my opinions and views on this topic.

Why we should not store comma – separated or delimited list?

We cannot enforce uniqueness inside the list, e.g. list can be like this: 5, 4, 8, 8, 8, 6.

We cannot ensure right data-type for all the values, e.g. list can be like this: 5, 8, 7, temp, abc, $, 8.

Do not possible to check referential integrity.

It is tough to search any data or value from the list and sometimes we have to use regular expressions to search the string.

We cannot perform any DML operation on a single value from the list without fetching the whole list.

Getting all the records in sorted order is tough.

Counting elements from the list is terrible.

When we are storing integer data into the list, it takes twice the space than binary integers.

We cannot use list value in the join conditions.

When we store the comma separated value, the overall maintenance of this list and data is complicated.

The simplest solution of this problem, is to create a separate table, or you can use data type like XML, JSON.

Should we allow NULL or We should apply NOT NULL ?

As a Database Architect, I would also like to express my opinions and views on this topic.

First single line statement, Should NULL or Should NOT NULL is depending on the situation and importance of data.

NULL means the pointer set to 0x00 in the row header that means no data to access.
Any search query or engine can easily identify a data which has a NULL.

As per ANSI Standard NULL means UNKNOWN value whatever data-type like, integer, varchar or datetime.

I read many articles and found that many database professionals are suggesting that never go with NULL.
Means, you have to enter a value in each column or else they will allow (”) empty string as NULL values.
e.g. In SQL Server if you enter an empty string in datetime column, it inserts the default value like ‘1900-01-01 00:00:00.000’. – Which is wrong.
In this type of column NOT NULL will not work and we should allow NULL.

Another example is, CHAR data-type column. If we apply NOT NULL and someone enters empty string because of NOT NULL, it occupies the same size of bytes like other rows and which is wrong.

We have to identify all columns and take decisions that in which columns we should allow NULL and in which columns we should apply NOT NULL.

The Next is about performance,
IS NULL or IS NOT NULL condition is always faster than any other string operation.

We should also apply NOT NULL constraint on a required column like Password column, but it doesn’t mean that user can enter (”) empty string again.

Many programming languages don’t recognize a NULL value so for that reason, many Database Professionals applied NOT NULL constraint and putting (”)Empty string as a NULL value.
But this is not an enough reason to avoid a NULL because NULL is created for a purpose to express the unknown data which can be valid.

Don’t make all columns nullable, and Don’t make them all NOT NULL. Use NULL only when necessary where the unknown is a valid condition. Everywhere else, use NOT NULL.