I am also adding a Database Design category for this post, because what I am feeling that you guys must know about this before designing tables or indexes.

You can create Indexes on VARCHAR columns, but you should not create indexes on big size of VARCHAR Columns.

Indexes also require storing data as key for accessing associated records. There is a size limitation of index key values.

Before SQL Server 2016, Index key size limitation was 900 bytes for Clustered and Non-Clustered index column.
SQL Server 2016 increased this limitation for Non-Clustered Index and now it is up to 1700 bytes, but for Clustered Index column it is still 900 bytes.

Let me demonstrate this.

Create a sample table with Nonclustered Index:

CREATE TABLE tbl_TestVarcharNonClustered
(
	ID INT
	,Name VARCHAR(2000)
)
GO

CREATE NONCLUSTERED INDEX idx_tbl_TestVarcharNonClustered_Name ON tbl_TestVarcharNonClustered(Name)
GO

You will get a below warning:
Because we have set Name VARCHAR(2000).

Warning! The maximum key length for a nonclustered index is 1700 bytes. The index 'idx_tbl_TestVarcharNonClustered_Name' has maximum length of 2000 bytes. For some combination of large values, the insert/update operation will fail.

Try to insert 1701 bytes data:

INSERT INTO tbl_TestVarcharNonClustered 
VALUES (1,REPLICATE('a',1701))
GO

You will get a below error:

Msg 1946, Level 16, State 3, Line 1
Operation failed. The index entry of length 1701 bytes for the index 'idx_tbl_TestVarcharNonClustered_Name' exceeds the maximum length of 1700 bytes for nonclustered indexes.

You can successfully run below INSERT, because it inserts only 1700 bytes:

INSERT INTO tbl_TestVarcharNonClustered 
VALUES (1,REPLICATE('a',1700))
GO

Now, create a sample table with Clustered Index:

CREATE TABLE tbl_TestVarcharClustered
(
	ID INT
	,Name VARCHAR(2000)
)
GO

CREATE CLUSTERED INDEX idx_tbl_TestVarcharClustered_Name ON tbl_TestVarcharClustered(Name)
GO

You will get a below warning:
Because we have set Name VARCHAR(2000) on cluster index column.

Warning! The maximum key length for a clustered index is 900 bytes. The index 'idx_tbl_TestVarcharClustered_Name' has maximum length of 2000 bytes. For some combination of large values, the insert/update operation will fail.

Try to insert a over size data like more than 900 bytes:

INSERT INTO tbl_TestVarcharClustered 
VALUES (1,REPLICATE('a',1001))
GO

You will get a below error:

Msg 1946, Level 16, State 3, Line 1
Operation failed. The index entry of length 1001 bytes for the index 'idx_tbl_TestVarcharClustered_Name' exceeds the maximum length of 900 bytes for clustered indexes.

You can successfully run below INSERT, because it inserts only 900 bytes:

INSERT INTO tbl_TestVarcharClustered 
VALUES (1,REPLICATE('a',900))
GO

In this post, I am sharing the list of common practices for Database Designing which should follow by Database Developers and Application Developers.

• Foreign keys should have indexes on them, and it should often cover multiple columns base on the queries you need to execute.
• You should make sure about the referential integrity by enforcing the Foreign Keys.
  Choose right column for CLUSTER Index.
• You should apply proper indexes and do not create duplicate indexes and avoid creation of unnecessary indexes.
• You should not put auto-number data type as a PRIMARY KEY to join a FOREIGN KEY in a parent/child table relationship.
• If your system is running large number of INSERT/DELETE, you should avoid frequent update of database Statistics.
• You should choose the write data-type for your data columns.
• Please separate big size of columns from main table and store in a different table.
• You should apply proper column name.
• Use proper data types for Latitude, Longitude, IP-Address, Network Address, Mac Address.
• Use Natural Primary key rather than Surrogate Primary key. Natural Primary keys are keys base on an external meaningful data. Surrogate Primary key are keys base on an auto-increment number or values which have no meaning outside the system.
• You must use Stored Procedures or Stored Functions.
• Please avoid too much database normalization and try to design your database to achieve Database Normalization + Database Denormalization. You should design this combination to achieve the performance.
• Avoid inappropriate or excessive use of cursors.
• Avoid inappropriate use of Database Transactions (BEGIN…COMMIT).
• Try to avoid the use of OR condition because an AND condition restricts the data set, whereas an OR condition grows it.
• You should do detailed performance analysis on queries.
• Try to use the Prepared SQL Statement which caches the compiled query and the execution plan for better performance. It will also better protect you against SQL injection attacks.
• Please do not simplify your complex queries through the views.
• Please do not use DISTINCT frequently. Many developers are using DISTINCT to hide duplicate data, but you should stop and remove the duplicate data.
• Try to avoid Correlated Subqueries.
• Do not store comma delimited list in a table.
• Do not store JSON data in TEXT COLUMN, you should use JSON data type for it.
• Please do not use comma separated table names in SELECT query, use explicit JOINs conditions to avoid internal Cartesian products.

 
The list is not final yet, you guys can add/share a comment and make it final…

As a Database Architecture, this is my responsibility to write and share my knowledge on which one is best option to store images either into the Database System or into the File System?

I found this is most asking questions by Database Professional.
Let me discuss the all truth about these both option so Database Professional can take the right decision.

” The truth about to store images into the File System “

Store image into the File System and store the only path of that image into the Database System.

You can serve this images or pictures as static files across the network.

The Operating System can easily cache the image data for better access performance.

The management of files in the File System is very important, You have to manage proper folder structure, different schedule for taking the backup of all files.

If you are planning to migrate your application from one server to another at that time you also have to take the backup of not only image files, but you have to also maintain the same folder structure and path structure.

This is not advisable to store more than 3000 files into one folder, after some number of limit you should create a new folder for further files and this is advisable for performance aspect.

You can write an automated program to create the new folder structure.

The thumb rule of RDBMS is data integrity, If you are storing images in the file system, the image data disconnects with a database system.

You should also take care about a security of file directory, if someone deletes your image files, your application should throw an error like “Image is not found”.

” The truth about to store images into the Database System “

You can create a separate table or column for BLOB data type in which you can store images.

You can manage proper data integrity within the database system.

You need only one system to manage and perform any action like migration, replication.

You can take a full database backup with included all the image data, but this will take more time depends on the size of image data.

You can also commit image data like any other data.

The storing images into database system are very resource consuming.

Imagine, If you stored millions of images in the database and each size is around 60 KB, you have to use your SELECT statement properly otherwise this will create performance issues. You cannot put SELECT * in such a condition.

The image data is more secure in the database system because you can easily manage different privileges for users.

You can also create a dedicated server, database, schema for image data just like any other file server system.

If you are using SQL Server 2008, one new FileStream data type gives a very good optimized approach for storing files in a database system.

” Conclusion… “

You should take decision basis on your system complexity and scalability. I would suggest you go with File System. The Database Storage is not advisable, but still, people are using it.

Above are my views and concerns, but you should take a decision base on all the above facts.

In this post, I am sharing about the effective storage of IP-Address into Database System.

Now let’s talk about four part of IP-Address.

E.g, 152.145.10.25

Many database developers or database designer store this IP-Address into one column which is VARCHAR(15).

What do you think? Is this effective storage of IP-Address?.

Now imagine that you are playing with 10,0,00,00 online customers in which all this IP-Address of the customer requires for analysis.

If we stored all IP-Addresses into one column, you have to perform string manipulation for finding a particular range of IP-Address.
The string operation is always costly and not advisable.

Effective storage of IP-Address is very simple:

Just divide your full-length IP-Address into four part.

E.g.

Full IP-Address: 152.145.10.25

Four Part :

152

145

10

25

Now create four tinyint/smallint column to store this four different part.

column like,

ip1 tinyint

ip2 tinyint

ip3 tinyint

ip4 tinyint

The full IP-Address split into four different integer part.

Now the task is very easy:
You can also apply an index on all this part and perform range analysis very quickly.

Yes, I agree that during insertion you also require splitting for storing in the different columns, but this is only a one time process after that you can easily use different parts for various analysis.

During select, you can also concat this four part and make full IP-Address for display purpose.

Problem :

Email Address is a ubiquitous field and column for every application or system. As per the application size and scope database system stores lots of user Email Addresses.

E.g. 10,00,000 email address is stored in a database system and it is also increasing day by day.
Usually, any database user saves all this email address into one column. Most probably there is one column name is EmailAddress Varchar(250) and user store email address like ” myprofile@yahoo.co.in ” into this column. – Great and usually 70% to 75% user doing this.

Once database size and number of users are going to increase day by day at that time, this storage will not perform as per expectation and decrease the performance of the database.

How this will decrease the performance of the database?

• As we assumed that email address stored into one column like ” myprofile@yahoo.co.in”.
• Now, we require to find that, how many total numbers of users from only “Yahoo” domain?
• How many total number of users from only “India Yahoo”?
• Require to set some additional offer only for “Gmail” domain.

Friends, now think about above questions.

In my career path I have also stored email address into one column, but gradually I came to know that this is a bad idea because now a day Email address is not only for user name or only for storage purpose.
This is very important for promotional and marketing activity.

When you have stored in one column and let’s imagine, you want to find only ” Gmail ” domain out of 10,0,00,000 email address.

As a database developer, you can think that you require one string operation on all stored email address in which you will find the part which is after @”. But this is not a solution where string searching and parsing is required, which slows down the performance of relevant select statement.

Solution :
The solution is simple,

Just divide your Email address into two part or three part.

E.g.

• Single Part : ” myprofile@yahoo.co.in “.
• Two Part : ” myprofile” and ” yahoo.co.in “.
• Three Part : “myprofile” and ” yahoo ” and ” co.in “.

Now, I am going to explain this using one MySQL Demo.

I have created above two table into MySQL. I have divided Email Address into three part.
Below is a sample data.
tbl_User table Data:
tbl_Email table Data:

Examine the first email address:

User Anvesh has anvesh@gmail.com, Roy has roy@yahoo.co.in
As you can find the EmailDomainID reference column in, a User master table which is indicating that which email address is associated with User.
Now a thing is going to easy:
How ?

If I require finding how many users with Gmail, I need to find the only EmailDomainID=1.

You can also apply easy extension filter like Yahoo from co.in and .com.
This will give you 80% to 85% better performance than to store Email into one column.

Also, This will reduce the size of data, E.g, if we store ” Gmail ” keyword 10,00,000 times into the database, then it consume more space than to store only integer ” 1 “.

Yes, I also agree that, during every insert, you have to split email address for storing purpose, but this is a good instead of parsing string to count only domain name.

This structure is quite complex to manage at every insert and every email existence check, but it gives better performance.

Below is simple string operation.

Full Email Address is : anvesh@yahoo.co.in

Now parse only ” anvesh ” from anvesh@yahoo.co.in

SELECT SUBSTRING
   ('anvesh@yahoo.CO.IN',1,
      (POSITION('@' IN 'anvesh@yahoo.CO.IN') -1)
   )

Now parse only ” yahoo ” from anvesh@yahoo.co.in

SELECT SUBSTRING(
 SUBSTRING('anvesh@yahoo.CO.IN'
                ,(POSITION('@' IN 'anvesh@yahoo.CO.IN') + 1))
                ,1
                ,POSITION('.' IN SUBSTRING('anvesh@yahoo.CO.IN'
                      ,(POSITION('@' IN 'anvesh@yahoo.CO.IN')))
                         )-2
          )

Now parse only ” co.in ” from anvesh@yahoo.co.in

SELECT SUBSTRING((SUBSTRING('anvesh@yahoo.CO.IN'
      ,(POSITION('@' IN 'anvesh@yahoo.CO.IN') + 1)))
      ,POSITION('.' IN SUBSTRING('anvesh@yahoo.CO.IN'
      ,(POSITION('@' IN 'anvesh@yahoo.CO.IN') + 1)))+1)

The above code is required whenever you insert the email address into the database because from the application end we receive one full email address, but for effective storage, you have to parse like above code.

You can also suggest three different input parameters from the application end where above all codes can shift to at application level.

Click here to find details about string functions of MySQL.