Like SQL Server, PostgreSQL data page size is also 8KB.

This post is very similar to a few of the SQL Server published articles.

SQL Server: The Internal Structure of the Data Page

SQL Server: What happened, when Row Overflow and Data Exceeding 8KB?

PostgreSQL uses 8kb fixed size data page to store tuples of Table.

It does not allow tuples to span in multiple pages because not possible to store large field value directly into the 8kb page.
This is one type of limitation of PostgreSQL.

What is TOAST (The Oversized-Attribute Storage Technique)?

The TOAST is one type of Technique to store and manage oversized or overflow data pages.

This is a background process of PostgreSQL, and It also compresses the value and broken into multiple physical pages.

If row size is exceeding in the main table, It creates new a TOAST table to manage and store oversized rows.

The mechanism is accomplished by splitting up the large column entry into 2KB bytes and storing them as chunks in the TOAST tables.
It then stores the length and a pointer to the TOAST entry back where the column stored.

Because of how the pointer system implemented, most TOAST table column types limited to a max size of 1GB.

The advantage of TOAST table is, whenever data require fetching from TOAST table, only It needs to refer TOAST table otherwise, It does not call unnecessary.

How to find whether a TOAST table is associated with your main tables?

You can easily select reltoastrelid from pg_class. Like any other table ObjectID is also assigned to TOAST Table.

You can execute below script:

SELECT relname,reltoastrelid 
FROM pg_class 
WHERE relname='Table_Name';

Tell me something about the SQL Server Pages.

In SQL Server, the page size is 8 KB. This means SQL Server databases have 128 pages per megabyte. Each page begins with a 96-byte header that is used to store system information about the page. This information includes the page number, page type, the amount of free space on the page, and the allocation unit ID of the object that owns the page.

What is a row offset of pages?

For each data row there is a 2-byte entry, saving the offset.

A row offset table starts at the end of the page, and each row offset table contains one entry for each row on the page.

What is the ROW_OVERFLOW_DATA allocation unit?

When the total row size of all fixed and variable columns in a table exceeds the 8,060 byte limitation, SQL Server dynamically moves one or more variable length columns to pages in the ROW_OVERFLOW_DATA allocation unit, starting with the column with the largest width.

What is a IN_ROW_DATA allocation unit?

It is an allocation unit for data page and index page.

What is a LOB_DATA allocation unit?

It is an allocation unit for text, ntext, image, xml, varchar (max), nvarchar (max), varbinary (max) data types.

Why 24-byte pointer is require with the ROW_OVERFLOW_DATA allocation unit?

When a combination of varchar, nvarchar, varbinary, sql_variant, or CLR user-defined type columns exceeds this 8KB limit, the Database Engine moves the record column with the largest width to another page in the ROW_OVERFLOW_DATA allocation unit, while maintaining a 24-byte pointer on the original page.

What is the size of the data page header?

The size of the data page header is 96-byte.

What are the different types of pages and what do they store?

• Page types include
• Data page
• Index page
• Text/Image page
• Page Free Space
• Global Allocation Map (GAM) – extent allocation information
• Shared Global Allocation Map (SGAM) – mixed extent with at least one unused page information
• Bulk Changed Map – extents that got changed by a bulk logged operation
• Differential Changed Map – extents that got changed since last BACKUP DATABASE statement
• Index allocation map

What is an Extent?

Extents are the basic unit in which space is managed. An extent is eight physically contiguous pages, or 64 KB. This means SQL Server databases have 16 extents per megabyte.

What are the different types of Extents?

• Uniform extents are owned by a single object; all eight pages in the extent can only be used by the owning object.
• Mixed extents are shared by up to eight objects. Each of the eight pages in the extent can be owned by a different object.

What are the different types of database files in SQL Server?

• Primary data files: The primary data file is the starting point of the database and points to the other files in the database. Every database has one primary data file. The recommended file name extension for primary data files is .mdf.
• Secondary data files: Secondary data files make up all the data files, other than the primary data file. Some databases may not have any secondary data files, while others have several secondary data files. The recommended file name extension for secondary data files is .ndf.
• Log files: Log files hold all the log information that is used to recover the database. There must be at least one log file for each database, although there can be more than one. The recommended file name extension for log files is .ldf.

Explained something about SQL Server Access Methods.

• SQL doesn’t directly retrieve from disk, it makes requests to buffer manager which serves up the page in cache before rendering out.
• When scan starts, SQL Server can use look-ahead mechanisms to qualify rows and index entries on a page. The retrieving of rows that meet specified criteria is known as a qualified retrieval.

Tell us something about the SQL Server Database Transactions.

• Provides support for Atomicity, Consistency, Isolation and Durability.
• Log records are always written to disk before the data pages where changes were made are actually written.
• Writes to the transaction log are synchronous while writes to data pages can be asynchronous.

What is the sparse file?

A database snapshot that is created by a user stores its data in one or more sparse files. Sparse file technology is a feature of the NTFS file system.

What is the Database Filegroups?

Database objects and files can be grouped together in filegroups for allocation and administration purposes.

What are the different type of Filegroups?

• Primary: The primary filegroup contains the primary data file and any other files not specifically assigned to another filegroup.
• User-defined: User-defined filegroups are any filegroups that are specified by using the FILEGROUP keyword in a CREATE DATABASE or ALTER DATABASE statement.

What are Page Splits?

When there is not enough room on a page for a new row, a server splits the page, allocates a new page, and moves some rows to the new page.

Page splits are considered as a very bad performance and there are number of techniques which is used to reduce the risk of page splits.

In this post, I am sharing about the effective storage of IP-Address into Database System.

Now let’s talk about four part of IP-Address.

E.g, 152.145.10.25

Many database developers or database designer store this IP-Address into one column which is VARCHAR(15).

What do you think? Is this effective storage of IP-Address?.

Now imagine that you are playing with 10,0,00,00 online customers in which all this IP-Address of the customer requires for analysis.

If we stored all IP-Addresses into one column, you have to perform string manipulation for finding a particular range of IP-Address.
The string operation is always costly and not advisable.

Effective storage of IP-Address is very simple:

Just divide your full-length IP-Address into four part.

E.g.

Full IP-Address: 152.145.10.25

Four Part :

152

145

10

25

Now create four tinyint/smallint column to store this four different part.

column like,

ip1 tinyint

ip2 tinyint

ip3 tinyint

ip4 tinyint

The full IP-Address split into four different integer part.

Now the task is very easy:
You can also apply an index on all this part and perform range analysis very quickly.

Yes, I agree that during insertion you also require splitting for storing in the different columns, but this is only a one time process after that you can easily use different parts for various analysis.

During select, you can also concat this four part and make full IP-Address for display purpose.

Problem :

Email Address is a ubiquitous field and column for every application or system. As per the application size and scope database system stores lots of user Email Addresses.

E.g. 10,00,000 email address is stored in a database system and it is also increasing day by day.
Usually, any database user saves all this email address into one column. Most probably there is one column name is EmailAddress Varchar(250) and user store email address like ” myprofile@yahoo.co.in ” into this column. – Great and usually 70% to 75% user doing this.

Once database size and number of users are going to increase day by day at that time, this storage will not perform as per expectation and decrease the performance of the database.

How this will decrease the performance of the database?

• As we assumed that email address stored into one column like ” myprofile@yahoo.co.in”.
• Now, we require to find that, how many total numbers of users from only “Yahoo” domain?
• How many total number of users from only “India Yahoo”?
• Require to set some additional offer only for “Gmail” domain.

Friends, now think about above questions.

In my career path I have also stored email address into one column, but gradually I came to know that this is a bad idea because now a day Email address is not only for user name or only for storage purpose.
This is very important for promotional and marketing activity.

When you have stored in one column and let’s imagine, you want to find only ” Gmail ” domain out of 10,0,00,000 email address.

As a database developer, you can think that you require one string operation on all stored email address in which you will find the part which is after @”. But this is not a solution where string searching and parsing is required, which slows down the performance of relevant select statement.

Solution :
The solution is simple,

Just divide your Email address into two part or three part.

E.g.

• Single Part : ” myprofile@yahoo.co.in “.
• Two Part : ” myprofile” and ” yahoo.co.in “.
• Three Part : “myprofile” and ” yahoo ” and ” co.in “.

Now, I am going to explain this using one MySQL Demo.

I have created above two table into MySQL. I have divided Email Address into three part.
Below is a sample data.
tbl_User table Data:
tbl_Email table Data:

Examine the first email address:

User Anvesh has anvesh@gmail.com, Roy has roy@yahoo.co.in
As you can find the EmailDomainID reference column in, a User master table which is indicating that which email address is associated with User.
Now a thing is going to easy:
How ?

If I require finding how many users with Gmail, I need to find the only EmailDomainID=1.

You can also apply easy extension filter like Yahoo from co.in and .com.
This will give you 80% to 85% better performance than to store Email into one column.

Also, This will reduce the size of data, E.g, if we store ” Gmail ” keyword 10,00,000 times into the database, then it consume more space than to store only integer ” 1 “.

Yes, I also agree that, during every insert, you have to split email address for storing purpose, but this is a good instead of parsing string to count only domain name.

This structure is quite complex to manage at every insert and every email existence check, but it gives better performance.

Below is simple string operation.

Full Email Address is : anvesh@yahoo.co.in

Now parse only ” anvesh ” from anvesh@yahoo.co.in

SELECT SUBSTRING
   ('anvesh@yahoo.CO.IN',1,
      (POSITION('@' IN 'anvesh@yahoo.CO.IN') -1)
   )

Now parse only ” yahoo ” from anvesh@yahoo.co.in

SELECT SUBSTRING(
 SUBSTRING('anvesh@yahoo.CO.IN'
                ,(POSITION('@' IN 'anvesh@yahoo.CO.IN') + 1))
                ,1
                ,POSITION('.' IN SUBSTRING('anvesh@yahoo.CO.IN'
                      ,(POSITION('@' IN 'anvesh@yahoo.CO.IN')))
                         )-2
          )

Now parse only ” co.in ” from anvesh@yahoo.co.in

SELECT SUBSTRING((SUBSTRING('anvesh@yahoo.CO.IN'
      ,(POSITION('@' IN 'anvesh@yahoo.CO.IN') + 1)))
      ,POSITION('.' IN SUBSTRING('anvesh@yahoo.CO.IN'
      ,(POSITION('@' IN 'anvesh@yahoo.CO.IN') + 1)))+1)

The above code is required whenever you insert the email address into the database because from the application end we receive one full email address, but for effective storage, you have to parse like above code.

You can also suggest three different input parameters from the application end where above all codes can shift to at application level.

Click here to find details about string functions of MySQL.