In this post, I am sharing one script to find the size of Indexes in SQL Server.

The first step for performance optimization is to measure the size of tables and indexes which helps you to find out more about fragmentation.

I have already shared a couple of articles on SQL Server Indexes like: find duplicate indexes, unused indexes and fragmentation of indexes.

You can also access these articles after below T-SQL.

SELECT
	OBJECT_SCHEMA_NAME(i.OBJECT_ID) AS SchemaName
	,OBJECT_NAME(i.OBJECT_ID) AS TableName
	,i.name AS IndexName
	,8 * SUM(au.used_pages) AS IndexSizeInKB
FROM sys.indexes AS i
JOIN sys.partitions AS p 
	ON p.OBJECT_ID = i.OBJECT_ID 
		AND p.index_id = i.index_id
JOIN sys.allocation_units AS au 
	ON au.container_id = p.partition_id
GROUP BY 
	i.OBJECT_ID
	,i.index_id
	,i.name
ORDER BY OBJECT_NAME(i.OBJECT_ID),i.index_id

Other related articles:

SQL Server: Find the total row count and size of the Tables

SQL Server: Script to find Redundant and Duplicate Indexes

SQL Server: Script to find usage of Database Indexes

SQL Server: Script to find Index Average Fragmentation in Percentage

SQL Server 2014 introduced sys.column_store_row_groups system view for finding the information on the Columnstore row groups.

Using this system view, you can find information like total number of rows physically stored, and a column for the number of rows marked as deleted.

If you find a high volume of deleted rows, you should decide to rebuild it.

T-SQL Script:

SELECT
	OBJECT_NAME([csrg].[object_id]) AS TableName
	,[i].[name] AS IndexName
	,[csrg].[row_group_id]
	,[csrg].[delta_store_hobt_id]
	,[csrg].[state_description]
	,[csrg].[total_rows]
	,[csrg].[deleted_rows]
	,[csrg].[size_in_bytes]
FROM sys.column_store_row_groups AS csrg
LEFT OUTER JOIN sys.indexes AS i
	ON [csrg].[object_id] = [i].[object_id]
		AND [csrg].[index_id] = [i].[index_id]
ORDER BY [TableName],[IndexName]
GO

In the previous two posts, I shared basic about Forwarding Records of Heap Table and shared a script to find the count of Forwarded Records in SQL Server.

In this post, I am sharing a practical demonstration on Forwarding Records like generate Forwarding Records and then Remove those Forwarding Records.

Please check the below full demonstration:

Create a sample HEAP table:

CREATE TABLE tbl_Heap
(
	Col1 INT IDENTITY(1,1)
	,Col2 VARCHAR(1000)	
)
GO

Insert sample records with VARCHAR(5) size:

INSERT INTO tbl_Heap (Col2) VALUES
('dbrnd')
GO 1000

Now, Update the data from VARCHAR(5) to VARCHAR(31):

UPDATE tbl_Heap
SET Col2 = 'Database Research & Development'
GO

Check the count of Forwarding Records:

For removing Forwarding Records, we should REBUILD the table:

ALTER TABLE tbl_Heap REBUILD
GO

Now, Check the new count of Forwarding Records:
The count is 0 now.

In the previous post, I shared basic about Forwarded Records in Heap table of SQL Server.
In this post, I am sharing a script to check the count of Forwarded Records of a Table.

In the below script, you can add a filter of your database and table for getting the number of Forwarding Records.

SELECT
	OBJECT_NAME(object_id) as TableName
	,index_type_desc
	,page_count
	,avg_page_space_used_in_percent
	,avg_record_size_in_bytes
	,forwarded_record_count
FROM sys.dm_db_index_physical_stats
(
	DB_ID('database_name')
	,OBJECT_ID('table_name')
	,NULL
	,NULL
	,'DETAILED'
)
GO

In this post, I am sharing an important note on Forwarded Records in the Heap table of SQL Server. Whenever DBA or Developer is facing performance issue, they are always talking about the table or index fragmentation, but no-one is discussing the Forwarded Records.

What is heap Table

Simple Example to understand Forwarded Records:

You create a heap table, and one of the column types is VARCHAR(200).
You inserted 100000 records and added a small size of the string in VARCHAR(200) like “dbrnd” value which is only VARCHAR(5).

Now next, you require updating the value from “dbrnd” to “Database Research & Development” which is now VARCHAR(31).
Previously, we stored our data based on VARCHAR(5), but while updating we need VARCHAR(31).

Before the update, all data correctly save in data page and for new update no more free space available on the data page. So SQL Server internally moves new data into the new page and assigned a new Relative Identifier (RID).

Now old page contains forwarding pointer to new RID and new records also back pointer to old records.

– Called as Forwarded Records and which is also one type of internal fragmentation.

The biggest drawback of Forwarded Record is, it requires high I/O because SQL Server has to jump between the old data page and new data page for getting data.

Few more following topics, which I will publish from tomorrow:
How to find the Forwarded Records?
How to remove Forwarded Records?

Few of SQL Server DBAs are frequently executing Shrink on database or data files. Why? because they are gaining more free space after shrink.

You can find N number of theory on, why you should not shrink your database or data file in SQL Server?

Instead of big theory, I am sharing full practical demonstration on this. You should not shrink your database or data files otherwise it generates massive fragmentation for your tables and indexes. Still, you can shrink your log file, but after full backup only.

Create a test database and use it:

CREATE DATABASE Test_Shrink
GO

Use Test_Shrink
GO

Create a sample table:

CREATE DATABASE Test_Shrink
GO

Use Test_Shrink
GO

Create second sample table with clustered index:

CREATE TABLE tbl_test2
(
    id INT IDENTITY(1,1)
    ,tag CHAR (7000) DEFAULT 'dbrnd'
)
GO
CREATE CLUSTERED INDEX pk_test2 ON tbl_test2 (id);
GO

Insert few sample records in both the tables:

INSERT INTO tbl_test1 DEFAULT VALUES;
GO 1500
 
INSERT INTO tbl_test2 DEFAULT VALUES;
GO 1500

Check the average fragmentation of Index:

SELECT
    [avg_fragmentation_in_percent]
FROM sys.dm_db_index_physical_stats 
(
    DB_ID (N'Test_Shrink'), OBJECT_ID (N'tbl_test2'), 1, NULL, 'LIMITED'
)
GO

--The result is:
--0.33%

Drop one sample table to test Shrink:

DROP TABLE tbl_test1;
GO

Shrink your database:

DBCC SHRINKDATABASE (Test_Shrink);
GO

Check the fragmentation of index:
You can find below result like 99.6% fragmentation for Index. Before shrink database, it was just 0.33%.

SELECT
    [avg_fragmentation_in_percent]
FROM sys.dm_db_index_physical_stats 
(
    DB_ID (N'Test_Shrink'), OBJECT_ID (N'tbl_test2'), 1, NULL, 'LIMITED'
)
GO

--The result is:
--99.6%

Now, REBUILD your index to remove the fragmentation:

ALTER INDEX pk_test2 ON tbl_test2 REBUILD
GO

Check the fragmentation of index:

SELECT
    [avg_fragmentation_in_percent]
FROM sys.dm_db_index_physical_stats 
(
    DB_ID (N'Test_Shrink'), OBJECT_ID (N'tbl_test2'), 1, NULL, 'LIMITED'
)
GO

--The result is:
-- 0%

What is HOT UPDATE (Heap Only Tuple) and How It helps to increase the performance of UPDATE Query?
In this post, I am going to explain the answer of this question.

This is one of the internal of PostgreSQL, but I am sure that most of you guys don’t know about PostgreSQL HOT.

PostgreSQL requires HOT because of MVCC architecture.
Please read below articles, before start to knowing about the HOT.

What is MVCC?

What is Fillfactor in PostgreSQL?

What is Vacuum in PostgreSQL?

In the MVCC architecture, When we update any record, internally MVCC architecture inserts a new record with new values and marks old record as deleted or discarded.
When we vacuum, we can clean that deleted records and utilize that space.

What is HOT (Hot Only Tuple)?

Now imagine that, we have total 500 data pages and we are updating record of page number 3, but unfortunately new record insert in page number 500.

It looks like very big internal fragmentation because of MVCC architecture.
But actually this is not happening.

PostgreSQL has a concept of HOT, With a HOT dead tuple space can be automatically reclaimed at the time of INSERT or UPDATE if no changes are made to indexed columns. This allows for more consistent performance.

HOT means, creating a new update tuple if possible on the same page as the old tuple, and maintaining a chain of updated tuples linking a new version to the old tuple.

HOT updates can only reuse dead tuples from previous transactions directly, not from the same or concurrent ones.
If you have big updates, changing large portions of the table at once, you might want to split them up in a couple of chunks.

HOT updates only work when the changed columns are not involved in indexes in any way because the indexes pointing the the old tuples need to point to the new version of it as of transaction id.

You might be blocking HOT updates with indexes on updated columns. If those are expendable, you might get better overall performance without them.

Let’s understand PostgreSQL HOT practically.

Create a table with Sample data:

CREATE TABLE tbl_Employees
(
	EmpID INT
	,EmpName CHARACTER VARYING 	
	,EmpDepartment CHARACTER VARYING
	,EmpSalary INT
);
 
INSERT INTO tbl_Employees
VALUES 
(1,'Anvesh','Database',90000)
,(2,'Jenny','JAVA',65000)
,(3,'Martin','PHP',85000)
,(4,'Roy','PHP',94000)
,(5,'Eric','PHP',70000)
,(6,'Rajesh','Animation',50000)
,(7,'Mahi','Database',40000)
,(8,'Sofia','JAVA',60000);

Delete one record:

DELETE FROM tbl_Employees WHERE EmpID = 7;

Insert one New Record:

INSERT INTO tbl_Employees
VALUES 
(7,'Junik','Database',10000);

Please check If any HOT Update or not on table:

SELECT pg_stat_get_tuples_hot_updated('tbl_employees'::regclass::oid);
/*
Result:
pg_stat_get_tuples_hot_updated
------------------------------
0
*/

Please update the statestic of that table:

ANALYZE tbl_Employees;

Now execute one more update command:

UPDATE tbl_Employees SET EmpName = 'Robort' WHERE EmpID=7;

Please check If any HOT Update or not on table:

SELECT pg_stat_get_tuples_hot_updated('tbl_employees'::regclass::oid);
/*
Result:
pg_stat_get_tuples_hot_updated
------------------------------
1
*/

Again, do more update:

UPDATE tbl_Employees SET EmpName = 'Manish' WHERE EmpID=8;
UPDATE tbl_Employees SET EmpName = 'Cerlin' WHERE EmpID=4;

Please check If any change in number of HOT Update of a table:

SELECT pg_stat_get_tuples_hot_updated('tbl_employees'::regclass::oid);
/*
Result:
pg_stat_get_tuples_hot_updated
------------------------------
3
*/

Now Apply, Index on EmpName Column:

CREATE INDEX idx_tbl_employees_empname 
ON tbl_employees USING BTREE (EmpName);

After Index, execute few more update:

UPDATE tbl_Employees SET EmpName = 'Kulin' WHERE EmpID=3;
UPDATE tbl_Employees SET EmpName = 'Priya' WHERE EmpID=5;

After Index, Please check If any change in number of HOT Update of a table:
The result is same because If column has an Index, HOT Update does not work.

SELECT pg_stat_get_tuples_hot_updated('tbl_employees'::regclass::oid);
/*
Result:
pg_stat_get_tuples_hot_updated
------------------------------
3
*/

You can also check number of HOT update of current transaction:

SELECT pg_stat_get_xact_tuples_hot_updated('tbl_employees'::regclass::oid);

This is very important task for PostgreSQL DBA to check the fragmentation level of Table.

Please do not forget about that the PostgreSQL is based on MVCC architecture. Which is good in one way, but bad in another way.

Using below article, You must read about MVCC and also access other related articles.

What is Multi Version Concurrency Control (MVCC)

PostgreSQL: Short note on VACUUM, VACUUM FULL and ANALYZE

PostgreSQL: Script to find total Live Tuples and Dead Tuples (Row) of a Table

PostgreSQL provides pgstattuple module to get all tuples information of a Table. You can find information like live_tuple, dead_tuple, free_space and other.

Using this information you can find fragmentation of table which you can remove using VACUUM / VACUUM FULL command.

You must install the pgstattuple to find tuples related information. You can also use pgstatindex() to find information related to indexes.

Load pgstattuple module:

CREATE EXTENSION pgstattuple;

(
     TranID SERIAL
     ,TransactionDate TIMESTAMPTZ
     ,TransactionName TEXT
 );

INSERT INTO tbl_ItemTransactions 
(TransactionDate, TransactionName)
SELECT x, 'dbrnd' 
FROM generate_series('2014-01-01 00:00:00'::timestamptz, '2016-08-01 00:00:00'::timestamptz,'2 seconds'::interval) a(x);

SELECT *FROM pgstattuple('public.tbl_itemtransactions');

table_len          | 2125627392
tuple_count        | 40737601
tuple_len          | 1873929646
tuple_percent      | 88.16
dead_tuple_count   | 45
dead_tuple_len     | 7787
dead_tuple_percent | 0.88
free_space         | 9923
free_percent       | 1.96

I have received a few emails related to PostgreSQL default cluster index and I have received from the developers of Microsoft SQL Server.
If you are Microsoft SQL Server developer, you know that Primary key of the Table is working as Cluster Index and resides in the same data page where data actually stored.

In PostgreSQL, There is no concept like: Table Primary Key means default Cluster Index of that table.
In PostgreSQL, we have one CLUSTER command which is similar to Cluster Index.

Once you create your table primary key or any other Index, you can execute the CLUSTER command by specifying that Index name to achieve the physical order of the Table Data.

CLUSTER instructs PostgreSQL to cluster the table specified by table_name based on the index specified by index_name. The index must already have been defined on table_name.

A clustering on an index forces the physical ordering of the data and you can have only one clustered index per table and should carefully pick which index you will use to cluster.

When a table is clustered, it is physically reordered based on the index information. Clustering is a one-time operation: when the table is subsequently updated, the changes are not clustered. That is, no attempt is made to store new or updated rows according to their index order.

If you wish, you can periodically recluster by issuing the CLUSTER command again and again.

A executing of CLUSTER command requires an ACCESS EXCLUSIVE lock on the table because It reorders your data. You can also get some free space by issuing CLUSTER because It rearranges the data in a particular order.

When you execute the CLUSTER command, It creates a temporary copy of the table therefore, you need free space on disk at least equal to the original table size.

When an index scan is used, a temporary copy of the table is created that contains the table data in the index order. Temporary copies of each index on the table are created as well. Therefore, you need free space on disk at least equal to the sum of the table size and the index sizes.

It is advisable to set maintenance_work_mem to a reasonably large value before clustering.

Once you are done with your CLUSTER, you should execute the ANALYZE command on that table because planner records statistics about the ordering of tables.

Syntax of Cluster:
First time you must execute CLUSTER using the Index Name.

CLUSTER table_name USING index_name;

Cluster the table:
Once you have executed CLUSTER with Index, next time you should execute only CLUSTER TABLE because It knows that which index already defined as CLUSTER.

CLUSTER table_name;

Cluster all tables of database:

CLUSTER;

In this post, I am sharing SQL Server Database Administrator script for rebuilding all Indexes of a Database.

To Rebuild or Reorganize the Indexes is a very common task for DBA.
If Index fragmentation is increasing, periodically it requires executing Rebuild Index for better performance.

If we want to Rebuild all Indexes of a Database, we should build and generate the script because using default command we can only perform Rebuild operation on table level.

SQL Server: Script to find Index Average Fragmentation in Percentage

In below script, I commented Fill Factor related changes, and If anyone wants, they uncomment it.
Please SELECT your database and execute the below script.

Note: Database REBUILD INDEX is a very costly operation and It requires an exclusive lock on table so please aware yourself before executing it.

DECLARE @DatabaseName SYSNAME   = DB_NAME()
DECLARE @TableName VARCHAR(256) 
--DECLARE @FILLFACTOR INT = 85
DECLARE @SQL NVARCHAR(MAX) =

 'DECLARE curAllIndex CURSOR FOR SELECT TABLE_SCHEMA +
 ''.'' + TABLE_NAME AS TABLENAME  
 FROM ' + @DatabaseName + '.INFORMATION_SCHEMA.TABLES WHERE
 TABLE_TYPE = ''BASE TABLE'''  
 
BEGIN 
  EXEC sp_executeSQL @SQL  
  OPEN curAllIndex
  FETCH NEXT FROM curAllIndex INTO @TableName  
  WHILE (@@FETCH_STATUS = 0) 
  BEGIN  
       /* -- For using FillFactor setting. 
	   SET @DynamicSQL = 'ALTER INDEX ALL ON ' + @TableName +  
	   ' REBUILD WITH (FILLFACTOR = ' + CONVERT(VARCHAR,@FILLFACTOR) + ')' 
	   */
	   SET @SQL = 'ALTER INDEX ALL ON ' + @TableName +  
	   ' REBUILD ' 
       PRINT @SQL       
       EXEC sp_executeSQL @SQL 
       FETCH NEXT FROM curAllIndex INTO @TableName  
   END   
   CLOSE curAllIndex  
   DEALLOCATE curAllIndex 
END