Are you SQL Developer or SQL DBA? Whatever you are, you must know about the SQL Server DMVs (Dynamic Management Views).

Last week, I was taking interview for SQL DBAs. I had asked few questions like:

What do you know about the SQL DMVs?
How can we effectively use it?
Do you know, Where does SQL Server stores the DMVs data?

I had found most of the SQL DBA given the wrong answer for the third question, that is about the storage of DMVs.
Most of the time, I got one type of answer like SQL Server stores DMVs data in master & msdb database.

This is an absolutely wrong answer.

The correct answer is: SQL Server doesn’t store DMV/DMF related data into the physical disk, but it stores directly into the memory. You can read this data directly from the process space of sqlservr.exe.

Because of this reason, most of the DBAs are suggesting that you should not restart the SQL Server. It will drive a SQL Server to poor performance.
Because it clears all query cache, buffer cache and wait type statistics.

This is one of the most important script for SQL Server Database Administrator.
In this post, I am sharing TSQL script to find memory utilization for each SQL Server database.

Generally, The majority percentage of the memory is utilized by the SQL Server buffer pool.

As a Database Administrator, this is our responsibility to find which database is consuming the most buffer pool memory.

We can use sys.dm_os_buffer_descriptors DMV to find information on every cached page of the Buffer pool.

SELECT
	CASE 
		WHEN database_id = 32767 
			THEN 'Resource DB' 
		ELSE DB_NAME (database_id) 
	END AS DatabaseName
	,COUNT (1) AS Buffer_PageCount
	,(COUNT (1) * 8)/1024 AS Used_MemoryInMB 	
FROM sys.dm_os_buffer_descriptors
GROUP BY database_id
ORDER BY db_name(database_id)

What is the max degree of parallelism option?

When SQL Server runs on a computer with more than one microprocessor or CPU, it detects the best degree of parallelism, that is, the number of processors employed to run a single statement, for each parallel plan execution. You can use the max degree of parallelism option to limit the number of processors to use in parallel plan execution.

What is MAXDOP in SQL Server?

Maximum Degree of Parallelism (MAXDOP) is an option in SQL server which enables you to configure SQL Server to use available CPU/CPUs to run a single statement in parallel execution plan. MAXDOP configuration highly depends on SQL Server edition, CPU type and operating system.

In my experience, I had to play with MAXDOP for certain statements coming from different servers in order to find best option of MAXDOP.

What is the MAXDOP query hint option?

You can override the max degree of parallelism value in queries by specifying the MAXDOP query hint in the query statement.

T-SQL script to change max degree of parallelism Option.

Setting maximum degree of parallelism to 0 allows SQL Server to use all the available processors up to 64 processors.
To suppress parallel plan generation, set max degree of parallelism to 1. Set the value to a number greater than 1 to restrict the maximum number of processors used by a single query execution.

sp_configure 'show advanced options', 1;
GO
RECONFIGURE WITH OVERRIDE;
GO
sp_configure 'max degree of parallelism', 8;
GO
RECONFIGURE WITH OVERRIDE;
GO

SQL Server 2014: What is Hekaton?

SQL Server 2014: What is Hekaton?

SQL Server 2014: Create Memory Optimized File Group and Table.

SQL Server 2014: Create Memory Optimized File Group and Table

What is a Read-Ahead mechanism?

The Database Engine supports a performance optimization mechanism called read-ahead. Read-ahead anticipates the data and index pages needed to fulfill a query execution plan and brings the pages into the buffer cache before they are actually used by the query.

What are the types of Read-Ahead?

There are two types of Read-Ahead, one is Reading Data Pages and second is Reading Index Pages.

How internally pages are modified and written to the disk?

First page modification happens in the buffer cache and When a page is modified in the buffer cache, it is not immediately written back to disk;instead, the page is marked as dirty.

This means that a page can have more than one logical write made before it is physically written to disk.

For each logical write, a transaction log record is inserted in the log cache that records the modification. The log records must be written to disk before the associated dirty page is removed from the buffer cache and written to disk. SQL Server uses a technique known as write-ahead logging.

What is Bookmark Lookup?

When a SQL query requests a small number of rows, the optimizer can use the nonclustered index, if available, on the column(s) in the WHERE clause to retrieve the data. If the query refers to columns that are not part of the nonclustered index used to retrieve the data, then navigation is required from the index row to the corresponding data row in the table to access these columns.This operation is called a bookmark lookup.

What is a COLD, DIRTY or CLEAN Buffer?

The cold buffer cache has a couple of variants within SQL Server.    Typically it refers to the buffer pool right after restart.   The data cache is not loaded (cold) and requires physical reads to populate the cache.

Modified buffers that have not been written to disk are termed DIRTY buffers and are typically written to disk by checkpoint processing.

A clean buffer is a data page in memory that is NOT MODIFIED.

What is the use of DBCC DROPCLEANBUFFERS?

DBCC DROPCLEANBUFFERS removes the UNMODIFIED buffers from the buffer pool (no longer hashed) and places them on the free list.

This is often used in basic performance testing to clear the data cache and force the data to be retrieved from disk as a physical read instead of a logic read.

What is affinity I/O mask option?

The affinity I/O mask option binds SQL Server disk I/O to a specified subset of CPUs. In high-end SQL Server online transactional processing (OLTP) environments, this extension can enhance the performance of SQL Server threads issuing I/Os.

What should be the different values for affinity I/O mask?

• A 1-byte affinity I/O mask covers up to 8 CPUs in a multiprocessor computer.
• A 2-byte affinity I/O mask covers up to 16 CPUs in a multiprocessor computer.
• A 3-byte affinity I/O mask covers up to 24 CPUs in a multiprocessor computer.
• A 4-byte affinity I/O mask covers up to 32 CPUs in a multiprocessor computer.
• To cover more than 32 CPUs, configure a four-byte affinity I/O mask for the first 32 CPUs and up to a four-byte affinity64 I/O mask for the remaining CPUs.

What Is Eager Writing in SQL Server?

Eager writing is a long back implemented idea in SQL server to prevent flooding of the buffer pool with pages that are newly created from minimally logged activities, and physically written to disk.

In SQL Server 2014, eager write doesn’t force dirty pages physical write as quickly as previous versions.   This allows dirty pages to spend their life time in RAM, used by SQL and flush them to free list without ever a physical write to disk as long memory is available. This will significantly increase the performance up to 300% as compared to previous versions.

Generally, what kind of operations are recorded in the transaction log file?

• The start and end of each transaction.
• Every data modification (insert, update, or delete).
• Every extent and page allocation or deallocation.
• Creating or dropping a table or index.
• Rollback operations are also logged.

What happened if FILEGROWTH is enable and disable?

• If the FILEGROWTH setting is enabled for the log and space is available on the disk, the file is extended by the amount specified in growth_increment and the new log records are added to the extension.
• If the FILEGROWTH setting is not enabled, or the disk that is holding the log file has less free space than the amount specified in growth_increment, an 9002 error is generated.

Are you suggesting to truncate an old transaction log?

Yes,

If the log records were never deleted from the transaction log, the logical log would grow until it filled all the available space on the disks holding the physical log files. At some point in time, old log records no longer necessary for recovering or restoring a database must be deleted to make way for new log records.

Do you think truncate log operation reduces the size of the physical log file?

Log truncation does not reduce the size of a physical log file, it reduces the size of the logical log and marks as inactive the virtual logs that do not hold any part of the logical log.

A log shrink operation removes enough inactive virtual logs to reduce the log file to the requested size.

How SQL Server virtual address space for the buffer pool?

When SQL Server starts, it computes the size of virtual address space for the buffer pool based on a number of parameters such as amount of physical memory on the system, number of server threads and various startup parameters. SQL Server reserves the computed amount of its process virtual address space for the buffer pool, but it acquires only the required amount of physical memory for the current load.

What is min server memory and max server memory?

The min server memory and max server memory configuration options establish upper and lower limits to the amount of memory used by the buffer pool of the Microsoft SQL Server Database Engine.

buffer pool never drops its memory allocation below the level specified in min server memory, and never acquires more memory than the level specified in max server memory.

An organization requires to add more memory in the running SQL Server so is it require to restart SQL Server service?

No, it is not required to restart a SQL Server because the buffer manager supports Hot Add Memory, which allows users to add physical memory without restarting the server.

What is a dynamic management view to monitor pages in the buffer cache?

You can use the sys.dm_os_buffer_descriptors view to monitor the pages in the buffer cache.

What is a Torn Page Protection?

Torn page protection is primarily a way of detecting page corruptions due to power failures.

It is managing a 2-bit signature is placed at the end of each 512-byte sector at the page and base on this signature value it detects the torn pages.

Why Checksum Protection is a very important?

Checksum protection is a stronger data integrity checking.

A checksum is calculated for the data in each page that is written, and stored in the page header. Whenever a page with a stored checksum is read from disk, the Database Engine recalculates the checksum for the data in the page and raises error 824 if the new checksum is different from the stored checksum.

What is the role of Named Pipes protocol?

Named Pipes is a protocol developed for local area networks. A part of memory is used by one process to pass information to another process, so that the output of one is the input of the other. The second process can be local or remote.

What is a Page Header?

Page number 0- the file header (page type 15). It holds the file header information. There will be only one header page for each data file and that reside in the 0th position. The header page store the file level information like file size, minimum size, max size and file growth.

What is PFS?

PFS page is the second page in the data file followed by file header. PFS pages are used to track page level allocation. PFS page repeat after every 8088 pages.

How many GAM pages will be there in a 7GB data file?

A GAM page can hold information of around 64000 extents. That is, a GAM page can hold information of (64000X8X8)/1024 = 4000 MB approximately. In short, a data file of size 7 GB will have two GAM pages.

How many SGAM pages will be there in a 7GB data file?

A SGAM page can hold information of 64000 extents. That is, a SGAM page can hold information of (64000X8X8)/1024 = 4000 MB. In short, a data file of size 7 GB will have two SGAM page.

How SQL Server engine will decide which extents need to be added into a differential backup?

A differential backup process scans through DCP page to identify the extents which are modified after the last full backup and add those extents in the Differential backup.

What is Boot Page?

There is one special data page that exists only once per database. It is the database boot page. The database boot page is always stored in page 9 of file 1, the first file in the primary file group.

The database boot page contains a broad variety of data about the database itself. Here you can find the internal database version, the compatibility level, the database name and the date of the last log backup.

What is an IAM page?

IAM stands for Index Allocation Map: To catalog all the pages that belong to a single allocation unit, SQL Server utilizes a special type of pages, the Index Allocation Map or IAM pages. Each allocation unit has at least one IAM Page and depending on the size of the table there might be many.

What is a Fill Factor?

The fill-factor option is provided for fine-tuning index data storage and performance. When an index is created or rebuilt, the fill-factor value determines the percentage of space on each leaf-level page to be filled with data, reserving the remainder on each page as free space for future growth.

What are Virtual Log files? How these are created?

SQL Server Database Engine divides each physical log file internally into a number of virtual log files. Virtual log files have no fixed size, and there is no fixed number of virtual log files for a physical log file. The Database Engine chooses the size of the virtual log files dynamically while it is creating or extending log files. The Database Engine tries to maintain a small number of virtual files.

Important DMV to find memory related information. (From SQL Server 2008)

select
(physical_memory_in_use_kb/1024)Memory_usedby_Sqlserver_MB,
(locked_page_allocations_kb/1024 )Locked_pages_used_Sqlserver_MB,
(total_virtual_address_space_kb/1024 )Total_VAS_in_MB,
process_physical_memory_low,
process_virtual_memory_low
from sys. dm_os_process_memory;

What is Locked pages in Memory (LPIM) concept?

Locked pages in memory is a privilege given to SQL Server service account which allows SQL Server not to trim its memory excessively when SQLOS ask SQL Server to do so.

When OS will face memory pressure and in turn inform to SQLOS and then SQLOS will ask SQL Server to trim it consumption. Now if LPIM privilege is not there SQL Server will start trimming its memory consumption.

To avoid this it is better to give optimum value to SQL Server memory and leave enough RAM for OS and then give LPIM privilege.

What is WOW?

When SQL Server 32 bit runs on 64 bit version of windows it is called WOW.

How you resolved out of memory error?

Make sure you have set OPTIMUM value for MAX SERVER MEMORY SETTING. This is one which should be adopted as best practice.

Try to find out whether it is internal memory pressure or external. By that I mean if SQL Server is facing memory crunch it will be internal if there is less physical RAM on system it will be external.

Try to find if it is SQL Server VAS pressure. Generally in error message if you get like ‘Failed Virtual Allocate Bytes: FAIL_VIRTUAL_RESERVE ‘ this points to fact that OOM error came because there was VAS pressure.

What is the difference between Bufferpool and Virtual memory (VAS)?

No buffer pool and VAS are not the same thing. In SQL Server direct physical memory access is not allowed any memory request which comes is first mapped to process VAS and then if SQL Server find memory free it would map this VAS address to physical memory and then memory becomes committed. Bufferpool is physical memory VAS is virtual memory.

How to find data page information?

• Execute DBCC IND to check allocated pages of a Table.
• Execute DBCC PAGE to find more information about the allocated pages.

What is a Global Allocation Map (GAM)?

GAM has a bit for every extent and it records what extents have been allocated for any use.

It stores information about both Uniform and Mixed extents.

If the bit value is 0, extent is in use and If the bit value is 1, extent is free.

It can hold information about the total 64000 extents and approximately size is 4000 MB.

What is a Shared Global Allocation Map (SGAM)?

SGAM has a bit for every extent and it records what extents are currently being used as mixed extent and have at least one unused page.

If the bit value is 0, either extent is not a mixed extent or it is a mixed extent and all its pages being used.

If the bit value is 1, it is a mixed extent and having an at least one unused page.

What is a Differential Changed Map (DCM)?

This tracks the extents that have changed since the last BACKUP DATABASE statement. If the bit for an extent is 1, the extent has been modified since the last BACKUP DATABASE statement. If the bit is 0, the extent has not been modified.

What is a Bulk Changed Map (BCM)?

This tracks the extents that have been modified by bulk logged operations since the last BACKUP LOG statement. If the bit for an extent is 1, the extent has been modified by a bulk logged operation after the last BACKUP LOG statement. If the bit is 0, the extent has not been modified by bulk logged operations.

What is RAID and what are different types of RAID configurations?

RAID stands for Redundant Array of Independent Disks. RAID defines data storage schemes to divide and replicate data among various disks so that data reliability and I/O performance can be increased.

What is a Connection in SQL Server?

A connection is established when the user is successfully logged in. The user can then submit one or more Transact-SQL statements for execution. A connection is closed when the user explicitly logs out, or the connection is terminated.

What is a SQL Batch?

A SQL batch is a set of one or more Transact-SQL statements sent from a client to an instance of SQL Server for execution. It represents a unit of work submitted to the Database Engine by users.

What is a Fiber?

A fiber is a lightweight thread that requires fewer resources than a Windows thread and can switch context when in user mode. One Windows thread can be mapped to many fibers.

What is a Worker Thread?

The worker thread represents a logical thread in SQL Server that is internally mapped (1:1) to either a Windows thread or, if lightweight pooling is turned ON, to a fiber.

What is the use of lightweight pooling parameter?

The server configuration option lightweight pooling controls whether an instance of SQL Server uses Windows threads or fibers.

The default for this option is 0. This indicates that the instance of SQL Server schedules a Windows thread per worker thread.

If lightweight pooling is set to 1, SQL Server uses fibers instead of Windows threads.

What is the priority boost option?

By default, each instance of SQL Server is a priority of 7, which is referred to as the normal priority. The priority boost configuration option can be used to increase the priority of the threads from an instance of SQL Server to 13. This is referred to as high priority.

What is the importance of Buffer in SQL Server?

The primary purpose of a SQL Server database is to store and retrieve data.

The disk I/O operations can consume many resources and take a relatively long time to finish, SQL Server focuses on making I/O highly efficient.

Buffer management is a key component in achieving this efficiency.

It caches the repeated data and execution plan in the buffer pool to reduce the disk I/O.

What is the size of the buffer in a memory?

A buffer is an 8-KB page in memory, the same size as a data or index page.

How an internal buffer manager manages the buffer cache?

The buffer manager manages the functions for reading data or index pages from the database disk files into the buffer cache and writing modified pages back to disk.

A page remains in the buffer cache until it has either not been referenced for some time or the buffer manager needs the buffer area to read in more data. Data is written back to disk only if it is modified. Data in the buffer cache can be modified multiple times before being written back to disk.

What is a ramp-up?

The interval between SQL Server startup and when the buffer cache obtains its memory target is called ramp-up.

The ramp-up depends on the number and type of client requests and it is transforming single-page read requests into aligned eight-page requests.

What is ROW LOCATOR?

If you define a NON CLUSTERED index on a table ,then the index row of a nonclustered index contains a pointer to the corresponding data row of the table. This pointer is called a row locator.

What is an Active Log?

The section of the log file from the MinLSN to the last-written log record is called the active portion of the log, or the active log. This is the section of the log required to do a full recovery of the database. No part of the active log can ever be truncated. All log records must be truncated from the parts of the log before the MinLSN.

What is “Write Ahead Transaction Log”?

SQL Server uses a write-ahead log (WAL), which guarantees that no data modifications are written to disk before the associated log record is written to disk.

Data modifications are not made directly to disk, but are made to the copy of the page in the buffer cache. The modification is not written to disk until a checkpoint occurs in the database.

What is a Clustered table?

Clustered tables are tables that have a clustered index. The data rows are stored in order based on the clustered index key.

What is a Heap Table?

Heaps are tables that have no clustered index. The data rows are not stored in any particular order, and there is no particular order to the sequence of the data pages.

What is a Worktables?

The relational engine may need to build a worktable to perform a logical operation specified in an SQL statement. Worktables are internal tables that are used to hold intermediate results. Worktables are generated for certain GROUP BY, ORDER BY, or UNION queries.

What is a COMPATIBILITY_LEVEL option of the Database?

This lets SQL Server know with which version of SQL Server to make the database compatible.

What is AUTO_SHRINK option of the Database?

This option either turns on or off the feature that will automatically shrink your database files when free space is available. In almost all cases, this should be set to OFF.

What is AUTO_UPDATE_STATISTICS option of the Database?

When turned ON, as it should be in most cases, the optimizer will automatically keep statistics updated, in response to data modications.

What is READ_WRITE option of the Database?

This is the default option and the one to use if you want users to be able to update the database. We could also set the database to READ_ONLY to prevent any users making updates to the database.

What is Lazy writer?

Lazy writer can adjust the number of buffers in the buffer pool if dynamic memory management is enabled. The lazy writer is a system process that keeps free buffers available by removing infrequently used pages from the buffer cache.

Ensure that a specified number of buffers are free in the Buffer Pool so they can be allocated for use by the server.

How you explicitly recompile your execution plan?

• You can use sp_recompile to explicitly recompile your execution plan.
• You can also use WITH RECOMPILE to recompile the stored procedure.

What is buffer pool?

SQL Server builds a buffer pool in memory to hold pages read from the database. You can minimize physical I/O to the database files by maximizing the size of the buffer pool.

What is a query wait option?

Use the query wait option to specify the time in seconds (from 0 through 2147483647) that a query waits for resources before timing out. If the default value of -1 is used, or if –1 is specified, then the time-out is calculated as 25 times of the estimated query cost.

You can change query wait value using sp_configure.

How SQL Server executes a single SELECT statement?

• The parser scans the SELECT statement and breaks it into logical units such as keywords, expressions, operators, and identifiers.
• A query tree, sometimes referred to as a sequence tree, is built describing the logical steps needed to transform the source data into the format required by the result set.
• The query optimizer analyzes different ways the source tables can be accessed and create an optimized version of the query tree is called the execution plan.
• The relational engine starts executing the execution plan.
• The relational engine processes the data and returns the result set to the client.

What is Query Plan?

The bulk of the execution plan is a re-entrant, read-only data structure used by any number of users. This is referred to as the query plan. No user context is stored in the query plan. There are never more than one or two copies of the query plan in memory.

What is Execution Context?

Each user that is currently executing the query has a data structure that holds the data specific to their execution, such as parameter values. This data structure is referred to as the execution context.

How SQL Server uses procedure cache?

When any SQL statement is executed, it first looks through the procedure cache to verify that an existing execution plan for the same SQL statement exists.

It reuses any existing plan it finds, saving the overhead of recompiling the SQL statement

Tell me something about the SQL Server Pages.

In SQL Server, the page size is 8 KB. This means SQL Server databases have 128 pages per megabyte. Each page begins with a 96-byte header that is used to store system information about the page. This information includes the page number, page type, the amount of free space on the page, and the allocation unit ID of the object that owns the page.

What is a row offset of pages?

For each data row there is a 2-byte entry, saving the offset.

A row offset table starts at the end of the page, and each row offset table contains one entry for each row on the page.

What is the ROW_OVERFLOW_DATA allocation unit?

When the total row size of all fixed and variable columns in a table exceeds the 8,060 byte limitation, SQL Server dynamically moves one or more variable length columns to pages in the ROW_OVERFLOW_DATA allocation unit, starting with the column with the largest width.

What is a IN_ROW_DATA allocation unit?

It is an allocation unit for data page and index page.

What is a LOB_DATA allocation unit?

It is an allocation unit for text, ntext, image, xml, varchar (max), nvarchar (max), varbinary (max) data types.

Why 24-byte pointer is require with the ROW_OVERFLOW_DATA allocation unit?

When a combination of varchar, nvarchar, varbinary, sql_variant, or CLR user-defined type columns exceeds this 8KB limit, the Database Engine moves the record column with the largest width to another page in the ROW_OVERFLOW_DATA allocation unit, while maintaining a 24-byte pointer on the original page.

What is the size of the data page header?

The size of the data page header is 96-byte.

What are the different types of pages and what do they store?

• Page types include
• Data page
• Index page
• Text/Image page
• Page Free Space
• Global Allocation Map (GAM) – extent allocation information
• Shared Global Allocation Map (SGAM) – mixed extent with at least one unused page information
• Bulk Changed Map – extents that got changed by a bulk logged operation
• Differential Changed Map – extents that got changed since last BACKUP DATABASE statement
• Index allocation map

What is an Extent?

Extents are the basic unit in which space is managed. An extent is eight physically contiguous pages, or 64 KB. This means SQL Server databases have 16 extents per megabyte.

What are the different types of Extents?

• Uniform extents are owned by a single object; all eight pages in the extent can only be used by the owning object.
• Mixed extents are shared by up to eight objects. Each of the eight pages in the extent can be owned by a different object.

What are the different types of database files in SQL Server?

• Primary data files: The primary data file is the starting point of the database and points to the other files in the database. Every database has one primary data file. The recommended file name extension for primary data files is .mdf.
• Secondary data files: Secondary data files make up all the data files, other than the primary data file. Some databases may not have any secondary data files, while others have several secondary data files. The recommended file name extension for secondary data files is .ndf.
• Log files: Log files hold all the log information that is used to recover the database. There must be at least one log file for each database, although there can be more than one. The recommended file name extension for log files is .ldf.

Explained something about SQL Server Access Methods.

• SQL doesn’t directly retrieve from disk, it makes requests to buffer manager which serves up the page in cache before rendering out.
• When scan starts, SQL Server can use look-ahead mechanisms to qualify rows and index entries on a page. The retrieving of rows that meet specified criteria is known as a qualified retrieval.

Tell us something about the SQL Server Database Transactions.

• Provides support for Atomicity, Consistency, Isolation and Durability.
• Log records are always written to disk before the data pages where changes were made are actually written.
• Writes to the transaction log are synchronous while writes to data pages can be asynchronous.

What is the sparse file?

A database snapshot that is created by a user stores its data in one or more sparse files. Sparse file technology is a feature of the NTFS file system.

What is the Database Filegroups?

Database objects and files can be grouped together in filegroups for allocation and administration purposes.

What are the different type of Filegroups?

• Primary: The primary filegroup contains the primary data file and any other files not specifically assigned to another filegroup.
• User-defined: User-defined filegroups are any filegroups that are specified by using the FILEGROUP keyword in a CREATE DATABASE or ALTER DATABASE statement.

What are Page Splits?

When there is not enough room on a page for a new row, a server splits the page, allocates a new page, and moves some rows to the new page.

Page splits are considered as a very bad performance and there are number of techniques which is used to reduce the risk of page splits.