The Visibility Map is one of the important architectural notes of PostgreSQL.

The Visibility Map associates with each Table and Index.

It stores in separate relation alongside with main relation and It starts with the file node number plus a suffix _vm.

The Visibility Map uses to keep track of which pages contain only tuples that are known to be visible to all active transactions.

It stores one bit per page. If it set bit, that means no free space on the page and all tuples can be visible to running transaction.

If you do not know about the MVCC (Multiversion Concurrency Control), please visit below an article.

What is Multi Version Concurrency Control (MVCC)

Why VACUUM requires in MVCC because VACUUM removes dead row versions in tables and marks the space available for future reuse.

With the Visibility Map, VACUUM will be now able to perform partial scans of the table data, skipping pages which marks as fully visible.

Now Database Administrator happy because partial scans require fewer I/O operations for VACUUM.

The Visibility map bits are only set by vacuum but are cleared by any data-modifying operations on a page.

What is the max degree of parallelism option?

When SQL Server runs on a computer with more than one microprocessor or CPU, it detects the best degree of parallelism, that is, the number of processors employed to run a single statement, for each parallel plan execution. You can use the max degree of parallelism option to limit the number of processors to use in parallel plan execution.

What is MAXDOP in SQL Server?

Maximum Degree of Parallelism (MAXDOP) is an option in SQL server which enables you to configure SQL Server to use available CPU/CPUs to run a single statement in parallel execution plan. MAXDOP configuration highly depends on SQL Server edition, CPU type and operating system.

In my experience, I had to play with MAXDOP for certain statements coming from different servers in order to find best option of MAXDOP.

What is the MAXDOP query hint option?

You can override the max degree of parallelism value in queries by specifying the MAXDOP query hint in the query statement.

T-SQL script to change max degree of parallelism Option.

Setting maximum degree of parallelism to 0 allows SQL Server to use all the available processors up to 64 processors.
To suppress parallel plan generation, set max degree of parallelism to 1. Set the value to a number greater than 1 to restrict the maximum number of processors used by a single query execution.

sp_configure 'show advanced options', 1;
GO
RECONFIGURE WITH OVERRIDE;
GO
sp_configure 'max degree of parallelism', 8;
GO
RECONFIGURE WITH OVERRIDE;
GO

SQL Server 2014: What is Hekaton?

SQL Server 2014: What is Hekaton?

SQL Server 2014: Create Memory Optimized File Group and Table.

SQL Server 2014: Create Memory Optimized File Group and Table

What is a Read-Ahead mechanism?

The Database Engine supports a performance optimization mechanism called read-ahead. Read-ahead anticipates the data and index pages needed to fulfill a query execution plan and brings the pages into the buffer cache before they are actually used by the query.

What are the types of Read-Ahead?

There are two types of Read-Ahead, one is Reading Data Pages and second is Reading Index Pages.

How internally pages are modified and written to the disk?

First page modification happens in the buffer cache and When a page is modified in the buffer cache, it is not immediately written back to disk;instead, the page is marked as dirty.

This means that a page can have more than one logical write made before it is physically written to disk.

For each logical write, a transaction log record is inserted in the log cache that records the modification. The log records must be written to disk before the associated dirty page is removed from the buffer cache and written to disk. SQL Server uses a technique known as write-ahead logging.

What is Bookmark Lookup?

When a SQL query requests a small number of rows, the optimizer can use the nonclustered index, if available, on the column(s) in the WHERE clause to retrieve the data. If the query refers to columns that are not part of the nonclustered index used to retrieve the data, then navigation is required from the index row to the corresponding data row in the table to access these columns.This operation is called a bookmark lookup.

What is a COLD, DIRTY or CLEAN Buffer?

The cold buffer cache has a couple of variants within SQL Server.    Typically it refers to the buffer pool right after restart.   The data cache is not loaded (cold) and requires physical reads to populate the cache.

Modified buffers that have not been written to disk are termed DIRTY buffers and are typically written to disk by checkpoint processing.

A clean buffer is a data page in memory that is NOT MODIFIED.

What is the use of DBCC DROPCLEANBUFFERS?

DBCC DROPCLEANBUFFERS removes the UNMODIFIED buffers from the buffer pool (no longer hashed) and places them on the free list.

This is often used in basic performance testing to clear the data cache and force the data to be retrieved from disk as a physical read instead of a logic read.

What is affinity I/O mask option?

The affinity I/O mask option binds SQL Server disk I/O to a specified subset of CPUs. In high-end SQL Server online transactional processing (OLTP) environments, this extension can enhance the performance of SQL Server threads issuing I/Os.

What should be the different values for affinity I/O mask?

• A 1-byte affinity I/O mask covers up to 8 CPUs in a multiprocessor computer.
• A 2-byte affinity I/O mask covers up to 16 CPUs in a multiprocessor computer.
• A 3-byte affinity I/O mask covers up to 24 CPUs in a multiprocessor computer.
• A 4-byte affinity I/O mask covers up to 32 CPUs in a multiprocessor computer.
• To cover more than 32 CPUs, configure a four-byte affinity I/O mask for the first 32 CPUs and up to a four-byte affinity64 I/O mask for the remaining CPUs.

What Is Eager Writing in SQL Server?

Eager writing is a long back implemented idea in SQL server to prevent flooding of the buffer pool with pages that are newly created from minimally logged activities, and physically written to disk.

In SQL Server 2014, eager write doesn’t force dirty pages physical write as quickly as previous versions.   This allows dirty pages to spend their life time in RAM, used by SQL and flush them to free list without ever a physical write to disk as long memory is available. This will significantly increase the performance up to 300% as compared to previous versions.

Generally, what kind of operations are recorded in the transaction log file?

• The start and end of each transaction.
• Every data modification (insert, update, or delete).
• Every extent and page allocation or deallocation.
• Creating or dropping a table or index.
• Rollback operations are also logged.

What happened if FILEGROWTH is enable and disable?

• If the FILEGROWTH setting is enabled for the log and space is available on the disk, the file is extended by the amount specified in growth_increment and the new log records are added to the extension.
• If the FILEGROWTH setting is not enabled, or the disk that is holding the log file has less free space than the amount specified in growth_increment, an 9002 error is generated.

Are you suggesting to truncate an old transaction log?

Yes,

If the log records were never deleted from the transaction log, the logical log would grow until it filled all the available space on the disks holding the physical log files. At some point in time, old log records no longer necessary for recovering or restoring a database must be deleted to make way for new log records.

Do you think truncate log operation reduces the size of the physical log file?

Log truncation does not reduce the size of a physical log file, it reduces the size of the logical log and marks as inactive the virtual logs that do not hold any part of the logical log.

A log shrink operation removes enough inactive virtual logs to reduce the log file to the requested size.

How to find data page information?

• Execute DBCC IND to check allocated pages of a Table.
• Execute DBCC PAGE to find more information about the allocated pages.

What is a Global Allocation Map (GAM)?

GAM has a bit for every extent and it records what extents have been allocated for any use.

It stores information about both Uniform and Mixed extents.

If the bit value is 0, extent is in use and If the bit value is 1, extent is free.

It can hold information about the total 64000 extents and approximately size is 4000 MB.

What is a Shared Global Allocation Map (SGAM)?

SGAM has a bit for every extent and it records what extents are currently being used as mixed extent and have at least one unused page.

If the bit value is 0, either extent is not a mixed extent or it is a mixed extent and all its pages being used.

If the bit value is 1, it is a mixed extent and having an at least one unused page.

What is a Differential Changed Map (DCM)?

This tracks the extents that have changed since the last BACKUP DATABASE statement. If the bit for an extent is 1, the extent has been modified since the last BACKUP DATABASE statement. If the bit is 0, the extent has not been modified.

What is a Bulk Changed Map (BCM)?

This tracks the extents that have been modified by bulk logged operations since the last BACKUP LOG statement. If the bit for an extent is 1, the extent has been modified by a bulk logged operation after the last BACKUP LOG statement. If the bit is 0, the extent has not been modified by bulk logged operations.

What is RAID and what are different types of RAID configurations?

RAID stands for Redundant Array of Independent Disks. RAID defines data storage schemes to divide and replicate data among various disks so that data reliability and I/O performance can be increased.

What is a Connection in SQL Server?

A connection is established when the user is successfully logged in. The user can then submit one or more Transact-SQL statements for execution. A connection is closed when the user explicitly logs out, or the connection is terminated.

What is a SQL Batch?

A SQL batch is a set of one or more Transact-SQL statements sent from a client to an instance of SQL Server for execution. It represents a unit of work submitted to the Database Engine by users.

What is a Fiber?

A fiber is a lightweight thread that requires fewer resources than a Windows thread and can switch context when in user mode. One Windows thread can be mapped to many fibers.

What is a Worker Thread?

The worker thread represents a logical thread in SQL Server that is internally mapped (1:1) to either a Windows thread or, if lightweight pooling is turned ON, to a fiber.

What is the use of lightweight pooling parameter?

The server configuration option lightweight pooling controls whether an instance of SQL Server uses Windows threads or fibers.

The default for this option is 0. This indicates that the instance of SQL Server schedules a Windows thread per worker thread.

If lightweight pooling is set to 1, SQL Server uses fibers instead of Windows threads.

What is the priority boost option?

By default, each instance of SQL Server is a priority of 7, which is referred to as the normal priority. The priority boost configuration option can be used to increase the priority of the threads from an instance of SQL Server to 13. This is referred to as high priority.

What is the importance of Buffer in SQL Server?

The primary purpose of a SQL Server database is to store and retrieve data.

The disk I/O operations can consume many resources and take a relatively long time to finish, SQL Server focuses on making I/O highly efficient.

Buffer management is a key component in achieving this efficiency.

It caches the repeated data and execution plan in the buffer pool to reduce the disk I/O.

What is the size of the buffer in a memory?

A buffer is an 8-KB page in memory, the same size as a data or index page.

How an internal buffer manager manages the buffer cache?

The buffer manager manages the functions for reading data or index pages from the database disk files into the buffer cache and writing modified pages back to disk.

A page remains in the buffer cache until it has either not been referenced for some time or the buffer manager needs the buffer area to read in more data. Data is written back to disk only if it is modified. Data in the buffer cache can be modified multiple times before being written back to disk.

What is a ramp-up?

The interval between SQL Server startup and when the buffer cache obtains its memory target is called ramp-up.

The ramp-up depends on the number and type of client requests and it is transforming single-page read requests into aligned eight-page requests.

What is ROW LOCATOR?

If you define a NON CLUSTERED index on a table ,then the index row of a nonclustered index contains a pointer to the corresponding data row of the table. This pointer is called a row locator.

What is an Active Log?

The section of the log file from the MinLSN to the last-written log record is called the active portion of the log, or the active log. This is the section of the log required to do a full recovery of the database. No part of the active log can ever be truncated. All log records must be truncated from the parts of the log before the MinLSN.

What is “Write Ahead Transaction Log”?

SQL Server uses a write-ahead log (WAL), which guarantees that no data modifications are written to disk before the associated log record is written to disk.

Data modifications are not made directly to disk, but are made to the copy of the page in the buffer cache. The modification is not written to disk until a checkpoint occurs in the database.

What is a Clustered table?

Clustered tables are tables that have a clustered index. The data rows are stored in order based on the clustered index key.

What is a Heap Table?

Heaps are tables that have no clustered index. The data rows are not stored in any particular order, and there is no particular order to the sequence of the data pages.

What is a Worktables?

The relational engine may need to build a worktable to perform a logical operation specified in an SQL statement. Worktables are internal tables that are used to hold intermediate results. Worktables are generated for certain GROUP BY, ORDER BY, or UNION queries.

What is a COMPATIBILITY_LEVEL option of the Database?

This lets SQL Server know with which version of SQL Server to make the database compatible.

What is AUTO_SHRINK option of the Database?

This option either turns on or off the feature that will automatically shrink your database files when free space is available. In almost all cases, this should be set to OFF.

What is AUTO_UPDATE_STATISTICS option of the Database?

When turned ON, as it should be in most cases, the optimizer will automatically keep statistics updated, in response to data modications.

What is READ_WRITE option of the Database?

This is the default option and the one to use if you want users to be able to update the database. We could also set the database to READ_ONLY to prevent any users making updates to the database.

What is Lazy writer?

Lazy writer can adjust the number of buffers in the buffer pool if dynamic memory management is enabled. The lazy writer is a system process that keeps free buffers available by removing infrequently used pages from the buffer cache.

Ensure that a specified number of buffers are free in the Buffer Pool so they can be allocated for use by the server.

How you explicitly recompile your execution plan?

• You can use sp_recompile to explicitly recompile your execution plan.
• You can also use WITH RECOMPILE to recompile the stored procedure.

What is buffer pool?

SQL Server builds a buffer pool in memory to hold pages read from the database. You can minimize physical I/O to the database files by maximizing the size of the buffer pool.

What is a query wait option?

Use the query wait option to specify the time in seconds (from 0 through 2147483647) that a query waits for resources before timing out. If the default value of -1 is used, or if –1 is specified, then the time-out is calculated as 25 times of the estimated query cost.

You can change query wait value using sp_configure.

How SQL Server executes a single SELECT statement?

• The parser scans the SELECT statement and breaks it into logical units such as keywords, expressions, operators, and identifiers.
• A query tree, sometimes referred to as a sequence tree, is built describing the logical steps needed to transform the source data into the format required by the result set.
• The query optimizer analyzes different ways the source tables can be accessed and create an optimized version of the query tree is called the execution plan.
• The relational engine starts executing the execution plan.
• The relational engine processes the data and returns the result set to the client.

What is Query Plan?

The bulk of the execution plan is a re-entrant, read-only data structure used by any number of users. This is referred to as the query plan. No user context is stored in the query plan. There are never more than one or two copies of the query plan in memory.

What is Execution Context?

Each user that is currently executing the query has a data structure that holds the data specific to their execution, such as parameter values. This data structure is referred to as the execution context.

How SQL Server uses procedure cache?

When any SQL statement is executed, it first looks through the procedure cache to verify that an existing execution plan for the same SQL statement exists.

It reuses any existing plan it finds, saving the overhead of recompiling the SQL statement

Tell me something about the SQL Server Pages.

In SQL Server, the page size is 8 KB. This means SQL Server databases have 128 pages per megabyte. Each page begins with a 96-byte header that is used to store system information about the page. This information includes the page number, page type, the amount of free space on the page, and the allocation unit ID of the object that owns the page.

What is a row offset of pages?

For each data row there is a 2-byte entry, saving the offset.

A row offset table starts at the end of the page, and each row offset table contains one entry for each row on the page.

What is the ROW_OVERFLOW_DATA allocation unit?

When the total row size of all fixed and variable columns in a table exceeds the 8,060 byte limitation, SQL Server dynamically moves one or more variable length columns to pages in the ROW_OVERFLOW_DATA allocation unit, starting with the column with the largest width.

What is a IN_ROW_DATA allocation unit?

It is an allocation unit for data page and index page.

What is a LOB_DATA allocation unit?

It is an allocation unit for text, ntext, image, xml, varchar (max), nvarchar (max), varbinary (max) data types.

Why 24-byte pointer is require with the ROW_OVERFLOW_DATA allocation unit?

When a combination of varchar, nvarchar, varbinary, sql_variant, or CLR user-defined type columns exceeds this 8KB limit, the Database Engine moves the record column with the largest width to another page in the ROW_OVERFLOW_DATA allocation unit, while maintaining a 24-byte pointer on the original page.

What is the size of the data page header?

The size of the data page header is 96-byte.

What are the different types of pages and what do they store?

• Page types include
• Data page
• Index page
• Text/Image page
• Page Free Space
• Global Allocation Map (GAM) – extent allocation information
• Shared Global Allocation Map (SGAM) – mixed extent with at least one unused page information
• Bulk Changed Map – extents that got changed by a bulk logged operation
• Differential Changed Map – extents that got changed since last BACKUP DATABASE statement
• Index allocation map

What is an Extent?

Extents are the basic unit in which space is managed. An extent is eight physically contiguous pages, or 64 KB. This means SQL Server databases have 16 extents per megabyte.

What are the different types of Extents?

• Uniform extents are owned by a single object; all eight pages in the extent can only be used by the owning object.
• Mixed extents are shared by up to eight objects. Each of the eight pages in the extent can be owned by a different object.

What are the different types of database files in SQL Server?

• Primary data files: The primary data file is the starting point of the database and points to the other files in the database. Every database has one primary data file. The recommended file name extension for primary data files is .mdf.
• Secondary data files: Secondary data files make up all the data files, other than the primary data file. Some databases may not have any secondary data files, while others have several secondary data files. The recommended file name extension for secondary data files is .ndf.
• Log files: Log files hold all the log information that is used to recover the database. There must be at least one log file for each database, although there can be more than one. The recommended file name extension for log files is .ldf.

Explained something about SQL Server Access Methods.

• SQL doesn’t directly retrieve from disk, it makes requests to buffer manager which serves up the page in cache before rendering out.
• When scan starts, SQL Server can use look-ahead mechanisms to qualify rows and index entries on a page. The retrieving of rows that meet specified criteria is known as a qualified retrieval.

Tell us something about the SQL Server Database Transactions.

• Provides support for Atomicity, Consistency, Isolation and Durability.
• Log records are always written to disk before the data pages where changes were made are actually written.
• Writes to the transaction log are synchronous while writes to data pages can be asynchronous.

What is the sparse file?

A database snapshot that is created by a user stores its data in one or more sparse files. Sparse file technology is a feature of the NTFS file system.

What is the Database Filegroups?

Database objects and files can be grouped together in filegroups for allocation and administration purposes.

What are the different type of Filegroups?

• Primary: The primary filegroup contains the primary data file and any other files not specifically assigned to another filegroup.
• User-defined: User-defined filegroups are any filegroups that are specified by using the FILEGROUP keyword in a CREATE DATABASE or ALTER DATABASE statement.

What are Page Splits?

When there is not enough room on a page for a new row, a server splits the page, allocates a new page, and moves some rows to the new page.

Page splits are considered as a very bad performance and there are number of techniques which is used to reduce the risk of page splits.

In the previous day, I shared some of the essential key concepts of the Cassandra Architecture.
In this post, I am sharing the basic architecture of reading and writing operations of Cassandra.

Cassandra Ring:

Cassandra is using a consistent hashing algorithm to treat all nodes of the cluster equally.

A Cassandra cluster is visualised as a Ring in which different nodes are participating with the same name. It can exchange state information with a maximum of three other nodes.
A token range assigns to each node which determines its position in the cluster.

Cassandra is using Gossip Protocol so whenever any node starts, it obtains information about the other nodes by exchanging information with each other.

A Partitioner is responsible for preparing the set of data for each node.
Cassandra has three different types of Partitioners, Murmur3Partitioner (default), RandomPartitioner and a ByteOrderedPartitioner.

Each node is performing replications of data basis on defined replication strategy. Basically there two types of replication strategy, one is Simple Strategy and second is Network Topology Strategy.

A Simple Strategy works in the clockwise, and Network Topology works for multiple data centres.

How it executes Write Operations?

A Cassandra is Masterless distributed architecture, and there is no any Master and Slave mechanism like any other distributed system.

At any given point in time, the client can connect to any node and that connected node called as a coordinator.
Based on the partition key and replication strategy, Coordinator forwards and replicates data to all applicable nodes.

Each node processes the request individually.
Every node first writes data into the commit log and then writes into the memtable.

The commit log ensures the durability because memtable is dealing with memory so whenever data is lost then we can use commit log to restore the data.
Whenever memtable is flushed or full, it is written to the SSTable (Sorted String Table) of the disk.

Every SSTable creates three files on the disk which include a bloom filter, a key index and a data file. Over a period, a number of SSTables are created.

How It executes Read Operations?

As Cassandra is Masterless architecture so clients can connect to any node of the cluster ring.
Like write operation, chosen node is called as coordinator and is responsible for returning the requested data.

A row key must be supplied for every read operation. The coordinator uses the row key to determine the first replica.

If replica has a different version of the data, coordinator returns the latest version to the client by issuing read repair command with the older version of the data.

Each read request fetches the data from memtable and SSTables after that it merged this data and returned to the coordinator.
Internally SSTables are using a Bloom Filter to check requested row key whether it exists in SSTable or not.