We all are aware about the ACID Properties of Database System (Atomicity, Consistency, Isolation, Durability).
If you don’t know about the ACID, please visit below article.

ACID Properties in Database System (Atomicity, Consistency, Isolation, Durability)

In last one decade, lots of new Database Products available in the Market. The types are: RDBMS Products, NoSQL Products, BIGDATA, MPP Products.

My Question is, how do you choose best database product for your project?

Traditional Answer, The first preference of Database Designer would be ACID Properties oriented Databases.

But most of the NoSQL Databases are not providing full support of ACID Properties, then also NoSQL Databases are popular and People are using it for different purposes.

Because most of the NoSQL Databases are not ACID oriented, but they are CAP oriented.

What is CAP theorem (Consistency, Availability, Partition)?

CAP Published by Eric Brewer in 2000, the theorem is a set of basic requirements that describe any distributed system like: NoSQL Cassandra, MongoDB, CouchDB.

Consistency:

All nodes can see the same data at the same time. The data are the same across the cluster, so you can read or write any node.

Each transaction start and end in consistent stats so that all nodes can access the same data. Actually, It is achieved by update all nodes before reading.

Availability:

Here, The meaning of Availability is to access cluster in any situation like: server goes down, server delay. The system should always respond to a request, even if data is not in sync or inconsistent state, It prints the message like “This cluster data are not updated”.

Partition:

Here, the meaning of Partition is to avoid any type of system failure and system should run with available nodes or data. It does not allow any single point of failure because data replicated across all the clusters.

The Partition can work with Consistency (CP) or can work with Availability (AP) means, you can never find any Database System which follows the full CAP theorem.

Storage System with Partition and Consistency (CP):
Redis, MongoDB, AppFabric Caching, and MemcacheDB.
Storage System with Partition and Availability (AP):
Cassandra, DynamoDB, and CouchDB.

In my overall career, I worked on multiple RDBMS/NoSQL systems. As per the ANSI SQL, SQL statements are terminated by semicolons. If you are working on MySQL or PostgreSQL then a semicolon is a mandatory between multiple SQL statements.

Most of the SQL Server DB guys are confused between “GO” and semicolon. “GO” is a default identifier which uses as a batch separator where a semicolon is a terminator of a SQL Statement.

The semicolon is a not a mandatory in SQL Server because by default SQL statements are terminated with semicolons.
But we must terminate the statement by using the semicolon which is the best practice.
In some cases, I got a wrong error message for those SQL Statements which are not ended by semicolons.

If you are a programmer and know the C, C++ or JAVA, you know that statement terminator is mandatory for the compiler.

Microsoft technologies are good, but they are changing and upgrading their versions very speedy. So in the future, there may be a chance of announcement like “Semicolon as a statement terminator is compulsory in SQL Server”, and after that, you can assume what will happen. You have to update all your DB Codes.

I would suggest, please don’t wait for anything and give explicitly semicolon at the end of your SQL Statement.

SQL Server: How to change the default “GO” Batch Separator?

In this post, I am sharing an important note on Forwarded Records in the Heap table of SQL Server. Whenever DBA or Developer is facing performance issue, they are always talking about the table or index fragmentation, but no-one is discussing the Forwarded Records.

What is heap Table

Simple Example to understand Forwarded Records:

You create a heap table, and one of the column types is VARCHAR(200).
You inserted 100000 records and added a small size of the string in VARCHAR(200) like “dbrnd” value which is only VARCHAR(5).

Now next, you require updating the value from “dbrnd” to “Database Research & Development” which is now VARCHAR(31).
Previously, we stored our data based on VARCHAR(5), but while updating we need VARCHAR(31).

Before the update, all data correctly save in data page and for new update no more free space available on the data page. So SQL Server internally moves new data into the new page and assigned a new Relative Identifier (RID).

Now old page contains forwarding pointer to new RID and new records also back pointer to old records.

– Called as Forwarded Records and which is also one type of internal fragmentation.

The biggest drawback of Forwarded Record is, it requires high I/O because SQL Server has to jump between the old data page and new data page for getting data.

Few more following topics, which I will publish from tomorrow:
How to find the Forwarded Records?
How to remove Forwarded Records?

One of the Important news for Database Community.
The UBER taxi number one company switched from PostgreSQL to MySQL.

There are a few major reasons behind for this migration.
As per the official document of UBER, they were facing a few notable issues with the PostgreSQL.

• Inefficient architecture for writes
• Inefficient data replication
• Issues with table corruption
• Poor replica MVCC support
• Difficulty upgrading to newer releases

Here, I am sharing the official link on UBER database migration. I completely read this article and agree with it on a few of the topics.
You guys must read carefully because we should understand the pros & cons of any RDBMS.

As a database professional, I work with both databases on a regular basis, and I typically use all the features of both databases for design and implementation.
I have got many replies to write about the core differences between PostgreSQL and MySQL.

As innovation on these databases has progressed, each development community has actively made changes to address their respective sets of perceived disadvantages.
The result that it has gotten more difficult to objectively determine which database is likely to be better suited for a given application.

MySQL is controlled by Oracle, whereas Postgres is available under an open-source license from the PostgreSQL Global Development Group.

PostgreSQL is closer to the ANSI SQL standard, MySQL is closer to the ODBC standard.

One of the most notable differences between MySQL and PostgreSQL is the fact that you can’t do nested subqueries of subselects in MySQL. MySQL doesn’t support CTE, window functions, full outer joins and arrays.

Compare to PostgreSQL, MySQL has its own large user community and various type of materials.

MySQL uses nonstandard ‘#’ to begin a comment line and PostgreSQL use ‘–‘ to begin a comment line.

MySQL provides different type of table storage engines like: InnoDB, MyIsam, Memory. PostgreSQL does not provide different type of table stroage engines.

PostgreSQL is case-sensitive for string comparisons. The field “DbRnd” is not the same as the field “dbrnd”. This is a big change for many users from MySQL and other small database systems.

PostgreSQL has some nice features like: generate_series, custom aggregate functions, arrays etc. other hand MySQL has also some nice feature like: session variables in queries, FORCE INDEX, etc.

In PostgreSQL, when you are inside a transaction almost any operation can be undone. There are some irreversible operations (like creating or destroying a database or tablespace), but normal table modifications can be backed out by issuing a ROLLBACK via its Write-Ahead Log design.

MySQL doesn’t support any sort of rollback when using MyISAM. With InnoDB, the server has an implicit commit that occurs even if the normal auto-commit behavior is turned off. This means that any single table alteration or similar change is immediately committed.

Postgres obliterates MySQL in all manner of query-level goodness, and we can use Postgres for data mining and batch analytics databases, where we run lots of very large, hand-coded queries that can take advantage of Postgres’ advantages in join methods (MySQL only supports Nested Loop and Nested Loop over Index, while Postgres supports numerous methods), generally richer querying capabilities, and other good stuff like partial indexes can be used extensively.

PostgreSQL uses a robust locking model called MVCC that limits situations where individual clients interfere with each other. A short summary of the main benefit of MVCC would be “readers are never blocked by writers”.

MySQL’s InnoDB implements MVCC using a rollback segment and InnoDB databases supports all four SQL standard transaction isolation levels.

PostgreSQL uses cost-based query optimization methods in order to get good performance for many different types of joins. Query costs are assessed based on planner statistics collected when tables are analyzed combined with adjustable planner costs and advanced features such as the Genetic Query Optimizer.

MySQL doesn’t have this level of sophistication in its planner, and the tunables for Controlling Query Optimizer Performance are crude. Developers instead do things like explicitly provide index hints to make sure joins are executed correctly.

PostgreSQL has different replication options than MySQL. MySQL has no point-in-time recovery.

One operation that PostgreSQL is known to be slow performing is doing a full count of rows in a table, typically using this SQL:

SELECT COUNT(*) FROM your_table_name;

The reason why this is slow is related to the MVCC implementation in PostgreSQL. The fact that multiple transactions can see different states of the data means that there can be no straightforward way for “COUNT(*)” to summarize data across the whole table; PostgreSQL must walk through all rows, in some sense.

In MySQL, MyISAM tables cache the row count information, making this type of count operation almost instant. But if you’re using InnoDB instead, this is no longer the case and on InnoDB can’t assume that a full row count will be fast.

In the past decade, lots of analysis and research one for the Parallel Database System and Parallel Query Processing.

The success of Parallel Database System depends on the relational database model and good CPU / DISK performance. Without a good CPU and DISK performance, we should not use the Parallel Query Processing.

In general,
When we are executing any SQL Query, It creates the execution plan and submits to the internal query executor. A query executor collects all execution plans and executes in sequence.
For example, your one SQL Query requires 100,00,0000 times manipulation, so using a single CPU thread system It takes more time to execute.

In the Parallel Query Processing,
More than one process worker running in the background and responsible for the same task or multiple tasks in the sharing mode.

For example, If we say 4 different process worker are executing one single big SQL Query which requires 100,00,0000 times manipulation, It completes the whole process four times faster than using single process worker or single CPU thread for same SQL Query.

Basically, there are main three types of architecture of Parallel Database System.

Shared Disk:

This approach is not that much popular where the same disk shared among the different background process workers.

Shared Nothing:

With a shared nothing system, each processor owns a portion of the database, and only that portion may be directly accessed or manipulated by that processor. A two-phase commit protocol is required to coordinate a transaction commit which involves multiple nodes.

A shared nothing system is based on the concept of declustering. Declustering a relation includes distributing its tuples among multiple nodes according to some distribution criteria such as applying a hash function to the key attribute of each tuple.

In the context of query processing, the main advantage of a shared nothing system is its scalability.
One of the best example of this architecture is: GreenPlum Relational Database.

Shared Everything:

In a shared everything system, main memory, in addition to disks, is also shared across all the processors, making system management and load balancing much easier.

First, there are no communication delays because messages are exchanged through the shared memory, and synchronization can be accomplished by cheap, low level mechanisms.

Second, load balancing is much easier because the operating system can automatically allocate the next ready process to the first available processor.

Tomorrow, You can access one good article about Performance Test of Parallel Query Processing using PostgreSQL 9.6 Database System.

The BRIN index is very new type of index in a Database System.

Currently we are using B-Tree indexes most, but BRIN Indexes are much faster than B-Tree Indexes.
It is not correct to say that BRIN is a 100% replacement of B-Tree but in some situation, it executes 10 times faster than B-Tree Index.

Let me provide you a small note on B-Tree as well,

The structure of B-Tree looks like a Binary Search Tree.
If you have a large table, overall branching of BTree Index is very large because it required to store thousands of keys in a B-Tree node.

The B-Tree also stored all the key values in their leaf nodes so for every search key lookups require to scan both the leaf node and B-Tree mapping node.

If we have very large table with the B-Tree index and we are selecting a large number of records, B-Tree index lookup will slow down the query performance.
The B-Tree index also takes more database space and maintenance is also very costly for a large database table.

What is BRIN (Block Range Index):

The BRIN is a new database indexing technique, mainly design for very large table.

We know about the Table Partitioing concept, which we require to implement on a very large table.
The BRIN index also works like Horizontal Table Partitioning, by creating a different block of data.
If we are using BRIN, we do not require to implement Table Partitioing explicitly.

The BRIN index creates different block of data and it defines range using MinMax function.
Every block has one Min value and Max value associated which is used to identify a related block of data.
A block range is a group of pages that are physically adjacent in the table, for each block range, some summary info is stored by the index.

The BRIN index is a very lightweight because it stores only summery of the data block. Unlike B-Tree indexes, it doesn’t require to store all key node value in index page.

An Even entire form of BRIN indexes can easily store in the memory because of its compact form and it reduces the scanning of the disk.
As it is stored only summary of block so it is also 10 times faster than B-Tree for INSERT and UPDATE operations.

The creation of the BRIN indexes on a large table are also much faster than B-Tree indexes.
The efficiency of BRIN index is also depends on the physical order of table data.

The BRIN indexes are scanning the tuples bases on a bitmap and it returns only those tuples which are matched in block range summary information so It also avoids scanning of the large parts of the table.

The B-Tree indexes are still superior for random searches, and we should use the BRIN indexes for specific type of queries which are mostly used in BIGDATA, Reporting, Data Warehouse.

Please visit next post, for one practical example of BRIN Index in PostgreSQL 9.5.

” Database Administrator = 24*7 Responsible person for Database Servers. “

If any surprise call or alert happens, the DBA never shock because mostly all the systems are running 24*7 and A day in the life of a DBA is also for 24 hours.

There is no any standard working time for the DBA and it should have enough potential to solve the online problems.

“A day with a bad morning, which starts with alerts and calls”

If any call or alarm occurs in the morning time/midnight time, most of the DBA called this day as a “Black Day”.

DBA has already set the different auto notification and alert system to track the system performance. Also, has handy DBA scripts which use in any immediate situations.

But again after all this, A black day is very critical for DBA because it has to plan lots of things to get out from the emergency situation.

“A day with the Security Loopholes, which improves data security”

For every organization and system, the security of data is a preeminent concern.

DBA has to build strong security policies by applying for proper user permissions.

DBA can also enable the different type of encryption algorithms on database or objects.

This is not a new thing that after all the settled implementations, DBA never face any Security Loopholes, A day with Security Loopholes is a very critical for DBA because it has to monitor different auditing tables and logs to find out the root cause.

“A day with the Performance Analysis and Tuning, which helps to build a good customer ranking”

The Database Performance Analysis and Tuning are also one of the important exercise of DBA.

Whenever a database performance issue occurs, that day is also very critical for DBA, and it may require to perform some immediate action on that.

DBA has to keep a monitor on database performance in different aspects like internal fragmentations, external fragmentations, disk i/o, network access time, statistical metadata.

“A day with the Planning and Meeting, which increases the decision power”

DBA continuously involves in different planning and meeting with the various team members.

Before any database or server level modification, DBA has to coordinate with the various team members.

DBA should perform all impact analysis before actual changes on the database server, and these changes may relate to the database schema modification or database maintenance related stuff.

“A day with the Load Balancing / High Availability, which increases the system performance/availability”

DBA has to monitor, manage and planning the different Load Balancing/High Availablity implementations for increasing the system performance/availability.

Before any disaster happens, DBA has to be made a proper plan to implement the Load Balancing/High Availablity like Cluserting, Replication, Mirroring.

The day with this exercise is very challenging because DBA have to play with multiple database servers to achieve HA/DR.

DBA should check the status of all running servers and makes sure that all data synchronised between the servers.

“A day with the System Upgradation, which will make the system up to date”

Technology is growing like anything; everyday changes are happening in the technology market.

Frequently, all the vendors of different RDBMS/NoSQL products are releasing different patches or editions of their RDBMS products.

Either DBA has to register itself for product notification or has to find news about the new versions and patches.

DBA has to upgrade the system with the new patches and makes sure that the system is up to date which may increase the overall performance of the system.

“A day with the dedicated Resource Monitoring, which helps to minimize the system downtime”

DBA has to monitor the performance of different resources continuously.

It has to identify usage of dedicated resources because if a number of sessions/connections increases, it requires to increase the size of dedicated resources like Disk, Memory, Network bandwidth.

DBA can use the different monitoring tools and scripts for the continuous supervision of the system resources, which save the DBA day from the unwanted downtime.

“A day with the Gathering of Statistics, which requires further system enhancement”

To gather a different statistical data is also one of the day to day exercise of DBA.

DBA is collecting different information like the ratio of database size, some active connections base on various time zones, the completion time of scheduled database jobs, the end time of database transactions.

Bases on different statistics, DBA can propose further system enhancement plans to his Manager/CTO.

“A day with the self-improvement, which helps to grow personally”

DBA can never find free time from its schedule, but it has to learn the new thing and keep itself updated.

DBA personal growth is also significant because lots of new theories implemented in the market which helps DBA for a better morning.

“Are you DBA ?”

Please share your experience through the comments !

Other Interesting Laugh@dbrnd posts:

In this post, I am sharing an advice for all Database Administrators on their database maintenance task.

In this area, Database administrators have to understand two terminologies one is Recovery Time Objective (RTO) and the second is Recovery Point Objective (RPO).

Whenever you are performing database administration task, you should analyse and suggest require RTO & RPO to concern client or organization.

Whenever DBAs are taking action on Disaster Recovery or Point in time Recovery, in these most of the situations Database Downtime is required.

The RTO & RPO help us to measure the system downtime and if the system downtime is required, what are the chances of data loss.

Without properly identifying RTO & RPO, DBAs are taking a huge risk to recover a Database from the disaster.

What is Recovery Time Objective (RTO)?

Recovery Time Objective means to measure the downtime of the system and also require to measure that how long system can be accessible again.

The DBA has to measure the desired downtime, including defined downtime and how much additional downtime is required to complete the maintenance task.

There are different approaches to define the RTO like, DBA can perform the same maintenance task on the replicate test environment and can be defined require RTO for the production database.

If DBAs are doing recovery of the large database, there is also some possibility of failure so DBA has to also calculate the time of failure for a maintenance task.

When you are determining an acceptable RTO, you should be focusing on individual applications rather than entire servers. The stopwatch starts on your RTO the instant an important application is no longer usable, and won’t stop until it can be used again as normal by all users who require it.

What is Recovery Point Objective (RPO)?

Recovery Point Objective means to measure the acceptable loss of data during the recovery operation.

Before making system down, DBAs have to monitor the load of application and make sure that there are no any running connections with the application.
DBAs have to also monitor the background job and services, which are accessing the application.

If the system has a large number of running connections, there are different approaches like point your application to the secondary server and make production server free for the maintenance task.

After the maintenance DBA has to recover backup data from the secondary server to the primary server.

If the system downtime is very high, the volume of data loss is also very high.
The DBA can measure RPO like if the bulk operation is running in the system, you can easily measure that how much data are inserted in a single minute and based on this DBA can calculate the possibility of data loss.

Once you’ve figured out how much data loss and downtime will cost or impact your business, you’re then able to formulate objectives for how to mitigate those costs—which is the exact role and nature of RTOs and RPOs.

Table Scan:

It scans every row for finding a particular node or data. If the planner fails to use a proper index, it has to do full Table Scanning.
If table has a less number of records, Table Scan is an excellent choice for query optimizer.

For example, If Table has around 1000 records, Table Scan is faster because unnecessary Index Scanning operation is not required.
But, If the Table has a large number of records, Table Scan requires high CPU and other resources for scanning the whole table.

Index Scan:

It is scanning of each row, whenever you are retrieving more than 50% or 70% of the table data, internally planner performing full Index Scan.

Generally, Index pages and Data pages are stored separately.
Index scanning is also dependent on the ordering of the Index data because If matched key found from the top pages, Index Scan is not required.

The Row Lookups also involved in Index Scanning because matched Index key has to fetch data from the data page, so Index Scanning is very costly for the big table.

Index Seek:

Index Seek is opposite the Table Scan and Index Scan.
It touches the only qualified row index rather than scanning whole Index Pages, and after that, it fetches only qualified data page.

When we are fetching less number of records with the specific filter, the planner has to choose Index Seek to fetch and scan only qualified data.

Index Seek operation is much faster than Index Scan, and also it does not require more CPU and I/O resources.