We all are aware about the ACID Properties of Database System (Atomicity, Consistency, Isolation, Durability).
If you don’t know about the ACID, please visit below article.

ACID Properties in Database System (Atomicity, Consistency, Isolation, Durability)

In last one decade, lots of new Database Products available in the Market. The types are: RDBMS Products, NoSQL Products, BIGDATA, MPP Products.

My Question is, how do you choose best database product for your project?

Traditional Answer, The first preference of Database Designer would be ACID Properties oriented Databases.

But most of the NoSQL Databases are not providing full support of ACID Properties, then also NoSQL Databases are popular and People are using it for different purposes.

Because most of the NoSQL Databases are not ACID oriented, but they are CAP oriented.

What is CAP theorem (Consistency, Availability, Partition)?

CAP Published by Eric Brewer in 2000, the theorem is a set of basic requirements that describe any distributed system like: NoSQL Cassandra, MongoDB, CouchDB.

Consistency:

All nodes can see the same data at the same time. The data are the same across the cluster, so you can read or write any node.

Each transaction start and end in consistent stats so that all nodes can access the same data. Actually, It is achieved by update all nodes before reading.

Availability:

Here, The meaning of Availability is to access cluster in any situation like: server goes down, server delay. The system should always respond to a request, even if data is not in sync or inconsistent state, It prints the message like “This cluster data are not updated”.

Partition:

Here, the meaning of Partition is to avoid any type of system failure and system should run with available nodes or data. It does not allow any single point of failure because data replicated across all the clusters.

The Partition can work with Consistency (CP) or can work with Availability (AP) means, you can never find any Database System which follows the full CAP theorem.

Storage System with Partition and Consistency (CP):
Redis, MongoDB, AppFabric Caching, and MemcacheDB.
Storage System with Partition and Availability (AP):
Cassandra, DynamoDB, and CouchDB.

I am telling you personally that I hate the use of DISTINCT.

DISTINCT used by those people, who are not sure about their data set or SELECT statement or JOINS.

Whenever I get any query with DISTINCT, immediately I suggest to remove it.

I asked to many developers that, does it makes sense to use DISTINCT or do we have any alternative?
Second, I asked that Why are you required to use DISTINCT.

Guys, what are you thinking? What was that answer?

Very simple, they keep quiet for a minute and were telling like: we are using to avoid duplicate result set.

This is not the correct answer for the use of DISTINCT.

Actually, most of developers are using DISTINCT because they want to hide their mistakes. They are writing incorrect joins and hiding duplication of data.

This is not only for database developer, DBA is a same responsible for this. The DBA must stop the duplication of data.

The DBA should give the answer that how duplicates data inserted in the database and who is responsible for that duplication.

Do you think, the use of DISTINCT is good for query performance?

My answer is simply no. Why, because it fetches unnecessary duplicate data and returns unique from it.

You should know the logical sequence of SQL Query execution. In the logical sequence, DISTINCT execute after SELECT means It checks all the data first and after that It avoids the duplicate data.

As I have raised a question for DISTINCT, few developers asked me to use GROUP BY instead of DISTINCT.

My simple answer was, Please do not use anything to remove duplicates from your result set.

Explicitly, you should remove the duplicate data and then also If you require to count few unique master group data, you can use GROUP BY instead of DISTINCT.

In this post, I am sharing the list of common practices for Database Designing which should follow by Database Developers and Application Developers.

• Foreign keys should have indexes on them, and it should often cover multiple columns base on the queries you need to execute.
• You should make sure about the referential integrity by enforcing the Foreign Keys.
  Choose right column for CLUSTER Index.
• You should apply proper indexes and do not create duplicate indexes and avoid creation of unnecessary indexes.
• You should not put auto-number data type as a PRIMARY KEY to join a FOREIGN KEY in a parent/child table relationship.
• If your system is running large number of INSERT/DELETE, you should avoid frequent update of database Statistics.
• You should choose the write data-type for your data columns.
• Please separate big size of columns from main table and store in a different table.
• You should apply proper column name.
• Use proper data types for Latitude, Longitude, IP-Address, Network Address, Mac Address.
• Use Natural Primary key rather than Surrogate Primary key. Natural Primary keys are keys base on an external meaningful data. Surrogate Primary key are keys base on an auto-increment number or values which have no meaning outside the system.
• You must use Stored Procedures or Stored Functions.
• Please avoid too much database normalization and try to design your database to achieve Database Normalization + Database Denormalization. You should design this combination to achieve the performance.
• Avoid inappropriate or excessive use of cursors.
• Avoid inappropriate use of Database Transactions (BEGIN…COMMIT).
• Try to avoid the use of OR condition because an AND condition restricts the data set, whereas an OR condition grows it.
• You should do detailed performance analysis on queries.
• Try to use the Prepared SQL Statement which caches the compiled query and the execution plan for better performance. It will also better protect you against SQL injection attacks.
• Please do not simplify your complex queries through the views.
• Please do not use DISTINCT frequently. Many developers are using DISTINCT to hide duplicate data, but you should stop and remove the duplicate data.
• Try to avoid Correlated Subqueries.
• Do not store comma delimited list in a table.
• Do not store JSON data in TEXT COLUMN, you should use JSON data type for it.
• Please do not use comma separated table names in SELECT query, use explicit JOINs conditions to avoid internal Cartesian products.

 
The list is not final yet, you guys can add/share a comment and make it final…

One of the Important news for Database Community.
The UBER taxi number one company switched from PostgreSQL to MySQL.

There are a few major reasons behind for this migration.
As per the official document of UBER, they were facing a few notable issues with the PostgreSQL.

• Inefficient architecture for writes
• Inefficient data replication
• Issues with table corruption
• Poor replica MVCC support
• Difficulty upgrading to newer releases

Here, I am sharing the official link on UBER database migration. I completely read this article and agree with it on a few of the topics.
You guys must read carefully because we should understand the pros & cons of any RDBMS.

As a database professional, I work with both databases on a regular basis, and I typically use all the features of both databases for design and implementation.
I have got many replies to write about the core differences between PostgreSQL and MySQL.

As innovation on these databases has progressed, each development community has actively made changes to address their respective sets of perceived disadvantages.
The result that it has gotten more difficult to objectively determine which database is likely to be better suited for a given application.

MySQL is controlled by Oracle, whereas Postgres is available under an open-source license from the PostgreSQL Global Development Group.

PostgreSQL is closer to the ANSI SQL standard, MySQL is closer to the ODBC standard.

One of the most notable differences between MySQL and PostgreSQL is the fact that you can’t do nested subqueries of subselects in MySQL. MySQL doesn’t support CTE, window functions, full outer joins and arrays.

Compare to PostgreSQL, MySQL has its own large user community and various type of materials.

MySQL uses nonstandard ‘#’ to begin a comment line and PostgreSQL use ‘–‘ to begin a comment line.

MySQL provides different type of table storage engines like: InnoDB, MyIsam, Memory. PostgreSQL does not provide different type of table stroage engines.

PostgreSQL is case-sensitive for string comparisons. The field “DbRnd” is not the same as the field “dbrnd”. This is a big change for many users from MySQL and other small database systems.

PostgreSQL has some nice features like: generate_series, custom aggregate functions, arrays etc. other hand MySQL has also some nice feature like: session variables in queries, FORCE INDEX, etc.

In PostgreSQL, when you are inside a transaction almost any operation can be undone. There are some irreversible operations (like creating or destroying a database or tablespace), but normal table modifications can be backed out by issuing a ROLLBACK via its Write-Ahead Log design.

MySQL doesn’t support any sort of rollback when using MyISAM. With InnoDB, the server has an implicit commit that occurs even if the normal auto-commit behavior is turned off. This means that any single table alteration or similar change is immediately committed.

Postgres obliterates MySQL in all manner of query-level goodness, and we can use Postgres for data mining and batch analytics databases, where we run lots of very large, hand-coded queries that can take advantage of Postgres’ advantages in join methods (MySQL only supports Nested Loop and Nested Loop over Index, while Postgres supports numerous methods), generally richer querying capabilities, and other good stuff like partial indexes can be used extensively.

PostgreSQL uses a robust locking model called MVCC that limits situations where individual clients interfere with each other. A short summary of the main benefit of MVCC would be “readers are never blocked by writers”.

MySQL’s InnoDB implements MVCC using a rollback segment and InnoDB databases supports all four SQL standard transaction isolation levels.

PostgreSQL uses cost-based query optimization methods in order to get good performance for many different types of joins. Query costs are assessed based on planner statistics collected when tables are analyzed combined with adjustable planner costs and advanced features such as the Genetic Query Optimizer.

MySQL doesn’t have this level of sophistication in its planner, and the tunables for Controlling Query Optimizer Performance are crude. Developers instead do things like explicitly provide index hints to make sure joins are executed correctly.

PostgreSQL has different replication options than MySQL. MySQL has no point-in-time recovery.

One operation that PostgreSQL is known to be slow performing is doing a full count of rows in a table, typically using this SQL:

SELECT COUNT(*) FROM your_table_name;

The reason why this is slow is related to the MVCC implementation in PostgreSQL. The fact that multiple transactions can see different states of the data means that there can be no straightforward way for “COUNT(*)” to summarize data across the whole table; PostgreSQL must walk through all rows, in some sense.

In MySQL, MyISAM tables cache the row count information, making this type of count operation almost instant. But if you’re using InnoDB instead, this is no longer the case and on InnoDB can’t assume that a full row count will be fast.

In online forums, I am observing one bad query practice. People are using multiple tables in the SELECT query by separating using commas.
Bad Query Like:

SELECT * FROM a, b, c WHERE a.id = b.id AND b.id = c.id;

Many of the Database Professionals are thinking that It is working like JOINs.

I am writing this article in the “Database Designing” category, means It applies to all database professionals for knowing the truth of this kind of bad SQL Query.

When you write a query like above sample, internally Query Planner creates a virtual join between these tables and this is not a strait forward joins creation. A Query Planner has to check all possible combinations like: first join A TO B and then join C, join A TO C and then join B, join B TO C and then join A.

When you write such types of bad queries, Query Planner has to do additional work for joining the tables correctly and return the correct result.
But actually It checks the data by creating a full Cartesian Product between these tables which is very costly and slow down the query performance.

We have taken example of three tables only, now let’s assume if we have 10 tables and people wrote a SELECT query using comma separated tables.
So please I am requesting, never use such a bad query and try to write explicit JOINs in the SELECT Query.

Another interesting point is:

When I discussed this point with our team, one of database developers said It is good that Query Planner is deciding the order of JOINS. If we give explicit joins, It may not be in the correct order all the times so let’s Query Planner to do.

My simple answer was:
I am 5% agree with you, but do not forget that we are enough professional and can define the JOINs order correctly. We should not rely on Query Planner because It requires more internal I/O to just prepare a joins of the table.

Correct Query should like:

SELECT a.* FROM a JOIN b ON a.id=b.id JOIN c ON b.id=c.id

In the past decade, lots of analysis and research one for the Parallel Database System and Parallel Query Processing.

The success of Parallel Database System depends on the relational database model and good CPU / DISK performance. Without a good CPU and DISK performance, we should not use the Parallel Query Processing.

In general,
When we are executing any SQL Query, It creates the execution plan and submits to the internal query executor. A query executor collects all execution plans and executes in sequence.
For example, your one SQL Query requires 100,00,0000 times manipulation, so using a single CPU thread system It takes more time to execute.

In the Parallel Query Processing,
More than one process worker running in the background and responsible for the same task or multiple tasks in the sharing mode.

For example, If we say 4 different process worker are executing one single big SQL Query which requires 100,00,0000 times manipulation, It completes the whole process four times faster than using single process worker or single CPU thread for same SQL Query.

Basically, there are main three types of architecture of Parallel Database System.

Shared Disk:

This approach is not that much popular where the same disk shared among the different background process workers.

Shared Nothing:

With a shared nothing system, each processor owns a portion of the database, and only that portion may be directly accessed or manipulated by that processor. A two-phase commit protocol is required to coordinate a transaction commit which involves multiple nodes.

A shared nothing system is based on the concept of declustering. Declustering a relation includes distributing its tuples among multiple nodes according to some distribution criteria such as applying a hash function to the key attribute of each tuple.

In the context of query processing, the main advantage of a shared nothing system is its scalability.
One of the best example of this architecture is: GreenPlum Relational Database.

Shared Everything:

In a shared everything system, main memory, in addition to disks, is also shared across all the processors, making system management and load balancing much easier.

First, there are no communication delays because messages are exchanged through the shared memory, and synchronization can be accomplished by cheap, low level mechanisms.

Second, load balancing is much easier because the operating system can automatically allocate the next ready process to the first available processor.

Tomorrow, You can access one good article about Performance Test of Parallel Query Processing using PostgreSQL 9.6 Database System.

The BRIN index is very new type of index in a Database System.

Currently we are using B-Tree indexes most, but BRIN Indexes are much faster than B-Tree Indexes.
It is not correct to say that BRIN is a 100% replacement of B-Tree but in some situation, it executes 10 times faster than B-Tree Index.

Let me provide you a small note on B-Tree as well,

The structure of B-Tree looks like a Binary Search Tree.
If you have a large table, overall branching of BTree Index is very large because it required to store thousands of keys in a B-Tree node.

The B-Tree also stored all the key values in their leaf nodes so for every search key lookups require to scan both the leaf node and B-Tree mapping node.

If we have very large table with the B-Tree index and we are selecting a large number of records, B-Tree index lookup will slow down the query performance.
The B-Tree index also takes more database space and maintenance is also very costly for a large database table.

What is BRIN (Block Range Index):

The BRIN is a new database indexing technique, mainly design for very large table.

We know about the Table Partitioing concept, which we require to implement on a very large table.
The BRIN index also works like Horizontal Table Partitioning, by creating a different block of data.
If we are using BRIN, we do not require to implement Table Partitioing explicitly.

The BRIN index creates different block of data and it defines range using MinMax function.
Every block has one Min value and Max value associated which is used to identify a related block of data.
A block range is a group of pages that are physically adjacent in the table, for each block range, some summary info is stored by the index.

The BRIN index is a very lightweight because it stores only summery of the data block. Unlike B-Tree indexes, it doesn’t require to store all key node value in index page.

An Even entire form of BRIN indexes can easily store in the memory because of its compact form and it reduces the scanning of the disk.
As it is stored only summary of block so it is also 10 times faster than B-Tree for INSERT and UPDATE operations.

The creation of the BRIN indexes on a large table are also much faster than B-Tree indexes.
The efficiency of BRIN index is also depends on the physical order of table data.

The BRIN indexes are scanning the tuples bases on a bitmap and it returns only those tuples which are matched in block range summary information so It also avoids scanning of the large parts of the table.

The B-Tree indexes are still superior for random searches, and we should use the BRIN indexes for specific type of queries which are mostly used in BIGDATA, Reporting, Data Warehouse.

Please visit next post, for one practical example of BRIN Index in PostgreSQL 9.5.

” Database Administrator = 24*7 Responsible person for Database Servers. “

If any surprise call or alert happens, the DBA never shock because mostly all the systems are running 24*7 and A day in the life of a DBA is also for 24 hours.

There is no any standard working time for the DBA and it should have enough potential to solve the online problems.

“A day with a bad morning, which starts with alerts and calls”

If any call or alarm occurs in the morning time/midnight time, most of the DBA called this day as a “Black Day”.

DBA has already set the different auto notification and alert system to track the system performance. Also, has handy DBA scripts which use in any immediate situations.

But again after all this, A black day is very critical for DBA because it has to plan lots of things to get out from the emergency situation.

“A day with the Security Loopholes, which improves data security”

For every organization and system, the security of data is a preeminent concern.

DBA has to build strong security policies by applying for proper user permissions.

DBA can also enable the different type of encryption algorithms on database or objects.

This is not a new thing that after all the settled implementations, DBA never face any Security Loopholes, A day with Security Loopholes is a very critical for DBA because it has to monitor different auditing tables and logs to find out the root cause.

“A day with the Performance Analysis and Tuning, which helps to build a good customer ranking”

The Database Performance Analysis and Tuning are also one of the important exercise of DBA.

Whenever a database performance issue occurs, that day is also very critical for DBA, and it may require to perform some immediate action on that.

DBA has to keep a monitor on database performance in different aspects like internal fragmentations, external fragmentations, disk i/o, network access time, statistical metadata.

“A day with the Planning and Meeting, which increases the decision power”

DBA continuously involves in different planning and meeting with the various team members.

Before any database or server level modification, DBA has to coordinate with the various team members.

DBA should perform all impact analysis before actual changes on the database server, and these changes may relate to the database schema modification or database maintenance related stuff.

“A day with the Load Balancing / High Availability, which increases the system performance/availability”

DBA has to monitor, manage and planning the different Load Balancing/High Availablity implementations for increasing the system performance/availability.

Before any disaster happens, DBA has to be made a proper plan to implement the Load Balancing/High Availablity like Cluserting, Replication, Mirroring.

The day with this exercise is very challenging because DBA have to play with multiple database servers to achieve HA/DR.

DBA should check the status of all running servers and makes sure that all data synchronised between the servers.

“A day with the System Upgradation, which will make the system up to date”

Technology is growing like anything; everyday changes are happening in the technology market.

Frequently, all the vendors of different RDBMS/NoSQL products are releasing different patches or editions of their RDBMS products.

Either DBA has to register itself for product notification or has to find news about the new versions and patches.

DBA has to upgrade the system with the new patches and makes sure that the system is up to date which may increase the overall performance of the system.

“A day with the dedicated Resource Monitoring, which helps to minimize the system downtime”

DBA has to monitor the performance of different resources continuously.

It has to identify usage of dedicated resources because if a number of sessions/connections increases, it requires to increase the size of dedicated resources like Disk, Memory, Network bandwidth.

DBA can use the different monitoring tools and scripts for the continuous supervision of the system resources, which save the DBA day from the unwanted downtime.

“A day with the Gathering of Statistics, which requires further system enhancement”

To gather a different statistical data is also one of the day to day exercise of DBA.

DBA is collecting different information like the ratio of database size, some active connections base on various time zones, the completion time of scheduled database jobs, the end time of database transactions.

Bases on different statistics, DBA can propose further system enhancement plans to his Manager/CTO.

“A day with the self-improvement, which helps to grow personally”

DBA can never find free time from its schedule, but it has to learn the new thing and keep itself updated.

DBA personal growth is also significant because lots of new theories implemented in the market which helps DBA for a better morning.

“Are you DBA ?”

Please share your experience through the comments !

Other Interesting Laugh@dbrnd posts:

Don’t make us Angry !
“Are you storing comma Separated List in Database”
Get lost !

I request to all Database Professionals,
“Please share your experience through the comments”

Database Design: Storing a comma separated list in a Database, Is a Bad Practice?

GIF reference taken from giphy.com.