In this post, I am sharing a script to check the list of running queries or statements which are waiting in the Resource Queues of Greenplum.

The Workload management of Greenplum means – Set priorities of database user’s queries using Resource Queues. If you create RQ with the limit of 10 concurrent queries and user is trying to execute more than 10 queries at a time then few RQ keeps few queries in the waiting mode.

Using below query, you can get the list of running queries which are in the waiting mode.

SELECT
	rolname
	,rsqname
	,pid
	,granted
	,current_query
	,datname
FROM pg_roles, gp_toolkit.gp_resqueue_status
,pg_locks, pg_stat_activity
WHERE pg_roles.rolresqueue=pg_locks.objid
	AND pg_locks.objid=gp_toolkit.gp_resqueue_status.queueid
	AND pg_stat_activity.procpid=pg_locks.pid
	AND pg_stat_activity.usename=pg_roles.rolnam

In this post, I am sharing a script to find the average running time and average resource utilization of running queries in Greenplum.

I am using the table of GPCC which are available in GPPERFMON database. So if you don’t configure GPCC, you can’t use below query.

Using below query, you can find the total average run time, average CPU, average Memory occupied by long-running queries which took more than 20 minutes and checking for only last 10 days.
As per your requirement, you can change the minutes and days filter value.

select
	sh.ctime
	,query_text
	,username
	,db as DatabaseName
	,rsqname as ResourceQueueName
	,avg(tfinish-tstart) as AverageRunTime
	,count(*) as TotalExecution
	,round(avg(100 - cpu_idle)::numeric,2) as AverageCPUUsed
	,round(max(100 - cpu_idle)::numeric,2) as MaxCPUUsed
	,round(avg(mem_actual_used)/power(1024,3)::numeric,2) as AverageMemoryUsed
	,round(max(mem_actual_used)/power(1024,3)::numeric,2) as MaxMemoryUsed
from system_history sh, queries_history qh
where sh.ctime between date_trunc('day',localtimestamp - interval '10 days')
	and date_trunc('day',localtimestamp)
	and sh.ctime=qh.ctime
	and db not in ('gpperfmon')
	and date_part('hour',tfinish - tstart)*60 + date_part('minute',tfinish - tstart) > 20
group by sh.ctime,query_text,username, db,rsqname;

In this post, I am sharing a few scripts for checking the size of Database, Schema and Table of Greenplum Server.

Here, I am doing this exercise using the views of the gp_toolkit schema of Greenplum.

Check the size of database in MB, GB and TB:

SELECT sodddatname, (sodddatsize/1048576) AS Sizein_MB
FROM gp_toolkit.gp_size_of_database;
 
SELECT sodddatname, (sodddatsize/1073741824) AS Sizein_GB
FROM gp_toolkit.gp_size_of_database;
 
SELECT sodddatname, (sodddatsize/1073741824)/1024 AS Sizein_TB
FROM gp_toolkit.gp_size_of_database;

Checkt the size of database in MB, GB and TB:

SELECT sosdnsp, (sosdschematablesize/1048576) AS Sizein_MB
FROM gp_toolkit.gp_size_of_schema_disk;
 
SELECT sosdnsp, (sosdschematablesize/1073741824) AS Sizein_GB
FROM gp_toolkit.gp_size_of_schema_disk;
 
SELECT sosdnsp, (sosdschematablesize/1073741824)/1024 AS Sizein_TB
FROM gp_toolkit.gp_size_of_schema_disk;

Check the size of any Table or Index:

SELECT pg_size_pretty(pg_total_relation_size('schema_name.table_name'));

In this post, I am sharing a script to check the Tablespace and Filespace information in Greenplum.

In a big database system, DBAs are creating different tablespaces for storage purpose so using below script we can get the information of tablespaces.

SELECT
	spcname as tblspc
	,fsname as filespc
	,fsedbid as seg_dbid
	,fselocation as datadir
FROM pg_tablespace pgts
	,pg_filespace pgfs
	,pg_filespace_entry pgfse
WHERE pgts.spcfsoid=pgfse.fsefsoid
AND pgfse.fsefsoid=pgfs.oid
ORDER BY tblspc, seg_dbid

In this post, I am sharing two different scripts to check the full size of Greenplum DCA.
To monitor the disk space is a ubiquitous task of DBA. DBA can use below script and can automate the report on disk size notification/alert.

There are different options to check the disk size of the server; below I prepared two queries using gpperfmon database and gp_toolkit schema.

Use gpperfmon Database:

select
	hostname
	,filesystem
	,round(total_bytes/power(1024,4)::numeric,2) as TotalSize_inTB
	,round(bytes_used/power(1024,4)::numeric,2) as UsedSize_inTB
	,round(bytes_available/power(1024,4)::numeric,2) as AvailableSize_inTB
from diskspace_now
where filesystem in ('/data1','/data2')
ORDER BY 4 DESC

Use gp_toolkit Schema:

select
	distinct dfhostname
	,dfdevice
	,dfspace
from gp_toolkit.gp_disk_free

As Greenplum is a MPP architecture, so distribution of data in all segments is the first stuff.

You can distribute your table data using Distributed BY , and if you are not sure about a particular column, you can create your table using Distributed Randomly.

But tables which are distributed randomly, are not good for table performance because query optimizer will take more time for Randomly distributed tables.
Greenplum DBA can use below script to find randomly distributed tables.

with cte AS
(
	SELECT                 
		pgn.nspname as SchemaName                 
		,pgc.relname as TableName
		,pga.attname as DistributionType
	FROM
	(
		SELECT gdp.localoid,
			CASE
			WHEN ( Array_upper(gdp.attrnums, 1) > 0 ) THEN
			Unnest(gdp.attrnums)
			ELSE NULL
			END AS attnum
		FROM gp_distribution_policy gdp
		ORDER BY gdp.localoid
	) AS distrokey
	INNER JOIN pg_class AS pgc
		ON distrokey.localoid = pgc.oid
	INNER JOIN pg_namespace pgn
		ON pgc.relnamespace = pgn.oid
	LEFT OUTER JOIN pg_attribute pga
		ON distrokey.attnum = pga.attnum
	AND distrokey.localoid = pga.attrelid      
)
select
	SchemaName
	,TableName      
	,COALESCE(DistributionType,'DISTRIBUTED RANDOMLY') AS DistributionType
from cte
where COALESCE(DistributionType,'DISTRIBUTED RANDOMLY')='DISTRIBUTED RANDOMLY'
ORDER BY SchemaName,TableName

In this post, I am sharing a script to find a list of Append Only or Append Optimized tables of Greenplum.

Before Greenplum 4.3, when you create Append only table, you can only perform insert and select on that table.
But from Greenplum 4.3, now you can perform UPDATE as well but still Append Only is efficient storage.

The Append Only tables are faster and also it reduces the disk usage by compressing the data size. You can use below script to find a list of Append only tables.

SELECT   
	b.nspname||'.'||a.relname as TableName
	,case c.columnstore
		when 'f' then 'Row Orientation'        
		when 't' then 'Column Orientation'
	end as TableStorageType
	,case COALESCE(c.compresstype,'')
		when '' then 'No Compression'        
		else c.compresstype
	end as CompressionType
FROM pg_class a
,pg_namespace b
,(select relid,segrelid,segidxid,columnstore,compresstype from pg_appendonly) c
WHERE b.oid=a.relnamespace
	and a.oid=c.relid

In this post, I am sharing a script to find the total usage of work spill files for each running query in Greenplum.

Whenever we are doing hash aggregation operation like GROUP BY or DISTINCT on billions of data, query executor holds this data in temporary files which called work spill files.

We should optimize our query to reduce the maximum use of work spill files because it degrades the overall performance and also requires more disk space.

Using below script, we can monitor the usage of work spill file for each running query and we can also optimize the long running queries.

SELECT
	gwupq.datname
	,psa.usename
	,gwupq.procpid
	,gwupq.sess_id
	,sum(gwupq.size)/1024::float as total_spill_size_kb
	,sum(gwupq.numfiles) as total_spill_file
FROM  gp_toolkit.gp_workfile_usage_per_query as gwupq
inner join pg_stat_activity psa
	on gwupq.procpid = psa.procpid
	and gwupq.sess_id = psa.sess_id
GROUP BY 1,2,3,4;

The Greenplum is a based on MPP (Massive Parallel Processing) architecture.
There are multiple segments running in nothing shared mode that means your data should equally distribute across all segments.

If table data is not equally distributed, we cannot achieve the good performance of parallel processing system.

The Skewness of the table means that table data is not equally distributed across the segments and workload is not divided properly between the segments.

You can find skewness of data by checking gp_segment_id for each record.

The record count of segments should be very near to each other like 90% to 95%, and if you find a big difference in a count or 0 counts for few segments that mean your data is not properly distributed.

SELECT gp_segment_id, count(*)
FROM table_name
GROUP BY gp_segment_id;

Other two gp_toolkit views to get the information of Skewness of data:

gp_toolkit.gp_skew_coefficients: This view shows data distribution skew by calculating the coefficient of variation (CV) for the data stored on each segment.
gp_toolkit.gp_skew_idle_fractions: This view shows data distribution skew by calculating the percentage of the system that is idle during a table scan, which is an indicator of processing data skew.

In this post, I am sharing a script to find the total usage of work spill files for all the segments of Greenplum.

Whenever we are doing hash aggregation operations like GROUP BY or DISTINCT on billions of data, query executor holds this data in temporary files which called work spill files.

We should optimize our query to reduce the maximum use of work spill files because it degrades the overall performance and also requires more disk space.
Using below query, you can check the total number of spill files and their size for each segment.

SELECT
                gwe.datname as DatabaseName
                ,psa.usename as UserName
                ,gwe.procpid as ProcessID
                ,gwe.sess_id as SessionID
                ,sc.hostname as HostName
                ,sum(size)/1024::float as SizePerHost
                ,sum(numfiles) NumOfFilesPerHost
FROM  gp_toolkit.gp_workfile_entries as gwe
inner join pg_stat_activity as psa
                on psa.procpid = gwe.procpid
                and psa.sess_id = gwe.sess_id,
gp_segment_configuration as sc
,pg_filespace_entry as fe
,pg_database as d
WHERE fe.fsedbid=sc.dbid
                AND gwe.segid=sc.content
                AND gwe.datname=d.datname
                AND sc.role='p'
group by
                gwe.datname
                ,psa.usename
                ,gwe.procpid
                ,gwe.sess_id
                ,sc.hostname
ORDER BY
                gwe.datname
                ,psa.usename
                ,gwe.procpid
                ,gwe.sess_id
                ,sc.hostname