Pivotal Greenplum Workload Manager Documentation | Pivotal

TableofContents
TableofContents
PivotalGreenplumWorkloadManagerDocumentation
AboutGreenplumWorkloadManager
InstallingGreenplumWorkloadManager
ManagingGreenplumWorkloadManagerServices
UsingtheGreenplumWorkloadManagerCommandLine
UsingtheWorkloadManagerGraphicalInterface(gptop)
UsingWorkloadManagerRules
UnderstandingRules
AddingRules
ManagingRules
ExampleRules
BestPracticesforRules
Caveats
QueryingWorkloadManagerRecordData
QueryingWorkloadManagerEventData
ManagingResourceQueues
ConfiguringWorkloadManagerComponents
Troubleshooting
WorkloadManagerDatumReference
©CopyrightPivotalSoftwareInc,2013-2017
1
1
2
3
4
6
8
11
13
14
19
21
23
25
26
27
29
31
34
36
37
1.7.0
PivotalGreenplumWorkloadManagerDocumentation
DocumentationforPivotalGreenplumWorkloadManager.
AboutGreenplumWorkloadManager
InstallingGreenplumWorkloadManager
ManagingGreenplumWorkloadManagerServices
UsingtheGreenplumWorkloadManagerCommandLine
UsingtheWorkloadManagerGraphicalInterface(gptop)
UsingWorkloadManagerRules
QueryingWorkloadManagerRecordData
QueryingWorkloadManagerEventData
ManagingResourceQueues
ConfiguringGreenplumWorkloadManagerComponents
Troubleshooting
WorkloadManagerDatumReference
©CopyrightPivotalSoftwareInc,2013-2017
2
1.7.0
AboutGreenplumWorkloadManager
GreenplumWorkloadManagerisamanagementtoolforGreenplumDatabaseyoucanusetomonitorandmanagequeriesandtomanageresource
queues.
YoucanuseGreenplumWorkloadManagertoperformtaskslikethese:
MonitorGreenplumDatabasequeriesandhostutilizationstatistics
Logwhenaqueryexceedsathreshold
ThrottletheCPUusageofaquerywhenitexceedsathreshold
Terminateaquery
Detectmemory,CPU,ordiskI/Oskewoccurringduringtheexecutionofaquery
Createdetailedrulestomanagequeries
Add,modify,ordeleteGreenplumDatabaseresourcequeues
WorkloadManagerArchitecture
GreenplumWorkloadManagerisasetofGreenplumDatabase-specificpluginsdeployedonanextensiblePivotalframework.Alloftheapplicationlogicis
isolatedintheseplugins.WorkloadManagerprovidesthefollowingplugins:
Agentplugins:
PublishinformationaboutactiveGreenplumDatabasequeries
Publishinformationaboutpostgresprocesses
Advertisequeryterminationcapability
Advertisequerythrottlingcapability
Advertisethresholdloggingcapability
Configurationmanagementplugins:
QuerythestateoftheGreenplumDatabaseclusterperiodically
InformtheframeworkoftheGreenplumDatabaseclusterstateandsizeallowing gp-wlm toautomaticallygrowwhenthedatabaseisexpanded.
Deployconfigurationsthroughoutthecluster.
Command-lineinterfaceplugins:
Add,modify,ordeleterules
Monitorqueriesandskew
ManageGreenplumDatabaseresourcequeues
Rulesengineplugins:
Provideextendedfunctionalityusedduringrulescreation
Theruntimeframeworkloadsthesepluginsatexecutiontime.
©CopyrightPivotalSoftwareInc,2013-2017
3
1.7.0
InstallingGreenplumWorkloadManager
Prerequisites
RedHatEnterpriseLinux(RHEL)64-bit5.5+or6orCentOS64-bit5.5+or6
GreenplumDatabaseversion4.3.x
PivotalGreenplumCommandCenterinstallerforyourplatform
Note:TheGreenplumWorkloadManagerinstallersareincludedinthePivotalGreenplumCommandCenterinstalleryoudownloadfromPivotal
Network .Theinstallerfile, gp-wlm-<version>-<platform>.bin ,isintheGreenplumCommandCenterinstallationdirectory, /usr/local/greenplum-cc-web ,by
default.
RunningtheGreenplumWorkloadManagerInstaller
GreenplumWorkloadManagerisinstalledontheGreenplumDatabasemasternode.Itautomaticallydistributesthesoftwaretoallsegmentserversinthe
databasecluster.TheinstallerdetectstheinstalledWorkloadManagerversion,ifany,andperformsanupgradeifnecessary.Runtheinstallerwiththe
--force optiontoforcereinstallationofthecurrentversion.
Thepackageinstallerhasthefollowingsyntax:
./gp-wlm-<version>-<platform>.bin--help
./gp-wlm-<version>-<platform>.bin--install=<DIR>[--force][--install-concurrency=<COUNT>]
[--no-remove-old][--skip-health-check][--dbname-records=<database_name>]
[--tool-manifest=<FILE>]
Options
--help
DisplayscommandsyntaxfortheWorkloadManagerinstaller.
--install=DIR
The --install optionisrequired.ItspecifiesthedirectorywhereGreenplumWorkloadManagerwillbeinstalled,forexample /home/gpadmin .
--force
Ifthe --install optionpointstoanexistingGreenplumWorkloadManagerinstall,theinstallerwillcheckthecurrentlyinstalledversionandperforman
upgradeonlyifthecurrentversionisolderthantheversionbeinginstalled.Ifthe --force optionisspecified,theinstallerwillallowinstallingthesame
versionofGreenplumWorkloadManagerontopofitself.Notethat --force doesnotallowyoutodowngradeGreenplumWorkloadManagertoanearlier
version.
--install-concurrency=COUNT
Themaximumnumberofhoststobootstrapatonce.Thedefaultcountiscomputedbytheinstaller.Thisoptionplacesalimitonthenumberof
processestheinstallercanfork.
--no-remove-old
Bydefault,theinstallerremovesallpreviousinstallationdirectoriesafteranupgrade.The --no-remove-old optionpreventstheinstallerfromremovingold
installationdirectories.
--skip-health-check
DonotperformaclusterhealthcheckafterWorkloadManagerinstallationcompletes.Thisoptionisnotrecommended.
--dbname-records
Thenameofthedatabasewherethe gp_wlm_records tableiscreated.Thedefaultispostgres.The template0 and template1 databasesmaynotbe
specified.Thedatabasemustexistatinstalltime.ThesamedatabasemustbespecifiedwhenupgradingtoanewWorkloadManagerrelease.
--tool-manifest
filename
Theoptional --tool-manifest optionspecifiesatextfilecontainingalistofcommandsandtheirabsolutepaths.WorkloadManagernormallyfindsstandard
systemcommandsonthepath.Ifyourenvironmenthasincompatibleimplementationsofthesecommandsonthepath,createamanifestfilethat
providestheabsolutepathtoastandardversion.
©CopyrightPivotalSoftwareInc,2013-2017
4
1.7.0
Followingisanexampletoolsmanifestfile:
stat=/home/gpadmin/bin/stat
readlink=/bin/readlink
ssh=/home/me/bin/myssh
Theinstallercreatesa gp-wlm-data directoryintheinstallationdirectoryandinstallstheGreenplumWorkloadManagerreleaseintoit.Asymboliclink
gp-wlm intheinstallationdirectorylinkstothespecificGreenplumWorkloadManagerreleasedirectory.
1. LogintotheGreenplummasterhostasthe gpadmin user.
2. EnsurethattheGreenplumWorkloadManagerinstallerisexecutable.
$chmod+xgp-wlm-<version>-<platform>.bin
3. RuntheGreenplumWorkloadManagerinstaller.Specifytheabsolutepathtoaninstallationdirectorywhereyouhavewritepermission.For
example:
$./gp-wlm-<version>-<platform>.bin--install=/home/gpadmin/
ThiscommandinstallsGreenplumWorkloadManagerinthe gp-wlm-data subdirectoryonallofthesegmentsandcreatesthe gp-wlm symboliclink.
Forexample,theabovecommandinstallsWorkloadManagerin /home/gpadmin/gp-wlm-data/gp-wlm-release andcreatesthesymboliclink /home/gpadmin/gpwlm .
Note:Inrarecases,theinstallercanfailduringthe cluster-health-check phase.Iftheclusterisreportednothealthy,re-runtheinstallerwiththe -force option.
4. ToaddtheWorkloadManagerexecutablestoyourpath,source <INSTALL_DIR>/gp-wlm/gp-wlm_path.sh inyourshell.
$source<INSTALL_DIR>/gp-wlm/gp-wlm_path.sh
Youcanaddthe source commandtoyour ~/.bash_profile or ~/.bashrc scripttoincludetheWorkloadManagerexecutablesinyourpathwheneveryou
login.
5. (Optional)Toenablethevmemdatums,seetheinstructionsintheVmemsectionoftheWorkloadManagerDatumReference.
UninstallingGreenplumWorkloadManager
TouninstallGreenplumWorkloadManager,runthefollowingcommand:
$<INSTALL_DIR>/gp-wlm/bin/uninstall--symlink<INSTALL_DIR>/gp-wlm
©CopyrightPivotalSoftwareInc,2013-2017
5
1.7.0
ManagingGreenplumWorkloadManagerServices
GreenplumWorkloadManagerinstallsandrunsfourservicesonallsegmenthostsintheGreenplumcluster:
agent
cfgmon
rabbitmq
rulesengine
Theservicescanbemanagedusingthe INSTALLDIR/gp-wlm/bin/svc-mgr.sh command.Thecommandhasthefollowingsyntax:
INSTALLDIR/gp-wlm/bin/svc-mgr.sh\
--service=SVCNAME\
--action=ACTION
SVCNAME
maybe agent , cfgmon , rabbitmq , rulesengine ,or all .If SVCNAME specifiesanindividualservice,onlythatserviceismodified.Specify all
tomanipulateallservices.
The ACTION parameteraffectsonlythelocalsystem,unlessitisprefixedwith cluster- ,inwhichcaseitrunsonallhostsinthecluster.Theactionsare:
start / cluster-start
stop / cluster-stop
–StartanyoftheWorkloadManagerservicesthatarenotrunning.
–StopanyWorkloadManagerservicesthatarerunning.
status / cluster-status
–Determineiftheservicesarerunning.
restart / cluster-restart
enable / cluster-enable
–RestarttheWorkloadManagerservices.
–EnableandstartWorkloadManagerservices.
disable / cluster-disable
–StopanddisableWorkloadManagerservices.
Ifyousourcethe INSTALLDIR/gp-wlm/gp-wlm_path.sh fileinyourshell,theWorkloadManagerscriptsareinyourpath.Otherwise,youmustprovidethefull
pathtotheutilityinthe gp-wlm/bin directory.
Whenaserviceisstopped,itwillnotberestarteduntilthe start actionisinvoked,orthelocalmachinereboots,whichevercomesfirst.
Whenaserviceisdisabled,itwillnotberestarteduntilthe enable actionisinvoked.Thisispersistentacrossreboot.
ThefollowingexamplechecksthestatusofallWorkloadManagerservicesonthelocalhost:
[gpadmin@mdw~]$svc-mgr.sh--service=all--action=status
RabbitMQisrunningoutofthecurrentinstallation.(PID=22541)
agent(pid22732)isrunning...
cfgmon(pid22858)isrunning...
rulesengine(pid22921)isrunning...
CheckingtheHealthofGreenplumWorkloadManagerServices
Atanytime,thehealthofGreenplumWorkloadManagerservicescanbeverifiedacrosstheclusterbyinvokingthe cluster-health-check utility.Thistool
confirmsthatallservicesarerunningacrossthecluster,andthatmessagesarebeingreceivedfromeachmachineinthecluster.Followingisthesyntax
for cluster-health-check :
INSTALLDIR/gpwlm/bin/cluster-health-check--symlink=/absolute/path/to/installation/symlink
[--max-concurrency=N]
[--max-cluster-checks=N]
[--help]
Options: -c or --max-concurrency
The max-concurrency optionspecifiesthenumberofhoststocheckatonce.Thedefaultisacomputedvaluebasedonthenumberofhostsinthecluster:
20iftherearefewerthan100hosts,50ifthereare100to199hosts,and75ifthereare200ormorehosts.
-m or --max-cluster-checks
Thenumberoftimestocheckforahealthycluster.Thedefaultis1.
©CopyrightPivotalSoftwareInc,2013-2017
6
1.7.0
-s or --symlink
Theabsolutepathtothe gp-wlm directorylinkedtotheinstalledWorkloadManagerrelease.Required.
-h or --help
Displaycommandusageinformationandexit.
Ifthecommandreportsanerrorcommunicatingwithoneormoreservices,theclustermayberestartedwiththiscommand:
INSTALLDIR/gp-wlm/bin/svc-mgr.sh--action=cluster-restart--service=all
ThiscommandstopsandthenrestartseachoftheWorkloadManagerservicesoneachsegmenthost.
©CopyrightPivotalSoftwareInc,2013-2017
7
1.7.0
UsingtheGreenplumWorkloadManagerCommandLine
TheGreenplumWorkloadManagercommandlineutility, gp-wlm ,providesaccesstoWorkloadManagercapabilities.Theutilitymayberunbyentering
commandsinteractivelyorbyspecifyingequivalentactionsusingcommand-lineoptions.Thecommand-lineoptionsareusefulforscripting,sincethey
requirenointeractiveuserinput.
Togethelpininteractivemode,issuethecommand: help
Togethelpforcommandlineinvocation,issuethecommand: gp-wlm-help
Belowisthe gp-wlm commandsyntax:
Usage:gp-wlm[-g|gptop]
[--rq-add=<queue-name>with<queue-settings>]
[--rq-delete=<queue-name>]
[--rq-modify=<queue-name>with<queue-settings>][--rq-show=all]
[--rq-useradd=<user>to<queue-name>]
[--rq-userdel=<user>from<queue-name>]
[--rule-add=[transient]<name><rule>]
[--rule-delete=all|<name>][--rule-dump=<path>][--rule-import=<path>]
[--rule-modify=[transient]<name><rule>][--rule-restore=<path>]
[--rule-show=all|<name>[<host><domain>]]
[--describe=<datum>]
[--config-show<component><setting>][--config-describe<component><setting>]
[--config-modify<component><setting>=<value>]
[--set-domain=<domain>][--set-host=<host>][--schema-path=<path>]
[--version][--help][--usage]
The gp-wlm command-lineoptionshaveparallelcommandsinthe gp-wlm interactivemode.Theoptiondescriptionsbelowlinktotheinteractivemode
commandsforadditionalusageinformationandexamples.
Options
-g or --gptop
Startsthe gptop graphicaluserinterface.SeeUsingtheWorkloadManagerGraphicalInterface(gptop)formoreabout gptop .
--rq-add
Addsaresourcequeuewiththespecifiednameandsettings.SeeResourceQueueSettingsfortheformatandcontentofthe<queue-settings>
argument.
--rq-delete
Deletestheresourcequeuewiththespecifiedname.SeeDeletingaResourceQueueforinformationaboutdeletingresourcequeues.
--rq-modify
Changesthesettingsfortheresourcequeuewiththespecifiednametothespecifiedsettings.SeeResourceQueueSettingsfortheformatand
contentofthe<queue-settings>argument.
--rq-show=all
Outputsalistofresourcequeuesandsettings.SeeDisplayingResourceQueuesforanexampleoftheoutput.
--rq-useradd
Addsadatabaseroletoaresourcequeue.Arolecanbelongtooneresourcequeueatatime.SeeAddingUserstoResourceQueuesfordetailsand
examples.
--rq-userdel
Deletesadatabaserolefromaresourcequeue.SeeDeletingaUserfromaResourceQueuefordetailsandexamples.
--rule-add
Addsaruletotherulesengine.SeeAddingRulesfordetailsaboutthepartsofaruleandexamples.
--rule-delete
Deletesarulewithaspecifiednameor,byusingthereservedname all ,allcurrentrules.SeeDeletingRulesfordetailsandexamples.
--rule-dump
Savesthecurrentsetofpermanentrulestoanamedfile.SeeSavingRulestoDiskfordetailsandexamples.
©CopyrightPivotalSoftwareInc,2013-2017
8
1.7.0
--rule-import
Addsrulessavedinanexternalfiletothecurrentruleset.SeeImportingRulesfordetailsandexamples.
--rule-modify
Modifiesarulebyreplacingtheruleexpressionormakingatransientrulepermanent.SeeModifyingRulesfordetails.
--rule-restore
Restorerulesfromanexternalfile,replacingthecurrentrulesintherulesengine.SeeRestoringRulesfordetails.
--rule-show
Displayarulebynameor,byusingthereservedname all ,allcurrentrules.SeeDisplayingRulesfordetailsandexamples.
--config-show
ShowthecurrentvalueofasettingforaWorkloadManagercomponent.SeeConfiguringWorkloadManagerComponentsfordetailsaboutthe
configurationcommands.
--config-describe
DescribethepurposeofasettingforaWorkloadManagercomponentanditsvalueconstraints.
--config-modify
OverridethevalueofasettingforaWorkloadManagercomponent.Thecomponentisautomaticallyrestartedafterasettingisupdated.
--set-domain
Setthedomain,orclustername,forthe gp-wlm interactivesession.Itisrecommendedtousethedefaultdomain.
--set-host
Setthehostwherethe gp-wlm sessionruns.Thedefaultisthemachinewhereyourun gp-wlm .Itisrecommendedtoonlyrun gp-wlm onthe
Greenplummasterhost.
--schema-path
Thepathtotheschemafiles.Thedefaultpath, INSTALLDIR/schema ,shouldnotbechanged.
--usage
Displaysusageinformationforthe gp-wlm command.
--help
Displaysusageinformationforthe gp-wlm command.
--describe
Displaysadescriptionofadatum.Forexample:
$gp-wlm--describe=datid:numbackends
--version or -v
Displaysthe gp-wlm version.
Usinggp-wlminInteractiveMode
1. Startgp-wlmatthecommandline:
$gp-wlm
The gp-wlm commandpromptdisplaysthenameofthehostwhere gp-wlm isrunningandthenameoftheGreenplumDatabaseclusterordomain.
Enter help attheinteractivepromptforausagemessage.
Whenusingthe gp-wlm command-line:
Entereachcommandonasingleline.Commandsareexecutedimmediately.
Enterthe help commandtoviewalistofWorkloadManagercommands.
Enter describe <datum> toviewadescriptionofadatum.
Whileenteringacommand,gethelpwithcommandsyntaxbypressingthetabkeytoshowvalidoptions.Thisisespeciallyusefulwhenconstructinga
rule.Inthefollowingpartialexample,userentryisinbold.
©CopyrightPivotalSoftwareInc,2013-2017
9
1.7.0
mdw/gpdb-cluster>rule<tab>
adddeletedumpmodifyrestoreshow
mdw/gpdb-cluster>ruleadd<tab>
<rule-name>transient
mdw/gpdb-cluster>ruleaddtransient<tab>
<rule-name>
mdw/gpdb-cluster>ruleaddtransientmyrule<tab>
gpdb_recordhost:pg_terminate_backend
mdw/gpdb-cluster>ruleaddtransientmyrulegpdb_record(<tab>
<dt>)gpdb_segment_rolemessagequery_startusename</dt>
current_queryhostpidsession_id
...
Enterthequitcommandattheprompttoexitthe gp-wlm interactivemode.
SettingtheWorkloadManagerTargetHostandDomain
Usethe sethost and setdomain commandstosetthedefaulthostanddomainfortheWorkloadManagersession.
Itisrecommendedtoonlyrunthe gp-wlm toolontheGreenplumDatabasemasternodeandtoleavethehostanddomainattheirdefaultvalues.
Thedefaulthostisthenameofthemachinewhereyouexecute gp-wlm .ThehostnamemustberesolvableinDNS.Youcanspecifydifferenthostand
clusternamesonthe gp-wlm commandlinebysupplyingthe --set-host and --set-domain commandlineoptions.
Example:
mdw/gpdb-cluster>sethostsmdw
smdw/gpdb-cluster>setdomaingpdbsys
smdw/gpdbsys>
©CopyrightPivotalSoftwareInc,2013-2017
10
1.7.0
UsingtheWorkloadManagerGraphicalInterface(gptop)
TheWorkloadManagerGraphicalinterface, gptop ,isacursesinterfacethatyoucanusetomonitorlivedatafortherulesengine,hoststatistics,active
GreenplumDatabasequeries,anddatabaseskew.
Youcanstart gptop fromthecommandlinebyrunning gptop inaterminal.Ifyouarealreadyusinginteractive gp-wlm ,enterthe gptop commandto
enterthemonitor.
Note:IfyouusethePuTTYssh/telnetclientforWindowstorun gptop ,youmayexperienceproblemswithfunctionkeysandline-drawingcharacterswith
thedefaultsettings.Tosupportfunctionkeys,inthePuTTYConfigurationwindow,chooseConnection>Dataandenter xterm-color or putty inthe
Terminal-typestringfield.Toenablecorrectline-drawingcharacters,chooseWindow>TranslationandsetRemotecharactersettoUsefont
encoding.
Whenyoufirststart gptop ,theGPDBQueriespane(seebelow)isselected.Atanytime,youcanpresstheF2keytogetapaneselectionmenu.Usethe
Tab,Left-Arrow,orRight-Arrowkeystomakeaselection.PressF2tocloseanopenmenuwithoutmakingaselection.
Anasterisk( * )nexttoacolumnheadingindicatesthattherowsaresortedbythatcolumn.Tochangethesortorder,presstheF3key,thenchoosethe
numberofthecolumnyouwanttosortbyfromthepop-upmenu.
Press q orchooseFile>Exittoleave gptop .
The gptop monitoringfeaturesareundertheMonitormenu.TheMonitormenuhasfouroptions:
GPDBQueries–ShowsactiveGreenplumDatabasequeries
GPDBSkew–Showsskewstaticsforactivequeries
Hydra–Showsstatisticsfromtherulesengine
SysData–Showsperformancestatisticsforeachhostinthecluster
GPDBQueries
TheGPDBQueriesmonitordisplaysalineforeachactiveGreenplumDatabasequery.
SessID
Thesessionidforthequery.
Time
Thenumberofsecondssincethequerybeganexecuting.
User
ThenameoftheGreenplumDatabaserolethatsubmittedthequery.
ClientAddr
Thenetworkaddressfromwhichthequerywassubmitted.
DatName
Thedatabasenamethequeryisrunningagainst.
Query
Thetextofthequery.
GPDBSkew
TheGPDBSkewmonitorshowscalculatedskewstatisticsforactiveGreenplumDatabasequeries.Statisticsarecalculatedoneachhostinthesystem
andthensenttothemasterwheretheyaresummarized.YoucanselectahostandpressEntertoseestatisticsforthehost.Thecalculatedskewvalueis
thecubedstandarddeviationacrossthecluster.Valuescloserto0.0indicatelessskew.TheGPDBSkewmonitorshowsthefollowingcolumnsforeach
activequery:
SessID
TheGreenplumDatabasesessionIDforthequery.
©CopyrightPivotalSoftwareInc,2013-2017
11
1.7.0
Time
Thenumberofsecondssincethequerystarted.
User
TheGreenplumDatabaserolethatsubmittedthequery.
CPU-Skew
AmeasureofCPUskewcalculatedasthecubedstandarddeviationofthetotalCPUforeachhostforthequery.
MEM-Skew
Ameasureofmemoryskewcalculatedasthecubedstandarddeviationofthetotalresidentsizepercentforeachhostforthequery.
READ-Skew
AmeasureofdiskreadI/Oskewcalculatedasthecubedstandarddeviationofthebytesreadpersecondforeachhostforthequery.
WRITE-Skew
AmeasureofdiskwriteI/Oskewcalculatedasthecubedstandarddeviationofthebyteswrittenpersecondforeachhostforthequery.
©CopyrightPivotalSoftwareInc,2013-2017
12
1.7.0
UsingWorkloadManagerRules
Rulestriggeractionswhentheymatchevents.Theagentpluginsonthesegmenthostscollectstatisticsandassociateddata.Therulesenginematches
themtorules,andperformsthespecifiedactionsintheagentplugins.
UnderstandingRules
AddRuleCommandSyntax
ManagingRules
ExampleRules
BestPracticesforRules
Caveats
©CopyrightPivotalSoftwareInc,2013-2017
13
1.7.0
UnderstandingRules
ThistopicprovidesanintroductiontoWorkloadManagerrulesincludinghowtowritethemandhowtheybehaveinaGreenplumDatabaseclusterwith
WorkloadManager.
RulesOverview
AWorkloadManagerrulespecifiesanactiontoexecutewhenaspecifiedconditionisdetectedintheGreenplumDatabasecluster.Administratorswrite
WorkloadManagerrulestoinvestigateproblemqueries,throttlequeriesthatconsumetoomuchCPU,orsimplyterminatequeriesthatcoulddisruptthe
databasesystem.
The rulesengine serviceoneachGreenplumhostevaluatesrulesagainstfacts,calleddatums,collectedfromtheGreenplumhostoperatingsystemsand
databaseprocesses.Atregularintervals,datumsarecollectedandsubmittedtothe rulesengine serviceoneachhost.Whentherulesenginematchesa
rule,itperformsitsaction.
Arulehasanactionexpressionandaconditionexpressionseparatedbythe WHEN keyword.Itcanbereadas“do<action-exp>WHEN<condition-exp>”.
Hereisarulethatterminatesanysessionthathasrunforover120seconds:
pg_terminate_backend()whensession_id:host:pid:runtime>120
Intheaboverule,theactionexpressionis pg_terminate_backend() andtheconditionexpressionis session_id:host:pid:runtime>
.
120
Theterm session_id:host:pid:runtime isascopeddatum; runtime isthenameofthedatumand session_id:host:pid isthescope.Thisscopeddatumspecifiesthe
elapsedexecutiontimeforaqueryexecutorprocessonasegmenthost.Thecolon-delimitedsectionsofthescopeanddatumidentifythesourceofthe
value:
session_id
host
pid
–IDofaGreenplumDatabasequery
–thenameofasegmenthost
–processIDofaqueryexecutorprocessrunningonthehost
runtime
–elapsedtimesincethequeryexecutorprocessstarted
Youcreaterulesusingthe ruleadd commandinaninteractive gp-wlm sessionorwiththe --rule-add command-lineoption.Eachrulehasauniquename
usedformanagingtherulewithcommandssuchas rulemodify or ruledelete .
Arulemayalsobelabeled transient ,whichmeanstheruleisactiveonlyuntilitisdeletedorWorkloadManagerisrestarted.
Fordetailsaboutthe ruleadd commandsyntaxandusageseeAddRules.
ForreferenceinformationaboutWorkloadManagercommandsthatmanageexistingrules(modify,delete,dump,import,andrestore),seeManaging
Rules.
Thenextsectionsprovidemoredetailedinformationaboutthecomponentsofarule:actionexpressions,conditionexpressions,datums,andscopes.
ActionExpression
TheactiontoperformwhenaruleistriggeredisspecifiedwithoneofthefollowingWorkloadManageractions:
gpdb_record
–recordacustommessageanddetailsofthedatabasequeryprocessinthe gp_wlm_records databasetable.
host:throttle_gpdb_query
host:pg_cancel_backend
pg_terminate_backend
–throttleaGreenplumDatabasequeryonaspecifiedhost.
–cancelthecurrentqueryinasessiononahostbycallingthePostgreSQL pg_cancel_backend() function.
–terminateasessionbycallingthePostgreSQL pg_terminate_backend() function.
Arule’sconditionexpressionalwaysidentifiesasinglequeryexecutorprocessonasingleGreenplumsegmenthost.Whenarule’sactionexecutes,itwill
haveinitscontextthequery’ssessionID,asegmenthostname,andtheprocessIDofasinglequeryexecutorprocessonthehost.
EachoftheactionsrespondstothesingleGreenplumDatabasequeryexecutorprocessidentifiedbytheconditionexpression.SeeRuleActionsfor
referenceinformationfortheactions.
©CopyrightPivotalSoftwareInc,2013-2017
14
1.7.0
Actionexpressionsarewrittenasfunctionsandcanhavezeroormorearguments,specifiedwith key=value pairsinparenthesesaftertheactionname:
<action-name>(<arg1>=<value1>,<arg2>=<value2>,...)
gpdb_record
The gpdb_record actionwritesthetextspecifiedinits message argumenttoalogfile,alongwithdetailsofthedatabasequeryprocessidentifiedinthe
rule’sconditionexpression.Forexample,the gpdb_record actioncanlogamessagewhenanyqueryprocessexceeds120seconds:
gpdb_record(message='queryruntimeexceeds120seconds')whensession_id:host:pid:runtime>120
The gp_wlm_records externalGreenplumDatabasetableprovidesSQLqueryaccesstotheloggedrecords.(SeeQueryingthegp_wlm_recordsTable
moreinformation.)
for
The gpdb_record actionhasseveralarguments,butonlythe message argumentisrequiredtobespecifiedintherule.Hereisthefulllistofargumentsfor
thisaction:
message
–Informativestringdescribingthereasonforrecording
current_query
-Thetextofthecurrentquery
gpdb_segment_role
host
pid
-Roleofthedatabaseinstance: GPDB_MASTER or GPDB_SEGMENT
-Thehostnameofthesegment
-Thepostgresprocessassociatedwiththequery
query_start
session_id
usename
-Querystarttime
-Sessionidofthequery
-Nameoftheuserloggedintothisbackend
Withtheexceptionof message ,avalueforeachoftheseargumentsisinferredfromthematchedqueryprocess. gpdb_record logsarecordthatincludes
thesuppliedmessage,alloftheseinferredvalues,thetextoftherule,andcontextvaluesfromtheconditionexpression.
host:throttle_gpdb_query
The host:throttle_gpdb_query actionholdsaquerytoamaximumshareofCPUonahost,specifiedinthe max_cpu argumentasapercentageofCPU
utilization.
The host: prefixonthe host:throttle_gpdb_query actionisascope.The host: scopeindicatesthattheactionwillbeperformedonlyonthehostmachines
wheretherule’sconditionismatched.The host:throttle_gpdb_query actioniscurrentlytheonlyscopedaction.(Datumsusedintheconditionexpressionare
allscoped.SeeDatumsandScopesbelowfordetails.)
This host:throttle_gpdb_query rulethrottlesaqueryonahostto30%CPUutilization:
host:throttle_gpdb_query(max_cpu=30)whensession_id:host:total_cpu>20
The session_id:host:total_cpu scopeddatumisthetotalpercentageofCPUusedbyallqueryexecutorprocessesonahostworkingonthesamequery.
Notethatthisruleestablishesarangebetween20%and30%CPUutilization.ThrottlingonahostbeginswhentotalCPUutilizationforthequeryexceeds
20%andendswhenitdropsbelow20%.ThrottlingkeepstheCPUutilizationfromexceeding30%.Setting max_cpu argumenthigherthantherule’s
triggerthresholdpreventsrapidlyalternatingbetweenthrottlingenabledandthrottlingdisabledstatesthatcouldoccurifthethresholdandmaximum
CPUareequal.
pg_cancel_backend
The host:pg_cancel_backend actioncancelsaqueryonahost.Itexecutesthe pg_cancel_backend() PostgreSQLfunctiononthesessionmatchedbythe
conditionexpression.
Thefollowingrulecancelsthecurrentqueryinasessionthatexceeds75%totalCPUutilizationonanysegmenthostandhasrunformorethanfive
minutes:
©CopyrightPivotalSoftwareInc,2013-2017
15
1.7.0
host:pg_cancel_backend()whensession_id:host:total_cpu>75andsession_id:host:pid:runtime>300
Whenarulecancelsaquery,WorkloadManagerlogstheeventinalogfileonthesegmenthost.Theseeventrecordscanbequeriedusingthe
gp_wlm_events databaseview.TheviewdependsonGreenplumexternaltablesoneachsegmenthostandmustfirstbesetupusing manage-event-tables.sh
script.SeeQueryingWorkloadManagerEventDatafordetails.
pg_terminate_backend
The pg_terminate_backend actionexecutesthePostgreSQL pg_terminate_backend() functiononthesessionmatchedbytheconditionexpression.Thisisan
unscopedactionbecauseasessionmustbeterminatedonallsegments.
Thefollowingruleterminatesasessionthatexceeds75%totalCPUutilizationonanysegmenthostandhasrunformorethanfiveminutes:
pg_terminate_backend()whensession_id:host:total_cpu>75andsession_id:host:pid:runtime>300
Whenaruleterminatesaquery,WorkloadManagerlogstheeventinalogfileoneachsegmenthost.Theseeventrecordscanbequeriedusingthe
gp_wlm_events databaseview.TheviewdependsonGreenplumexternaltablesoneachsegmenthostandmustfirstbesetupusing manage-event-tables.sh
script.SeeQueryingWorkloadManagerEventData fordetails.
ConditionExpression
Theconditionexpression(predicate)ofaruleisaBooleanexpressionthatidentifiesGreenplumDatabasequeriesyouwanttoactupon.
Datumscanbecomparedtovaluesusingthefollowingoperators.
Operator
ValueFormat
Description
=
Anumberfornumericdatumsoraquotedstringfor
strings.
Matchesonlywhenthevaluesareexactlyequal.
!=
Anumberfornumericdatumsoraquotedstringfor
strings.
Matcheswhenthevaluesarenotequal.
=~
Regularexpressionontherightsideenclosedin
slashes( / ). datum =~ /sel.*by/
Performsaregularexpressionmatchbetweenthestringvalueandthespecified
regex.Posixregularexpressionsyntaxisused.
>
Number
Greaterthan
<
Number
Lessthan
>=
Number
Greaterthanorequalto
<=
Number
Lessthanorequalto
Expressionscanbearbitrarilycomplex,joiningmultiplecomparisonswithBooleanANDandORoperatorsandparenthesestoenforceprecedence.For
example:
host:pid:cpu_util>50or(host:pid:cpu_util>30andsession_id:host:pid:usename="fred")
IncludingClause
The including keywordintroducesacomma-separatedlistofdatumstoaddtothecontextwhenaruletriggers.Anydatumreferencedinthecondition
expressionisautomaticallyaddedtothecontext.Toaddcontextvaluesfordatumsnotusedintheconditionexpression,listthedatumsafterthe
including keyword.
Datumsinthe including clausearespecifiedwithoutscopes.Iftherulescompilercannotinferthescopefromscopesalreadyboundintherule,therule
failscompilationwithanerrormessage.
Thefollowingruleaddsthe host:pid:long_name and host:pid:avg_cpu_util datumstothecontext:
gpdb_record(message="CPUover50%")whenhost:pid:cpu_util>50includinglong_name,avg_cpu_util
©CopyrightPivotalSoftwareInc,2013-2017
16
1.7.0
The host:pid:cpu_util datumisinthecontextbecauseitisreferencedintheconditionclause.
Whena gpdb_record actiontriggers,thecontextdatumsareaddedtothe context_args columnofthe gp_wlm_events table.Whena host:pg_cancel_backend or
pg_terminate_backend actiontriggers,thecontextdatumsareaddedtothe context columnofthe gp_wlm_events view.
Theadditionaldatumvaluescanprovideusefulinformationwheninvestigatingrecordedmessagesandterminationevents.
DatumsandScopes
Datumsaredataitemscollectedbytheagent,andincludeoperatingsystemstatistics,OSprocessstatistics,anddatabasequerydata.
WorkloadManagerprovidesarichsetofdatumstouseinconditionexpressionssothatyoucantargetqueriesandqueryprocesseswithveryspecific
characteristics.Forexample,arulecouldtargetqueriesexecutedwithacertaindatabaserolethataccessacertaintableanduseover30%ofCPUonany
host.
Thenameofadatumisprefixedbyitsscope,whichprovidescontextforthedatum.The host:pid scopeofthe host:pid:cpu_util datum,forexample,means
thatthe cpu_util datumisthepercentageofCPUusedbyanOSprocess( pid )executingonaspecifichost( host ).The session_id:host:pid scopeforthe
session_id:host:pid:usename datumindicatesthatthe usename datumisthedatabaseroleexecutingaGreenplumDatabasesegmentqueryprocess.The
session_id istheidofthequeryand host isthesegmenthostwherethequeryexecutorprocess, pid ,isexecuting.
Datumsinthe including listofarulearespecifiedwithoutscopes.Therulescompilersearchesforincludeddatumsinscopesalreadyboundinthe
conditionexpressionandfailsifthescopecannotbeinferred.
Rulesmustbewritteninawaytoidentifyasinglequeryexecutorprocessonahost.Thefollowingrulerecordsamessagewhentheresidentmemorysize
foranyprocessexceeds20%.The host:pid scopedoesnotincludea session_id ,soanadditionalrexexptermisaddedtotheconditionexpressionmatch
anyquery.Thisensuresthatthe host:pid:resident_size_pct datumisfromaqueryexecutorprocessandthattheactionhasaknownquerywhenitexecutes.
Withoutthe session_id:host:pid:usename comparison,thisrulewouldfailtocompile.
ruleaddmem_high_segment_useage_20
gpdb_record(message="MEM:highsegmentpctusage-20%")when
host:pid:resident_size_pct>20
andsession_id:host:pid:usename=~/.*/
WorkloadManagerDatumReferencelistsallofthedatums,theirscopes,andtheirdataformats.
Scope
The datid scopeisfordatumsthatarevaluesfromasingledatabaseintheGreenplumDatabasesystem.The datid:datname datum,for
example,canbeusedtorestrictaruletoaspecificdatabase:
datid
...anddataid:datname='my_db'
Datumswith datid scopemustbecombinedintheconditionexpressionwithotherdatumsthatidentifyaqueryprocess.
gpdb
Scope
The gpdb scopeisfordatumsfromtheentireGreenplumDatabasesystem.Thereiscurrentlyjustonesuchdatum:
gpdb:total_master_connections ,whichisthetotalnumberofclientconnectsforalldatabasesinthesystem.Thisdatumcouldbeusedto
preventarulefromtriggeringuntilaspecifiednumberofconnectionsisexceeded.
host
Scope
The host scopeappliestodatumsthatarevaluesfromasinglehostintheGreenplumcluster.Theseincludethecurrentdateandtimevalues
fromthehostandthehost’stotalCPUutilization.
Scope
The host:segment_id ScopeisusedfordatumsfromasingleGreenplumsegment.Itisusedfordatumsthatreportthevirtualmemory(vmem)
usageforasegment.
host:segment_id
Scope
The host:pid scopeisfordatumsreferringtoanyoperatingsystemprocessonahost.Thesedatumsincludethememory,CPU,andI/Ostatistics
availablefromLinuxforOSprocesses.Datumswith host:pid scopecanbeusedtonarrowaruletoqueryprocessesusingmorehostresources
thanexpected.
host:pid
Scope
A session_id istheGreenplumcluster-wideIDforadatabasequery.Thedatumswith session_id scopeareCPUanddiskI/Oskewstatistics
forasinglequerythatWorkloadManagercalculatesfromthe host:pid datumsfromallqueryexecutorprocessesonallsegmenthostsforthe
session_id
©CopyrightPivotalSoftwareInc,2013-2017
17
1.7.0
query.
Scope
The session_id:host scopeincludesdatumsthatareaggregatedmemory,CPU,andI/Ostatisticsforallprocessesonallhostsrunningaquery.
session_id:host
Scope
The session_id:host:segment_id scopeincludesdatumsthatreporttheamountofvirtualmemory(vmem)consumedbyaGreenplumsegment
forasession.
session_id:host:segment_id
Scope
The session_id:host:pid scopeisusedfordatumsthattakevaluesfromaqueryexecutorprocessonasinglesegmenthost.
session_id:host:pid
©CopyrightPivotalSoftwareInc,2013-2017
18
1.7.0
AddingRules
AddRuleCommandSyntax
The ruleadd commandaddsarule.Hereisthesyntaxfortheruleaddcommand:
ruleadd[transient]<name><action-name>(<action-args>)when<expression>[including<datum_list>]
transient
Rulesmaybepersistentortransient.Apersistentruleremainsactiveuntilitisdeleted,whileatransientruledisappearswhentherulesengine
serviceisshutdownonallhosts.Rulesarepersistentbydefault;youmustincludethe transient keywordtocreateatransientrule.
<name>
Auniquenamefortherule.Thename all isreserved.
<action-name>
Theactiontoperform.Oneofthefollowing:
host:throttle_gpdb_query
host:pg_cancel_backend
pg_terminate_backend
gpdb_record
–specifyamaximumallowedCPUutilizationpercentageforaGreenplumDatabasequery.
-cancelthecurrentqueryonahostbycallingthePostgreSQL host:pg_cancel_backend() function.
–terminateasessionbycallingthePostgreSQL pg_terminate_backend() function.
–recordaneventaboutaqueryinthe gp_wlm_records table.
<action-args>
Argumentsthatpassvaluestotheaction,ifneeded.Anargumentisspecifiedasan arg-name=value pair.Multipleargumentsareseparatedby
commas.
<expression>
ABooleanexpressionthatfilterstargetsfortheaction.Theexpressionreferencesoneormoredatumstofilterthefactsthattriggertheaction.The
expressionmaycontainPosixregularexpressions(regex).
including<datum-list>
Anoptional,comma-separatedlistofdatumstoaddtothecontextwhentheruletriggers.Withoutan including clause,theactioncontext
containsonlyvaluesfordatumsreferencedinthe expression clause.Addthe including clausetoaddvaluesforadditionaldatumstothe
actioncontext.
Datumsinthe<datum_list>arespecifiedwithoutscopeprefixes.IftheWorkloadManagerrulecompilercannotfindadatuminanycurrentlybound
scope,addingtherulefailswithanerrormessage.
When gpdb_record , host:pg_cancel_backend ,and pg_terminate_backend actionsaretriggered,thedatumsin<datum-list>areaddedtothecontext
argumentscolumnsinthe gp_wlm_records tableor gp_wlm_events view.
RuleActions
host:throttle_gpdb_query
ThrottleaGreenplumDatabasequeryonaspecifiedhost.
Arguments:
max_cpu
pid
-HoldprocesstoamaximumofthispercentageCPUutilization.
-Theprocesstothrottle.
session_id
-Thesessiontothrottle.
The max_cpu argumentisrequired.The pid and session_id argumentscanbeinferredfromthesession_idinthewhenclauseandarenormallyomitted.
©CopyrightPivotalSoftwareInc,2013-2017
19
1.7.0
host:pg_cancel_backend
Cancelaqueryonahost.ThisactioncallsthePostgreSQL pg_cancel_backend administrativefunction.
Arguments:
session_id
–ThesessionIDofthequerytoterminate.
Theargumentisnormallyomitted,allowingthesessionIDtobeinferredbyusingthesession_idintherule’swhenclause.WorkloadManagerthen
determineswhichsessiontocancel.TheactionsendsaSIGINTsignaltothesessionprocess,whichcancelsthecurrentquery.See
http://www.postgresql.org/docs/9.3/static/functions-admin.html formoredetails.
Thefollowingexamplecancelsthecurrentqueryinanysessionthathasbeenexecutingformorethan20seconds:
mdw/gpdb-cluster>ruleaddcancel_queryhost:pg_cancel_backend()
whensession_id:host:pid:runtime>20
pg_terminate_backend
Terminateasessiononallhosts.ThisactioncallsthePostgreSQL pg_terminate_backend administrativefunction.
Arguments:
session_id
–ThesessionIDtoterminate.
Theargumentisnormallyomitted,allowingthesessionIDtobeinferredbyusingthesession_idmatchedbyrule’swhenclause.WorkloadManagerthen
determineswhichpidtoterminate.Seehttp://www.postgresql.org/docs/9.3/static/functions-admin.html formoredetails.
Thefollowingexampleterminatesanysessionthathasbeenexecutingformorethan20seconds:
mdw/gpdb-cluster>ruleaddcancel_sessionpg_terminate_backend()
whensession_id:host:pid:runtime>20
gpdb_record
Logsamessagetothe gp_wlm_records tablewhenaruleismatched.
Arguments:
current_query
-Thetextofthecurrentquery
gpdb_segment_role
host
-Thehostnameofthesegment
message
pid
-Roleofthedatabaseinstance: GPDB_MASTER or GPDB_SEGMENT
–Informativestringdescribingthereasonforrecording.
-Thepostgresprocessassociatedwiththequery
query_start
session_id
usename
-Querystarttime
-Sessionidofthequery
-Nameoftheuserloggedintothisbackend
Onlythemessageargumentmustbesupplied;allotherargumentscanbeinferredfromtherule’swhenclause.
Thefollowingexamplelogsallqueries:
mdw/gpdb_cluster>ruleaddrecord_querygpdb_record(message="all")whensession_id:host:pid:usename=~/.*/
SeeQueryingthegp_wlm_recordsTable
forinformationaboutthe gp_wlm_records
©CopyrightPivotalSoftwareInc,2013-2017
table.
20
1.7.0
ManagingRules
Usingcommandsdescribedinthistopic,rulescanbedisplayed,deleted,modified,andsavedtoorrestoredfromdisk.Eachofthecommandshasa
gp-wlm command-lineequivalent.
DisplayingRules
Usethe rule
commandtoseeexistingrules.Youcanshowallexistingrulesorspecifyasinglerulebyname.
show
ruleshow{all|rule-name}
The ruleshow
commandinthisexamplelistsallregisteredrules:
all
mdw/gpdb-cluster>ruleshowall
---Name--------------Expression----------record_querygpdb_record(message="all")whensession_id:host:pid:usename=~/.*/
cancel_querypg_terminate_backend()whensession_id:host:pid:runtime>20
throttle_queryhost:throttle_gpdb_query(max_cpu=20)whensession_id:host:pid:current_query=~/.*selectcount.*/
Thisexamplelistsasinglerulebyname:
mdw/gpdb-cluster>ruleshowthrottle_query
---Name--------------Expression----------throttle_queryhost:throttle_gpdb_query(max_cpu=20)whensession_id:host:pid:current_query=~/.*selectcount.*/
DeletingRules
Theruledeletecommandremovesarule.
ruledeleterule-name
Todeleteallrulesatonce,use ruledeleteall :
ruledeleteall
Iftherearenorules,thiscommandreturnsanerror.
ModifyingaRule
Usethe rulemodify commandtoaltertheexpressionforanexistingrule.Youmayalsoremovethetransientkeywordfromtheruledeclarationtoconvert
ittoapersistentrule.Conversionfrompersistenttotransientisnotcurrentlysupported.
rulemodify[transient]nameaction-name(action-args)
whenexpression
Thisexamplemodifiesthe cancel_query ruletoalterthenumberofsecondsaqueryrunsonahosttotriggertherulefrom20to25:
mdw/gpdb-cluster>rulemodifycancel_querypg_terminate_backend()whensession_id:host:pid:runtime>25
SavingRulestoDisk
The ruledump commandsavesallpersistentrulesintheclustertoatextfile,oneruleperline.
©CopyrightPivotalSoftwareInc,2013-2017
21
1.7.0
ruledumppath
Ifyoudonotprovidethefullpathtothefile,thefileiswrittenrelativetothedirectorywhereyoustartedthe gp-wlm session.Theuserrunning gp-wlm
musthavepermissiontowritethefileatthespecifiedlocation.Ifthefileexists,the ruledump commandoverwritesit.
Thefollowingexamplesavesrulestothe /home/gpadmin/rules/20150910-1 file.Ifthe /home/gpadmin/rules directorydoesnotexist,anerrorisreported.
mdw/gpdb-cluster>ruledump/home/gpadmin/rules/20150910-1
ImportingRulesfromDisk
The ruleimport commandimportsrulesfromafileintotheactivesetofrules.Importedrulesreplaceexistingruleswiththesamenames.Existingrules
withnamesnotpresentinthefileareunchanged.
ruleimportpath
RestoringRulesfromDisk
The rule
commandrestoresallrulesfromafile,replacinganyexistingrules.Itisequivalentto rule
restore
delete=all
followedby ruleimport
.
path
rulerestorepath
©CopyrightPivotalSoftwareInc,2013-2017
22
1.7.0
ExampleRules
Thissectionprovidesexamplesofruleswrittenforvariouspurposes.
Note:Rulesmustbeenteredonasingleline,buttherulesshowninthissectionarewrappedforreadability.
Recordhighcpuutilizationqueries
Thefollowingruleinvokesthe gpdb_record actionwhenthegpadminuserrunsaqueryanditstotalcpuutilizationonahostexceeds100%.
ruleaddsimplegpdb_record(message="Toomuchcpuforgpadmin")
whensession_id:host:total_cpu>100
andsession_id:host:pid:usename='gpadmin'
Throttlethecpuutilizationofaquery
Thisruleinvokesthe host:throttle_gpdb_query actionwhenthecpuutilizationofaprocessexceedsathresholdandthequeryhasrunformorethan20
seconds.
ruleaddthrottlehost:throttle_gpdb_query(max_cpu=30)
whenhost:pid:cpu_util>20
andsession_id:host:pid:usename='gpadmin'
andsession_id:host:pid:runtime>20
Cancelanyquerywherethesessionhasrunlongerthan120seconds
Thisruleinvokesthe host:pg_cancel_backend actionwhena session_id:host:pid:runtime exceedstwominutes.
ruleaddkill_longhost:pg_cancel_backend()
whensession_id:host:pid:runtime>120
Throttleandevenoutskew
Thisruleinvokes host:throttle_gpdb_query whenthetotalcpuusageofaqueryonahostexceeds90%andthecurrentqueryisaselectontheskewtesttable.
ruleaddskewrulehost:throttle_gpdb_query(max_cpu=50)
whensession_id:host:total_cpu>100
andsession_id:host:pid:current_query=~/select.*skewtest/
Youcanobservetheeffectsofthisruleinthe gptop GPDBSkewpage.
Complexrule
Thisruleinvokes gpdb_record foraquerythatmeetsthefollowingcriteria:
aqueryhastotalCPUusagegreaterthan90%onahostandhasbeenrunningformorethan45seconds,or
hascpuskewgreaterthan20%,and
isaselectonatablethatcontains“test”initsname.
ruleaddcomborulegpdb_record(message="MyMessage")
when((session_id:host:total_cpu>90andsession_id:host:pid:runtime>45)
orsession_id:cpu_skew>20)
andsession_id:host:pid:current_query=~/select.*test/
©CopyrightPivotalSoftwareInc,2013-2017
23
1.7.0
TheruleshowshowyoucangroupBooleanexpressionswithparentheses.
Recordquerieswithhighmemoryusage
Thisrulerecordsamessagewhenaqueryprocessexceeds20%oftheresidentmemoryonahost.
ruleaddtransientmem_high_segment_useage_20
gpdb_record(message=”MEM:highsegmentpctusage-20%”)when
host:pid:resident_size_pct>20
andsession_id:host:pid:usename=~/.*/
Recordquerieswithmemory(rss)skewabove10%
Thisrulecallsthe gpdb_record actiontologwhenmemoryskewexceeds10%onahost.
ruleaddmem_skew_10gpdb_record(message="MEM:queryskew10")
whensession_id:resident_size_pct_skew>10
andsession_id:host:pid:usename=~/.*/
©CopyrightPivotalSoftwareInc,2013-2017
24
1.7.0
BestPracticesforRules
1. Avoidcreatingrulesthatmodifytheconditiontherule’sexpressionismatching.Forexample,considerthisrule:
host:throttle_gpdb_query(max_cpu=20)whenhost:pid:cpu_util>30andsession_id:host:pid:runtim>0
IfCPUusagegoesabove30%,theruletriggersandreducestheusageto20%.Whentheusagefallsbelow30%,theruleisnolongermatched,sothe
throttlingendsandusagecanagainclimbto30%.Thiscreatesanundesirablecyclicbehavior.Instead,createarulelikethefollowing:
host:throttle_gpdb_query(max_cpu=30)whenhost:pid:cpu_util>20
andsession_id:host:pid:runtime>0
Thisruletriggersat20%CPUutilizationandthrottlestheCPUto30%utilization.Thethrottlingcontinuesuntilutilizationdropsbelow20%.The
session_id:host:pid:runtime conditionistrueforanyrunningqueryandprovidesthenecessary session_id forthe throttle_gpdb_query action.
2. Avoidcreatingrulesthatterminateaquerybasedonskewalone.Considerthefollowingrule:
pg_terminate_backendwhensession_id:resident_size_pct_skew>10
Thisisapoorrulefortworeasons.First,itterminatesallquerieswhenskewisabove10,includingqueriesthatwerenotcontributingtoskew.
Second,wellbehavedqueriescantemporarilyexperienceskewhighenoughtoachievethiscondition.Forexample,ifthesegmentsdonot
completeaqueryatthesametime,skewcanappearneartheendofexecution.Aquerycouldrunnormallyacrossseveralnodesandthen,aseach
nodecompletesitsportionofthequery,itsresourceutilizationdrops,causingatemporaryincreaseinskewwhileothernodesarestillrunning.
3. Rulesthatmatchdatawith datid: scopewilltriggerforanydatabaseintheclusterunlessapredicateisaddedtoconfinethematchtoatarget
database.Forexample,thisruletriggerswheneverthenumberofconnectionstoanysingledatabaseexceeds10:
gpdb_record(message="exceeded10connections")
whensession_id:host:pid:runtime>0
anddatid:numbackends>10
Addapredicatetofilterforthedatabaseassociatedwiththesession:
gpdb_record(message="exceeded10connectionsonfoo")
whensession_id:host:pid:runtime>0
anddatid:datname="foo"
anddatid:numbackends>10
©CopyrightPivotalSoftwareInc,2013-2017
25
1.7.0
Caveats
RuleConditionsMustIncludeasession_id
TowritearulethatperformsaGreenplumDatabaseaction( gpdb_record , pg_terminate_backend , host:throttle_gpdb_query ),theconditionmustincludea
session_id ,evenwhentheintendedconditionisbasedsolelyonprocessinformation.Forexample,thefollowingruleappearstoterminateanyquerythat
usesmorethan20%ofsystemmemory:
pg_terminate_backend()whenhost:pid:resident_size_pct>20
However,becausethisrulecontainsno session_id ,WorkloadManagercannotinferthequerytoterminate,andtherulewillnotbeadded.Togetthe
desiredbehavior,addanalways-true session_id conditiontotherule,forexample:
pg_terminate_backend()whenhost:pid:program_size_pct>20
andsession_id:host:pid:runtime>0
AvoidRaceConditionsWhenUsingVmemDatums
Inrareconditions,ifmemoryallocatedforasegmentisclosetoexceeding gp_vmem_protect_limit or runaway_detector_activation_percent ,aquerythattriggers
theselimitsmaybekilledbythevmemprotectorbeforeWorkloadManagercancancelanotherquerythathasmetavmem-relatedWorkloadManager
rulecondition.
Forexample,queryQ1maybeanimportantquerythatconsumesasignificantamountofmemory.WorkloadManagerwantstoprotectQ1bykilling
otherlessimportantqueries,Q2andQ3,whichconsumelessmemory.Ifthetotalmemoryusageforasegmentrunningthesequeriesiscloseto
runaway_detector_activation_percent andWorkloadManagerdecidestokillQ2andQ3attimet,Q1maybekilledduetosegmentmemoryexceeding
runaway_detector_activation_percent attimet+1,andQ2andQ3maybekilledbyWorkloadManagerattimet+2basedonthedecisionmadeattimet.This
issuecanbeavoidedbydisabling runaway_detector_activation_percent andensuringaWorkloadManagerruletriggerswellbefore vmem_protect_limit canbe
reached.The host:segment_id:total_vmem_size_pct and session_id:host:segment_id:vmem_size_pct datumscanbeusedforthispurpose.Hereisanexamplerule:
cancel_Q2_vmem_exceedhost:pg_cancel_backend()when
host:segment_id:total_vmem_size_pct>65and
session_id:host:segment_id:vmem_size_pct>5and
session_id:host:pid:current_query=~/Q2/
Ifyouwouldliketousethesevmemdatums,besuretoenablethemasdescribedintheVmemsectionoftheWorkloadManagerDatumReference.
©CopyrightPivotalSoftwareInc,2013-2017
26
1.7.0
QueryingWorkloadManagerRecordData
The gp_wlm_records tablecontainsarecordofeventsdescribingwhere,why,andhowthe gpdb_record actionwastriggeredbyaruleontheGreenplum
cluster.
The gp_wlm_records tableiscreatedinthe postgres databasebydefault.Adifferentdatabasecanbespecifiedatinstallationtimewithwiththe
--dbname-records installationoption.
Thetablehasthefollowingstructure:
Column
Type
time
text
state
text
ident
text
hostname
text
query_start
text
message
text
pid
integer
session_id
integer
gpdb_segment_role
text
usename
text
current_query
text
rule
text
context_args
text
Theprimaryidentifierofeachentryinthetableisthe ident column.Thiscolumnstoresauniqueidentifierthatrepresentsaspecificrulethattriggeredon
aspecificnodeinthecluster.Ifaruletriggersonmorethanonenodeintheclusteratthesametime,eachnodeistreatedasaseparateeventand
receivesauniqueidentifier.
Followingaretwosampleentriesfromthe gp_wlm_records table.Inthisexample,arulewascreatedtotrackwhenaqueryrunsformorethan120
seconds:
=#\xon
Expandeddisplayison.
=#select*fromgp_wlm_records;
-[RECORD1]-----+----------------------------------------------------------------------------------------time|FriJun1714:30:272016
state|BEGIN
ident|36b3369d-0be8-4d98-b116-6d55f1caf122
hostname|sdw2
query_start|2016-06-1714:28:24.162044-07
message|Queryexceeds120seconds.
pid|98885
session_id|1112
gpdb_segment_role|GPDB_SEGMENT
usename|gpadmin
current_query|deletefromtestwheref1();
rule|gpdb_record(message="Queryexceeds120seconds.")whensession_id:host:pid:runtime>120
context_args|runtime=121
-[RECORD2]-----+----------------------------------------------------------------------------------------time|FriJun1714:31:072016
state|END
ident|36b3369d-0be8-4d98-b116-6d55f1caf122
hostname|
query_start|
message|
pid|
session_id|
gpdb_segment_role|
usename|
current_query|
rule|
context_args|
Intheaboveexample,the state columnrepresentswhenaquerybegantriggeringaruleonagivennodeandwhenitstopped.Thehostname column
storesthehostonwhichtheruletriggered.
©CopyrightPivotalSoftwareInc,2013-2017
27
1.7.0
©CopyrightPivotalSoftwareInc,2013-2017
28
1.7.0
QueryingWorkloadManagerEventData
WhenaWorkloadManagerrulesuccessfullyexecutesa pg_terminate_backend() or host:pg_cancel_backend() actiontocancelaGreenplumDatabasequery,the
eventisloggedtoafileonthehost.
The manage-event-tables.sh utilityscriptcreatesexternaltablestoaccessthelogfilesandaviewtoconsolidatetheexternaltablessothatyoucanquery
theseeventrecordsfromwithinadatabase.Theexternaltablesandviewmustfirstbecreatedusingthe manage-event-tables.sh script.Theexternaltables
arecreatedinthepostgresdatabasebydefault,butyoucanspecifyadifferentdatabasewhenyoucreatethetables.
SettingUpthegp_wlm_eventsView
The manage-event-tables.sh scriptcreates,recreates,ordropstheexternaltablesand gp_wlm_events view.
Toseethesyntax,logintotheGreenplummasterhostasthegpadminuserandrunthescriptwiththe -h ( --help )option:
$<INSTALL_DIR>/bin/manage-event-tables.sh--help
Managethegp-wlmexternaleventtables.
Commands:
--createCreatetheexternaltableandviews.
--dropDroptheexternaltableandviews.
-h,--helpDisplaythismessage.
Options:
-d,--dbname=NAMEUsedatabaseNAME.Defaultispostgres.
-q,--quietSilencestdout.
Tocreate(orrecreate)theexternaltablesandthe gp_wlm_events view,runthescriptwiththe --create flag.Ifyouwanttocreatethetablesandviewina
databaseotherthanpostgres,includethe --dbname option.
<INSTALL_DIR>/bin/manage-event-tables.sh--create--dbname=<database-name>
Todeletethetablesandviewsfromadatabaseotherthanpostgres,youmustincludethe --dbname optionwiththe --drop option.
Usingthegp_wlm_eventsView
The gp_terminate_backend and pg_cancel_backend eventsareloggedandaccessibleinthe pg_wlm_events view.
Thefollowingtabledescribesthecontentsofthe gp_wlm_events view.
Columnname
Type
Description
ident
text
Auniqueidentifierforeachrow.
time
timestamp
Thetimetheeventrecordwascreated.
rulename
text
Thenameofthetriggeredrule.
action
text
Thecomponentthattriggeredtheevent.
sess_id
integer
TheIDofthesessionthatmatchedthisruletrigger.
hostname
text
Thehostonwhichtheeventoccurred.
usename
text
Therolenamefromthesessionthatmatchedthisruletrigger.
current_query
text
Thetextofthecurrentqueryinthesession.
datname
text
Thedatabasenameofthesession.
application_name
text
Thenameoftheclientapplicationofthesessionthatmatchedthisruletrigger.
context
text
Acomma-delimitedlistofrule-specificcontextualdatums.
rule
text
Theruleexpression.
The sess_id , usename , current_query , datname ,and application_name columnsmatchcolumnswiththesamenamesinthe pg_stat_activity systemviewrow
fortheprocessthatmatchedtheruletrigger.Seepg_stat_activity .
Sincetheviewisbasedonexternaltables,eachtimeyourunaquery,theviewisrefreshedfromtheeventlogsontheGreenplumhosts.
©CopyrightPivotalSoftwareInc,2013-2017
29
1.7.0
Followingisanexampleof pg_cancel_backend and pg_terminate_backend rowsinthe gp_wlm_events view:
postgres=#select*fromgp_wlm_events;
-[RECORD1]----+-------------------------------------------------------------------------------ident|e7054d71-293b-4bce-a3bb-caafbcbc6758
time|2017-01-3119:31:02
rulename|test
action|pg_cancel_backend
sess_id|4200
hostname|localhost.localdomain
usename|pivotal
current_query|selectpg_sleep(10);
datname|postgres
application_name|psql
context|runtime=6,host=localhost.localdomain,session_id=4200,host=localhost.localdomain
rule|host:pg_cancel_backend()whensession_id:host:pid:runtime>3
-[RECORD2]----+-------------------------------------------------------------------------------ident|0c1f50dd-e1fc-4dd8-9829-7e450f74fde8
time|2017-01-3119:37:30
rulename|test2
action|pg_terminate_backend
sess_id|4226
hostname|localhost.localdomain
usename|pivotal
current_query|<IDLE>
datname|postgres
application_name|psql
context|runtime=8,session_id=4226
rule|pg_terminate_backend()whensession_id:host:pid:runtime>3
©CopyrightPivotalSoftwareInc,2013-2017
30
1.7.0
ManagingResourceQueues
Use rq commandstoshow,create,remove,andmodifyresourcequeues,andtomanagetherolesthatareassignedtoresourcequeues.
The queue-settings argumentcancontainthefollowingproperties:
PropertyName
Type
Description
active_statements
Integer
Limitsthenumberofqueriesthatcanbeexecutedbyrolesassignedtotheresourcequeue.Eitherthe
active_statements or max_cost propertymustbesetoneachresourcequeue.
max_cost
Float
Setsamaximumlimitonthetotalcostofqueriesthatcanbeexecutedbyrolesassignedtotheresource
queue.Thecostofaqueryisestimatedbythequeryplannerandismeasuredinunitsofdiskpagefetches.
Eitherthe active_statements or max_cost propertymustbesetoneachresourcequeue.
min_cost
Float
Setstheminimumestimatedquerycostforaquerytobemanagedbytheresourcequeue.Querieswith
estimatedcostsbelowthisthresholdareexecutedimmediately.
cost_overcommit
Boolean
Ifaresourcequeueislimitedbasedon MAX_COST ,aquerythatexceedsthe MAX_COST limitisallowedto
executeifthesystemisidleand COST_OVERCOMMIT is true .If COST_OVERCOMMIT issetto false ,queries
thatexceed MAX_COST arealwaysrejected.
priority
Enumeration
Setstherelativepriorityofqueriesexecutedbyrolesassignedtotheresourcequeue.Theallowedvalues,in
orderofincreasingpriority,are MIN , LOW , MEDIUM , HIGH ,and MAX .
memory_limit
Integer
(kilobytes)
Setsthetotalamountofmemorythatallactivestatementssubmittedtothequeuemayconsume.The
minimumis10240KB.Thereisnomaximum,butwhenaqueryexecutesitislimitedbythesegmenthost’s
physicalmemory.Settheparameterto-1fornolimit.
:Table1.ResourceQueueProperties
Specifytheresourcequeuepropertiesina parameter-name=value format,forexample:
mdw/gpdb-cluster>rqmodifymyrqwithactive_statements=10
Separatemultiplequeuesettingswithacomma.The queue-settings argumentmustnotcontainspaces.Thisexamplesetsthreeproperties:
mdw/gpdb-cluster>rqaddETLwith
active_statements=3,priority=LOW,memory_limit=524288
AGreenplumDatabaserole(loginuser)isassociatedwithasingleresourcequeue.Newlycreatedrolesareaddedtothe pg_default queueifanother
resourcequeueisnotspecified.
Queriessubmittedbyusersassociatedwitharesourcequeuearemanagedbythequeue.Thequeue’ssettingsdeterminewhetherqueriesareaccepted
orrejected,iftheyrunimmediatelyorwaitforresourcestobereturnedtothequeue,howmuchmemorytoallocatetothequery,andtherelativeamount
ofCPUthequerywillhave.
Resourcequeuessharethememoryallocatedtoeachsegment.Addinganewresourcequeueoralteringaqueue’ssettingsmayrequireadjustingother
resourcequeuestoavoidover-allocatingtheavailablememoryandcausingqueriestofail.SeetheWorkloadManagementsectionintheGreenplum
DatabaseAdministratorGuideforguidelinesonconfiguringresourcequeues.
AddingaNewResourceQueue
DeletingaResourceQueue
ModifyingaResourceQueue
DisplayingResourceQueues
AddingUserstoResourceQueues
DeletingaUserfromaResourceQueue
AddingaNewResourceQueue
The rqadd commandcreatesanewresourcequeue.Thecommandhasthefollowingsyntax:
©CopyrightPivotalSoftwareInc,2013-2017
31
1.7.0
rqaddqueue-namewithqueue-settings
Youmustsetoneorbothofthethresholdproperties— active_statements or max_cost —whenyoucreateanewresourcequeue.
ThefollowingexamplecreatesanETLqueuethatcanrunthreeconcurrentqueriesatlowCPUpriorityrelativetootherqueries.
mdw/gpdb-cluster>rqaddetlwithactive_statements=3,priority=low
DeletingaResourceQueue
Deleteanexistingresourcequeuebynameusingthe rqdelete command:
rqdeletequeue-name
Itisnotpossibletodeleteaqueuethathasrolesassignedtoit.
ModifyingaResourceQueue
Usethe rqmodify commandtoalterqueuesettings.Youcanaddnewsettingsorupdateexistingsettingsbyspecifyingpropertiesinthe queue-settings
argument.
rqmodifyqueue-namewithqueue-settings
ThefollowingexamplemodifiestheETLqueuetoruntwoconcurrentquerieswithamaximumof524288KB(512MB)ofmemory.Eachquerywillbe
allocated256MBofmemory.
rqmodifyetlwithactive_statements=2,memory_limit=524288
DisplayingResourceQueues
Usethe rqshow
commandtodisplayresourcequeues.Thisexampledisplayssettingsforthe etl and pg_default resourcequeues.
all
mdw/gpdb-cluster>rqshowall
rsqnameresnameressettingrestypeid
etlactive_statements21
etlmax_cost-12
etlmin_cost03
etlcost_overcommit14
etlprioritylow5
etlmemory_limit5242886
pg_defaultactive_statements101
pg_defaultmax_cost-12
pg_defaultmin_cost03
pg_defaultcost_overcommit04
pg_defaultprioritymedium5
pg_defaultmemory_limit-16
AddingUserstoResourceQueues
The rq
commandaddsausertoaresourcequeue.Theuserisremovedfromtheirpreviousresourcequeueasusersareassociatedwithonlyone
useradd
resourcequeue.Theuser’ssubsequentqueriesaremanagedbythenewresourcequeue.
rquseraddusertoqueue-name
©CopyrightPivotalSoftwareInc,2013-2017
32
1.7.0
DeletingaUserfromaResourceQueue
The rquserdel commanddeletesarolefromaresourcequeue.Theuserwillbeassociatedwiththedefaultqueue, pg_default .
rquserdeluserfromqueue-name
©CopyrightPivotalSoftwareInc,2013-2017
33
1.7.0
ConfiguringWorkloadManagerComponents
YoucanusetheGreenplumWorkloadManager config commandtoview,override,anddescribecertainWorkloadManagerconfigurationsettings.The
config commandmayberuninteractivelyina gp-wlm sessionorinbatchmodeatthecommandline.ThecommandmustberunontheGreenplum
masterhost.
SeeUsingtheGreenplumWorkloadManagerCommandLinefor gp-wlm command-linesyntaxandusage.
Note
The config commandworksonlywithsettingsthatcanbechangedbyusers.
Whenviewing,describing,orsettingthevalueofaconfigurationsetting,youmustspecifyitsWorkloadManagercomponent.Acomponentcanbean
individualservice,plugin,orcommand-linetoolthatisapartoftheWorkloadManagersystem.
Ininteractivemode,youcandouble-tapthetabcharactertoseewhichcomponentsandsettingsareavailablefortheshow,describe,andmodify
commands.
ViewingConfigurationValues
Toviewthecurrentvalueofaconfigurationsettingwhileina gp-wlm session,usethefollowingsyntax:
>configshow<component><setting>
Forexample,thefollowingcommandshowstheloggingleveloftherulesengineservice:
>configshowrulesenginelogging:log_level
Fromthecommandline,usethe --config-show option:
$gp-wlm--config-show='<component><setting>'
Forexample:
$gp-wlm--config-show='rulesenginelogging:log_level'
DescribingConfigurationValues
Usethe describe commandtoseeadescriptionofasettingandconstraintsforthesetting’svalues.
Ina gp-wlm session,thesyntaxis:
>configdescribe<component><setting>
Onthe gp-wlm commandline,usethe config-describe command-lineoption:
$gp-wlm--config-describe='<component><setting>'
Forexample,todescribetheloggingleveloftherulesengineinaninteractive gp-wlm session,usethiscommand:
>configdescriberulesenginelogging:log_level
Theoutputofthecommandlookslikethefollowing:
component:rulesengine
setting:logging:log_level
description:Thelogverbosityoftherulesenginedaemon
validvalues:err,warn,info,debug,trace
©CopyrightPivotalSoftwareInc,2013-2017
34
1.7.0
Hereisthesamecommandinbatchmodeatthecommandline:
$gp-wlm--config-describe='rulesenginelogging:log_level'
ModifyingConfigurationValues
Usethe configmodify commandtochangethevalueofaWorkloadManagerconfigurationsetting.Changingaconfigurationsettingautomatically
changesthesettingonallhostsinthecluster.
Inaninteractive gp-wlm session,usethissyntax:
>configmodify<component><setting>=<value>
Atthecommandline,usethe gp-wlm --config-modify option,withthefollowingsyntax:
$gp-wlm--config-modify='<component><setting>=<value>'
Thenewsettingispersisted,andwillbepreservedduringfutureWorkloadManagersoftwareupgrades.
Whenasettingforaserviceismodified,theaffectedserviceisautomaticallyrestartedoneveryhostinthecluster.However,thiscanonlyoccur
automaticallyifthecfgmonserviceisrunningontheGreenplummasteratthetimethesettingischanged.Ifthecfgmonserviceisnotrunning,thesetting
isstillupdatedpersistently,butthenewvalueisnotbroadcasttotherestoftheclusteruntilthecfgmonserviceisstarted.Thecfgmonserviceisalways
running,bydefault.
ConfigurableWorkloadManagerSettings
Thefollowingtablelistssettingsthatcanbeviewed,described,andconfiguredusingthe config command.
Component
Setting
Description
Type
Constraints
Default
agent
logging:log_level
Logverbosityofagent
String
ValidValues:err,warn,info,
debug,trace
info
cfgmon
logging:log_level
Logverbosityofcfgmon
String
ValidValues:err,warn,info,
debug,trace
info
gpdb_stats
collect_frequency
HowoftentocollectGPDBstatisticsinformation
Float
Validrange:0.1-60.0seconds
1.0
publish_frequency
HowoftentopublishGPDBstatisticsinformation
Float
Validrange:0.1-60.0seconds
4.0
publish_idle_sessions
PublishinformationaboutidleGreenplumDatabase
sessions
Boolean
‘true’or'false’
'true’
engine:rule_frequency
Frequencyofruleevaluationinseconds
Float
Validrange:0.1-60.0seconds
2.0
logging:log_level
Logverbosityofrulesengine
String
ValidValues:err,warn,info,
debug,trace
info
logging:log_level
Logverbosityofsystemdataplugin
String
ValidValues:err,warn,info,
debug,trace
info
publish_idle_processes
PublishinformationaboutidleGreenplumDatabase
sessions
Boolean
'true’or'false’
'true’
rulesengine
systemdata
©CopyrightPivotalSoftwareInc,2013-2017
35
1.7.0
Troubleshooting
Youmaycollectalllogsacrosstheclusterusingasinglecommand.Tocreateatarballofalllogsinthecurrentdirectory,invoke:
bin/gather-cluster-logs.sh--symlink<LN>
where LN isthepathtothe gp-wlm symboliclinktotheGreenplumWorkloadManagerinstallationdirectory.
©CopyrightPivotalSoftwareInc,2013-2017
36
1.7.0
WorkloadManagerDatumReference
ThistopicliststhedatumscollectedbyGreenplumWorkloadManager.ThesedatumscanbeusedinWorkloadManagerrulestoselectfactsthattrigger
anaction.Inrules,prefixdatumswiththeirscope,forexample:
host:cpu_util>35
Thiswillmatchanyhostwithgreaterthan35%CPUutilization.Thefollowingexpressionmatchesasingle postgres processonanyhostusingmorethan
35%CPU:
host:pid:cpu_util>35andhost:pid:name='postgres'
Thedatumsarearrangedinthefollowingcategories:
Connections–numberofbackendconnectionsandconnectionstothemaster
Identification–namesofusers,hosts,databases,ports,processes,andsoon
Transactions–informationaboutthecurrenttransaction,querieswithintransactions,andnumbersoftransactionscommittedandrolledbackinthe
database
Date/Time–dateandtimedatumsforahost
CPU–CPUutilizationforhosts,processes,andsessions
Memory–memoryutilizationforprocessesandqueries
Vmem-vmemutilizationforsegmentsandsessions
Spill–numberofspillfilescreatedandtotalspillfilesizeforaquery
I/O–diskread/writestatisticsfordatabases,processes,andqueries
Skew–diskread/writeskewandmemoryskewforqueries
Connections
Scope
Datum
Datatype
Description
datid
numbackends
integer
Numberofbackends
gpdb
total_master_connections
integer
Totalnumberofconnectionstothemastersegmentacrossalldatabases
Identification
Scope
Datum
Data
type
Description
session_id:host:pid
usename
string
Nameoftheuserloggedintothisbackend
datid
datname
string
Nameofthisdatabase
host:pid
long_name
string
Bydefault,thisistheabsolutepathtotheprocessexecutable,butmaybeoverriddenbythe
processitselftostatusinformationinutilitieslikeps(1)
host:pid
name
string
Thefilenameoftheexecutable
host:pid
state
string
Kernelstateofthisprocess;seethemanpageforproc(5)formoreinformation
session_id:host:pid
application_name
string
Nameoftheapplicationthatisconnectedtothisbackend
session_id:host:pid
client_addr
string
IPaddressoftheclientconnectedtothisbackend
session_id:host:pid
client_port
integer
TCPportnumberthattheclientisusingforcommunicationwiththisbackend
session_id:host:pid
datid
integer
OIDofthedatabasethisbackendisconnectedto
session_id:host:pid
datname
string
Nameofthedatabasethisbackendisconnectedto
session_id:host:pid
gpdb_segment_role
string
ThecurrentroleofthisGreenplumDatabasesegment(MASTER,SEGMENT,MIRROR)
session_id:host:pid
usesysid
integer
OIDoftheuserloggedintothisbackend
©CopyrightPivotalSoftwareInc,2013-2017
37
1.7.0
Transactions
Scope
Datum
Datatype
Description
datid
xact_commit
integer
Numberoftransactionsinthisdatabasethathavebeencommitted
datid
xact_rollback
integer
Numberoftransactionsinthisdatabasethathavebeenrolledback
session_id:host:pid
backend_start
string
Timewhenthisprocesswasstarted,i.e.,whentheclientconnectedtotheserver
session_id:host:pid
current_query
string
Textofthisbackend’scurrentquery.
session_id:host:pid
query_start
string
Timewhenthecurrentlyactivequerywasstarted
session_id:host:pid
runtime
integer
Timeelapsedsincethequerystarted
session_id:host:pid
xact_start
string
Timewhenthisprocess’currenttransactionwasstarted
Date/Time
Note:DateandtimevaluesarestoredinUTCstandardtimeandconvertedtothelocaltimezonefordisplay.Usethe SHOWTIMEZONE and
SETTIMEZONE
commandsin psql toviewandsetthelocaltimezone.
Scope
Datum
Datatype
Description
host
day
integer
Dayas0-30
host
day_of_week
integer
Dayas0-6
host
day_of_week_string
string
Mon,Tue,…
host
month
integer
Monthas0-11
host
year
integer
Numericyear
host
hour
integer
Houras0-23
host
minute
integer
Minuteas0-59
CPU
Scope
Datum
Data
type
Description
host
node_cpu_util
float
CurrentCPUutilizationonthishost,normalizedbynumberofactiveCPUs
host:pid
avg_cpu_util
float
AverageCPUutilizationconsumedbythisprocessoverthelasttwopollingintervals
host:pid
cpu_util
float
PercentageoftotalCPUutilizationconsumedbythisprocess
session_id
cpu_skew
float
CPUutilizationskewacrossthecluster.Calculatedasthecubedstandarddeviationof
session_id:host:total_cpufromallhostsrunningacertainquery
session_id:host
total_cpu
float
Totalcpuutilizationofallprocessesrunningacertainqueryonahost
Memory
Scope
Datum
Datatype
Description
host
mem_avail
integer
Totalavailablememoryonthishost(free+buffers+cached)(kB)
host
mem_avail_pct
float
Availablememoryonthishostaspercentageoftotal
host
mem_buffers
integer
Memoryinbuffersonthishost(kB)
host
mem_cached
integer
Cachedmemoryonthishost(kB)
host
mem_free
integer
Freememoryonthishost(kB)
host
mem_free_pct
float
Freememoryonthishostaspercentageoftotal
host
mem_total
integer
Totalmemoryonthishost(kB)
host:pid
data_size_bytes
integer
Thesizeofdata+stackmemoryregioninthisprocess(bytes)
host:pid
dirty_size_bytes
integer
Thesizeofdirtypagesusedinthisprocess(bytes)
©CopyrightPivotalSoftwareInc,2013-2017
38
1.7.0
host:pid
Scope
host:pid
library_size_bytes
Datum
program_size_bytes
integer
Datatype
integer
Thesizeoflibrarymemoryregioninthisprocess(bytes)
Description
Thetotalprogramsize(bytes)
host:pid
program_size_pct
float
Thesizeofthisprocessasapercentageoftotalsystemmemory
host:pid
resident_size_bytes
integer
Thesizeofresidentmemoryconsumedbythisprocess(bytes)
host:pid
resident_size_pct
float
Thesizeofthisprocess’residentmemoryasapercentageoftotalsystemmemory
host:pid
shared_size_bytes
integer
Thesizeofallsharedpagesusedbythisprocess(bytes)
host:pid
text_size_bytes
integer
Thesizeofcodememoryregioninthisprocess(bytes)
session_id:host
total_resident_size_pct
float
Totalresidentmemorypercentageofallprocessesrunningacertainqueryonahost
Vmem
TousethevmemdatumsinWorkloadManager,youmustfirstrunthe gp_session_state.sql scriptincludedwithGreenplumDatabase.Thisisaone-time
task.Executethescriptwiththefollowingcommand:
psql-dpostgres-f$GPHOME/share/postgresql/contrib/gp_session_state.sql
Scope
Datum
Data
type
Description
host:segment_id
total_vmem_size_mb
integer
TotalvmemusageforthisGreenplumsegmentinmegabytes
host:segment_id
total_vmem_size_pct
float
TotalvmemusageforthisGreenplumsegmentasapercentageoftotal
session_id:host:segment_id
vmem_size_mb
integer
Totalvmemusedbythesessiononthissegment
session_id:host:segment_id
vmem_size_pct
float
Thepercentageofthissegment’s gp_vmem_protect_limit consumedbythis
session
Spill
Scope
Datum
Datatype
Description
session_id:host:pid
spillfile_count_across_cluster
integer
Totalnumberofspillfilescreatedforthisqueryacrossthecluster
session_id:host:pid
spillfile_size_across_cluster
integer
Totalsizeofspillfilescreatedforthisqueryacrossthecluster
I/O
Scope
Datum
Datatype
Description
datid
blks_hit
integer
NumberoftimesdiskblockswerefoundalreadyinthePostgreSQLbuffercache
datid
blks_read
integer
Numberofdiskblocksreadinthisdatabase
host:pid
disk_read_bytes
integer
Totalnumberofbytesreadfromdiskbythisprocess
host:pid
disk_read_bytes_per_sec
float
Thenumberofbytesreadfromdiskpersecondbythisprocess
host:pid
disk_write_bytes
integer
Totalnumberofbyteswrittentodiskbythisprocess
host:pid
disk_write_bytes_per_sec
float
Thenumberofbyteswrittentodiskpersecondbythisprocess
host:pid
read_bytes
integer
Totalnumberofbytes(disk,network,IPC)readbythisprocess
host:pid
read_bytes_per_sec
float
Thenumberofbytesreadpersecond(disk+net+IPC)bythisprocess
host:pid
reads
integer
Totalnumberofreadsystemcallsmadebythisprocess
host:pid
reads_per_sec
float
Thenumberoftotalread(2)callspersecondbythisprocess
host:pid
write_bytes
integer
Totalnumberofbytes(disk,network,IPC)writtenbythisprocess
host:pid
write_bytes_per_sec
float
Thenumberofbyteswrittenpersecond(disk+net+IPC)bythisprocess
host:pid
writes
integer
Totalnumberofwritesystemcallsmadebythisprocess
host:pid
writes_per_sec
float
Thenumberoftotalwrite(2)callspersecondbythisprocess
session_id:host
total_disk_read_bytes_per_sec
integer
Totaldiskreadbytes-per-secondofallprocessesrunningacertainqueryonahost
©CopyrightPivotalSoftwareInc,2013-2017
39
1.7.0
session_id:host
Scope
total_disk_write_bytes_per_sec
Datum
integer
Datatype
Totaldiskwritebytes-per-secondofallprocessesrunningacertainqueryonahost
Description
Skew
Scope
Datum
Data
type
Description
session_id
disk_read_bytes_per_sec_skew
float
Diskreadskewacrossthecluster.Calculatedasthecubedstandarddeviationof
session_id:host:total_disk_read_bytes_per_secfromallhostsrunningacertainquery
session_id
disk_write_bytes_per_sec_skew
float
Diskwriteskewacrossthecluster.Calculatedasthecubedstandarddeviationof
session_id:host:total_disk_write_bytes_per_secfromallhostsrunningacertainquery
session_id
resident_size_pct_skew
float
Residentmemoryutilizationskewacrossthecluster.Calculatedasthecubedstandard
deviationofsession_id:host:total_resident_size_pctfromallhostsrunningacertainquery
©CopyrightPivotalSoftwareInc,2013-2017
40
1.7.0