cancel
Showing results for 
Search instead for 
Did you mean: 

pgpool load balance issue black_function_list not working sometimes

Highlighted
Level 3 Adventurer

pgpool load balance issue black_function_list not working sometimes

We are currently load testing our application using pgpool select load balancing.  Under load we are getting the following errors when the app calls nextval.  

 

Capture.PNG

 

pgpoolError.PNGCapture.PNGCapture.PNG

 

20 REPLIES 20
EDB Team Member

Re: pgpool load balance issue black_function_list not working sometimes

Hi davejung,

 

Thank you for reporting the issue with us. 

 

We are working on it and we will udpate you soon. Meanwhile, could you please confirm on the pgpool version.

Highlighted
Level 3 Adventurer

Re: pgpool load balance issue black_function_list not working sometimes

This has been tested using

 - edb .net client 3.6.2

 - pgpool 3.7 (same behavior in 4.0.1 as well)

 - epas 10

 - rhel 7.x (lastest updates)

Highlighted
Level 3 Adventurer

Re: pgpool load balance issue black_function_list not working sometimes

Hi,

 

I verified the pgpool with same version, but not getting any such issues. nextval is going to primary and select commands are coming to stanby.

I think there is some issue with your pgpool settings in pgpool.conf file. Make sure if following things are configured properly in your conf file:

 

white_function_list = 'get_.*,select_.*'
black_function_list = 'nextval,currval,lastval,setval'

database_redirect_preference_list = 'postgres:standby' ## comma separated list of pairs of database and node id.

 

In my case I am getting below status with above settings in my local environment:

 

postgres=# show pool_nodes;
node_id | hostname | port | status | lb_weight | role | select_cnt | load_balance_node | replication_delay 
---------+----------------+------+--------+-----------+---------+------------+-------------------+-------------------
0 | 172.16.106.240 | 5432 | up | 0.500000 | primary | 0 | false | 0
1 | 172.16.106.239 | 5432 | up | 0.500000 | standby | 1 | true | 0
(2 rows)

Kindly verify this at your end.

Hope this helps.

 

 

--

Regards

Ajinkya Bangale

 

 

 

Highlighted
Level 3 Adventurer

Re: pgpool load balance issue black_function_list not working sometimes

I will make the suggested changes and restart.  Remember that we are not saying this does not work but under load we are seeing only some of the queries failing.  We will report back later today.

 

Dave

Highlighted
Level 3 Adventurer

Re: pgpool load balance issue black_function_list not working sometimes

We made the changes and confirmed the pgpool.conf file.  We are still seeing the errors but not all requests are experiencing this issue

Highlighted
Level 3 Adventurer

Re: pgpool load balance issue black_function_list not working sometimes

Hi davejung,

 

 

Please attach pgpool.conf file you are using, that would be helpful for debugging an issue. 

 

Let us know in case of any issues further.

Highlighted
Level 3 Adventurer

Re: pgpool load balance issue black_function_list not working sometimes

# ----------------------------
# pgPool-II configuration file
# ----------------------------
#
# This file consists of lines of the form:
#
#   name = value
#
# Whitespace may be used.  Comments are introduced with "#" anywhere on a line.
# The complete list of parameter names and allowed values can be found in the
# pgPool-II documentation.
#
# This file is read on server startup and when the server receives a SIGHUP
# signal.  If you edit the file on a running system, you have to SIGHUP the
# server for the changes to take effect, or use "pgpool reload".  Some
# parameters, which are marked below, require a server shutdown and restart to
# take effect.
#


#------------------------------------------------------------------------------
# CONNECTIONS
#------------------------------------------------------------------------------

# - pgpool Connection Settings -

listen_addresses = '*'
                                   # Host name or IP address to listen on:
                                   # '*' for all, '' for no TCP/IP connections
                                   # (change requires restart)
port = 5454
                                   # Port number
                                   # (change requires restart)
socket_dir = '/tmp'
                                   # Unix domain socket path
                                   # The Debian package defaults to
                                   # /var/run/postgresql
                                   # (change requires restart)
listen_backlog_multiplier = 2
                                   # Set the backlog parameter of listen(2) to
								   # num_init_children * listen_backlog_multiplier.
                                   # (change requires restart)
serialize_accept = off
                                   # whether to serialize accept() call to avoid thundering herd problem
                                   # (change requires restart)

# - pgpool Communication Manager Connection Settings -

pcp_listen_addresses = '*'
                                   # Host name or IP address for pcp process to listen on:
                                   # '*' for all, '' for no TCP/IP connections
                                   # (change requires restart)
pcp_port = 9898
                                   # Port number for pcp
                                   # (change requires restart)
pcp_socket_dir = '/tmp'
                                   # Unix domain socket path for pcp
                                   # The Debian package defaults to
                                   # /var/run/postgresql
                                   # (change requires restart)

# - Backend Connection Settings -

#backend_hostname0 = 'localhost'
                                   # Host name or IP address to connect to for backend 0
#backend_port0 = 5432
                                   # Port number for backend 0
#backend_weight0 = 1
                                   # Weight for backend 0 (only in load balancing mode)
#backend_data_directory0 = '/var/lib/pgsql/data'
                                   # Data directory for backend 0
# backend_flag0 = 'ALLOW_TO_FAILOVER'
                                   # Controls various backend behavior
                                   # ALLOW_TO_FAILOVER, DISALLOW_TO_FAILOVER
				   # or ALWAYS_MASTER
# - Backend Connection Settings -
backend_hostname0 = 'server1.lab'
backend_port0 = 5444
backend_weight0 = 1
backend_data_directory0 = '/var/lib/edb/as10/data'
backend_flag0 = 'ALWAYS_MASTER'

backend_hostname1 = 'server2.thn.lab'
backend_port1 = 5444
backend_weight1 = 1
backend_data_directory1 = '/var/lib/edb/as10/data'
backend_flag1 = 'DISALLOW_TO_FAILOVER'

# - Authentication -

enable_pool_hba = on
                                   # Use pool_hba.conf for client authentication
pool_passwd = 'pool_passwd'
                                   # File name of pool_passwd for md5 authentication.
                                   # "" disables pool_passwd.
                                   # (change requires restart)
authentication_timeout = 60
                                   # Delay in seconds to complete client authentication
                                   # 0 means no timeout.

# - SSL Connections -

ssl = off
                                   # Enable SSL support
                                   # (change requires restart)
#ssl_key = './server.key'
                                   # Path to the SSL private key file
                                   # (change requires restart)
#ssl_cert = './server.cert'
                                   # Path to the SSL public certificate file
                                   # (change requires restart)
#ssl_ca_cert = ''
                                   # Path to a single PEM format file
                                   # containing CA root certificate(s)
                                   # (change requires restart)
#ssl_ca_cert_dir = ''
                                   # Directory containing CA root certificate(s)
                                   # (change requires restart)


#------------------------------------------------------------------------------
# POOLS
#------------------------------------------------------------------------------

# - Concurrent session and pool size -

num_init_children = 300
                                   # Number of concurrent sessions allowed
                                   # (change requires restart)
max_pool = 4
                                   # Number of connection pool caches per connection
                                   # (change requires restart)

# - Life time -

child_life_time = 300
                                   # Pool exits after being idle for this many seconds
child_max_connections = 0
                                   # Pool exits after receiving that many connections
                                   # 0 means no exit
connection_life_time = 0
                                   # Connection to backend closes after being idle for this many seconds
                                   # 0 means no close
client_idle_limit = 0
                                   # Client is disconnected after being idle for that many seconds
                                   # (even inside an explicit transactions!)
                                   # 0 means no disconnection


#------------------------------------------------------------------------------
# LOGS
#------------------------------------------------------------------------------

# - Where to log -

log_destination = 'syslog,stderr'
                                   # Where to log
                                   # Valid values are combinations of stderr,
                                   # and syslog. Default to stderr.

# - What to log -

log_line_prefix = '%t: pid %p: '   # printf-style string to output at beginning of each log line.
# log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d,app=%a,client=%h'

log_connections = off
                                   # Log connections
log_hostname = off
                                   # Hostname will be shown in ps status
                                   # and in logs if connections are logged
log_statement = off
                                   # Log all statements
log_per_node_statement = off
                                   # Log all statements
                                   # with node and backend informations
log_standby_delay = 'none'
                                   # Log standby delay
                                   # Valid values are combinations of always,
                                   # if_over_threshold, none

# - Syslog specific -

syslog_facility = 'LOCAL0'
                                   # Syslog local facility. Default to LOCAL0
syslog_ident = 'pgpool'
                                   # Syslog program identification string
                                   # Default to 'pgpool'

# - Debug -

#log_error_verbosity = default          # terse, default, or verbose messages

#client_min_messages = notice           # values in order of decreasing detail:
                                        #   debug5
                                        #   debug4
                                        #   debug3
                                        #   debug2
                                        #   debug1
                                        #   log
                                        #   notice
                                        #   warning
                                        #   error

#log_min_messages = warning             # values in order of decreasing detail:
                                        #   debug5
                                        #   debug4
                                        #   debug3
                                        #   debug2
                                        #   debug1
                                        #   info
                                        #   notice
                                        #   warning
                                        #   error
                                        #   log
                                        #   fatal
                                        #   panic

#------------------------------------------------------------------------------
# FILE LOCATIONS
#------------------------------------------------------------------------------

pid_file_name = '/var/run/pgpool/pgpool.pid'
                                   # PID file name
                                   # Can be specified as relative to the"
                                   # location of pgpool.conf file or
                                   # as an absolute path
                                   # (change requires restart)
logdir = '/var/log/pgpool'
                                   # Directory of pgPool status file
                                   # (change requires restart)


#------------------------------------------------------------------------------
# CONNECTION POOLING
#------------------------------------------------------------------------------

connection_cache = on
                                   # Activate connection pools
                                   # (change requires restart)

                                   # Semicolon separated list of queries
                                   # to be issued at the end of a session
                                   # The default is for 8.3 and later
reset_query_list = 'ABORT; DISCARD ALL'
                                   # The following one is for 8.2 and before
#reset_query_list = 'ABORT; RESET ALL; SET SESSION AUTHORIZATION DEFAULT'


#------------------------------------------------------------------------------
# REPLICATION MODE
#------------------------------------------------------------------------------

replication_mode = off
                                   # Activate replication mode
                                   # (change requires restart)
replicate_select = off
                                   # Replicate SELECT statements
                                   # when in replication mode
                                   # replicate_select is higher priority than
                                   # load_balance_mode.

insert_lock = on
                                   # Automatically locks a dummy row or a table
                                   # with INSERT statements to keep SERIAL data
                                   # consistency
                                   # Without SERIAL, no lock will be issued
lobj_lock_table = ''
                                   # When rewriting lo_creat command in
                                   # replication mode, specify table name to
                                   # lock

# - Degenerate handling -

replication_stop_on_mismatch = off
                                   # On disagreement with the packet kind
                                   # sent from backend, degenerate the node
                                   # which is most likely "minority"
                                   # If off, just force to exit this session

failover_if_affected_tuples_mismatch = off
                                   # On disagreement with the number of affected
                                   # tuples in UPDATE/DELETE queries, then
                                   # degenerate the node which is most likely
                                   # "minority".
                                   # If off, just abort the transaction to
                                   # keep the consistency


#------------------------------------------------------------------------------
# LOAD BALANCING MODE
#------------------------------------------------------------------------------

load_balance_mode = on
                                   # Activate load balancing mode
                                   # (change requires restart)
ignore_leading_white_space = on
                                   # Ignore leading white spaces of each query
white_function_list = ''
                                   # Comma separated list of function names
                                   # that don't write to database
                                   # Regexp are accepted
black_function_list = 'nextval,setval,currval,lastval'
                                   # Comma separated list of function names
                                   # that write to database
                                   # Regexp are accepted

black_query_pattern_list = 'SELECT\s.*_seq\.(nextval|currval|setval|lastval).*'
                                   # Semicolon separated list of query patterns
                                   # that should be sent to primary node
                                   # Regexp are accepted

database_redirect_preference_list = ''
								   # comma separated list of pairs of database and node id.
								   # example: postgres:primary,mydb[0-4]:1,mydb[5-9]:2'
								   # valid for streaming replicaton mode only.

app_name_redirect_preference_list = ''
								   # comma separated list of pairs of app name and node id.
								   # example: 'psql:primary,myapp[0-4]:1,myapp[5-9]:standby'
								   # valid for streaming replicaton mode only.
allow_sql_comments = off
								   # if on, ignore SQL comments when judging if load balance or
								   # query cache is possible.
								   # If off, SQL comments effectively prevent the judgment
								   # (pre 3.4 behavior).

disable_load_balance_on_write = 'transaction'   # Load balance behavior when write query is issued
                                                # in an explicit transaction.
                                                # Note that any query not in an explicit transaction
                                                # is not affected by the parameter.
                                                # 'transaction' (the default): if a write query is issued,
                                                # subsequent read queries will not be load balanced
                                                # until the transaction ends.
                                                # 'trans_transaction': if a write query is issued,
                                                # subsequent read queries in an explicit transaction
                                                # will not be load balanced until the session ends.
                                                # 'always': if a write query is issued, read queries will
                                                # not be load balanced until the session ends.

#------------------------------------------------------------------------------
# MASTER/SLAVE MODE
#------------------------------------------------------------------------------

master_slave_mode = on
                                   # Activate master/slave mode
                                   # (change requires restart)
master_slave_sub_mode = 'stream'
                                   # Master/slave sub mode
                                   # Valid values are combinations stream, slony
                                   # or logical. Default is stream.
                                   # (change requires restart)

# - Streaming -

sr_check_period = 0
                                   # Streaming replication check period
                                   # Disabled (0) by default
sr_check_user = 'efm'
                                   # Streaming replication check user
                                   # This is necessary even if you disable
                                   # streaming replication delay check with
                                   # sr_check_period = 0
sr_check_password = 'password'
                                   # Password for streaming replication check user
sr_check_database = 'edb'
                                   # Database name for streaming replication check
delay_threshold = 0
                                   # Threshold before not dispatching query to standby node
                                   # Unit is in bytes
                                   # Disabled (0) by default

# - Special commands -

follow_master_command = ''
                                   # Executes this command after master failover
                                   # Special values:
                                   #   %d = node id
                                   #   %h = host name
                                   #   %p = port number
                                   #   %D = database cluster path
                                   #   %m = new master node id
                                   #   %H = hostname of the new master node
                                   #   %M = old master node id
                                   #   %P = old primary node id
                                   #   %r = new master port number
                                   #   %R = new master database cluster path
                                   #   %% = '%' character

#------------------------------------------------------------------------------
# HEALTH CHECK GLOBAL PARAMETERS
#------------------------------------------------------------------------------

health_check_period = 0
                                   # Health check period
                                   # Disabled (0) by default
health_check_timeout = 20
                                   # Health check timeout
                                   # 0 means no timeout
health_check_user = 'enterprisedb'
                                   # Health check user
health_check_password = 'password'
                                   # Password for health check user
health_check_database = ''
                                   # Database name for health check. If '', tries 'postgres' frist, then 'template1'

health_check_max_retries = 0
                                   # Maximum number of times to retry a failed health check before giving up.
health_check_retry_delay = 1
                                   # Amount of time to wait (in seconds) between retries.
connect_timeout = 10000
                                   # Timeout value in milliseconds before giving up to connect to backend.
								   # Default is 10000 ms (10 second). Flaky network user may want to increase
								   # the value. 0 means no timeout.
								   # Note that this value is not only used for health check,
								   # but also for ordinary conection to backend.

#------------------------------------------------------------------------------
# HEALTH CHECK PER NODE PARAMETERS (OPTIONAL)
#------------------------------------------------------------------------------
#health_check_period0 = 0
#health_check_timeout0 = 20
#health_check_user0 = 'nobody'
#health_check_password0 = ''
#health_check_database0 = ''
#health_check_max_retries0 = 0
#health_check_retry_delay0 = 1
#connect_timeout0 = 10000

#------------------------------------------------------------------------------
# FAILOVER AND FAILBACK
#------------------------------------------------------------------------------

failover_command = ''
                                   # Executes this command at failover
                                   # Special values:
                                   #   %d = node id
                                   #   %h = host name
                                   #   %p = port number
                                   #   %D = database cluster path
                                   #   %m = new master node id
                                   #   %H = hostname of the new master node
                                   #   %M = old master node id
                                   #   %P = old primary node id
                                   #   %r = new master port number
                                   #   %R = new master database cluster path
                                   #   %% = '%' character
failback_command = ''
                                   # Executes this command at failback.
                                   # Special values:
                                   #   %d = node id
                                   #   %h = host name
                                   #   %p = port number
                                   #   %D = database cluster path
                                   #   %m = new master node id
                                   #   %H = hostname of the new master node
                                   #   %M = old master node id
                                   #   %P = old primary node id
                                   #   %r = new master port number
                                   #   %R = new master database cluster path
                                   #   %% = '%' character

fail_over_on_backend_error = on
                                   # Initiates failover when reading/writing to the
                                   # backend communication socket fails
                                   # If set to off, pgpool will report an
                                   # error and disconnect the session.

search_primary_node_timeout = 300
                                   # Timeout in seconds to search for the
                                   # primary node when a failover occurs.
                                   # 0 means no timeout, keep searching
                                   # for a primary node forever.

#------------------------------------------------------------------------------
# ONLINE RECOVERY
#------------------------------------------------------------------------------

recovery_user = 'nobody'
                                   # Online recovery user
recovery_password = ''
                                   # Online recovery password
recovery_1st_stage_command = ''
                                   # Executes a command in first stage
recovery_2nd_stage_command = ''
                                   # Executes a command in second stage
recovery_timeout = 90
                                   # Timeout in seconds to wait for the
                                   # recovering node's postmaster to start up
                                   # 0 means no wait
client_idle_limit_in_recovery = 0
                                   # Client is disconnected after being idle
                                   # for that many seconds in the second stage
                                   # of online recovery
                                   # 0 means no disconnection
                                   # -1 means immediate disconnection


#------------------------------------------------------------------------------
# WATCHDOG
#------------------------------------------------------------------------------

# - Enabling -

use_watchdog = on
                                    # Activates watchdog
                                    # (change requires restart)

# -Connection to up stream servers -

trusted_servers = ''
                                    # trusted server list which are used
                                    # to confirm network connection
                                    # (hostA,hostB,hostC,...)
                                    # (change requires restart)
ping_path = '/bin'
                                    # ping command path
                                    # (change requires restart)

# - Watchdog communication Settings -

wd_hostname = 'server1.thn.lab'
                                    # Host name or IP address of this watchdog
                                    # (change requires restart)
wd_port = 9000
                                    # port number for watchdog service
                                    # (change requires restart)
wd_priority = 1
									# priority of this watchdog in leader election
									# (change requires restart)

wd_authkey = ''
                                    # Authentication key for watchdog communication
                                    # (change requires restart)

wd_ipc_socket_dir = '/tmp'
									# Unix domain socket path for watchdog IPC socket
									# The Debian package defaults to
									# /var/run/postgresql
									# (change requires restart)


# - Virtual IP control Setting -

delegate_IP = '192.168.1.100'
                                    # delegate IP address
                                    # If this is empty, virtual IP never bring up.
                                    # (change requires restart)
if_cmd_path = '/sbin'
                                    # path to the directory where if_up/down_cmd exists 
                                    # (change requires restart)
if_up_cmd = 'ip addr add $_IP_$/24 dev eth0 label eth0:0'
                                    # startup delegate IP command
                                    # (change requires restart)
if_down_cmd = 'ip addr del $_IP_$/24 dev eth0'
                                    # shutdown delegate IP command
                                    # (change requires restart)
arping_path = '/usr/sbin'
                                    # arping command path
                                    # (change requires restart)
arping_cmd = 'arping -U $_IP_$ -w 1'
                                    # arping command
                                    # (change requires restart)

# - Behaivor on escalation Setting -

clear_memqcache_on_escalation = on
                                    # Clear all the query cache on shared memory
                                    # when standby pgpool escalate to active pgpool
                                    # (= virtual IP holder).
                                    # This should be off if client connects to pgpool
                                    # not using virtual IP.
                                    # (change requires restart)
wd_escalation_command = ''
                                    # Executes this command at escalation on new active pgpool.
                                    # (change requires restart)
wd_de_escalation_command = ''
									# Executes this command when master pgpool resigns from being master.
									# (change requires restart)

# - Watchdog consensus settings for failover -

failover_when_quorum_exists = on
									# Only perform backend node failover
									# when the watchdog cluster holds the quorum
									# (change requires restart)

failover_require_consensus = on
									# Perform failover when majority of Pgpool-II nodes
									# aggrees on the backend node status change
									# (change requires restart)

allow_multiple_failover_requests_from_node = off
									# A Pgpool-II node can cast multiple votes
									# for building the consensus on failover
									# (change requires restart)

# - Lifecheck Setting -

# -- common --

wd_monitoring_interfaces_list = ''  # Comma separated list of interfaces names to monitor.
									# if any interface from the list is active the watchdog will
									# consider the network is fine
									# 'any' to enable monitoring on all interfaces except loopback
									# '' to disable monitoring
									# (change requires restart)


wd_lifecheck_method = 'heartbeat'
                                    # Method of watchdog lifecheck ('heartbeat' or 'query' or 'external')
                                    # (change requires restart)
wd_interval = 10
                                    # lifecheck interval (sec) > 0
                                    # (change requires restart)

# -- heartbeat mode --

wd_heartbeat_port = 9694
                                    # Port number for receiving heartbeat signal
                                    # (change requires restart)
wd_heartbeat_keepalive = 2
                                    # Interval time of sending heartbeat signal (sec)
                                    # (change requires restart)
wd_heartbeat_deadtime = 30
                                    # Deadtime interval for heartbeat signal (sec)
                                    # (change requires restart)
heartbeat_destination0 = '10.200.0.92'
                                    # Host name or IP address of destination 0
                                    # for sending heartbeat signal.
                                    # (change requires restart)
heartbeat_destination_port0 = 9694 
                                    # Port number of destination 0 for sending
                                    # heartbeat signal. Usually this is the
                                    # same as wd_heartbeat_port.
                                    # (change requires restart)
heartbeat_device0 = ''
                                    # Name of NIC device (such like 'eth0')
                                    # used for sending/receiving heartbeat
                                    # signal to/from destination 0.
                                    # This works only when this is not empty
                                    # and pgpool has root privilege.
                                    # (change requires restart)

# heartbeat_destination1 = 'server3'
# heartbeat_destination_port1 = 9694
# heartbeat_device1 = ''

# -- query mode --

wd_life_point = 3
                                    # lifecheck retry times
                                    # (change requires restart)
wd_lifecheck_query = 'SELECT 1'
                                    # lifecheck query to pgpool from watchdog
                                    # (change requires restart)
wd_lifecheck_dbname = 'template1'
                                    # Database name connected for lifecheck
                                    # (change requires restart)
wd_lifecheck_user = 'nobody'
                                    # watchdog user monitoring pgpools in lifecheck
                                    # (change requires restart)
wd_lifecheck_password = ''
                                    # Password for watchdog user in lifecheck
                                    # (change requires restart)

# - Other pgpool Connection Settings -

other_pgpool_hostname0 = 'server2'
                                    # Host name or IP address to connect to for other pgpool 0
                                    # (change requires restart)
other_pgpool_port0 = 5454
                                    # Port number for other pgpool 0
                                    # (change requires restart)
other_wd_port0 = 9000
                                    # Port number for other watchdog 0
                                    # (change requires restart)
# other_pgpool_hostname1 = 'server'
# other_pgpool_port1 = 5454
# other_wd_port1 = 9000


#------------------------------------------------------------------------------
# OTHERS
#------------------------------------------------------------------------------
relcache_expire = 0
                                   # Life time of relation cache in seconds.
                                   # 0 means no cache expiration(the default).
                                   # The relation cache is used for cache the
                                   # query result against PostgreSQL system
                                   # catalog to obtain various information
                                   # including table structures or if it's a
                                   # temporary table or not. The cache is
                                   # maintained in a pgpool child local memory
                                   # and being kept as long as it survives.
                                   # If someone modify the table by using
                                   # ALTER TABLE or some such, the relcache is
                                   # not consistent anymore.
                                   # For this purpose, cache_expiration
                                   # controls the life time of the cache.

relcache_size = 256
                                   # Number of relation cache
                                   # entry. If you see frequently:
                                   # "pool_search_relcache: cache replacement happend"
                                   # in the pgpool log, you might want to increate this number.

check_temp_table = on
                                   # If on, enable temporary table check in SELECT statements.
                                   # This initiates queries against system catalog of primary/master
                                   # thus increases load of master.
                                   # If you are absolutely sure that your system never uses temporary tables
                                   # and you want to save access to primary/master, you could turn this off.
                                   # Default is on.

check_unlogged_table = on
                                   # If on, enable unlogged table check in SELECT statements.
                                   # This initiates queries against system catalog of primary/master
                                   # thus increases load of master.
                                   # If you are absolutely sure that your system never uses unlogged tables
                                   # and you want to save access to primary/master, you could turn this off.
                                   # Default is on.

#------------------------------------------------------------------------------
# IN MEMORY QUERY MEMORY CACHE
#------------------------------------------------------------------------------
memory_cache_enabled = off
								   # If on, use the memory cache functionality, off by default
memqcache_method = 'shmem'
								   # Cache storage method. either 'shmem'(shared memory) or
								   # 'memcached'. 'shmem' by default
                                   # (change requires restart)
memqcache_memcached_host = 'localhost'
								   # Memcached host name or IP address. Mandatory if
								   # memqcache_method = 'memcached'.
								   # Defaults to localhost.
                                   # (change requires restart)
memqcache_memcached_port = 11211
								   # Memcached port number. Mondatory if memqcache_method = 'memcached'.
								   # Defaults to 11211.
                                   # (change requires restart)
memqcache_total_size = 67108864
								   # Total memory size in bytes for storing memory cache.
								   # Mandatory if memqcache_method = 'shmem'.
								   # Defaults to 64MB.
                                   # (change requires restart)
memqcache_max_num_cache = 1000000
								   # Total number of cache entries. Mandatory
								   # if memqcache_method = 'shmem'.
								   # Each cache entry consumes 48 bytes on shared memory.
								   # Defaults to 1,000,000(45.8MB).
                                   # (change requires restart)
memqcache_expire = 0
								   # Memory cache entry life time specified in seconds.
								   # 0 means infinite life time. 0 by default.
                                   # (change requires restart)
memqcache_auto_cache_invalidation = on
								   # If on, invalidation of query cache is triggered by corresponding
								   # DDL/DML/DCL(and memqcache_expire).  If off, it is only triggered
								   # by memqcache_expire.  on by default.
                                   # (change requires restart)
memqcache_maxcache = 409600
								   # Maximum SELECT result size in bytes.
								   # Must be smaller than memqcache_cache_block_size. Defaults to 400KB.
                                   # (change requires restart)
memqcache_cache_block_size = 1048576
								   # Cache block size in bytes. Mandatory if memqcache_method = 'shmem'.
								   # Defaults to 1MB.
                                   # (change requires restart)
memqcache_oiddir = '/var/log/pgpool/oiddir'
				   				   # Temporary work directory to record table oids
                                   # (change requires restart)
white_memqcache_table_list = ''
                                   # Comma separated list of table names to memcache
                                   # that don't write to database
                                   # Regexp are accepted
black_memqcache_table_list = ''
                                   # Comma separated list of table names not to memcache
                                   # that don't write to database
                                   # Regexp are accepted
Highlighted
Level 3 Adventurer

Re: pgpool load balance issue black_function_list not working sometimes

Hi davejung,

 

By looking at pgpool.conf provided, we come to know that 'server1.lab' is the master server and 'server2.thn.lab' is standby.

If our understanding is correct, could you please change the parameter as below 

 

backend_hostname0 = 'server1.lab'
backend_port0 = 5444
backend_weight0 = 1

backend_weight0 = 0
backend_data_directory0 = '/var/lib/edb/as10/data'
backend_flag0 = 'ALWAYS_MASTER'

 

Kindly restart the pgpool service and check if you are getting the result as expected.

 

Note: For a query to be load balanced, all the following requirements must be met

  • the query must not be in an explicitly declared transaction (i.e. not in a BEGIN ~ END block)
  • However, if following conditions are met, load balance is possible even if in an explicit transaction
    • transaction isolation level is not SERIALIZABLE
    • transaction has not issued a write query yet (until a write query is issued, load balance is possible. Here "write query" means non SELECT DML or DDL. SELECTs having write functions as specified in black or white function list is not regarded as a write query

Let us know in case of any issues further.

Highlighted
Level 3 Adventurer

Re: pgpool load balance issue black_function_list not working sometimes

Thanks for the reply and we will make the change and test again.

 

NOTE:  Load balancing is working in our scenario.  In fact, we want to prevent the nextval function from being sent to the standby not the opposite.

 

Dave

Highlighted
Level 3 Adventurer

Re: pgpool load balance issue black_function_list not working sometimes

Setting the master server (server1) to 

 

backend_weight0 = 0

 

does not work b/c  it sends all traffiic to the standby server (server2).  Again, the problem is that the blacklisted functions "nextval" for example is being sent to the standby under load.  I even tried to add the additional blacklist_pattern 

 

black_query_pattern_list = 'SELECT\s.*_seq\..*' 

in an attempt to force the query.

 

Highlighted
Level 3 Adventurer

Re: pgpool load balance issue black_function_list not working sometimes

Hi davejung,

 

Thanks for the update provided on the issue.

 

We need to know the background of how 'SELECT nextval()' is getting called from application end.

 

We are able to reproduce this issue in the case where 'nextval()' is wrapped by function along with blacklist function provided by you and if the 'nextval()' as a standalone function call, it went on primary only.

 

So to analyze the usage of 'nexval()' function call would be useful here.

 

Kindly let us know the further information as requested above.

Highlighted
Level 3 Adventurer

Re: pgpool load balance issue black_function_list not working sometimes

We will have our developers look at the code surrounding "nextval".  In the meantime, can you provide an example of this?  Most of our calls to nextval are done as insert triggers to a table for example,

 

create function table_trigger()
  returns trigger
security definer
language edbspl
as $$
BEGIN
    IF :NEW.ID IS NULL THEN
      SELECT TABLE_SEQ.NEXTVAL INTO :NEW.ID FROM DUAL;
    END IF;
END
$$;

But these would be insert queries so I do not think it would apply

 

Highlighted
Level 3 Adventurer

Re: pgpool load balance issue black_function_list not working sometimes

So we are not wrapping the nextval function but we are calling it from our application using the following pattern

 

SELECT UNSUCCESSFULLOGINATTEMPTS_SEQ.nextval As entityId FROM Dual

and

SELECT recurringrules_seq.nextval INTO series FROM Dual;

 

The above was taken directly from the edb server logs.

 

That said, I added the additional black_list_pattern which should have worked?

Highlighted
Level 3 Adventurer

Re: pgpool load balance issue black_function_list not working sometimes

I think I may have stumbled upon what is going on and its has something to do with connection pooling at the .net client but I need further testing to confirm.  What I believe is happening is that connections in the client pool are being reused that have estabished (idle) sessions connected to the standby DB.  My theory is that a connection/session is load balanced to the standby then returned to the client connection pool.  When another process resues that connection and issues a nextval statement it is not reevaluated by pgpool and sent directly to the standby.  Considering we only get these errors when we increase the load on the application of when the connections in the pool are reused faster without time close the connection.  We also do not get errors when we set

 

disable_load_balance_on_write = always

 

In our current configuration we have pooling enabled at the client and at pgpool.  To test this theory, I am going to

1. Disable connection pooling at pgpool and enable at client

2. Disable connection pooling at client and enable at pgpool

3. Drop idle sessions immediately

 

What are your thoughts on this theory?

Highlighted
Level 3 Adventurer

Re: pgpool load balance issue black_function_list not working sometimes

Hi davejung,

 

We have gone through the details about nexval() function call. Plesae find the below suggesstion.

  1. If nextval() wrapped by function call like below
    create function table_trigger()
      returns trigger
    security definer
    language edbspl
    as $$
    BEGIN
        IF :NEW.ID IS NULL THEN
          SELECT TABLE_SEQ.NEXTVAL INTO :NEW.ID FROM DUAL;
        END IF;
    END
    $$;
    Then that function itself should be black listed.
  2. If nextval is not fuction wrapped and it is called in specific SQL pattern like below (in terms Oracle) then there are two ways to black list nextval call
    SELECT UNSUCCESSFULLOGINATTEMPTS_SEQ.nextval As entityId FROM Dual
    and
    SELECT recurringrules_seq.nextval INTO series FROM Dual;
    • You can have /*NO LOAD BALANCE*/ comment before the SELECTstatement. This will disable the load balance of the particular query and will send it to the master node (the primary node in Master Slave mode) 
    • Try to put regular expression in blacklist function to match with nextval function call
  3. As said earlier that you have also added the black list pattern for SQL like 
black_query_pattern_list = 'SELECT\s.*_seq\..*' 

 We suspect that this is failing to have pattaern match SQL template provided above, regex could be someting like 'SELECT *_seq\.*'

 

Let us know if this is helpful for you. 

 

Highlighted
Level 3 Adventurer

Re: pgpool load balance issue black_function_list not working sometimes

Hi davejung,

 

Sure, let's have some test for the above theory you described.

 

I would like to add one more point is , just disable connection pooling from both application and pgpool level and try to load balance the SQLs which are causing issue with connection pooling enabled. So it will give us fair idea about does connection pooling come in picture/causing issue while load balancing.

 

Another thing is you said that if isable_load_balance_on_write set to 'always' then everything goes well which is obvious. Because, if you set this parameter to 'always', once pgpool found write SQL, subsequent read queries are not load balanced until the session ends regardless they are in explicit transactions or not.

 

Also please share your thoughts on suggestions were made in previous post. Let us know in case of any issues.

Highlighted
Level 3 Adventurer

Re: pgpool load balance issue black_function_list not working sometimes

I have tested all scenarios and unfortunately in all cases some of the nextval queries are being sent to the standby.  For example, on app login we perform a seq.nextval to insert into the transactions table (SELECT transactions_seq.nextval as newItemId FROM Dual) as seen by the edb log file.  If that call fails you will be denied access.  When you attempt to login it works randomly and when it fails we see the above query in the standy db server logs.

 

I have also verified the regEx pattern using https://regex101.com/ Moreover, I used a broader pattern of "SELECT.*" which would match everything thus nothing is sent to the standby and it fails.  I have learned that the exclusion pattern has to match the entire query not just a piece of it.

 

This feature feels like its just not ready for primetime.  I am not sure how other are using it in production successfully under load?

Highlighted
Level 3 Adventurer

Re: pgpool load balance issue black_function_list not working sometimes

Hi davejung,

 

Below is the test case I performed.

Step 1: Initially all call to currval() was load balanced (sending them to standby)

Step 2: Added the belwo regex in blacklist function

black_function_list = '[A-Za-z]*currval'

Step 3: Reload/Restarted pgpool to reflect changes.

Step 4: Executed SQL 'select currval('testing_id_seq') from testing;' via pgpool. This is now executing on primary, which was executing on slave before adding to blacklist.

 

Let us know if this is helpful

Highlighted
Level 3 Adventurer

Re: pgpool load balance issue black_function_list not working sometimes

I will test this but for your step 1 did you have 

black_function_list = 'currval,nextval,lastval,setval'

or was it empty?

 

I ask b/c regEx wise '[A-Za-z]*currval' and 'currval' will both match

 

select table.table_seq.currval as if from dual;

 

I will test and post back.  Again, we are finding that load balancing works but under load we are seeing nextval being sent to the standby

Highlighted
Level 3 Adventurer

Re: pgpool load balance issue black_function_list not working sometimes

Just to test regex in black function list, I made these changes (kept regex only in black function list)

 

>'[A-Za-z]*currval' and 'currval' will both match

table.table_seq.currval might not match with 'currval' only, we need regex for that. Still, you can test these both.

 

> Again, we are finding that load balancing works but under load we are seeing nextval being sent to the standby

There must be some SQLs which causing issue and these are hitting under load only. Try to find out these SQLs and test these without load.