In specific scenarios, parallel queries may be queued due to waiting for thread resources.
Concurrency control
You can specify the maximum number of parallel worker threads a tenant can provide on each node using the tenant-level variable Set parallel execution parameters. Before a parallel query starts, worker thread resources are reserved from all involved OBServer nodes. As long as any OBServer node cannot provide sufficient resources for the parallel query, the query will not be executed. The query is then requeued and waits for the next execution attempt to try again for thread resources until enough worker thread resources are available. Once the entire query is completed, the reserved worker thread resources are immediately released.
This process of "attempting to reserve worker thread resources - requeuing if resources are insufficient - gaining another execution opportunity - attempting to reserve worker thread resources again" is called parallel query queuing. The parallel execution resource manager is the module responsible for managing all worker thread resource reservations on the OBServer nodes.
To calculate the number of worker threads required for each parallel query, the parallel execution resource manager performs DFO partitioning on the query plan, simulates the scheduling DFO process, and calculates the maximum number of threads required by the query on each OBServer node based on parameters such as parallel hints and table parallelism. This set of thread counts is called the "resource vector."
The resource vector is a logical concept used to control concurrency and queuing. After reserving sufficient worker thread resources from the parallel execution resource manager through the resource vector, the parallel query is executed. During execution, although physical threads are continuously acquired and released as different DFOs are scheduled, the logical thread resources are not returned to the parallel execution resource manager. These resource vectors are only returned to the parallel execution resource manager after the parallel query is completely executed.
When a large number of concurrent queries reserve thread resources from the parallel execution resource manager, a first-come, first-served strategy is adopted until resources are exhausted and no query's resource requirements can be met. Subsequent queries are then requeued and retry to obtain resources during the next scheduling.
Worker thread allocation
Each OBServer node of the tenant has a parallel execution thread pool for executing parallel query tasks. When executing tasks, if the number of threads in the thread pool is insufficient, it dynamically expands. If a thread in the thread pool is idle for more than 10 minutes, it triggers an automatic reduction to 10 threads; if a thread is idle for more than 60 minutes, further reduction may trigger, potentially down to 0 threads.
Once a parallel query is scheduled for execution, each DFO can always obtain the required parallel thread resources from the parallel execution thread pool of the involved OBServer nodes. Note that by default, the number of threads allocated by each DFO on one OBServer node must not exceed the MIN_CPU value of the tenant × 10. If a DFO requests more resources than this value, it will be automatically reduced to the MIN_CPU value of the tenant × 10.
Two levels of resource control
Any parallel query undergoes two levels of resource control:
- Global control: Under the control of the execution resource manager, reserve a resource vector containing sufficient execution threads.
- Local control: Under the control of the parallel execution thread pool, allocate the desired number of physical threads.
Global control considers resource acquisition in distributed scenarios, while local control only considers resource allocation within the single-machine thread pool. Each performs its own role. The former ensures that a Query will definitely be able to execute after passing the check and will not encounter resource acquisition issues at runtime. The latter ensures that, in extreme cases, the DFO of a single Query does not request a number of physical threads far greater than what can be effectively utilized, causing thread resource waste. Once a parallel query passes the global control stage, it can execute smoothly, regardless of concurrency, without encountering a shortage of physical threads.
Parallel execution resource management
The parallel execution resource manager has a global perspective. You can view the thread reservation status of each OBServer node in a tenant through the GV$OB_PX_TARGET_MONITOR view. For detailed meanings of the view fields, see GV$OB_PX_TARGET_MONITOR.
obclient> SELECT * FROM GV$OB_PX_TARGET_MONITOR;
+--------------+----------+-----------+-----------+-----------------+--------------+-----------+-------------+------------------+-------------------+------------------------------+
| SVR_IP | SVR_PORT | TENANT_ID | IS_LEADER | VERSION | PEER_IP | PEER_PORT | PEER_TARGET | PEER_TARGET_USED | LOCAL_TARGET_USED | LOCAL_PARALLEL_SESSION_COUNT |
+--------------+----------+-----------+-----------+-----------------+--------------+-----------+-------------+------------------+-------------------+------------------------------+
| 192.xx.xx.xx | 19512 | 1004 | N | 555393108309134 | 192.xx.xx.xx | 19510 | 10 | 6 | 0 | 0 |
| 192.xx.xx.xx | 19512 | 1004 | N | 555393108309134 | 192.xx.xx.xx | 19512 | 10 | 0 | 0 | 0 |
| 192.xx.xx.xx | 19510 | 1004 | Y | 555393108309134 | 192.xx.xx.xx | 19510 | 10 | 6 | 6 | 1 |
| 192.xx.xx.xx | 19510 | 1004 | Y | 555393108309134 | 192.xx.xx.xx | 19512 | 10 | 0 | 0 | 1 |
+--------------+----------+-----------+-----------+-----------------+--------------+-----------+-------------+------------------+-------------------+------------------------------+
4 rows in set
At transient times, the global state seen by different OBServer nodes may be inconsistent, but the global state is synchronized every 500 milliseconds in the background. Overall, the state seen by each OBServer node will be largely consistent with little deviation.