Cluster locks services

Use Case
, ,
1

Originally asked by Yaakov Shamii on 25 December 2019 (original question)

Hi,

Can you please provide me all the services that can create cluster locks in the system?

such as - com.exalate.replication.in.RequestWorker.clusterLock ?

We need this for monitoring

2

Answer by Ariel Aguilar on 04 March 2021

Hi Yaakov,

The multi-threading model behind Exalate for on-premise Jira (in one cluster node) is the following (as far as back-end is concerned):

Upon Install / re-enabling of Exalate, it creates a number of thread pools:

  • eventScheduledExecutor (EventWorker); - 1 thread
  • requestScheduledExecutor (RequestWorker); - 1 thread
  • requestPollExecutor; - 1 thread
  • lifecycleScheduledExecutor; - 1 thread
  • statusWorkerExecutor; - 1 thread
  • errorWorkerExecutor; - 1 thread
  • expirationWarningScheduledExecutor; - 1 thread
  • clusterLockerWorkerExecutor; - 1 thread
  • connectionEventExecutor; - 1 thread
  • eventScheduledExecutor (router service); - 1 thread
  • ForkJoinThreadPool - n threads, where n is number of Available Processors

At any point in time only Exalate code (+ calls to Atlassian API originating from Exalate code) executes on these threads.

In addition to that:

  • Exalate code that executes on the issue updating thread (spawned and controlled by Jira):
    • SyncListener - event listener targeted at issue events, sprint events and link events. Generally, it collects context info (which can only be collected from the Jira thread) and passes the info to eventScheduledExecutor thread
    • Escalate Now, Unexalate Now post functions
  • Exalate code that integrates with HTTP (threads spawned and controlled by Jira):
    • Exalate REST endpoints (/rest/issuehub/*)
    • Exalate servlets: (/plugins/servlet/exalate/*)
    • Controllers for Exalate UI (threads spawned and controlled by Jira)
  • Exalate code that executes upon issue view (threads spawned and controlled by Jira):
    • Is exalated condition, Is non-exalated condition
    • remote issues tab controller
    • sync panel controller
  • Exalate JQL functions (only executed during search if used in the query, spawned and controlled by Jira):
    • Is Issue Under Sync, Is Issue Sync Stuck, Is Issue Under Sync With Instance, Is Issue Under Sync By Connection

In addition to that, Exalate executes groovy scripts, where customers may decide to spawn additional threads (However, in these cases, it’s the customer’s responsibility to stop such threads).

As for a synchronization lifecycle, here are the threads involved:

  1. the Jira thread which is updating the issue invokes the SyncListener
  2. the SyncListener collects info from the event and passes over to the “eventScheduledExecutor (router service)”
  3. “eventScheduledExecutor (router service)” interacts with the clusterLockerWorkerExecutor thread so that it takes a cluster lock for the “eventScheduledExecutor (router service)” to execute exclusively on one node
  4. once that the cluster lock is confirmed, “eventScheduledExecutor (router service)” runs the outgoing sync script for the designated connection and puts an outgoing sync transaction (Sync Event) into the outgoing sync queue
  5. “eventScheduledExecutor (router service)” once again interacts with the clusterLockerWorkerExecutor thread so that cluster lock is released
  6. eventScheduledExecutor (EventWorker) wakes up every 5 seconds to:
    1. interact with the clusterLockerWorkerExecutor thread so that it takes a cluster lock for the “eventScheduledExecutor (EventWorker)” to execute exclusively on one node
    2. check if there is any outgoing sync transactions (sync events) to be processed, process them
    3. interact with the clusterLockerWorkerExecutor thread so that it releases the cluster lock held for the “eventScheduledExecutor (EventWorker)”

At certain states, outgoing sync transactions (sync events) send HTTP(S) requests to the remote instance, putting incoming sync transactions (sync requests) to the incoming sync queue of the remote instance.

Then there, the following threads get involved:

  1. requestScheduledExecutor (RequestWorker) wakes up every 5 seconds to:
    1. interact with the clusterLockerWorkerExecutor thread so that it takes a cluster lock for the “requestScheduledExecutor (RequestWorker)” to execute exclusively on one node
    2. check if there is any incoming sync transactions (sync requests) to be processed, process them
      At certain states, (CREATE_ISSUE, UPDATE_ISSUE, ADD_LINK), Exalate would ask the Jira API to create / update the issue being synced
    3. interact with the clusterLockerWorkerExecutor thread so that it releases the cluster lock held for the “requestScheduledExecutor (RequestWorker)”

Throughout the entire process, Exalate uses the ForkJoinThreadPool to delegate ad-hoc asynchronous operations (like queries to the DB, or REST calls) to.

We expect, that memory consumption should grow while either “eventScheduledExecutor (router service)” or any other Exalate thread fails to obtain a cluster lock, however we are not expecting any additional threads other than the ones I’ve highlighted to be spawned due to cluster lock, or internet connectivity problems: at all times Exalate would use 1 thread for outgoing sync processing (“eventScheduledExecutor (EventWorker)” - including sending the sync request to the remote), 1 thread for request processing (“requestScheduledExecutor (RequestWorker)”) and so on.

Please let us know if you have any additional questions on the thread model of Exalate, or need any further clarifications.

Kind regards,

Ariel