Implementing Mutexes using SObjects

In the previous posts I explored the need for platform-level Mutexes and what adding these might mean for the platform. In this post I’ll start exploring some of the complexity of implementing Mutexes using the available tools on the platform (basically SObjects). I’ll start with a naive implementation, identify some problems and then try and find some solutions.

Before I get started what sort of use case am I trying to solve. Let’s imagine we have a workload that can be broken down into a series of tasks or jobs that have to be processed in a FIFO manner system wide. That is regardless of which user we want a single unified queue that is processed single threaded. This might be the case where we are syncing to an external platform for example.

In order to ensure these operations we will expose an enqueue API that looks something like this:

public interface Queue {
  void enqueue(Callable task);
}

When enqueue is called the task is written to the queue (serialised to an SObject). Only one “thread” can access (read or write) the queue at a time so internally in enqueue we would need a Mutex. If there is Queueable processing the Queue then nothing else is done in enqueue. If there is not a Queueable then one is started. So another Mutex may well be required here.

Our Mutex API would look like this:

public class Mutexes {
  static boolean lock(String name) {}
  static void unlock(String name) {}
  static void maintain(String name) {}
}

As we require Mutexes to persist across contexts (and be lockable between contexts) the most obvious option is to use SObjects. Assume we create a Mutex SObject. We can use the standard Name field to represent the name of the lock. We will need a Checkbox to represent the status of the lock. And we will need a Datetime to store the last update time on the lock.

Given this we might assume the lock method could look like this:

static boolean lock(String name) {
  Mutex__c m = MutexSelector.selectByNameForUpdate(name); // Assume we have a selector class
  if (m == null) {
    m =  new Mutex__c(
      Name = name,
      Status__c = ‘Locked’,
      Last_Updated__c = System.now()
    ); 
    insert m;
    return true;
  }
  if (m.Status__c == ‘Locked’) {
    return false;
  }
  m.Status__c = ‘Locked’;
  m.Last_Updated__c = System.now();
  update m;
  return true;
}

Unfortunately this has some issues. One issue is that if two threads both call lock at the same time when there is no existing lock two new Mutex__c objects will be created when there should only be one. There are various options to solve this. One would be to have a custom field for Name that is unique. We could try/catch around the insert and return false if we failed to insert.

The next issue is that it is possible for a lock to be locked and never unlocked. We only expect locks to be held for as long as a context takes to run. In the case of chains of Queueables we have the maintain method to allow the lock to be maintained for longer. But if a context hits a limits exception and does not unlock then no other process will ever be able to lock the lock

The way round this is to allow locking of locked Mutexes if the Last_Updated__c is old enough (this amount of time could be configured). This will prevent locks being held for and processes stalling.

unlock and maintain are relatively simple methods that will similarly use for update to hold locks for the duration of their transactions.

This whole thing feels like a fairly fragile implementation that would be far better served at a platform level