Salesforce Winter’24 - Queueable Deduplication

Given my previous blog posts about the lack of Mutexes in Apex and the problems this causes the introduction of Queueable deduplication would seem to solve for some of my use cases. I’ll explore the new feature and how it might be used to solve our queueing use case. Tl;dr - this new feature does not remove the need for Mutexes and I can’t actually see where it can be safely used without Mutexes or locks of some sort

In the Winter ‘24 release notes there is an entry for Ensure that Duplicate Queueable Jobs Aren’t Enqueued. This describes a new mechanism for adding a signature to a Queueable. The platform will ensure that there is only one Queueable with the same signature running at once. Any attempt to enqueue a second Queueable with the same signature throws a catchable exception. Thread safe, multi-user safe, multi app-server safe. Pretty amazing and clearly using the sort of Mutexes we might want for our own code behind the scenes.

However this is not actually a good enough primitive to build on. If we start with our queue example. We want to be able to process a queue of items asynchronously. The queue is represented by sObject records but that is not too important here. The semantics we expect are that we enqueue items. If there is not a process running to process the queue we start one. Otherwise we do not. At the end of the Queueable that processes the items we check for more items to process and if there are more we enqueue another Queueable. We need some for of locking to prevent race conditions around the enqueue and check for more items to process steps.

If we use the new Queueable method it might seem that we can use this to ensure that there is only one Queueable running. But there is still a nasty race condition around the enqueue and check steps. If our Queueable is suspended after it has checked for more items to process but before the Queueable actually ends and another process can enqueue in this period we have a problem (no new Queueable will be started or chained so these items will sit unprocessed). So even with this is nice new API we still need our own locks.

But this is a more general problem. Even without the queueing we can consider the general use case. When we are enqueueing to perform some work we are going to be doing that in response to some sort of event. If another event happens whilst the queueable is still running perhaps we don’t want to run two processes in parallel (perhaps they will overwrite each other or cause locking). But we still want the second event to be processed: there is no guarantee that the already running Queueable will pick up changes from our new event. So we need a queue if we are going to use this new method for just about anything. Which means we need Mutexes.

So I have to conclude this new API is not much use on it’s own and if you implement your own locks you don’t need it.

Testing Queueable Finalizers Beyond The Limits

Finalizers on Queueables are fantastic. Finally we can catch the uncatchable. Even if your Queueable hits the governor limits and is terminated your Finalizer will still get run and can do something to handle the exception. A reasonably likely scenario is that the failed Queueable will be re-enqueued with some change to its scope so as it will complete.

An example Queueable is shown below

public with sharing class TestQueueable implements Queueable, Finalizer {
  private Callable function;
  private Integer scope;
  public TestQueueable(
    Callable function,
    Integer scope
  ) {
    this.function = function;
    this.scope = scope;
  }
  public execute(QueueableContext ctx) {
    System.attachFinalizer(this);
    Map<String, Object> params = new Map<String,Object> {
      ‘scope’ => scope
    };
   function.call(‘action’,params);
  }
  public execute(FinalizerContext ctx) {
    if (null != ctx.getException()) {
      System.enqueueJob(
        new TestQueueable(function, scope/2)
      );
    }
  }
}

As you can see this is about the most basic implementation of this pattern possible. As a simple form of dependency injection we use a Callable that will take a scope parameter which will control the amount of work done. If the callable throws we just retry with half the scope.

Like any good developer we now need to test this. A naive test might look something like this

@IsTest
private class TestQueueableTest {
@IsTest
private static void execute_whenNoException_doesNotReenqueue() {
Test.startTest();
System.enqueueJob(
new TestQueueable(new NoThrowCallable(),200);
);
Test.stopTest();
// Verify no re-enqueue
}

private NoThrowCallable implements Callable {
private call(action, args) {
}
}
}

This will test the happy path. In the real world we might want to have a mockable wrapper round System.enqueueJob to make this testing easier (and ot enqueue with no delay and appropriate depth control for initial enqueue). This would make the verification simple (with ApexMocks verify). I’ll consider that out of scope for this post as it does not impact the point.

But what about testing re-enqueue? Well that’s simple right - just throw an exception!

@IsTest
private class TestQueueableTest {
@IsTest
private static void execute_whenNoException_doesNotReenqueue() {
Test.startTest();
System.enqueueJob(
new TestQueueable(new ThrowCallable(),200);
);
Test.stopTest();
// Verifyre-enqueue
}

private ThrowCallable implements Callable {
private call(action, args) {
throw new MockException()
}
}
private MockException extends Exception {}
}

This won’t work. The exception will bubble to the top and be re-thrown at stopTest and the test will fail. Oh no! Ah, well that’s not a problem is it? We can catch exceptions like this

private class TestQueueableTest {
@IsTest
private static void execute_whenNoException_doesNotReenqueue() {
try {
Test.startTest();
System.enqueueJob(
new TestQueueable(new ThrowCallable(),200);
);
Test.stopTest();
} catch (Exception e) {
}
// Verifyre-enqueue
}

private ThrowCallable implements Callable {
private call(action, args) {
throw new MockException()
}
}
private MockException extends Exception {}
}

Now the test will complete and the verification section can run. Note that even without the try/catch in the test the Finalizer would run. We can verify this with debug logs. But what if we only want to re-enqueue on limits exceptions. That makes sense as we are reducing the scope to handle limits. Let’s say we altered our Queueable to look like this

public with sharing class TestQueueable implements Queueable, Finalizer {
  private Callable function;
  private Integer scope;
  public TestQueueable(
    Callable function,
    Integer scope
  ) {
    this.function = function;
    this.scope = scope;
  }
  public execute(QueueableContext ctx) {
    System.attachFinalizer(this);
    Map<String, Object> params = new Map<String,Object> {
      ‘scope’ => scope
    };
    try {
      function.call(‘action’,params);
    } catch (Exception e) {
      // Do something
    }
  }
  public execute(FinalizerContext ctx) {
    if (null != ctx.getException()) {
      System.enqueueJob(
        new TestQueueable(function, scope/2)
      );
    }
  }
}

Now any catchable Exception will be caught and the Finalizer won’t see the exception. But a limits exception cannot be caught and will cause re-enqueue. Great! But our previous test won’t work. We are throwing a catchable Exception. Let’s try again

private class TestQueueableTest {
  @IsTest
  private static void execute_whenNoException_doesNotReenqueue() {
    try {
      Test.startTest();
      System.enqueueJob(
        new TestQueueable(new ThrowCallable(),300);
      );
      Test.stopTest();
    } catch (Exception e) {
    }
    // Verifyre-enqueue
  }

  private ThrowCallable implements Callable {
    private call(action, args) {
      Integer scope = args.get(‘scope’);
      for (Integer i = 0;i<scope;i++) {
        List<Contact> contacts = [Select id from Contact Limit 1];
      }
    }
  }
}

So now if scope is large enough we’ll get a limits exception for too many SOQL queries. If it’s small enough we won’t. But this test can never succeed. The Limits exception will be re-thrown at Test.stopTest and cannot be caught. So the test will fail.

This means it is impossible to test finalizers with limit exceptions. Unless you know how! If so please get in touch 🙂