Implementing Process Locks in vRO Workflows

I discovered last week that a developer wasn’t aware that you could use a locking semaphore in vRO. They were writing some API calls and encountered a 409 (conflict) error when a workflow was attempting to concurrently update the same resource (a load balancer if I recall but can’t remember the exact details). I felt that there may be other developers out there who are also not aware so I have created two workflows that can be used to create and remove process locks in your vRO workflows.

vRO provides an always available Locking System that can be used to create a lock for a given id and owner. It does this by creating a record in the embedded PostgreSQL database. When an attempt is made to create another record in the database, a key constraint occurs and the lock fails.

You should consider using a process lock if you are updating any resources that could cause a conflict or data consistency problems. Some examples include, updating a load balancer, updating a vRA Reservation, or preventing a data collection task from running because there is already one in progress. It is also useful to create a lock between sections of code in your workflow, which could cause a problem if the workflow was executed multiple times, simultaneously. Note, that you don’t always need to lock the entire workflow, just between areas of code where updates are performed.

All code that I have provided or talked about in this post can be downloaded as a vRO package for your consumption here.

Create Process Lock

I have created a workflow that handles the locking process. The workflow includes an Action that uses the Locking function ‘lock‘ that will attempt to acquire a lock. If this is successful, the function will return true, otherwise false is returned. There is another function that could have been used called ‘LockAndWait‘, however, this will wait indefinitely if the lock cannot be acquired for some reason. The use of the ‘lock’ function provides more control on how best to handle the situation, such as introducing a timeout to prevent the workflow running indefinitely or taking corrective measures.

The workflow is mostly handled by an Action that creates the lock with some logic build around it.

/*global lockId lockOwner*/

/**
 * Creates a process lock using the vRO Locking System.
 * @author Gavin Stephens <gavin.stephens@simplygeek.co.uk>
 * @version 1.1.0
 * @function findVcVmByInstanceUuid
 * @param {string} lockId - What to lock.
 * @param {string} lockOwner - Who is creating the lock.
 * @returns {boolean} Returns true if the lock is created, otherwise returns false.
 */

function checkParams(lockId, lockOwner) {
    var inputErrors = [];
    var errorMessage;
    if (!lockId || typeof lockId !== "string") {
        inputErrors.push(" - lockId missing or not of type 'string'");
    }
    if (!lockOwner || typeof lockOwner !== "string") {
        inputErrors.push(" - lockOwner missing or not of type 'string'");
    }
    if (inputErrors.length > 0) {
        errorMessage = "Mandatory parameters not satisfied:\n" + inputErrors.join("\n");
        log.error(errorMessage);
    }
}

var logType = "Action";
var logName = "createLock"; // This must be set to the name of the action
var Logger = System.getModule("com.simplygeek.library.util").logger(logType, logName);
var log = new Logger(logType, logName);
var lockCreated;

try {
    checkParams(lockId, lockOwner);
    log.log("Creating lock on '" + lockId + "' for '" + lockOwner + "'");
    lockCreated = LockingSystem.lock(lockId,lockOwner);
    if (lockCreated) {
        log.log("Lock created for '" + lockOwner + "'");
    } else {
        log.log("Exisiting lock found for '" + lockOwner + "'");
    }
} catch (e) {
    log.error("Action failed to create lock.",e);
}

return lockCreated;

The Workflow

The workflow attempts to create a lock, and if this fails, will try to gain the lock after 10 seconds. It will attempt to do this 180 times (30 minutes). If it still fails on the final attempt, the existing lock will be released and a new lock created. I decided to do this because 99% of the time, any lock maintained for that long was usually a stale lock that had not been released. Most update operations where this is used aren’t anywhere near the threshold. The timeout values can be changed in the workflow and feel free to change the logic to suite your own requirements.

And here is an example log output with it used in my Data Collection workflow (whilst there was already an existing lock in place).

[] [I] [Workflow: Start Data Collection On Compute Resource] Trying to create lock: Attempt 1 of 180
[] [I] [Action: createLock] Creating lock on 'SITEB-CLS-CLOUD-01' for 'inventory'
[] [I] [Action: createLock] Exisiting lock found for 'inventory'
[] [I] [Workflow: Start Data Collection On Compute Resource] Trying to create lock: Attempt 2 of 180
[] [I] [Action: createLock] Creating lock on 'SITEB-CLS-CLOUD-01' for 'inventory'
[] [I] [Action: createLock] Exisiting lock found for 'inventory'
[] [I] [Workflow: Start Data Collection On Compute Resource] Trying to create lock: Attempt 3 of 180
[] [I] [Action: createLock] Creating lock on 'SITEB-CLS-CLOUD-01' for 'inventory'
[] [I] [Action: createLock] Exisiting lock found for 'inventory'
[] [I] [Workflow: Start Data Collection On Compute Resource] Trying to create lock: Attempt 4 of 180
[] [I] [Action: createLock] Creating lock on 'SITEB-CLS-CLOUD-01' for 'inventory'
[] [I] [Action: createLock] Lock created for 'inventory'

This works really well and prevents more than one of the same data collection task running on a compute resource.

Remove Process Lock

The removal of the process lock only requires a single Action to complete.

/*global lockId lockOwner*/

/**
 * Creates a process lock using the vRO Locking System.
 * @author Gavin Stephens <gavin.stephens@simplygeek.co.uk>
 * @version 1.1.0
 * @function removeLock
 * @param {string} lockId - What to unlock.
 * @param {string} lockOwner - Who is removing the lock.
 */

function checkParams(lockId, lockOwner) {
    var inputErrors = [];
    var errorMessage;
    if (!lockId || typeof lockId !== "string") {
        inputErrors.push(" - lockId missing or not of type 'string'");
    }
    if (!lockOwner || typeof lockOwner !== "string") {
        inputErrors.push(" - lockOwner missing or not of type 'string'");
    }
    if (inputErrors.length > 0) {
        errorMessage = "Mandatory parameters not satisfied:\n" + inputErrors.join("\n");
        log.error(errorMessage);
    }
}

var logType = "Action";
var logName = "removeLock"; // This must be set to the name of the action
var Logger = System.getModule("com.simplygeek.library.util").logger(logType, logName);
var log = new Logger(logType, logName);

try {
    checkParams(lockId, lockOwner);
    log.log("Removing lock on '" + lockId + "' for '" + lockOwner + "'");
    LockingSystem.unlock(lockId,lockOwner);
    log.log("Lock removed for '" + lockId + "'");
} catch (e) {
    log.error("Action failed to remove lock.",e);
}

[] [I] [Action: removeLock] Removing lock on 'SITEB-CLS-CLOUD-01' for 'inventory'
[] [I] [Action: removeLock] Lock removed for 'SITEB-CLS-CLOUD-01'

Do make sure you include locks to protect your update operations and if you require any help with anything, then please drop me a message via the Drift app.