Implementing Stateful Work-In-Progress Pattern with Durable Azure Functions

Implementing Stateful Work-In-Progress Pattern with Durable Azure Functions

In some cases, you need to gather chunks of data from multiple sources and submit the final information package to the server for processing. The work-in-progress (WIP) pattern enables you to gather lots of data over a long period of time before reviewing it and submitting the data collected for processing.

The key component of the WIP pattern is a persistent “work document” that you keep enriching with input over time and finally submit to the server after a review. The work document is not a single entity, and the pattern allows you to manage several work documents simultaneously. For example, you might want to create individual work documents for every customer requesting approval for credit purchases.

A work-in-progress application should support the following operations:

1. Create document: Create a new work document.
2. Read document: Get a single work document.
3. List documents: Get the list of work documents.
4. Filter documents: Get a filtered list of work documents.
5. Update document: Update a single work document.
6. Cancel document: Cancel an existing work document.
7. Submit document: Submit a completed work document.

Diagram

The work-in-progress workflow is made of the following resources:

1. Home: The root resource from where you can navigate to other resources.
2. WIP List: The list of work documents.
3. WIP Item: A work document.

Below is the list of actions that your workflow navigation should support:

1. List/List Filtered: Go to the WIP list resource.
2. Create: Create a new work document.
3. Get Status: Get the current state of the work document.
4. Cancel/Submit/Update: Operations allowed on a single work document.
5. Read: Retrieve a single work document.

The following diagram illustrates the work-in-progress workflow consisting of the resources and the navigation actions.

The workflow and operations outlined in the diagram are the minima you need to have a viable work-in-progress system. It is also important to note that too many dependencies between fields of a work document that must be submitted together can make the workflow hard to understand and debug.

You can also link work documents together to create a hierarchical workflow. For example, an employee work document can contain a link to the interview work document, with both documents following their lifecycles. It would help if you archived completed and canceled work documents for future reference. You can further extend this feature by adding support for reusing work documents in the future.

Work-In-Progress Pattern Example

Consider the use case of onboarding an employee. The onboarding process requires data entry from multiple parties and, in some cases, requires manual approvals that may take days to arrive. The WIP pattern could be used to handle this process. You can create a work document for each employee that records data such as human resource department feedback, the result of a background check, details on an employee contract, and so on. The various parties in the organization can edit the document when they have the data available. Eventually, someone with the proper permissions can submit the final copy after a review.

Implementation

The WIP pattern can be implemented with an Azure Durable Function. The Durable Functions allow you to model a workflow as a set of Azure Functions that can interact with each other. A function orchestrator enables you to execute the functions in the desired order and maintains their data and state. Azure Durable Functions consist of the following components:

1. Activity function: It implements the actual business logic and acts as a step in the workflow.
2. Orchestrator function: It invokes the activity functions and orchestrates them as a workflow. It can invoke one or more activity functions and wait for them to receive the result.
3. Client function: It invokes the orchestrator function. The end user of the workflow invokes the client function.

Azure Durable Functions supports many of the operations that a work-in-progress application requires through its built-in instance management APIs as follows:

It is worth noting that the management APIs are available as built-in HTTP endpoints that you can use. You can find the list of management APIs on the Microsoft documentation website. However, for ease of use, the Durable Function SDKs make the APIs available via the orchestration client binding, freeing you from dealing with raw HTTP requests and responses.

Prerequisites

You will use the following tools to build the sample application:

• Visual Studio or VS Code as the IDE.
• .NET 6 (or newer)
• Azure Functions Tools. They can be added by including the Azure development workload in your Visual Studio installation.
• An Azure subscription if you wish to deploy your application.

Demo App: Employee Onboarding

Let's implement the employee onboarding example using the Azure Durable Functions. For reference, you can download the source code of the demo application from my GitHub repository.

First, use this Microsoft guide to create an Azure Durable Function application with VS Code or Visual Studio. By following the steps mentioned in the guide create a durable function named OnboardingFx.

After your Function project is ready, you can start adding code to the OnboardingFx class. Delete the boilerplate code from the class and repopulate the class using the following instructions.

First, replace the using statements at the top of the file with the following using statements:

Next, define the types that make up your work document and the work document itself as follows:

The first function you will define is the client function that initiates the orchestration. The function is triggered by an HTTP POST request that accepts the metadata of the work document and kicks off the workflow. Copy the following code into the OnboardingFx class:

The client function returns a list of instance management of endpoints that are useful for checking the status of the instance created by the function.

The client function kicks off the orchestrator function named StartWorkflow (which you will create soon). The orchestrator is responsible for invoking the activity functions and driving the workflow. In our case, the orchestrator will wait for external events. Each event expects to receive a part of the work document in the payload.

Once all the event responses are received, and the complete work document is formed, the orchestrator triggers an activity function that notifies the approver to review and submit the document. The approver notifies the orchestrator of their decision by raising the SubmissionApproval event and providing true or false as the response. Irrespective of the decision, the workflow is marked as complete.

The following is the complete code of the orchestrator function which you should add to the OnboardingFx class:

We used a helper function named WaitForExternalEventAndSetCustomStatus that sets a custom status message for the orchestrator function after the event is processed. This status message will help us measure the progress of the workflow. The following is the definition of the function which you should copy into the OnboardingFx class:

Your orchestrator used the SubmitDocument activity to notify the approver of the availability of the work document. Copy the following definition of the activity function into the OnboardingFx class:

Finally, the following client function provides an endpoint for the end users to fetch all the instances (hence the work documents) that are currently active (pending or running state). You can customize this function as per your preference. Copy the following code into the OnboardingFx class:

Finally, you can specify a custom route prefix in the host.json file to substitute the default api prefix with a more useful value that represents your application as follows:

Demonstration

Launch the application using VS Code or Visual Studio by pressing the F5 key. In the debug terminal, you will find the HTTP endpoints of the orchestrator clients. You will first use the Start function to create a new work document. Note that the URLs of the HTTP triggered functions are listed in the command output as follows:

Use a REST client tool such as Postman to send the following HTTP request to the Start function endpoint:

HTTP Request:

Alternatively, instead of using a REST client tool, you can use these PowerShell scripts to follow along. Open a PowerShell shell, copy/paste, and run the PowerShell commands:

PowerShell:

The function returns the endpoints you can use to perform instance management operations such as terminating the workflow, fetching the status, and so on.

You will notice that the HTTP endpoints include a few query string parameters. These parameters help the durable function locate the state information in the underlying storage and reload the state in the workflow so that the requested operation can be performed. Following is the list of the query string parameters and their functions:

1. taskHub: The name of the task hub.
2. connection: The name of the connection app setting for the backend storage provider.
3. code: The authorization key required to invoke the durable function management API.

You will use the sendEventPostUri POST endpoint from the response to send the necessary events to the orchestrator function. Use your REST client tool to send the following POST request to the orchestrator. Remember to substitute the {instanceId}and {code} with the values you received in the previous response. Replace the placeholder {eventName} with InterviewFeedback, which is the name of the event expected by the orchestrator. Also, note that the body of the request is the same as what is expected by the orchestrator for the event.

HTTP Request:

PowerShell:

The following screenshot presents the output received from the orchestrator upon successful delivery of the event:

Use a similar process to post the rest of the events: BackgroundCheckFeedback, and ContractFeedback, with their desired payloads to the orchestrator.

PowerShell:

At any point in time through the execution of the workflow, you can send the following GET request to the Function to get the status of the instance (or work document):

HTTP Request:

PowerShell:

The following screenshot presents the output received from the endpoint right after the InterviewFeedback event was delivered:

Once all the events are received and the document assembled, you will find that the activity function logged the complete work document to the console as follows:

Now that you have a document in the system, test the second client function that returns the list of active instances/work documents in the application: GetInstances.

Send the following request to the function to get the filtered list of instances:

HTTP Request:

PowerShell:

The following screenshot presents the response to the request, which includes the application that is currently being processed:

You can now play the role of the approver and raise another event like before to register whether you accept or decline the document. Use your REST client to send the following HTTP request to the Function:

HTTP Request:

PowerShell:

The following screenshot presents the response received for the request:

Finally, you can check the status of the workflow by sending another GET request to the function, just like you did previously. The following screenshot presents the status of the workflow after completion:

PowerShell:

Conclusion

In this article, you learned about the Work-In-Progress pattern and discussed its use cases. You covered some common operations a WIP system must implement to be viable.

Azure Durable Functions is one of the easiest technologies to implement the WIP pattern, since it can seamlessly persist the workflow state and provides many out-of-the-box management APIs that can be used as is to fulfill the requirements of the WIP system.

You built a simple implementation of the WIP system using Azure Durable Functions and tested the various operations that your function supports.

Rahul Rai is a technology enthusiast and a Software Engineer at heart. He is a Microsoft Azure MVP with over 13 years of hands-on experience in cloud and web technologies. With so much leadership experience behind him and currently holding a Group Product Manager Role at LogicMonitor, he has successfully established and led engineering teams and designed enterprise applications to help solve businesses’ organizational challenges.  

Outside of his day job, Rahul ensures he’s still contributing to the Microsoft ecosystem by authoring books, offering free workshops, and frequently posting on his blog: https://thecloudblog.net to share his insights and help break down complex topics for other current and aspiring professionals.