Workato recently improved users’ batch process ability by creating “synthetic” datapills, which make it easy to enrich datasets and perform activities at the beginning and end of batch processes. To understand these new synthetic datapills, let’s first review what a datapill is in Workato.
Datapills are how Workato formats data for use in recipes and triggers. For example, let’s say you’re integrating contact data from Salesforce to Zendesk. The data from each Salesforce field (Name, Company, Address, etc.) would be its own datapill that can be used later in the recipe. Each pill is a discrete package that can be one of the following data types: strings, integers/numbers, dates, arrays/lists, or Boolean operators. In the screenshot below, you can see datapills that have been extracted from Salesforce under “App data” on the right. On the left, these datapills are being mapped to a custom email message that will be sent out during this integration process.
Typical datapills are visible in the App Data box on the right.
Complex processing when moving data in batches
When moving data in batches, occasionally it is required to perform additional processing to the data to properly load it into the downstream system. In these cases, complex logic had to be added to the recipe to handle certain batch processing use cases. Here are two examples:
- Adding a Sequencing Number: When creating a CSV from a source list, users often need to create an additional column to indicate the sequence number for each row in order to format the data according to the requirements for consumption by the downstream app. This was already doable in Workato, but it required logic to loop through each of the list items and add the sequence value using the index datapill from the FOREACH step output.
- Before and After Batch Processing: Sometimes processing steps are required before the first batch or after the last one. Until now, there was no easy way to easily identify when you’ve reached the first or last batch.
Synthetic datapills to the rescue
We are constantly looking for ways to enable users by simplifying the steps needed to build recipes and automate complex processes That’s why Workato recently added several synthetic datapills that make processing data in batches simpler. We call them “synthetic” because our platform creates them from the extracted data; they weren’t naturally existing in the source. These new datapills can be found when running a “Repeat Action” in a Workato recipe. They are:
- List Index: A field in a recipe that returns the row index when used in the in-line array mapper.
- Size: This field returns the list size when used in the in-line array mapper.
- Is first: Flags the first row batch coming in from an upstream system for batch processing.
- Is last: Flags the last row batch of a batch processing step.
The new synthetic datapills (in the teal box) are List Index, List Size, Is First, and Is Last.
These small but powerful bits of metadata streamline the creation of Workato recipes. Here’s an example of how synthetic datapills streamline recipe building.
New List Index datapill helps map out your data rows
Let’s say you’re synching financial information between Netsuite and Anaplan. Data is consumed by Anaplan in the form of a CSV file and requires a sequence number for each row, which must be added during the load. Without the synthetic datapill feature, the recipe first needs to loop through each list item to create a new source list — just to add a sequence number.
The List Index synthetic datapill solves this problem when used in the in-line array mapper. Below is a before and after comparison of the act of adding a row ID to a CSV file of income data.
Recipe logic is more compact using the new synthetic datapill, List Index.
Without List Index (the left example), the user needs to create a new list, loop through the original list to add a row ID field to the source to add to the new list. Finally, this new list is used to compose the CSV file.
With the List Index synthetic datapill, a single step can be used to create the column for row ID in the CSV file (the right example).
Helpful tools for streamlined batch processing
These new Synthetic Datapills simplify recipe construction and maintenance, protect data integrity by reducing the number of steps where errors could lurk, and increase user productivity.
They’re just one in a series of batch processing improvements Workato has made recently. We know digital workflows often require moving large volumes of data from one application to another, whether you’re searching for sales prospects or building ETL pipelines to a massive data warehouse, and these movements often require complex logic to handle. So we’ve added refinements such as Repeat in Batches that automate and streamline batch handling.
Curious to see Workato in action? Click here to request a custom demo. To learn more about Workato and our intelligent platform to integrate anything and automate everything across your organization, click here. Or for live, real-time demos and presentations on other handy features, check out our Product Hour videos.