The Expert Series explores common automation challenges businesses face and offers perspectives from our in-house automation experts on how to approach these issues. Read on as Sashikumar Kannappan, our APJ Senior Solutions Consultant, discusses why data orchestration tools alone aren’t enough to drive efficiency and productivity for data and business intelligence (BI) teams.
In the last decade, one of the most exciting changes in the data orchestration space has been the emergence of new tools like Fivetran, Stitch, and dbt. These tools empower data and BI teams with a turnkey ability to connect to applications and databases, making it much easier to configure and manage data pipelines.
Having worked closely with data and BI teams over the years, I’ve found that even with the adoption of these technologies, they’re still struggling with siloed data and fragmented systems. The reality is that companies in different industries and countries have unique needs which are often only covered by niche software vendors. And data orchestration tools often lack the flexibility to extract data from niche vendors.
Before we explore the challenges of using data orchestration tools any further, let’s define data orchestration and cover common use cases.
What is data orchestration?
Data orchestration is the process of automatically extracting data from multiple data sources (business apps, systems, databases, etc.), and aggregating, transforming, and synchronizing this data to make it available for downstream use.
For instance, data and BI teams can leverage processed data to create visualizations that help business teams stay on top of their goals. They can also perform analysis on the processed data for better decision making.
Here are some ways companies use data orchestration tools:
- Replicating customer information and product lists from Shopify in a centralized data warehouse like Snowflake
- Using dbt to run complex SQL queries and tapping that information to evaluate a customer’s lifetime value
- Generating dashboards with the data for a visual overview, and using the information to send customers customized promotions and content
Most of these use cases leverage connectors and templates built by data orchestration tools.
Related: What is API integration?
Challenges faced by data and BI teams today
Most data orchestration tools are focused on building full-fledged connectors for mainstream applications, like Shopify, Netsuite, and Snowflake, rather than niche apps.
Niche/industry-specific apps are also difficult to connect to. Here’s why:
- They often don’t support integration standards like APIs, and only allow single action tasks like downloading .zip files, which have to be manually uncompressed.
- Certain API structures prevent bulk data download or require scheduled data exports. This means data has to be manually downloaded, split, and uploaded into systems.
- After extracting the data, teams have to upload and sync the data back into the niche/industry-specific apps.
All of this is time consuming and prevents data and BI teams from focusing on high-value work. In fact, our conversations with data and BI leaders suggest that nearly half of their analysts’ time is spent on repetitive, manual tasks associated with data pipelines.
To use an example, I recently worked with a customer in the grocery industry on extracting call log data from their call center software. The goal was to retrieve the logs, filter them, and update them in Zendesk.
We were met with challenges from the get-go, as it was difficult to gain access and retrieve the call log files—an administrator had to manually download them. Compounding the problem, the customer’s data orchestration tools were unable to connect to email services like Outlook or Gmail, and, as a result, couldn’t identify and filter out emails.
This inability to perform configurations like filters or set conditions is a common downside of “one-click” turnkey connectors. Businesses need a more flexible solution that allows them to easily extract data from niche software, and eliminate the need for human intervention.
Related: Automation vs orchestration (here’s how they vary)
So, how can you improve your data orchestration process?
We’ve established that the biggest obstacle businesses face in data orchestration is gaining full access to data from their applications. An enterprise automation platform like Workato offers two advantages over data orchestration tools in this area:
1. Extensive connectivity: It can access a wide variety of interfaces, like webhooks, API, SOAP, CSV and more; in addition, it can automate bulk or batch data processing and even handle special scenarios, like picking up an excel spreadsheet attached in an email.
2. Automation capability: It can automate manual actions that your analyst performs today, such as unzipping and decrypting data files, executing data extraction scripts, and performing complex data cleansing and transformation.
In short, enterprise automation platforms can extract and transform data from your applications into any format that your business requires, while automating repetitive manual work.
Related: A guide to automating tasks
Unlocking the true value of integration-led automation
An enterprise automation platform offers capabilities that can truly transform your BI teams. We’ve seen first hand how automating data pipelines has freed up time for data analysts to pursue higher-value work, like building better data visualizations and performing deeper analysis to uncover more meaningful insights.
