Agile Data Engine - DataOps Platform for Data Warehousing

Integrated DataOps development and operations platform for building, deploying and running cloud-based data warehouses. Data modeling, ELT & orchestration capabilities combined in the same software.

Go to features

Agile Data Engine addresses 4 major demands

1 Speed & Quality

When you decide to build a cloud based data capability, typically you need to choose either quick or resilient approaches. We provide a way with no compromises.
2 Agility

Achieving true agility in data platform development is not a trivial undertaking. You need a combination of the right technologies with the right way of working.
3 Operational Analytics

Too often data platforms do not properly address the requirements of operational use cases. Either data is not not real-time business intelligent, or implementation is way too slow – or both
4 Future-proof

When making decisions about data architectures, cloud ecosystems and approaches, there are significant risks involved. Agile Data Engine is built to unlock people, technology and ecosystem lock-ins.

"Truly exceptional concept! I'm also happy to say that we've been running our own data platform on it for two years now and it's working like a charm."

Tommi Vihervaara, Finavia

Agile Data Engine combines best of both worlds from data management and software engineering

Multi-Cloud DataOps Platform for Cloud DW

Agile Data Engine supports your multi-domain data and multi-site teams with multi-tiered architecture and multi-phased continuous deployment to isolated Development, Test and Production environments.

Figure 1. Agile Data Engine with Snowflake

Agile Data Engine with Azure SQL Data Warehouse — Figure 2. Agile Data Engine with Azure Synapse SQL

"Agile Data Engine allows us to focus on what is important – producing value. We do not have to think about the release process when everything works automatically."

Ville Viitanen, Fortum

Features

Cloud infrastructure

Design/Dev/Test/Prod environment automation

Agile Data Engine has an automated and configurable cloud infrastructure set-up enabling Development/Test/Production environments’ isolation and fast start-up of the project.

Multi-cloud and multi-DBMS support

Agile Data Engine is built with multi-cloud and multi-DBMS support for easy transition from a cloud/DBMS service to another. It is also about being future-proof enabling support to-be-developed to new market leading cloud/DBMS services to come. Currently it supports AWS & Azure in cloud and Snowflake, Amazon Redshift, Azure Synapse SQL, Azure SQL Database & Google BigQuery in DBMS services.

Designer

Data modeling and transformations combined

Agile Data Engine has a unique entity-driven approach for implementing data models and data transformations together to enable modularity in data warehouse development. All data warehouse modeling methodologies are supported.

Data Vault automation

Data Vault 2.0 automation for generating data vault entity loads and accelerating data modeling.

Interactive end-to-end data lineage

Graphical data lineage view with data warehouse entities and loads, with metadata based search. Lineage helps data professionals in managing the whole data platform solution content and helps in troubleshooting and understanding the audit trail of the data loads.

Deployment Management

Continuous deployment

Continuous deployment framework manages the deployment pipeline of data warehouse solution packages from Design to Development/Test/Production runtime environments.

Data model deployment with automatic schema changes

Data model deployment is a fully automated process and the automatic change detection of schema happens in runtime based on the metadata. The framework supports multiple DBMS services (currently Snowflake, Amazon Redshift, Azure Synapse SQL, Azure SQL Database and Google BigQuery). No need for schema compare tools and incremental change scripts for data model and grants/revokes.

Metadata-driven workflow generation

Data workflows (as directed acyclic graphs) are dynamically generated based on entities’ load dependencies and scheduling information in the metadata repository.

Runtime

Flexibility for different data ingestion styles

There are a multitude of ways to ingest and integrate source data into cloud data platforms and Agile Data Engine enables the use of these based on source data and workload requirements. Regardless of style, ingestion to database platform is harmonized with Agile Data Engine DW input interface.

Metadata and data crawling from JDBC sources

Agile Data Engine provides functionality to crawl both metadata and data from JDBC supported database sources.

ELT processing and integration with other processing methods

Agile Data Engine utilises an ELT approach instead of ETL, which means that data processing/compute capacity is based on the cloud database platform and therefore the processing capacity can be scaled easily, when requirements change. Processing of data can also be done with other cloud-based processing technologies (e.g. Spark) and integrated into Agile Data Engine’s orchestration engine.

Data Smoke Tests

Integrated and customizable data smoke tests as part of the data workflows, that monitor data quality continuously and can be used to control the execution of workflows or to flag warnings.

Dependency and concurrency aware workflow orchestration

Agile Data Engine’s Dagger component utilises Apache Airflow open source software as the data orchestration platform. Airflow is integrated together with Agile Data Engine software, so Airflow DAG codes are dynamically generated based on the metadata.

New ADE documentation is here!

Faster time-to-delivery & experiments
Lower total cost of ownership
Data capability that adapts and scales

"We can develop at an entirely different rate. In practice, the improvement is in multiple percentages. We previously saw development cycle turnarounds in weeks – now they happen in days.”

Tomi Eronen, DNA

How can we help you?

1

Does your data warehousing team underperform and struggle to satisfy the increasingly challenging data requirements coming from business? Do you want to know how to achieve 4X value from your cloud data platform?
2

Do you have a need to migrate your existing data warehouse to cloud and need to do it as fast as possible, but without compromising the architecture?
3

Are you planning to build a resilient cloud data capability, but are overwhelmed by competing approaches and technological choices (e.g. data lake or data warehouse, Azure or AWS)?

Select country