DataOps for Cloud DW
Integrated DataOps development and operations platform for building, deploying and running cloud-based data warehouses. Data modeling, ELT & orchestration capabilities combined in the same software.
Go to featuresAgile Data Engine addresses 4 major demands
Truly exceptional concept! I'm also happy to say that we've been running our own data platform on it for two years now and it's working like a charm.
Agile Data Engine combines best of both worlds from data management and software engineering
Productivity leap with Agile Data Engine
-
Faster time-to-delivery & experiments
-
Lower total cost of ownership
-
Data capability that adapts and scales
Multi-Cloud DataOps Platform for Cloud DW
Agile Data Engine supports your multi-domain data and multi-site teams with multi-tiered architecture and multi-phased continuous deployment to isolated Development, Test and Production environments.
Agile Data Engine allows us to focus on what is important – producing value. We do not have to think about the release process when everything works automatically.
Design/Dev/Test/Prod environment automation
Agile Data Engine has an automated and configurable cloud infrastructure set-up enabling Development/Test/Production environments’ isolation and fast start-up of the project.
Multi-cloud and multi-DBMS support
Agile Data Engine is built with multi-cloud and multi-DBMS support for easy transition from a cloud/DBMS service to another. It is also about being future-proof enabling support to-be-developed to new market leading cloud/DBMS services to come. Currently it supports AWS & Azure in cloud and Snowflake, AWS Redshift, Azure SQL DW, Azure SQL DB & Google BigQuery (in preview) in DBMS services.
Data modeling and transformations combined
Agile Data Engine has a unique entity-driven approach for implementing data models and data transformations together to enable modularity in data warehouse development. All data warehouse modeling methodologies are supported.
Data Vault automation
Data Vault 2.0 automation for generating data vault entity loads and accelerating data modeling.
Interactive end-to-end data lineage
Graphical data lineage view with data warehouse entities and loads, with metadata based search. Lineage helps data professionals in managing the whole data platform solution content and helps in troubleshooting and understanding the audit trail of the data loads.
Continuous deployment
Continuous deployment framework manages the deployment pipeline of data warehouse solution packages from Design to Development/Test/Production runtime environments.
Data model deployment with automatic schema changes
Data model deployment is a fully automated process and the automatic change detection of schema happens in runtime based on the metadata. The framework supports multiple DBMS services (currently Snowflake, AWS Redshift, Azure SQL DW, Azure SQL DB). No need for schema compare tools and incremental change scripts for data model and grants/revokes.
Metadata-driven workflow generation
Data workflows (as directed acyclic graphs) are dynamically generated based on entities’ load dependencies and scheduling information in the metadata repository.
Flexibility for different data ingestion styles
There are a multitude of ways to ingest and integrate source data into cloud data platforms and Agile Data Engine enables the use of these based on source data and workload requirements. Regardless of style, ingestion to database platform is harmonized with Agile Data Engine DW input interface.
Metadata and data crawling from JDBC sources
Agile Data Engine provides functionality to crawl both metadata and data from JDBC supported database sources.
ELT processing and integration with other processing methods
Agile Data Engine utilises an ELT approach instead of ETL, which means that data processing/compute capacity is based on the cloud database platform and therefore the processing capacity can be scaled easily, when requirements change. Processing of data can also be done with other cloud-based processing technologies (e.g. Spark) and integrated into Agile Data Engine’s orchestration engine.
Data Smoke Tests
Integrated and customizable data smoke tests as part of the data workflows, that monitor data quality continuously and can be used to control the execution of workflows or to flag warnings.
Dependency and concurrency aware workflow orchestration
Agile Data Engine’s Dagger component utilises Apache Airflow open source software as the data orchestration platform. Airflow is integrated together with Agile Data Engine software, so Airflow DAG codes are dynamically generated based on the metadata.
How can we help you?
1
Does your data warehousing team underperform and struggle to satisfy the increasingly challenging data requirements coming from business? Do you want to know how to achieve 4X value from your cloud data platform?
2
Do you have a need to migrate your existing data warehouse to cloud and need to do it as fast as possible, but without compromising the architecture?
3
Are you planning to build a resilient cloud data capability, but are overwhelmed by competing approaches and technological choices (e.g. data lake or data warehouse, Azure or AWS)?
