Smarter | Data Engineering | Scheduling

datasqill /ˈdeɪtə skɪl/ is a server-based data engineering and scheduling solution for massive data processing in context of data warehousing, big data, data science and data integration.

datasqill helps customers to uncover full potential of their data solutions by providing transparent yet very powerful platform to develop, execute and monitor data transformations. datasqill brings it to the next level by modularizing routine operations, providing rich integration capabilities, full SQL support, reliable scheduling and high customizability. datasqill ensures high quality standards without sacrificing agility and development speed.

Download Flyer Download Demo Version

Push Down

Transformations get executed directly in the environment where data is stored, which allows using datasqill for performant and transparent data processing for ELT and ETL Scenarios. Resources of respective databases are optimally used and new features like In-Memory or MPP are seamlessly supported.

SQL Support

datasqill enables use of SQL as data transformation language in all project stages making development process more agile and debugging process more transparent as the code is directly executable. datasqill provides Apache Freemarker Template-based modules for most common database-native transformations (e.g. Insert, Update, Upsert, Merge, Delete) to automate and standardize routine operations.

Smart Orchestration

datasqill identifies dependencies between transformations based on the metadata and executes them in a correct order without needing to create special job nets thus saving developers effort and minimizing mistakes. Metadata, dependencies and transformation code are stored in historized Repository of datasqill thus enabling auditability and data lineage.

Modularization

datasqill provides a number of modules to streamline transformation routines, amongst them In-Database Transformation, Flat Files Handling, Web Services Communication, Integration with Reporting Tools and External Schedulers. Custom modules can be built using generic or database-native technologies (Apache Freemarker, Java, PL/SQL, Shell-Scripting, Lua, PL/pgSQL) and seamlessly integrated.

Data Historization

Out-of-the-Box Solution for data historization for databases allows efficient Data Management for scenarios like Slowly Changing Dimensions (SCD2), Data Staging and maintaining mapping tables in Data Warehousing without writing a single line of code.

Scheduling and Monitoring

datasqill allows flexible scheduled execution of transformation batches while considering inter-batch dependencies. Historical execution statistics are used for optimization of the future runs. Monitoring views provide visual insights into running batches and give operators control over executed processes.

Use Case: Data Warehousing and BI

datasqill is the ideal tool for implementation, execution and monitoring of ELTs and ETLs for Data Warehousing: staging source systems data, loading 3NF cores, Star-Schemas, Data Vaults and communication with reporting systems. Rich module library, well thought through concepts for security, code deployment and easy operations will ensure enterprise-grade standards without sacrificing agility.

Use Case: Orchestration and Scheduling

datasqill can orchestrate not only its own transformations but also trigger and control operations performed by other tools (e.g. other ETL Tools, Java Programs or Scripts). In this case customers would benefit from centralized environment that allows building and scheduling batches, managing dependencies (intra- and inter-batch) and monitoring execution, thus making operations much easier.

Use Case: Big Data

datasqill is able to control native Hadoop Clients like HDFS or Apache Sqoop as well as to execute transformations via Apache Hive or Spark, thus bringing transformation logic directly into the environment where data is stored which is especially important technique to process large amounts of data.

Use Case: Cloud

datasqill supports communication via web services, allows performing data transformations cross-environment and provides an overview of processes running In-the-Cloud and On-Premises, so that the customers have their transformation landscape under control disregarding of its distribution. datasqill server runs on Linux and can be run in any cloud.

Use Case: In-Memory / MPP Databases

datasqill runs native transformations in database with SQL or database-native scripting (PL/SQL, Lua), thus eliminating unnecessary layer in form of ETL application server and utilizing natural power of In-Memory or MPP Databases for transformations.

Use Case: Data Integration

Data integration from heterogeneous data sources (Databases, Flat Files, Web Services), parsing different formats (e.g. CSV, XML, JSON), transforming und exporting processed data can be easily automated with datasqill.

Transformations


Transformations (blue) are placed on the Worksheet, source and target objects (grey) are defined for each transformation.

Code Editor


Transformation logic is programmed in form of SQL Select-Statement. Functional module generates a target SQL at runtime and executes it against the database.

Monitor


Monitoring view allows supervising and controlling the running transformations.

Batch Editor


Transformations from multiple worksheets can be grouped into batches and scheduled for execution.

  • "datasqill attracted our attention by its concepts of native execution and development with SQL. Using SQL for analysis, implementation and testing allowed a smooth transition between project phase and more agile development. Further features like smart scheduling, standardized modularity and customizability have convinced us even more. datasqill supports short development cycles as opposed to many other ETL Solutions. This all made datasqill one of the pillars of our Data Warehouse. The achievements during implementation of our Data Warehouse were noted by CIO Magazine and Business Application Research Center (BARC) in Germany.""

    Head of BICC, M-net Telekommunikations GmbH

  • "With an introduction of the innovative orchestration solution datasqill we were able to improve stability of our ETL processes. We have reached new level of data quality. Today we can monitor and clearly see if there any issues in our loading processes and quickly react. datasqill and SoftQuadrat consultants have made valuable contribution to optize development process and ensure operational efficiency for our modern analytical data platform."

    Head of Business Intelligence Operations, Seven.One Entertainment Group GmbH

Contact us

Contact us via the contact form to download a free copy of datasqill for evaluation. We would be glad to show you how datasqill can improve your data transformation processes and address your data engineering challenges.

We are also interested in getting in contact with consulting companies that find datasqill interesting for their work.

datasqill was with by SoftQuadrat GmbH in Germany.

contactdatasqill.de
Tel.: +49 8095 875927

Our Partners

Exasol Snowflake Productive Data Eurodata Comesio Cologne Intelligence Blue Veery Brainer

I have read and understood the Privacy Policy. I confirm that my contact data can be stored to be used for communication and further inquiries regarding my message. Hint: You can always opt-out by contacting us via contactdatasqill.de.