Dozer

This week at Dozer #5

This week at Dozer #5

Abhishek Mishra's photo
Abhishek Mishra
·

3 min read

Table of contents

🌟 Welcome to this week's update on dozer! Dozer is excited to announce the release of version v0.1.17, which brings a host of new features and improvements. Our team has been working tirelessly to enhance the overall functionality, and we can't wait for you to experience all the new features. Here are the updates for this week.

Key enhancement of v0.1.17 🚀

Switched to Apache License #1404 📜:

In this release, Dozer has switched to the Apache License (The 2.0 version), making is 100% Open Source. This change provides a more permissive and flexible licensing option, encouraging collaboration and adoption by a wider range of developers and organizations.

Asyncify Connectors #1409 ⚡:

In this release, connectors have been refactored to work asynchronously, leading to performance improvements and smoother execution.

SQL Logic Test Framework #1326 🧪:

A new SQL Logic Test Framework has been introduced, allowing for more comprehensive testing of Dozer's SQL capabilities.

APIs for Monitoring and Status Updates #1329 📈:

We've added APIs for monitoring and status updates, enabling better insight into Dozer's performance and health.

Graceful API Thread Shutdown #1416 🛠️:

The API thread now shuts down gracefully, preventing potential issues with abrupt termination.

Dozer Lambda Functions: A Glimpse into the Future 🚀🔮 #1424, #1426 🐍🌐:

We're excited to share that we've released JavaScript and Python bindings for reading Dozer logs. These bindings will be part of our Dozer Lambda runtime, allowing you to create Lambda functions in JS and Python that react to Dozer events. Just imagine triggering a Twilio notification based on a SQL statement (data change condition)! This feature is currently experimental, and we'll publish a complete article with use cases and samples soon.

state diagram of dozer lambda runtime on data change condition

Note: DCC stands for Data Change Condition

Here's an example for JavaScript:

const dozer = require('..');

async function main() {
    runtime = dozer.Runtime();
    reader = await runtime.create_reader('../.dozer/pipeline', 'trips');

    for (let i = 0; i < 10; ++i) {
        let op = await reader.next_op();
        console.log(op);
    }

    process.exit(0);
}

main();

And for Python:

import dozer_log

async def main():
    reader = await dozer_log.LogReader.new('../.dozer/pipeline', 'trips')
    for _ in range(10):
        data = await reader.next_op()
        print(data)

if __name__ == '__main__':
    import asyncio
    asyncio.run(main())

In these examples, the console.log() and print() statements can be replaced with user-defined code.

Other Improvements & Fixes 🔧

  • Fixed e2e tests writing to directories that are not removed in #1407

  • Fixed type validation in #1405

  • Replaced std::thread::sleep with tokio::time::sleep in #1413

  • Consolidated to a single Tokio runtime for everything in #1417

  • Used the correct runtime for internal pipeline thread in #1418

  • Refactored as dozer-log crate in #1420

  • Used proper error message for PostgreSQL replication slot creation in #1422

  • Stored schema of sinks separately in #1423

  • Added deb generation to release process in #1432

  • Removed unnecessary workspace.exclude in Cargo.toml in #1430

  • Bumped h2 from 0.3.16 to 0.3.17 in #1427

Dozer v0.1.17 is available now. Check out the release notes here.

Looking Forward 🌈

As we continue to make strides in improving Dozer, we remain committed to enhancing its features and functionality. Your feedback is invaluable to us, so please don't hesitate to reach out with any suggestions or ideas. Together, we can make Dozer the best it can be!

Full Changelog:

Contact us 📬

RustAPIsDatabasesOpen SourceDeveloper