feat(py): A dbt adapter for Feldera (not a Feldera adapter, but a dbt adapter, that is wrapping Feldera)#5950
Conversation
…w count. TODO: Assert IVM by adding more data to source, perhaps via Kafka
|
I was going to review this but on first glance I don't understand it at all. Feldera adapters are written in Rust and go in the adapters crate. This is something else completely different written in Python and I don't see how it is a Feldera adapter at all. |
|
Hey @blp, This is not a Feldera adapter, this is a dbt adapter: dbt is a super popular CLI tool used by Data Engineers/Analysts. The idea of building a dbt adapter is to increase Feldera adoption amongst laymans. Here are some dbt adapters for other ETL/Database engines, they're all written in Python: https://github.com/dbt-labs/dbt-adapters I suppose whoever wrote the Python client for Feldera would be a more suitable reviewer, as this PR is using that client to bridge dbt to feldera. |
OK. I will remove myself from the reviewers list, because I am not the right person to look at this. |
|
@mdrakiburrahman Mihai should definitely look at this, but is traveling so might be slow to reply. In the meantime, tagging in @snkas who understands the python API well. |
|
@lalithsuresh - no rush! I still have a few TODOs to get this PR merge worthy, but I'm curious to get feedback on how I'm bridging the SQL to REST API |
|
also @abhizer |
…e FACT table UNION
|
When you are done with your rework please request another review. |
|
Hey @mihaibudiu, I think I got em all! Apologies if I missed any, please take a look at the latest iteration. |
mihaibudiu
left a comment
There was a problem hiding this comment.
I have approved, although there are still a couple of minor issues.
I will try to play with this stuff; for now I have only read the code.
|
Hey @gz, @mihaibudiu - Would you have permissions to merge? |
|
Since this has been around for a while, I assume everyone who wanted to comment has commented, so I will put this in the merge queue. |
A new dbt adapter for Feldera.
Small demo
Notes
Not for this PR, but we need to open a PR in the dbt repo to get this adapter docs exposed in dbt's public docs, sample PR: Onboarding dbt-fabricspark adapter, adding documentation of the adapter dbt-labs/docs.getdbt.com#7219
Certain features like dbt
snapshotsare not yet supported, we need to think through how this can be done in Feldera. I noticed the confluent Flink adapter doesn't have it, so perhaps it's fine to start without these luxuriesIntegration tests are part of the PR but not running in CI, will come with the existing CI infra that @gz noted in this PR discussion, task here: [dbt] Add CI support with existing Kafka broker infra #5966
Describe Manual Test Plan
Includes a
dbt-adventureworksIntegration test that runs both PyTest and dbt CLI with bash. Uses Feldera in docker.There is a
.scripts/run.sh alltarget that runs all targets of the new Python project locally, see thedbt-feldera/README.md:Checklist
Breaking Changes?
Mark if you think the answer is yes for any of these components:
Describe Incompatible Changes
N/A
Issue
Closes #5794