S3 Dag Bundle won't remove files removed on remote S3 #64775
Unanswered
ilgiznurgaliev
asked this question in
Q&A
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hello.
I'm using S3 Dag Bundle and it works perfectly except one thing. Files removed remotely (in S3 bucket) won't be removed in Airflow. My source S3 bucket contains some config files additionally to dag files. I use these config files to generate dags dynamically like that:
So number of dags have to be the same as number of yaml files in
configsfolder.When one of the yaml files removed from
configsfolder it's expected that it will be removed locally too and appropriate dag will be deactivated.Actual behavior: dags will appear when new config files created, but won't be deactivated when config files removed.
Is there any way to clean up files that were removed on remote S3 bucket? Any workaround would help. Would be best to allow something like
--deleteflag for S3 Bundle configuration, but currently it is not supported.Beta Was this translation helpful? Give feedback.
All reactions