-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Read DAG from file only later during _run_raw_task #1510
Commits on Jun 12, 2024
-
AIP-64: Add TaskInstance history table (apache#39951)
* AIP-64: Add TaskInstance history table This commit adds the taskinstance history table as a first step to implementing AIP-64 Co-authored-by: Jed Cunningham <[email protected]> Co-Authored-By: dstandish <[email protected]> * Remove rel from TI<->TIHistory * Add a couple comments * Update the history table at strategic points * Add history table to db cleanup * Remove history table update from ti.set_state * Lazily import the history table * Fix server default db migration for max_tries & map_index * Fix Backcompat for provider test in old airflow * Update the history table only when task completes * record state as failed when it's upd_for_retry * Only record the ti history when it's rerunning * fixup! Only record the ti history when it's rerunning * Don't use column.copy() since it's deprecated * Add test for task clearing * Refactor TI history recording * Update test * Use table to use UUID as PK. also removed onupdate cascade for TI &TIH table * Add unique constraint and use autoincrementing ID for PK * Add back onupdate cascade * Remove TaskIntanceHistory from lazy imports in models * Update comment --------- Co-authored-by: Jed Cunningham <[email protected]> Co-authored-by: dstandish <[email protected]> Co-authored-by: Jed Cunningham <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 89b32e6 - Browse repository at this point
Copy the full SHA 89b32e6View commit details -
Much smaller CI output for paralell tests (apache#40192)
The output of parallel tests especially for lowest-direct tests is very long because we are printing state every 10 seconds and there are many paralell test types (90+ for lowest direct). We do not need to print progress that often, and it has already been added in apache#39946 but one place to add it was missing - context manager still had the default 10 seconds refresh time. After this change the output will be printed every 20 seconds in the regular tests and every 2 minutes in "lowest-direct" tests (controlled by env variable) so the output should be much easier to find reasons for issues.
Configuration menu - View commit details
-
Copy full SHA for a90c07e - Browse repository at this point
Copy the full SHA a90c07eView commit details -
Bump minimum version of google auth (apache#40190)
The apache#39873 added an implicit dependency to google auth > 2.29.0 because it uses SubjectTokenSupplier added in that version. Our "Lowest-direct" tests caught it (yay!) so we should add the min requirement to the dependency.
Configuration menu - View commit details
-
Copy full SHA for 23a0152 - Browse repository at this point
Copy the full SHA 23a0152View commit details -
Configuration menu - View commit details
-
Copy full SHA for 14deaa2 - Browse repository at this point
Copy the full SHA 14deaa2View commit details -
Configuration menu - View commit details
-
Copy full SHA for c98cd54 - Browse repository at this point
Copy the full SHA c98cd54View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1372e10 - Browse repository at this point
Copy the full SHA 1372e10View commit details -
Configuration menu - View commit details
-
Copy full SHA for 794678f - Browse repository at this point
Copy the full SHA 794678fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 930db71 - Browse repository at this point
Copy the full SHA 930db71View commit details -
openlineage: add some debug logging around sql parser call sites (apa…
…che#40200) Signed-off-by: Maciej Obuchowski <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for f0b51cd - Browse repository at this point
Copy the full SHA f0b51cdView commit details -
Configuration menu - View commit details
-
Copy full SHA for c1ffe45 - Browse repository at this point
Copy the full SHA c1ffe45View commit details -
Configuration menu - View commit details
-
Copy full SHA for 28c1419 - Browse repository at this point
Copy the full SHA 28c1419View commit details -
Configuration menu - View commit details
-
Copy full SHA for a84d56d - Browse repository at this point
Copy the full SHA a84d56dView commit details -
Configuration menu - View commit details
-
Copy full SHA for c2a93ea - Browse repository at this point
Copy the full SHA c2a93eaView commit details -
Configuration menu - View commit details
-
Copy full SHA for 835f28c - Browse repository at this point
Copy the full SHA 835f28cView commit details
Commits on Jun 13, 2024
-
Ensures DAG params order regardless of backend (apache#40156)
* Ensures DAG params order regardless of backend Fixes apache#40154 This change adds an extra attribute to the serialized DAG param objects which helps us decide the order of the deserialized params dictionary later even if the backend messes with us. I decided not to limit this just to MySQL since the operation is inexpensive and may turn out to be helpful. I made sure the new test fails with the old implementation + MySQL. I assume this test will be executed with MySQL somewhere in the build actions? * Removes GitHub reference Co-authored-by: Jed Cunningham <[email protected]> * Serialize DAG params as array of tuples to ensure ordering Alternative to previous approach: We serialize the DAG params dict as a list of tuples which _should_ keep their ordering regardless of backend. Backwards compatibility is ensured because if `encoded_params` is a `dict` (not the expected `list`) then `dict(encoded_params)` still works. * Make backwards compatibility more explicit Based on suggestions by @uranusjr with an additional fix to make mypy happy. --------- Co-authored-by: Jed Cunningham <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 2149b4d - Browse repository at this point
Copy the full SHA 2149b4dView commit details -
local task job: add timeout, to not kill on_task_instance_success lis…
…tener prematurely (apache#39890) Signed-off-by: Maciej Obuchowski <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for fa65a20 - Browse repository at this point
Copy the full SHA fa65a20View commit details -
Resolve deprecations in
LatestOnlyOperator
tests (apache#40181)* Resolve deprecations in `LatestOnlyOperator` * Use explicit data_interval
Configuration menu - View commit details
-
Copy full SHA for feb8307 - Browse repository at this point
Copy the full SHA feb8307View commit details -
doc: metrics allow_list complet example (apache#40120)
Co-authored-by: raphaelauv <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 205ad57 - Browse repository at this point
Copy the full SHA 205ad57View commit details -
AIP-64: Add UI endpoint for task instance history (apache#40221)
* AIP-64: Add UI endpoint for task instance history This adds UI endpoint for task instance history Co-authored-by: Jed Cunningham <[email protected]> Co-Authored-By: dstandish <[email protected]> Co-Authored-By: Brent Bovenzi <[email protected]> * fixup! AIP-64: Add UI endpoint for task instance history --------- Co-authored-by: Jed Cunningham <[email protected]> Co-authored-by: dstandish <[email protected]> Co-authored-by: Brent Bovenzi <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 2272ea2 - Browse repository at this point
Copy the full SHA 2272ea2View commit details -
Fix Scheduler restarting due to too many completed pods in cluster (a…
…pache#40183) * Fix Scheduler restarting due to too many completed pods in cluster Currently, when a pod completes and is not deleted due to the user's configuration, the watcher keeps listing these pods and checking their status. We should instead stop watching the pod once it succeeds. To do that, pods are created with the executor done label set to False and changed to True when the pod completes. The watcher then watches only those pods that the pod executor done label is False closes: apache#22612 * Update airflow/providers/cncf/kubernetes/pod_generator.py Co-authored-by: Jed Cunningham <[email protected]> * Add back removed section * Don't add pod key label from get go * Update airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py Co-authored-by: Jed Cunningham <[email protected]> --------- Co-authored-by: Jed Cunningham <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 67798b2 - Browse repository at this point
Copy the full SHA 67798b2View commit details
Commits on Jun 14, 2024
-
Configuration menu - View commit details
-
Copy full SHA for d5a7544 - Browse repository at this point
Copy the full SHA d5a7544View commit details -
openlineage, redshift: do not call DB for schemas below Airflow 2.10 (a…
…pache#40197) Signed-off-by: Maciej Obuchowski <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 6f40984 - Browse repository at this point
Copy the full SHA 6f40984View commit details -
Configuration menu - View commit details
-
Copy full SHA for f0bae33 - Browse repository at this point
Copy the full SHA f0bae33View commit details -
add generation_config and safety_settings to google cloud multimodal …
…model operators (apache#40126)
Configuration menu - View commit details
-
Copy full SHA for e2b8f68 - Browse repository at this point
Copy the full SHA e2b8f68View commit details -
fix bigquery_to_gcs documentation (apache#40219)
Currently the documentation states Importing files and also suggest that the file would be imported from GCS to BigQuery instead of what the operator actually does which is exporting single BigQuery table to GCS
Configuration menu - View commit details
-
Copy full SHA for d8a3257 - Browse repository at this point
Copy the full SHA d8a3257View commit details -
Add executor field to the task instance API (apache#40034)
Return executor as part of TaskInstance queries and also enable filtering by executor field. Also use the changes to display the executor field on the TaskInstance Details web page. Co-authored-by: Vincent <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for c2959c9 - Browse repository at this point
Copy the full SHA c2959c9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2587295 - Browse repository at this point
Copy the full SHA 2587295View commit details -
Configuration menu - View commit details
-
Copy full SHA for bffb7b0 - Browse repository at this point
Copy the full SHA bffb7b0View commit details -
Add slightly more detailed guidance about upgrading to the docs (apac…
…he#40227) --------- Signed-off-by: merobi-hub <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 16b17f7 - Browse repository at this point
Copy the full SHA 16b17f7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3e88f47 - Browse repository at this point
Copy the full SHA 3e88f47View commit details -
Databricks: stop including user names in
list_jobs
(apache#40178)* Databricks: stop including user names in `list_jobs` The user's name is not used on the Airflow side, and this argument saves the lookup, which makes the request faster.
Configuration menu - View commit details
-
Copy full SHA for a1f9b7d - Browse repository at this point
Copy the full SHA a1f9b7dView commit details -
Handle db isolation for mapped operators and task groups (apache#39259)
* Handle db isolation for mapped operators and task groups * Update airflow/models/taskinstance.py
Configuration menu - View commit details
-
Copy full SHA for e69ab3a - Browse repository at this point
Copy the full SHA e69ab3aView commit details -
openlineage: execute extraction and message sending in separate proce…
…ss (apache#40078) Signed-off-by: Maciej Obuchowski <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 1a8d12f - Browse repository at this point
Copy the full SHA 1a8d12fView commit details -
refactor: Added get_extra_dejson method with nested parameter which a…
…llows you to specify if you want the nested json as string to be also deserialized. The extra_dejson property uses this method with nested set to False. (apache#39811) Co-authored-by: David Blain <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for ca73694 - Browse repository at this point
Copy the full SHA ca73694View commit details -
Chart: set
workers.safeToEvict
default to False (apache#40229)This is a safer default for our workers. This can be safe to set to true if you have a long enough `workers.terminationGracePeriodSeconds` set, but what is "long enough" is very situational, so I feel its better to default to not evicting worker pods.
Configuration menu - View commit details
-
Copy full SHA for 4c9f12d - Browse repository at this point
Copy the full SHA 4c9f12dView commit details -
Configuration menu - View commit details
-
Copy full SHA for f5079db - Browse repository at this point
Copy the full SHA f5079dbView commit details -
Update
Dag.test()
to run with an executor if desired (apache#40205)* Update `Dag.test()` to run with an executor if desired * Add missing parameter * Fix typo * Move `add_logger_if_needed` to local execution * Add `keep-env-variables` to `breeze testing db-tests`, `breeze testing non-db-tests` and `breeze shell` * Add documentation * Fix tests * Introduce `use-executor` flag * Update `debug` documentation * Fix test
Configuration menu - View commit details
-
Copy full SHA for 9595357 - Browse repository at this point
Copy the full SHA 9595357View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3014165 - Browse repository at this point
Copy the full SHA 3014165View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8eebe2b - Browse repository at this point
Copy the full SHA 8eebe2bView commit details
Commits on Jun 15, 2024
-
Change
httpx
torequests
infile_task_handler
(apache#39799)* Change httpx to requests in file_task_handler - httpx does not support CIDRs in NO_PROXY - simply, convert httpx to requests, issues done - related issue: apache#39794 * Add cidr no_proxy test test_log_handlers.py * Apply monkeypatch fixture --------- Co-authored-by: scott-py <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 1ddadf5 - Browse repository at this point
Copy the full SHA 1ddadf5View commit details -
Add dependency to httpx >= 0.25.0 everywhere (apache#40256)
Our "lowest-dependency" tests detectaed that weaviate client depends implicitly on httpx >= 0.19.0 (imports USE_CLIENT_DEFAULTS from httpx and it's missing < 0.19.0). Howeer this error is raised during importing of examples for weaviate in "Always" tests, and closer look at weaviate shows that it actually has >=0.25.0 and it makes sense for all our providers to bump httpx to 0.25.0 as minimum as well as add it to weaviate explicitly..
Configuration menu - View commit details
-
Copy full SHA for 35871f8 - Browse repository at this point
Copy the full SHA 35871f8View commit details -
Fix typing in telegram provider (apache#40255)
The python-telegram-bot new version has typing added and we should pass the right dict type to it.
Configuration menu - View commit details
-
Copy full SHA for bc4ca9d - Browse repository at this point
Copy the full SHA bc4ca9dView commit details -
Working fix for typing in telegram provider (apache#40258)
The python-telegram-bot new version has typing added and we should pass the right dict type to it. The apache#40255 was an unsuccessful attempt to fix it, this one actually fixes it.
Configuration menu - View commit details
-
Copy full SHA for 1451bac - Browse repository at this point
Copy the full SHA 1451bacView commit details -
Fix typo when uninstalling weaviate for Pydantic 1 tests (apache#40259)
Weaviate 4 requires Pydantic v2, and we currently limit weaviate to 4+. There was already removal of weaviate for Pydantic 1, but it had a typo which caused the weaviate-client not to be uninstalled.
Configuration menu - View commit details
-
Copy full SHA for cb372e5 - Browse repository at this point
Copy the full SHA cb372e5View commit details -
Add timeout to base python test (apache#40262)
The tests for Python Virtualev operator when running in parallel can take more time than 60 seconds (default timeout) we already set the timeout on python virtualenv operator level but it seems that the timeout should also be set on the base class for tests that are run from the base class. This should remove flakiness from those tests.
Configuration menu - View commit details
-
Copy full SHA for e58c048 - Browse repository at this point
Copy the full SHA e58c048View commit details -
Add pytest timeout also to PythonVirtualenvDecorator (apache#40263)
One more test that timeouts often when running in parallel - this one also missed extra timeout.
Configuration menu - View commit details
-
Copy full SHA for 30f6161 - Browse repository at this point
Copy the full SHA 30f6161View commit details -
Resolve deprecations in the tests for
Google MLEngine
operators (ap……ache#40261) * Resolve deprecations in `MLEngine` operators * Resolve deprecations in `MLEngine` operator utils
Configuration menu - View commit details
-
Copy full SHA for 1363043 - Browse repository at this point
Copy the full SHA 1363043View commit details -
Configuration menu - View commit details
-
Copy full SHA for 161fd55 - Browse repository at this point
Copy the full SHA 161fd55View commit details -
Remove dependency on special tests for tests finalization (apache#40264)
Tests finalization is run generally when all tests succeded in canary run. What the finalization does is: * updating constrainst * pushing them * updating image cache * summarizing warnings However special tests are really to test some special cases - back compatibility, lowest dependencies, latest boto etc. All those tests will only be run in a few selected PRs where we upgrade dependencies and they will not affect "regular" PRs, so we can safely update the constraints and update the cache without waiting for special tests. This will increase the frequency of updates to constraints - because now they might be quite delayed in case some special tests fail, but this is unnecessary holding the constraints update.
Configuration menu - View commit details
-
Copy full SHA for 1d7ede7 - Browse repository at this point
Copy the full SHA 1d7ede7View commit details
Commits on Jun 16, 2024
-
Configuration menu - View commit details
-
Copy full SHA for 5690439 - Browse repository at this point
Copy the full SHA 5690439View commit details -
Configuration menu - View commit details
-
Copy full SHA for bfe5fd7 - Browse repository at this point
Copy the full SHA bfe5fd7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 518a9e4 - Browse repository at this point
Copy the full SHA 518a9e4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 60c2d36 - Browse repository at this point
Copy the full SHA 60c2d36View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8414c90 - Browse repository at this point
Copy the full SHA 8414c90View commit details