Skip to content

Fix #73432: preserve last_finish_time in callback-only DagFileProcessorManager processing - #73771

Closed
chinmayasameeru wants to merge 1 commit into
apache:mainfrom
chinmayasameeru:fix/73432-dag-processor-infinite-loop
Closed

chinmayasameeru wants to merge 1 commit into
apache:mainfrom
chinmayasameeru:fix/73432-dag-processor-infinite-loop

Conversation

@chinmayasameeru

@chinmayasameeru chinmayasameeru commented Sep 27, 2026 •

Copy link
Copy Markdown

Fix #73432 — the dag processor can get stuck in an infinite loop after a Dag-level callback runs.

Here's what happens: when a Dag file gets processed in "callback-only" mode (meaning it was triggered by a Dag callback, not a regular scan), process_parse_results creates a new DagFileStat with last_finish_time=None. That stat overwrites the existing one in _file_stats, so on the next scan the file looks like it's never been parsed. The scheduler re-queues it, processes it again, creates another stat with last_finish_time=None, and the cycle repeats forever.

The fix is to preserve the existing last_finish_time when doing callback-only processing. I added a last_finish_time parameter to process_parse_results — when it's passed (from the existing stat), the is_callback_only branch uses it instead of the current finish_time. I updated both call sites:

  • handle_parsing_result: passes self._file_stats[file].last_finish_time — this is the main fix for the reported issue
  • process_unparsed_files: passes db_stat.last_finish_time to DagFileStat.from_db — this handles the case where a file is already in the DB

I couldn't run the full airflow test suite here (the test environment needs a database and a bunch of infra), but I did validate the Python syntax with py_compile and the logic is pretty straightforward — it's just passing a timestamp through instead of dropping it.

…rocessorManager processing

When a DagFileProcessorManager processes a file in callback-only mode
(is_callback_only=True), process_parse_results creates a new DagFileStat
with last_finish_time=None. This new stat shadows the existing stat's
timestamp in _file_stats, causing the file to appear as "never parsed"
on subsequent scans. The file gets re-queued every scan cycle, creating
an infinite loop.

The fix adds a last_finish_time parameter to process_parse_results. In
the is_callback_only branch, the passed last_finish_time (from the
existing stat) is preserved instead of using the current finish_time.
The handle_parsing_result call site passes
self._file_stats[file].last_finish_time, and the process_unparsed_files
path passes db_stat.last_finish_time to DagFileStat.from_db.

Closes apache#73432
@boring-cyborg

boring-cyborg Bot commented Sep 27, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

@Eason09053360 Eason09053360 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This duplicates #73433 (opened earlier by the issue reporter).
Also, ruff format fails on the function signature — please run prek install so the static checks run before pushing. :)

@henry3260 henry3260 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you reproduce the bug yourself and confirm this fix resolves it? Note that this duplicates #73433, and the lint checks are currently failing, I may close this PR unless those are addressed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dag processor reparses a file every loop forever after a Dag-level callback runs

3 participants