Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Airflow SmartSuggest [Don't merge] #1508

Closed
wants to merge 7 commits into from
Closed

Conversation

sunank200
Copy link

@sunank200 sunank200 commented Mar 5, 2024

Airflow SmartSuggest is an intelligent assistant designed to enhance the Apache Airflow experience by providing smart suggestions for anomalies detection for DAGs, tasks and airflow resources like workers, schedulers, triggerers alerting users for cost and performance optimizations. Airflow SmartSuggest also helps to analyze DAG code and logs providing recommendations around best DAG writing practices and error resolution directly on Airflow UI. It uses Ask-Astro APIs and statistical methods for the analysis. This tool aims to integrate directly into the Airflow UI, offering real-time alerts for anomalies, DAG code optimizations, error resolution, cluster profilings and metrics suggestions. Itl also includes security features to detect and alert users about sensitive information inadvertently included in their DAG code.

This is the page for anomaly detection:
Screenshot 2024-03-06 at 6 39 45 PM

This is the page for Airflow SmartSuggest code analysis:
Screenshot 2024-03-06 at 6 42 57 PM

Screenshot 2024-03-06 at 6 43 05 PM

This is the page for Airflow SmartSuggest log analysis:

Screenshot 2024-03-06 at 6 56 53 PM

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@sunank200 sunank200 changed the title Add airflow doctor workflow for task logs Add airflow doctor [Don't merge] Mar 5, 2024
@sunank200 sunank200 changed the title Add airflow doctor [Don't merge] Add Airflow SmartSuggest [Don't merge] Mar 6, 2024
Copy link

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the Stale label Apr 21, 2024
@github-actions github-actions bot closed this Apr 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants