Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Repository update fallback to default branch when requested too early #4146

Open
8 tasks done
Nezz opened this issue Oct 17, 2024 · 10 comments
Open
8 tasks done

Repository update fallback to default branch when requested too early #4146

Nezz opened this issue Oct 17, 2024 · 10 comments
Labels
issue:backend For issues with the backend/integration

Comments

@Nezz
Copy link

Nezz commented Oct 17, 2024

System Health details

System Information

version core-2024.10.2
installation_type Home Assistant OS
dev false
hassio true
docker true
user root
virtualenv false
python_version 3.12.4
os_name Linux
os_version 6.6.54-haos
arch x86_64
timezone Europe/Helsinki
config_dir /config
Home Assistant Community Store
GitHub API ok
GitHub Content ok
GitHub Web ok
HACS Data ok
GitHub API Calls Remaining 4789
Installed Version 2.0.1
Stage running
Available Repositories 1450
Downloaded Repositories 15
AccuWeather
can_reach_server ok
remaining_requests 25
Home Assistant Cloud
logged_in true
subscription_expiration November 5, 2024 at 02:00
relayer_connected true
relayer_region eu-central-1
remote_enabled true
remote_connected true
alexa_enabled false
google_enabled true
remote_server eu-central-1-6.ui.nabu.casa
certificate_status ready
instance_id 1c3b420602134da4af646762c2aef794
can_reach_cert_server ok
can_reach_cloud_auth ok
can_reach_cloud ok
Home Assistant Supervisor
host_os Home Assistant OS 13.2
update_channel stable
supervisor_version supervisor-2024.10.2
agent_version 1.6.0
docker_version 27.2.0
disk_total 30.8 GB
disk_used 10.4 GB
healthy true
supported true
host_connectivity true
supervisor_connectivity true
ntp_synchronized true
virtualization vmware
board ova
supervisor_api ok
version_api ok
installed_addons Mosquitto broker (6.4.1), Terminal & SSH (9.15.0), Zigbee2MQTT (1.40.2-1), Studio Code Server (5.17.2), AirCast (4.2.2), Home Assistant Google Drive Backup (0.112.1)
Dashboards
dashboards 2
resources 7
views 5
mode storage
Recorder
oldest_recorder_run September 17, 2024 at 07:57
current_recorder_run October 18, 2024 at 01:04
estimated_db_size 694.99 MiB
database_engine sqlite
database_version 3.45.3

Checklist

  • I'm running the newest version of HACS https://github.com/hacs/integration/releases/latest
  • I have enabled debug logging for my installation.
  • I have filled out the issue template to the best of my ability.
  • I have read https://hacs.xyz/docs/help/issues/
  • This issue is related to the backend (integration part) of HACS.
  • This issue only contains 1 issue (if you have multiple issues, open one issue for each issue).
  • This is a bug and not a feature request.
  • This issue is not a duplicate issue of currently open or issues pending release.

Describe the issue

When developing an integration, I often push to GitHub and then download it via HACS. Sometimes the zipball that HACS downloads returns a 404 because it probably takes some time to be available. When this happens, HACS does not display an error. It also prompts me to restart Home Assistant to update the integration, even though it failed to update.

Reproduction steps

  1. Install a custom integration using your own fork's default branch
  2. Push a commit for a custom integration to the branch
  3. Quickly update it via HACS

Repeat until the error below is logged as it requires a certain timing.

Debug logs

2024-10-18 00:51:35.045 ERROR (MainThread) [custom_components.hacs] Download failed - Got status code 404 when trying to download https://github.com/Nezz/eight_sleep/archive/refs/heads/2ab1381.zip
Traceback (most recent call last):
  File "/config/custom_components/hacs/base.py", line 709, in async_download_file
    raise HacsException(
custom_components.hacs.exceptions.HacsException: Got status code 404 when trying to download https://github.com/Nezz/eight_sleep/archive/refs/heads/2ab1381.zip
2024-10-18 00:51:35.049 ERROR (MainThread) [custom_components.hacs] [<Integration Nezz/eight_sleep>] Failed to download zipball
Traceback (most recent call last):
  File "/config/custom_components/hacs/repositories/base.py", line 631, in download_content
    await self.download_repository_zip()
  File "/config/custom_components/hacs/repositories/base.py", line 677, in download_repository_zip
    raise HacsException(f"[{self}] Failed to download zipball")
custom_components.hacs.exceptions.HacsException: [<Integration Nezz/eight_sleep>] Failed to download zipball

Note that HACS displays and tries to download the correct hash, but in this case it took ~10 minutes for GitHub to catch up and start serving the zipball correct raw file.

Diagnostics dump

No response

@Nezz Nezz added the issue:backend For issues with the backend/integration label Oct 17, 2024
@hacs-bot
Copy link

hacs-bot bot commented Oct 17, 2024

Make sure you have read the issue guidelines and that you filled out the entire template.

If you have an issue identical to this, do not add comments like "same here", "i have this too", instead add a 👍 reaction to the issue description. Thanks! 👍

@ludeeus ludeeus added Missing required issue information Added to issues with missing information from the issue template and removed issue:backend For issues with the backend/integration labels Oct 18, 2024
@Nezz
Copy link
Author

Nezz commented Oct 18, 2024

@ludeeus What information is missing?

@Nezz
Copy link
Author

Nezz commented Oct 18, 2024

I see that there is a fallback to downloading the individual files. I'll try to debug why that does not give us the latest files.

@ludeeus
Copy link
Member

ludeeus commented Oct 18, 2024

@ludeeus What information is missing?

Debug logs? Diagnostics?

@Nezz
Copy link
Author

Nezz commented Oct 18, 2024

Here are the debug logs:

2024-10-18 15:58:08.161 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Getting hacs.json for version=bab166b
2024-10-18 15:58:08.162 DEBUG (MainThread) [custom_components.hacs] Trying to download https://raw.githubusercontent.com/Nezz/eight_sleep/bab166b/hacs.json
2024-10-18 15:58:08.217 INFO (MainThread) [custom_components.hacs] Starting download, bab166b
2024-10-18 15:58:08.219 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Running pre installation steps
2024-10-18 15:58:08.220 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Pre installation steps completed
2024-10-18 15:58:08.221 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Running installation steps
2024-10-18 15:58:08.221 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Getting repository information
2024-10-18 15:58:08.783 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Running checks against main
2024-10-18 15:58:09.488 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Getting documentation for version=bab166b,filename=README.md
2024-10-18 15:58:09.488 DEBUG (MainThread) [custom_components.hacs] Trying to download https://raw.githubusercontent.com/Nezz/eight_sleep/bab166b/README.md
2024-10-18 15:58:09.964 DEBUG (SyncWorker_17) [custom_components.hacs] Backup for /config/custom_components/eight_sleep, created in /tmp/hacs_backup/eight_sleep
2024-10-18 15:58:09.967 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Local path is set to /config/custom_components/eight_sleep
2024-10-18 15:58:09.968 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Remote path is set to custom_components/eight_sleep
2024-10-18 15:58:09.968 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Version to install: bab166b
2024-10-18 15:58:09.972 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Downloading repository archive
2024-10-18 15:58:09.973 DEBUG (MainThread) [custom_components.hacs] Trying to download https://github.com/Nezz/eight_sleep/archive/refs/tags/bab166b.zip
2024-10-18 15:58:10.545 DEBUG (MainThread) [custom_components.hacs] Trying to download https://github.com/Nezz/eight_sleep/archive/refs/heads/bab166b.zip
2024-10-18 15:58:10.776 ERROR (MainThread) [custom_components.hacs] Download failed - Got status code 404 when trying to download https://github.com/Nezz/eight_sleep/archive/refs/heads/bab166b.zip
Traceback (most recent call last):
  File "/config/custom_components/hacs/base.py", line 709, in async_download_file
    raise HacsException(
custom_components.hacs.exceptions.HacsException: Got status code 404 when trying to download https://github.com/Nezz/eight_sleep/archive/refs/heads/bab166b.zip
2024-10-18 15:58:10.778 ERROR (MainThread) [custom_components.hacs] [<Integration Nezz/eight_sleep>] Failed to download zipball
Traceback (most recent call last):
  File "/config/custom_components/hacs/repositories/base.py", line 631, in download_content
    await self.download_repository_zip()
  File "/config/custom_components/hacs/repositories/base.py", line 677, in download_repository_zip
    raise HacsException(f"[{self}] Failed to download zipball")
custom_components.hacs.exceptions.HacsException: [<Integration Nezz/eight_sleep>] Failed to download zipball
2024-10-18 15:58:10.787 DEBUG (MainThread) [custom_components.hacs] <QueueManager> Checking out tasks to execute
2024-10-18 15:58:10.788 DEBUG (MainThread) [custom_components.hacs] <QueueManager> Starting queue execution for 17 tasks
2024-10-18 15:58:10.789 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Downloading __init__.py
2024-10-18 15:58:10.791 DEBUG (MainThread) [custom_components.hacs] Trying to download https://raw.githubusercontent.com/Nezz/eight_sleep/main/custom_components/eight_sleep/__init__.py
2024-10-18 15:58:10.800 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Downloading binary_sensor.py
2024-10-18 15:58:10.801 DEBUG (MainThread) [custom_components.hacs] Trying to download https://raw.githubusercontent.com/Nezz/eight_sleep/main/custom_components/eight_sleep/binary_sensor.py
2024-10-18 15:58:10.805 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Downloading config_flow.py
2024-10-18 15:58:10.805 DEBUG (MainThread) [custom_components.hacs] Trying to download https://raw.githubusercontent.com/Nezz/eight_sleep/main/custom_components/eight_sleep/config_flow.py
2024-10-18 15:58:10.814 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Downloading const.py
2024-10-18 15:58:10.815 DEBUG (MainThread) [custom_components.hacs] Trying to download https://raw.githubusercontent.com/Nezz/eight_sleep/main/custom_components/eight_sleep/const.py
2024-10-18 15:58:10.816 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Downloading manifest.json
2024-10-18 15:58:10.816 DEBUG (MainThread) [custom_components.hacs] Trying to download https://raw.githubusercontent.com/Nezz/eight_sleep/main/custom_components/eight_sleep/manifest.json
2024-10-18 15:58:10.818 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Downloading number.py
2024-10-18 15:58:10.818 DEBUG (MainThread) [custom_components.hacs] Trying to download https://raw.githubusercontent.com/Nezz/eight_sleep/main/custom_components/eight_sleep/number.py
2024-10-18 15:58:10.821 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Downloading __init__.py
2024-10-18 15:58:10.821 DEBUG (MainThread) [custom_components.hacs] Trying to download https://raw.githubusercontent.com/Nezz/eight_sleep/main/custom_components/eight_sleep/pyEight/__init__.py
2024-10-18 15:58:10.823 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Downloading constants.py
2024-10-18 15:58:10.823 DEBUG (MainThread) [custom_components.hacs] Trying to download https://raw.githubusercontent.com/Nezz/eight_sleep/main/custom_components/eight_sleep/pyEight/constants.py
2024-10-18 15:58:10.824 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Downloading eight.py
2024-10-18 15:58:10.825 DEBUG (MainThread) [custom_components.hacs] Trying to download https://raw.githubusercontent.com/Nezz/eight_sleep/main/custom_components/eight_sleep/pyEight/eight.py
2024-10-18 15:58:10.830 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Downloading exceptions.py
2024-10-18 15:58:10.830 DEBUG (MainThread) [custom_components.hacs] Trying to download https://raw.githubusercontent.com/Nezz/eight_sleep/main/custom_components/eight_sleep/pyEight/exceptions.py
2024-10-18 15:58:10.955 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Download of __init__.py completed
2024-10-18 15:58:11.148 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Download of binary_sensor.py completed
2024-10-18 15:58:11.149 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Download of config_flow.py completed
2024-10-18 15:58:11.149 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Download of eight.py completed
2024-10-18 15:58:11.151 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Download of const.py completed
2024-10-18 15:58:11.151 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Download of exceptions.py completed
2024-10-18 15:58:11.168 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Download of __init__.py completed
2024-10-18 15:58:11.180 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Download of constants.py completed
2024-10-18 15:58:11.181 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Download of manifest.json completed
2024-10-18 15:58:11.185 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Download of number.py completed
2024-10-18 15:58:11.958 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Downloading structs.py
2024-10-18 15:58:11.958 DEBUG (MainThread) [custom_components.hacs] Trying to download https://raw.githubusercontent.com/Nezz/eight_sleep/main/custom_components/eight_sleep/pyEight/structs.py
2024-10-18 15:58:11.989 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Download of structs.py completed
2024-10-18 15:58:12.149 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Downloading user.py
2024-10-18 15:58:12.150 DEBUG (MainThread) [custom_components.hacs] Trying to download https://raw.githubusercontent.com/Nezz/eight_sleep/main/custom_components/eight_sleep/pyEight/user.py
2024-10-18 15:58:12.155 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Downloading select.py
2024-10-18 15:58:12.155 DEBUG (MainThread) [custom_components.hacs] Trying to download https://raw.githubusercontent.com/Nezz/eight_sleep/main/custom_components/eight_sleep/select.py
2024-10-18 15:58:12.159 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Downloading sensor.py
2024-10-18 15:58:12.160 DEBUG (MainThread) [custom_components.hacs] Trying to download https://raw.githubusercontent.com/Nezz/eight_sleep/main/custom_components/eight_sleep/sensor.py
2024-10-18 15:58:12.163 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Downloading services.yaml
2024-10-18 15:58:12.164 DEBUG (MainThread) [custom_components.hacs] Trying to download https://raw.githubusercontent.com/Nezz/eight_sleep/main/custom_components/eight_sleep/services.yaml
2024-10-18 15:58:12.166 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Downloading strings.json
2024-10-18 15:58:12.167 DEBUG (MainThread) [custom_components.hacs] Trying to download https://raw.githubusercontent.com/Nezz/eight_sleep/main/custom_components/eight_sleep/strings.json
2024-10-18 15:58:12.204 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Downloading en.json
2024-10-18 15:58:12.204 DEBUG (MainThread) [custom_components.hacs] Trying to download https://raw.githubusercontent.com/Nezz/eight_sleep/main/custom_components/eight_sleep/translations/en.json
2024-10-18 15:58:12.239 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Download of select.py completed
2024-10-18 15:58:12.240 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Download of sensor.py completed
2024-10-18 15:58:12.240 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Download of user.py completed
2024-10-18 15:58:12.241 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Download of strings.json completed
2024-10-18 15:58:12.243 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Download of services.yaml completed
2024-10-18 15:58:12.244 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Download of en.json completed
2024-10-18 15:58:13.245 DEBUG (MainThread) [custom_components.hacs] <QueueManager> Queue execution finished for 17 tasks finished in 2.46 seconds
2024-10-18 15:58:13.276 DEBUG (SyncWorker_23) [custom_components.hacs] Backup dir /tmp/hacs_backup/ cleared
2024-10-18 15:58:13.283 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Installation steps completed
2024-10-18 15:58:13.283 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Running post installation steps
2024-10-18 15:58:13.283 INFO (MainThread) [custom_components.hacs] Reloading custom_component cache
2024-10-18 15:58:13.302 WARNING (SyncWorker_1) [homeassistant.loader] We found a custom integration helen_energy which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-10-18 15:58:13.312 WARNING (SyncWorker_1) [homeassistant.loader] We found a custom integration spook which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-10-18 15:58:13.321 WARNING (SyncWorker_1) [homeassistant.loader] We found a custom integration smartir which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-10-18 15:58:13.323 WARNING (SyncWorker_1) [homeassistant.loader] We found a custom integration eight_sleep which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-10-18 15:58:13.329 WARNING (SyncWorker_1) [homeassistant.loader] We found a custom integration spook_inverse which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-10-18 15:58:13.333 WARNING (SyncWorker_1) [homeassistant.loader] We found a custom integration somneo which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-10-18 15:58:13.340 WARNING (SyncWorker_1) [homeassistant.loader] We found a custom integration hacs which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-10-18 15:58:13.346 INFO (MainThread) [custom_components.hacs] Custom_component cache reloaded
2024-10-18 15:58:13.346 DEBUG (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Creating restart_required issue
2024-10-18 15:58:13.347 INFO (MainThread) [custom_components.hacs] <Integration Nezz/eight_sleep> Post installation steps completed
2024-10-18 15:58:13.351 DEBUG (MainThread) [custom_components.hacs] <HacsData async_write> Saving data

In this case
https://raw.githubusercontent.com/Nezz/eight_sleep/main/custom_components/eight_sleep/pyEight/user.py
is out of date and takes a while to change.

It seems that GitHub internally uses a 5 minute cache for raw files, but adding a token parameter would work around it for public repos:
https://github.com/orgs/community/discussions/46758#discussioncomment-6078032

@Nezz
Copy link
Author

Nezz commented Oct 18, 2024

Another, possibly more reliable solution is to use the commit hash to retrieve the files instead of the branch:
https://raw.githubusercontent.com/Nezz/eight_sleep/2b3b799/custom_components/eight_sleep/pyEight/user.py

This way when HACS tells me that commit X will be downloaded, it will actually download it:
image

@Nezz Nezz changed the title Integration update fails silently with "Failed to download zipball" Integration update downloads out-of-date version Oct 18, 2024
@ludeeus
Copy link
Member

ludeeus commented Oct 18, 2024

Yes, when its requested like this it should not try the default branch, that is a part of the old logic.
With that said, HACS i really designed for published things, not for development (there is git for that ;) )

@ludeeus ludeeus added issue:backend For issues with the backend/integration and removed Missing required issue information Added to issues with missing information from the issue template labels Oct 18, 2024
@ludeeus ludeeus changed the title Integration update downloads out-of-date version Integration update fallback to default branch when requested too early Oct 18, 2024
@ludeeus ludeeus changed the title Integration update fallback to default branch when requested too early Repository update fallback to default branch when requested too early Oct 18, 2024
@Nezz
Copy link
Author

Nezz commented Oct 18, 2024

Yes, it's not a typical use-case :)

By the way, I'm not sure if it's a timing thing or a change from GitHub, but the zipball from yesterday is still missing:
https://github.com/Nezz/eight_sleep/archive/refs/heads/2ab1381.zip

I get a 404 from the upstream too, which has not changed in 5 days 🤔
https://github.com/lukas-clarke/eight_sleep/archive/refs/heads/81bc2e7.zip

@ludeeus
Copy link
Member

ludeeus commented Oct 18, 2024

Oh! My bad..
You can not use the short commit, you need to use the full length

@Nezz
Copy link
Author

Nezz commented Oct 18, 2024

This URL with the full hash however works:
https://github.com/lukas-clarke/eight_sleep/archive/81bc2e727b7fefb00cd2cd9da27015711b1a9cae.zip

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
issue:backend For issues with the backend/integration
Projects
None yet
Development

No branches or pull requests

2 participants