Crashing with "panic: failed to migrate ds" #4588
Comments
|
Hi @shashank42 , thank you for reporting the issue. I will get back to you soon. |
|
@shashank42 can you please share the version number of |
|
For the current deployment https://github.com/glip-gg/rudderstack-helm Here is the repo I am using |
|
I think the issue is similar to this. I have taken the steps here, that is deleting all the tables inside jobsdb, however the errors have not stopped |
|
@shashank42 does the issue persist when you restart the postgres db pod? |
|
I just did a new deployment on a new cluster to check fresh setup case. There are still restarts, but less frequent. 
Postgress doesn't seem to run out memory at any point. |
|
@shashank42 can you check the resource (CPU, RAM) usage before and after you start the server? please also mention what resources(CPU, memory) have you provisioned for postgres? Context: with the limited info, we are guessing that enough resources might not have been allocated (assuming the server is not overloaded with unusual number of requests) which causes a query timeout |
|
@shashank42 can you please help with more information as mentioned above? |
|
The error logs indicate that the RudderStack backend is crashing and restarting. This could be due to a variety of reasons, including issues with the database connection, memory limits, or problems with the Docker entrypoint script. Here are some steps you can take to troubleshoot this issue: Check the Database Connection: The log message Computed db host to namaste-rudderstack-rudderstack-postgresql-0.namaste-rudderstack-rudderstack-postgresql-headless suggests that the application is trying to connect to a PostgreSQL database. Ensure that the database is running and accessible from the RudderStack backend. If the database is running on a different server, check that the network connection between the two servers is working correctly1. |
|
Thank you! I tried to resolve multiple times but I think kubernetes was an overkill. Just using Docker right now and it works great. I do have one issue where I am running out of memory (200gb) in local postgress after every few days so I clean and restart. Is there an option to have an expiry of local archive? I have these variables right now |
|
So sorry @shashank42 , I missed your reply. Good to hear that docker installation is working for you.
Let me check if there's a configuration for this |
|
Same issue here: `2024-11-25T09:37:05.896Z INFO jobsdb.gw jobsdb/jobsdb.go:1234 [JobsDB] gw_jobs_9225 is full by rows. Count: 100954, Size: 241360896 2024/11/25 09:40:56 notifying bugsnag: failed to migrate ds: pq: canceling statement due to user request; driver: bad connection `
|
|
@kalavt can you please describe how to reproduce your issue and how did it impact you? |
|
@shashank42 this is the config to set local archive expiry - |
|
Hi @gitcommitshow I've no idea how to reproduce the issue, `
rudder-server rudder-server-postgresql-0 what I can see is on rudder-server-o crashing, the postgresql-0 shows below logs might relevant: ` 2024-11-25 09:32:51.447 GMT [122614] ERROR: canceling autovacuum task ` let me know if you would like more details or logs. |
|
@gitcommitshow I think it's might relevant to below SQL, after DB Accumulated up with massive data, the performance would be very low, hence we got "panic": "failed to migrate ds: pq: canceling statement due to user request; driver: bad connection" and rudder-server kept crashing. it seems the MR fixed the issue, but not merged to 1.3.x release. |
|
@kalavt are you using rudder-server Please upgrade to at least |
|
Hi @gitcommitshow I'm currently on rudder-server 1.38.3, 
|
In that case, your version is patched already. But you still see this error, right? |
Describe the bug
Deployed using https://github.com/rudderlabs/rudderstack-helm
rudderstack backend keeps crashing and restarting with the error
While at the same time postgress machine logs the following error
While there are no errors on transformer
The text was updated successfully, but these errors were encountered: