Resolved: Why are we suddenly getting “Function execution took 60000 ms, finished with status: ‘error'” on Firebase callable functions / google cloud functions?

In this post, we will see how to resolve Why are we suddenly getting “Function execution took 60000 ms, finished with status: ‘error'” on Firebase callable functions / google cloud functions?

Question:

Our app is in production since the March 22nd. It makes use of Firebase callable functions. The execution of the functions were flawless until suddenly on March 28th, we started to observe many failed executions get the following debug messages in the cloud functions logs Function execution took 60000 ms, finished with status: 'error'.
While some executions of the same function failed, others worked as usual and completed in on average 250ms.
When checking the cloud platform infrastructure status, there were no indicators, of any outages or any other issues.
For instance we have this very simple function:
Note that the very first line of the function is creating a log warn entry. We were able to observe the following:

• For failed executions with the above error message, there was no log.warn entry.
• For succeeded executions, there was always an according log.warn entry.

In other words, it looks like failed executions, do not even bother to execute the functions body. All we see in the failure case is Function Execution started followed by Function execution took 60000 ms, finished with status: 'error'.
These are our logs, corroborating our observations:

• When filtering for the failure case: (resource.type="cloud_function" ) OR (resource.type="cloud_run_revision" ) textPayload="Function execution took 60000 ms, finished with status: 'error' we get:

• When listing all executions (resource.type="cloud_function" ) OR (resource.type="cloud_run_revision" ) we get:

Note that we have not deployed anything around the time the issues started to show.

Best Answer:

Seeing the same issue, most calls end up with status code 408 and have much higher latency than we usually see.
Think it’s related to this, which is very broadly phrased:

Summary: Google Cloud Networking experiencing elevated latency in multiple regions

https://status.cloud.google.com/incidents/nieR2aLyg1rwFKq1aWZU#2c2sBHWU84yPDJ8y1ar4

If you have better answer, please add a comment about this, thank you!

Source: Stackoverflow.com