Common Cloud Run errors and how to fix them with Cloud Logging

Key Concepts

• Cloud Run: A fully managed compute platform for deploying containerized applications.
• Cloud Logging: Google Cloud's log management system, offering storage, search, analysis, and monitoring of logs.
• Standard Out: The default output stream for programs, where applications typically send their log messages.
• Container Image: A lightweight, standalone, executable package of software that includes everything needed to run an application.
• Revision: An immutable version of a Cloud Run service, created each time a new deployment occurs.
• Startup Probe: A mechanism used by Cloud Run to check if a container instance is ready to receive traffic by pinging its defined port.
• Golang Panic: In Go, a runtime error that stops the normal execution of a program, similar to an unhandled exception.
• g-cloud run deploy command: The Google Cloud CLI command used to deploy services to Cloud Run.
• Cloud Console: The web-based user interface for managing Google Cloud resources.

This video demonstrates how to troubleshoot common development issues when deploying and running applications on Cloud Run, leveraging Google Cloud's log management system, Cloud Logging. Cloud Run automatically sends application logs to Cloud Logging, and any data sent to standard out by an application is automatically captured without requiring client libraries. The Cloud Console provides a convenient way to view logs, pre-filtered for specific Cloud Run services.

The video covers troubleshooting three primary types of errors:

1. The container image fails to build.
2. The wrong port is specified, causing the revision to be unhealthy.
3. An application crash occurs in production.

Error Type 1: Container Image Fails to Build

Scenario: An error is introduced into a Go application that normally returns "hello world" on its root route. The specific error involves attempting to print an undefined variable.

Process:

1. Modify the Go application to include an undefined variable in a print statement.
2. Execute the g-cloud run deploy command, deploying from source.
3. Result: The build fails because Golang does not permit undefined variables.
4. Troubleshooting: The error message can be found in the Cloud Build logs, specifically showing "undefined message". This indicates a compilation or build-time issue.

Error Type 2: Wrong Port Specified (Revision Not Healthy)

Scenario: After fixing the previous build error, the application's listening port is hardcoded to 8085, while Cloud Run's default traffic routing port remains 8080.

Process:

1. Comment out the previous error (undefined variable).
2. Change the application's listening port to 8085.
3. Deploy the application from source again using g-cloud run deploy, without changing Cloud Run's default port configuration (8080).
4. Cloud Run Behavior: During revision creation, Cloud Run initiates a container instance and performs a "startup probe" by pinging the default port (8080) to check if the application is ready.
5. Result: The revision fails to become ready and cannot serve traffic. The Cloud Console displays the error: "revision is not ready and cannot serve traffic. The user provided container failed to start and listen on the port defined by the port equals 8080 environment variable." This explicitly points to the port mismatch.
6. Troubleshooting:
   • A link is provided to Cloud Logging for more detailed error information.
   • The error is also visible in the Cloud Console under the Cloud Run service's logs, which are automatically filtered.
   • Important Detail: This error can take up to 4 minutes to appear due to the default startup probe timeout configuration, which can be modified.
7. Resolution: The issue is resolved by modifying the application code to dynamically use the environment variable provided by Cloud Run for the port, ensuring it listens on 8080.

Error Type 3: Crash in Production

Scenario: An unhandled exception (a "panic" in Golang) is deliberately introduced into a new route (/crash) of the application.

Process:

1. Add a new route /crash to the Go application that causes a panic.
2. Deploy the updated application to Cloud Run using g-cloud run deploy from source, specifying a region (e.g., US West 2).
3. Accessing the Application:
   • Visiting the root route (/) still shows "hello world".
   • Visiting the /crash route results in a "service unavailable" (HTTP 503) error.
4. Troubleshooting:
   • Navigate to the logs section for the Cloud Run service in the Cloud Console.
   • The logs show the 503 error.
   • Crucially, the print statement to standard out (if any were made before the panic) and the full stack trace for the panic are visible, providing detailed information about the crash.

Conclusion

The video effectively demonstrates how to troubleshoot three common development and production errors on Cloud Run: container build failures, unhealthy revisions due to port misconfigurations, and application crashes. It highlights the critical role of Cloud Logging in debugging these issues, emphasizing that logs sent to standard out are automatically captured and accessible in the Cloud Console, eliminating the need for specialized logging libraries. For further exploration of Cloud Logging's extensive features, users are encouraged to consult its official documentation.