Troubleshooting Guide
Common Cloud Run issues and their solutions.
Deployment Issues
Container Failed to Start
| Error | Cause | Solution |
|---|---|---|
Container failed to start | App not listening on PORT | Ensure app listens on $PORT env var |
Memory limit exceeded | OOM during startup | Increase --memory flag |
Container failed to start within 4 minutes | Slow initialization | Increase --timeout or optimize startup |
bash
# Check container logs
gcloud logging read "resource.type=cloud_run_revision AND textPayload:failed" --limit 20
# Increase memory
gcloud run services update SERVICE --region REGION --memory 2Gi
# Increase timeout
gcloud run services update SERVICE --region REGION --timeout 600Permission Denied
bash
# Grant Cloud Run admin role
gcloud projects add-iam-policy-binding PROJECT_ID \
--member user:YOUR_EMAIL \
--role roles/run.admin
# Grant Artifact Registry reader
gcloud projects add-iam-policy-binding PROJECT_ID \
--member serviceAccount:PROJECT_NUMBER-compute@developer.gserviceaccount.com \
--role roles/artifactregistry.readerImage Not Found
bash
# Verify image exists
gcloud artifacts docker images list us-central1-docker.pkg.dev/PROJECT/REPO
# Check image path format
# Correct: us-central1-docker.pkg.dev/project/repo/image:tag
# Wrong: gcr.io/project/image (old format, may still work)Runtime Issues
Cold Start Latency
| Solution | Impact |
|---|---|
--min-instances 1 | Eliminates cold starts, increases cost |
--startup-cpu-boost | Faster startup, slight cost increase |
| Reduce image size | Faster container pull |
| Use multi-stage builds | Smaller final image |
bash
# Keep instance warm
gcloud run services update SERVICE --region REGION --min-instances 1
# Enable CPU boost
gcloud run services update SERVICE --region REGION --startup-cpu-boostRequest Timeouts
bash
# Increase timeout (max 3600s for HTTP, 86400s for streaming)
gcloud run services update SERVICE --region REGION --timeout 900
# Check current timeout
gcloud run services describe SERVICE --region REGION \
--format 'value(spec.template.spec.timeoutSeconds)'Memory Errors (OOM)
bash
# Check memory usage in logs
gcloud logging read "resource.type=cloud_run_revision AND textPayload:memory" --limit 50
# Increase memory
gcloud run services update SERVICE --region REGION --memory 4Gi
# Memory limits
# Min: 128Mi
# Max: 32Gi (or 64Gi with specific configs)Concurrency Issues
SYMPTOMS
- Requests queuing
- 503 errors under load
- High latency spikes
bash
# Reduce concurrency (for CPU-bound workloads)
gcloud run services update SERVICE --region REGION --concurrency 10
# Increase max instances
gcloud run services update SERVICE --region REGION --max-instances 100
# Check current settings
gcloud run services describe SERVICE --region REGION \
--format 'yaml(spec.template.spec.containerConcurrency)'Networking Issues
VPC Connector Errors
bash
# Check connector status
gcloud compute networks vpc-access connectors describe CONNECTOR \
--region REGION
# Recreate connector
gcloud compute networks vpc-access connectors delete CONNECTOR --region REGION
gcloud compute networks vpc-access connectors create CONNECTOR \
--network default \
--region REGION \
--range 10.8.0.0/28Cannot Connect to Database
| Cause | Solution |
|---|---|
| No VPC connector | Create and attach VPC connector |
| Private IP only | Use Cloud SQL Proxy or VPC |
| Firewall rules | Allow Cloud Run IP ranges |
bash
# For Cloud SQL (recommended)
gcloud run services update SERVICE \
--region REGION \
--add-cloudsql-instances PROJECT:REGION:INSTANCECORS Errors
Add CORS headers in your application:
javascript
// Express.js example
const cors = require('cors');
app.use(cors({
origin: ['https://your-frontend.com'],
methods: ['GET', 'POST', 'PUT', 'DELETE'],
allowedHeaders: ['Content-Type', 'Authorization']
}));GPU Issues
GPU Not Available
bash
# Check quota
gcloud compute regions describe us-central1 \
--format 'value(quotas[?metric==NVIDIA_L4_GPUS])'
# Request quota increase
# Go to: https://console.cloud.google.com/iam-admin/quotasCUDA Errors
| Error | Solution |
|---|---|
CUDA out of memory | Reduce batch size or model size |
CUDA not available | Verify --gpu 1 flag in deploy |
Driver version mismatch | Use compatible CUDA base image |
bash
# Verify GPU attached
gcloud run services describe SERVICE --region REGION \
--format 'yaml(spec.template.spec.containers[0].resources)'Logging & Debugging
Enable Detailed Logs
bash
# Set log level in app
gcloud run services update SERVICE \
--region REGION \
--set-env-vars "LOG_LEVEL=debug"
# View structured logs
gcloud logging read 'resource.type="cloud_run_revision"' \
--format json \
--limit 100Debug Locally
bash
# Run container locally with same env
docker run -p 8080:8080 \
-e PORT=8080 \
-e K_SERVICE=local \
YOUR_IMAGE
# Test
curl http://localhost:8080Check Service Health
bash
#!/bin/bash
SERVICE_URL=$(gcloud run services describe SERVICE \
--region REGION --format 'value(status.url)')
echo "Testing $SERVICE_URL"
# Health check
curl -w "\nStatus: %{http_code}\nTime: %{time_total}s\n" \
-o /dev/null -s "$SERVICE_URL/health"
# Load test (requires hey)
hey -n 100 -c 10 "$SERVICE_URL/"Common Error Codes
| HTTP Code | Meaning | Common Cause |
|---|---|---|
| 403 | Forbidden | IAM permissions |
| 404 | Not Found | Wrong URL or service not deployed |
| 429 | Too Many Requests | Rate limiting / quota exceeded |
| 500 | Internal Error | Application crash |
| 502 | Bad Gateway | Container not responding |
| 503 | Unavailable | Scaling / cold start issue |
| 504 | Timeout | Request exceeded timeout |