SOP-004: Set Up GPU Workloads

DOCUMENT CONTROL

Field	Value
SOP ID	SOP-004
Version	1.0
Status	Active

Purpose

Deploy AI inference workloads on Cloud Run with NVIDIA L4 GPU support. Host LLMs like Llama 3.1, Mistral, and Gemma 2 with on-demand GPU access that scales to zero.

GPU Specifications

GPU Type	VRAM	Best For
NVIDIA L4	24GB	LLM inference, image generation

KEY BENEFITS

• 5-second startup for GPU instances
• Scale to zero - pay only when processing
• No GPU reservation required

Prerequisites

• Cloud Run GPU quota approved (request at signup form)
• Region with GPU support (us-central1, europe-west4)
• Container with CUDA support

Flowchart

┌─────────────────┐
│ Request GPU     │
│    Quota        │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Build GPU-      │
│ enabled Image   │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Deploy with     │
│   --gpu flag    │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Configure       │
│ Memory/CPU      │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Test Inference  │
└─────────────────┘

Procedure

Step 1: Request GPU Quota

1. Go to Cloud Run GPU signup
2. Fill out the request form
3. Wait for approval (typically 1-2 business days)

Step 2: Prepare GPU Container

Dockerfile for vLLM

FROM vllm/vllm-openai:latest

ENV MODEL_NAME="meta-llama/Llama-3.1-8B-Instruct"
ENV PORT=8080

CMD ["--model", "${MODEL_NAME}", "--port", "8080", "--host", "0.0.0.0"]

Dockerfile for Ollama

FROM ollama/ollama:latest

EXPOSE 8080

COPY entrypoint.sh /entrypoint.sh
RUN chmod +x /entrypoint.sh

ENTRYPOINT ["/entrypoint.sh"]

FastAPI + Transformers

# app.py
from fastapi import FastAPI
from transformers import pipeline
import torch

app = FastAPI()
generator = pipeline("text-generation",
                     model="meta-llama/Llama-3.1-8B-Instruct",
                     device="cuda")

@app.post("/generate")
async def generate(prompt: str):
    result = generator(prompt, max_length=500)
    return {"response": result[0]["generated_text"]}

Step 3: Build and Push Image

# Build for GPU
docker build -t gcr.io/YOUR_PROJECT/llm-service:latest .

# Push to registry
docker push gcr.io/YOUR_PROJECT/llm-service:latest

Step 4: Deploy with GPU

gcloud run deploy llm-service \
  --image gcr.io/YOUR_PROJECT/llm-service:latest \
  --region us-central1 \
  --gpu 1 \
  --gpu-type nvidia-l4 \
  --memory 24Gi \
  --cpu 8 \
  --timeout 900 \
  --concurrency 1 \
  --no-cpu-throttling \
  --allow-unauthenticated

IMPORTANT FLAGS

• --gpu 1 - Attach one GPU
• --gpu-type nvidia-l4 - Specify L4 GPU
• --concurrency 1 - One request per instance for GPU workloads
• --no-cpu-throttling - Keep CPU active for GPU operations

Step 5: Configure for Production

# Set min instances to avoid cold starts
gcloud run services update llm-service \
  --region us-central1 \
  --min-instances 1

# Add health check endpoint
gcloud run services update llm-service \
  --region us-central1 \
  --startup-probe-path /health \
  --startup-probe-initial-delay 30s

Supported Models

Model	VRAM Required	Recommended Config
Llama 3.1 8B	~16GB	24Gi memory, 8 CPU
Mistral 7B	~14GB	24Gi memory, 8 CPU
Gemma 2 9B	~18GB	24Gi memory, 8 CPU
Llama 3.1 70B	Requires quantization	Not recommended

Testing the Service

# Get service URL
SERVICE_URL=$(gcloud run services describe llm-service \
  --region us-central1 \
  --format 'value(status.url)')

# Test inference
curl -X POST "${SERVICE_URL}/generate" \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Explain Cloud Run in one sentence:"}'

Verification Checklist

• [ ] GPU quota approved for project
• [ ] Container starts successfully with GPU
• [ ] CUDA recognized inside container
• [ ] Model loads into GPU memory
• [ ] Inference returns expected results
• [ ] Cold start time acceptable (<30s)

Cost Optimization

# Scale to zero when not in use
gcloud run services update llm-service \
  --region us-central1 \
  --min-instances 0

# Use startup CPU boost for faster cold starts
gcloud run services update llm-service \
  --region us-central1 \
  --startup-cpu-boost

GPU Pricing

Component	Price
NVIDIA L4 GPU	~$0.70/GPU-hour
Memory (24Gi)	~$0.05/GiB-hour
CPU (8 vCPU)	~$0.14/vCPU-hour

Scales to zero = $0 when idle