WF-002: AI Inference Setup

DOCUMENT CONTROL

Field	Value
WF ID	WF-002
Version	1.0
Status	Active

Overview

Deploy a production-ready AI inference service on Cloud Run with GPU support. This workflow covers model selection, containerization, deployment, and optimization.

Architecture Diagram

┌───────────────────────────────────────────────────────────────────────┐
│                      AI INFERENCE ARCHITECTURE                        │
├───────────────────────────────────────────────────────────────────────┤
│                                                                       │
│   ┌─────────┐     ┌──────────────┐     ┌────────────────────────┐    │
│   │ Client  │────▶│  Cloud CDN   │────▶│      Cloud Run         │    │
│   │ Request │     │  (Optional)  │     │   ┌────────────────┐   │    │
│   └─────────┘     └──────────────┘     │   │  Your Service  │   │    │
│                                        │   │  ┌──────────┐  │   │    │
│                                        │   │  │   LLM    │  │   │    │
│                                        │   │  │ (GPU)    │  │   │    │
│                                        │   │  └──────────┘  │   │    │
│                                        │   └────────────────┘   │    │
│                                        └────────────────────────┘    │
│                                                    │                  │
│                                                    ▼                  │
│                                        ┌────────────────────┐        │
│                                        │   Cloud Storage    │        │
│                                        │   (Model Weights)  │        │
│                                        └────────────────────┘        │
│                                                                       │
└───────────────────────────────────────────────────────────────────────┘

Phase 1: Model Selection

Recommended Models for Cloud Run

Model	Size	VRAM	Use Case	License
Llama 3.1 8B	8B	16GB	General chat, coding	Meta
Mistral 7B	7B	14GB	Fast inference	Apache 2.0
Gemma 2 9B	9B	18GB	Instruction following	Google
Phi-3 Mini	3.8B	8GB	Lightweight tasks	MIT

MODEL SELECTION

For Cloud Run's L4 GPU (24GB VRAM), stick to models under 13B parameters or use quantized versions of larger models.

Phase 2: Containerize Model

Option A: Using vLLM (Recommended)

# Dockerfile.vllm
FROM vllm/vllm-openai:latest

# Download model at build time (faster cold starts)
RUN python -c "from vllm import LLM; LLM('meta-llama/Llama-3.1-8B-Instruct')"

ENV MODEL_NAME="meta-llama/Llama-3.1-8B-Instruct"

CMD ["python", "-m", "vllm.entrypoints.openai.api_server", \
     "--model", "meta-llama/Llama-3.1-8B-Instruct", \
     "--port", "8080", \
     "--host", "0.0.0.0", \
     "--max-model-len", "4096"]

Option B: Using Ollama

# Dockerfile.ollama
FROM ollama/ollama:latest

# Pre-pull model
RUN ollama pull llama3.1:8b

EXPOSE 8080

# Wrapper script
COPY <<EOF /start.sh
#!/bin/bash
ollama serve &
sleep 5
ollama run llama3.1:8b --keepalive -1 &
wait
EOF

RUN chmod +x /start.sh
CMD ["/start.sh"]

Option C: Custom FastAPI + Transformers

# app.py
import os
from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

app = FastAPI()

# Load model on startup
model_name = os.getenv("MODEL_NAME", "meta-llama/Llama-3.1-8B-Instruct")
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

class GenerateRequest(BaseModel):
    prompt: str
    max_tokens: int = 500
    temperature: float = 0.7

@app.post("/v1/completions")
async def generate(request: GenerateRequest):
    inputs = tokenizer(request.prompt, return_tensors="pt").to("cuda")

    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=request.max_tokens,
            temperature=request.temperature,
            do_sample=True
        )

    response = tokenizer.decode(outputs[0], skip_special_tokens=True)
    return {"text": response}

@app.get("/health")
async def health():
    return {"status": "healthy", "model": model_name}

Phase 3: Deploy to Cloud Run

Build and Push

# Authenticate with Artifact Registry
gcloud auth configure-docker us-central1-docker.pkg.dev

# Build with GPU support
docker build -t us-central1-docker.pkg.dev/PROJECT_ID/repo/llm-service:v1 .

# Push image
docker push us-central1-docker.pkg.dev/PROJECT_ID/repo/llm-service:v1

Deploy with GPU

gcloud run deploy llm-service \
  --image us-central1-docker.pkg.dev/PROJECT_ID/repo/llm-service:v1 \
  --region us-central1 \
  --gpu 1 \
  --gpu-type nvidia-l4 \
  --memory 24Gi \
  --cpu 8 \
  --timeout 900 \
  --concurrency 4 \
  --min-instances 0 \
  --max-instances 5 \
  --no-cpu-throttling \
  --port 8080 \
  --allow-unauthenticated

Phase 4: Optimize Performance

Reduce Cold Start Time

# Keep one instance warm
gcloud run services update llm-service \
  --region us-central1 \
  --min-instances 1 \
  --startup-cpu-boost

# Add startup probe
gcloud run services update llm-service \
  --region us-central1 \
  --startup-probe-path /health \
  --startup-probe-initial-delay 60s \
  --startup-probe-period 10s

Enable Response Streaming

# FastAPI streaming response
from fastapi.responses import StreamingResponse

@app.post("/v1/chat/completions/stream")
async def stream_generate(request: GenerateRequest):
    async def generate_stream():
        # Your streaming logic here
        for token in model.generate_stream(...):
            yield f"data: {token}\n\n"

    return StreamingResponse(
        generate_stream(),
        media_type="text/event-stream"
    )

Add Caching with Cloud CDN

# Create backend service with CDN
gcloud compute backend-services create llm-backend \
  --global \
  --enable-cdn \
  --cache-mode CACHE_ALL_STATIC

# Add Cloud Run as backend
gcloud compute backend-services add-backend llm-backend \
  --global \
  --network-endpoint-group=llm-neg \
  --network-endpoint-group-region=us-central1

Phase 5: Monitor and Scale

Set Up Monitoring

# Create alert for high latency
gcloud alpha monitoring policies create \
  --display-name "LLM High Latency" \
  --condition-display-name "P95 > 30s" \
  --condition-filter 'resource.type="cloud_run_revision" AND metric.type="run.googleapis.com/request_latencies"' \
  --condition-threshold-value 30000 \
  --condition-threshold-duration 300s

View Metrics

# Check current usage
gcloud run services describe llm-service \
  --region us-central1 \
  --format yaml

# View logs
gcloud logging read "resource.type=cloud_run_revision AND resource.labels.service_name=llm-service" \
  --limit 50

Success Criteria

• [ ] Model loads successfully on GPU
• [ ] Cold start time < 60 seconds
• [ ] Inference latency < 5s for 500 tokens
• [ ] Service scales based on demand
• [ ] Health check endpoint responding
• [ ] Monitoring alerts configured
• [ ] Cost tracking enabled

Cost Estimation

Scenario	Monthly Cost
Scale to zero (occasional use)	~$50
1 instance always on	~$550
2 instances always on	~$1,100
Burst to 5 instances (8hr/day)	~$900

Use GCP Pricing Calculator for accurate estimates.