claude-gen/fission-python/template/docs/DEPLOYMENT.md

# Deployment Guide

This guide covers deploying Fission Python functions to Kubernetes, including configuration tuning, troubleshooting, and best practices.

## Table of Contents

1. [Prerequisites](#prerequisites)
2. [Quick Start](#quick-start)
3. [Deployment Configuration](#deployment-configuration)
4. [Executors](#executors)
5. [Resource Tuning](#resource-tuning)
6. [Environments](#environments)
7. [Secrets Management](#secrets-management)
8. [Rolling Updates](#rolling-updates)
9. [Monitoring & Logging](#monitoring--logging)
10. [Troubleshooting](#troubleshooting)

## Prerequisites

- Kubernetes cluster (v1.19+)
- Fission installed (`kubectl apply -f https://github.com/fission/fission/releases/latest/download/fission-all.yaml`)
- `fission` CLI installed and configured
- `kubectl` configured to access cluster
- Docker registry access (for custom images if needed)

## Quick Start

Assuming you have a project set up:

```bash
# 1. Build the package (creates specs/ directory)
cd /path/to/project
./src/build.sh

# 2. Verify deployment configuration
fission spec verify --file=.fission/deployment.json

# 3. Deploy to Fission
fission deploy

# 4. Test deployed function
curl http://$FISSION_ROUTER/api/items
```

**That's it!** Fission will:
- Build package.zip from src/
- Create environment (if not exists)
- Create package
- Create functions from docstring metadata
- Set up HTTP triggers

## Deployment Configuration

### deployment.json vs fission.yaml

This template uses `deployment.json`, **not** `fission.yaml` or `fission.json`. The Fission Python builder extracts function metadata from Python docstrings directly.

### Key Sections

#### environments

Define build environment:

```json
{
  "environments": {
    "myproject-py": {
      "image": "ghcr.io/fission/python-env",
      "builder": "ghcr.io/fission/python-builder",
      "mincpu": 50,
      "maxcpu": 100,
      "minmemory": 50,
      "maxmemory": 500,
      "poolsize": 1
    }
  }
}
```

- `image` - Runtime image (Python + libraries)
- `builder` - Builder image (compiles dependencies)
- Resource limits in millicores (50 = 0.05 CPU) and MB

#### packages

Define how to build your code:

```json
{
  "packages": {
    "myproject": {
      "buildcmd": "./build.sh",
      "sourcearchive": "package.zip",
      "env": "myproject-py"
    }
  }
}
```

- `buildcmd` - Build script inside builder container
- `sourcearchive` - Generated by builder from `sourcepath`
- `env` - Links to environment definition

#### function_common

Default configuration for all functions:

```json
{
  "function_common": {
    "pkg": "myproject",
    "secrets": ["fission-myproject-env"],
    "configmaps": ["fission-myproject-config"],
    "executor": { ... },
    "mincpu": 50,
    "maxcpu": 100,
    "minmemory": 50,
    "maxmemory": 500
  }
}
```

- `pkg` - Package name to use
- `secrets` / `configmaps` - K8s resources to mount into functions
- `executor` - Execution strategy (poolmgr or newdeploy)

#### secrets / configmaps

**Placeholder definitions only**. These inform Fission what secret names to expect, but the actual values go in real K8s secrets:

```json
{
  "secrets": {
    "fission-myproject-env": {
      "literals": [
        "PG_HOST=localhost",
        "PG_PORT=5432"
      ]
    }
  }
}
```

Create the actual secret:

```bash
kubectl create secret generic fission-myproject-env \
  --from-literal=PG_HOST=prod-db.example.com \
  --from-literal=PG_PORT=5432 \
  --from-literal=PG_USER=myuser \
  --from-literal=PG_PASS=mypassword
```

## Executors

Fission supports two executor types:

### poolmgr (default)

Good for:
- High-concurrency HTTP functions
- Functions that should scale to zero
- Stateless request/response patterns

Configuration:

```json
"executor": {
  "select": "poolmgr",
  "poolmgr": {
    "concurrency": 1,        // Requests per pod
    "requestsperpod": 1,
    "onceonly": false
  }
}
```

- `concurrency` - How many concurrent requests each pod handles (usually 1 for Python due to GIL)
- `poolsize` from environment controls number of pods in pool

### newdeploy

Good for:
- Dedicated function instances
- Long-running or background jobs
- Functions needing stable network identity

Configuration:

```json
"executor": {
  "select": "newdeploy",
  "newdeploy": {
    "minscale": 1,       // Minimum pods (set to 0 for scale-to-zero)
    "maxscale": 5,       // Maximum pods
    "targetcpu": 80      // Scale up when CPU > 80%
  }
}
```

- `minscale` - Keep at least N pods running (0 = scale to zero)
- `maxscale` - Maximum pods for auto-scaling
- `targetcpu` - CPU threshold for scaling

## Resource Tuning

Resources are defined in millicores (m) and MB:

- `mincpu` / `maxcpu`: 1000 = 1 CPU core
- `minmemory` / `maxmemory`: in MB

**Example settings**:

| Function Type | mincpu | maxcpu | minmemory | maxmemory |
|--------------|--------|--------|-----------|-----------|
| Simple API | 50 | 100 | 128 | 256 |
| DB-intensive | 200 | 500 | 256 | 512 |
| ML inference | 1000 | 2000 | 1024 | 2048 |

**Tips**:
- Start conservatively, monitor, then adjust
- Function pods are killed if they exceed `maxmemory`
- CPU limits are enforced by Kubernetes scheduler
- Use `minmemory` >= 128 to avoid OOM kills

### Checking Current Usage

```bash
# Get function pods
kubectl get pods -n fission

# Describe pod for resource usage
kubectl describe pod <pod-name> -n fission

# See metrics (if metrics-server installed)
kubectl top pod <pod-name> -n fission
```

## Environments

You can have multiple deployment environments (dev, staging, prod):

### Using deployment.json variants

- `deployment.json` - Production (default)
- `dev-deployment.json` - Development (used with `fission deploy --dev`)

Example `dev-deployment.json`:

```json
{
  "namespace": "fission-dev",
  "function_common": {
    "secrets": ["fission-myproject-dev-env"],
    "configmaps": ["fission-myproject-dev-config"]
  }
}
```

### Switching Environments

```bash
# Deploy to dev
fission deploy --dev

# Deploy to prod (default)
fission deploy

# Specify namespace
fission deploy --namespace fission-staging
```

## Secrets Management

### Creating Secrets

```bash
# Basic secret from literals
kubectl create secret generic fission-myproject-env \
  --from-literal=PG_HOST=localhost \
  --from-literal=PG_PORT=5432

# From file
kubectl create secret generic fission-myproject-env \
  --from-file=secrets.properties

# With multiple namespaces
kubectl create secret generic fission-myproject-env \
  --namespace fission-dev \
  --from-literal=PG_HOST=dev-db.example.com
```

### Encrypted Secrets (Vault)

To encrypt sensitive values:

```python
# On your local machine (with PyNaCl installed)
from vault import encrypt_vault

key = "your-32-byte-hex-key-here..."  # 64 hex chars
encrypted = encrypt_vault("super-secret-password", key)
print(encrypted)  # vault:v1:base64...
```

Store the encrypted string in K8s secret:

```bash
kubectl create secret generic fission-myproject-env \
  --from-literal=PG_PASS='vault:v1:base64...'
```

Set `CRYPTO_KEY` in `helpers.py` to the hex key:

```python
CRYPTO_KEY = "e24ad6ceed96115520f6e6dc8a0da506ae9a706823d54f30a5b75447ecf477b6"
```

**Important**: Rotate keys periodically. When changing key, re-encrypt all secrets.

### Updating Secrets

```bash
# Edit secret
kubectl edit secret fission-myproject-env

# Update single key
kubectl set secret secret fission-myproject-env \
  --from-literal=PG_PASS='new-password'

# Roll function to pick up new secret
fission function update --name my-function
```

## Rolling Updates

### Deploy Changes

```bash
# Build and deploy
./src/build.sh
fission deploy

# Or deploy single function
fission function update --name my-function
```

### Zero-Downtime Deployments

Fission handles rolling updates automatically:
1. New package is built
2. New function pods are created with new code
3. Old pods continue serving traffic until new pods are ready
4. Old pods are terminated

**No downtime** by default for HTTP triggers.

### Canary Deployments

For canary deployments:
1. Deploy new version with different function name: `my-function-v2`
2. Route some traffic using ingress annotations or service mesh
3. Gradually shift traffic
4. Delete old function

## Monitoring & Logging

### Viewing Logs

```bash
# All function logs in namespace
kubectl logs -n fission -l fission-function=true --tail=100

# Specific function
kubectl logs -n fission -l fission-function/name=my-function --tail=100

# Follow logs
kubectl logs -n fission -l fission-function/name=my-function -f

# Container logs (if multiple containers)
kubectl logs -n fission -l fission-function/name=my-function -c builder
```

### Structured Logging

Use `logger` from `helpers.py` (already configured):

```python
logger.info("Processing request", extra={"user_id": user_id})
logger.error("Database error", exc_info=True, extra={"query": sql})
```

Logs are collected by the container runtime and available via `kubectl logs`.

### Metrics

Fission exposes Prometheus metrics:

```bash
# Get metrics endpoint
kubectl port-forward -n fission svc/fission-prometheus-server 9090:9090

# Or query via kubectl
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/fission/pods/*" | jq .
```

Metrics include:
- Request rate
- Error rate
- Response latency
- Pod counts

## Troubleshooting

### Deployment Fails

**Error**: `Error building package`

Check:
- `build.sh` is executable: `chmod +x src/build.sh`
- All dependencies in `requirements.txt` are valid
- Python syntax is correct: `python -m py_compile src/*.py`

**Error**: `Function not found after deploy`

Check:
- Fission docstring block is properly formatted (must be ````fission` with backticks)
- No YAML/JSON syntax errors in docstring
- Function file is in `src/` directory

### Function Not Responding

**Check pod status**:
```bash
kubectl get pods -n fission -l fission-function/name=my-function
```

**Pod stuck in Pending** - Insufficient resources or image pull error

**Pod stuck in ContainerCreating** - Volume mount issue or image pull

**Pod CrashLoopBackOff** - Application error. Check logs:
```bash
kubectl logs -n fission <pod-name> --previous
```

### Configuration Not Loading

**Secrets not available**:
```bash
# Check secret exists in correct namespace
kubectl get secret fission-myproject-env -n fission

# Verify secret is mounted
kubectl exec -it <pod-name> -n fission -- ls /secrets/default/
```

**ConfigMaps not available**:
```bash
kubectl get configmap fission-myproject-config -n fission
```

**Profusion parms not reading**:
- Ensure `SECRET_NAME` in helpers.py matches created secret name
- Path format: `/secrets/{namespace}/{secret-name}/{key}`

### Slow Performance

1. **Increase resources**: Raise `maxmemory` and `maxcpu`
2. **Connection pooling**: Use connection pooler like PgBouncer for heavy DB load
3. **Database queries**: Check slow queries, add indexes
4. **Cold starts**: Set `minscale: 1` with newdeploy executor to keep warm

### Database Connection Errors

**Error**: `could not connect to server: Connection refused`

- Verify database is reachable from cluster
- Check security groups/network policies
- Test connectivity from pod:
  ```bash
  kubectl exec -it <pod-name> -n fission -- nc -zv $PG_HOST $PG_PORT
  ```

**Error**: `password authentication failed`

- Verify credentials in secret
- Check PG_USER format (with `plaintext:` prefix for vault)

## Advanced Topics

### Custom Runtime Image

If you need system packages:

```dockerfile
FROM ghcr.io/fission/python-env:latest
RUN apk add --no-cache gcc libffi-dev
```

Build and push:
```bash
docker build -t myregistry/python-custom:latest .
docker push myregistry/python-custom:latest
```

Update `deployment.json`:
```json
"environments": {
  "myproject-py": {
    "image": "myregistry/python-custom:latest",
    ...
  }
}
```

### Environment Variables from ConfigMap

```json
"configmaps": {
  "fission-myproject-config": {
    "literals": [
      "LOG_LEVEL=DEBUG",
      "FEATURE_FLAG_X=true"
    ]
  }
}
```

Access in code:
```python
import os
log_level = os.getenv("LOG_LEVEL", "INFO")
```

### Lifecycle Hooks

Use `function_pre_remove` and `function_post_remove` in deployment hooks:

```json
"hooks": {
  "function_pre_remove": [
    {
      "type": "http",
      "url": "http://cleanup-service/cleanup",
      "timeout": 30000
    }
  ]
}
```

## Common Commands Reference

```bash
# List functions
fission function list

# Test function manually
fission function test --name my-function

# Update single function
fission function update --name my-function

# Delete function
fission function delete --name my-function

# View function pods
kubectl get pods -n fission -l fission-function/name=my-function

# View logs
kubectl logs -n fission -l fission-function/name=my-function -f

# Exec into pod
kubectl exec -it <pod-name> -n fission -- /bin/sh

# Describe function
fission function describe --name my-function

# Get function YAML
fission function get --name my-function -o yaml

# Check Fission version
fission version

# Check Fission status
kubectl get pods -n fission
```

## Further Reading

- [Fission Deployment Documentation](https://fission.io/docs/usage/deploy/)
- [Fission Executors](https://fission.io/docs/architecture/executor/)
- [Kubernetes Resource Management](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
- [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/)