ref: up

2026-03-18 20:21:56 +07:00
commit 29667cd92f
58 changed files with 8459 additions and 0 deletions
--- a/fission-python/template/docs/DEPLOYMENT.md
+++ b/fission-python/template/docs/DEPLOYMENT.md
@@ -0,0 +1,594 @@
+# Deployment Guide
+
+This guide covers deploying Fission Python functions to Kubernetes, including configuration tuning, troubleshooting, and best practices.
+
+## Table of Contents
+
+1. [Prerequisites](#prerequisites)
+2. [Quick Start](#quick-start)
+3. [Deployment Configuration](#deployment-configuration)
+4. [Executors](#executors)
+5. [Resource Tuning](#resource-tuning)
+6. [Environments](#environments)
+7. [Secrets Management](#secrets-management)
+8. [Rolling Updates](#rolling-updates)
+9. [Monitoring & Logging](#monitoring--logging)
+10. [Troubleshooting](#troubleshooting)
+
+## Prerequisites
+
+- Kubernetes cluster (v1.19+)
+- Fission installed (`kubectl apply -f https://github.com/fission/fission/releases/latest/download/fission-all.yaml`)
+- `fission` CLI installed and configured
+- `kubectl` configured to access cluster
+- Docker registry access (for custom images if needed)
+
+## Quick Start
+
+Assuming you have a project set up:
+
+```bash
+# 1. Build the package (creates specs/ directory)
+cd /path/to/project
+./src/build.sh
+
+# 2. Verify deployment configuration
+fission spec verify --file=.fission/deployment.json
+
+# 3. Deploy to Fission
+fission deploy
+
+# 4. Test deployed function
+curl http://$FISSION_ROUTER/api/items
+```
+
+**That's it!** Fission will:
+- Build package.zip from src/
+- Create environment (if not exists)
+- Create package
+- Create functions from docstring metadata
+- Set up HTTP triggers
+
+## Deployment Configuration
+
+### deployment.json vs fission.yaml
+
+This template uses `deployment.json`, **not** `fission.yaml` or `fission.json`. The Fission Python builder extracts function metadata from Python docstrings directly.
+
+### Key Sections
+
+#### environments
+
+Define build environment:
+
+```json
+{
+  "environments": {
+    "myproject-py": {
+      "image": "ghcr.io/fission/python-env",
+      "builder": "ghcr.io/fission/python-builder",
+      "mincpu": 50,
+      "maxcpu": 100,
+      "minmemory": 50,
+      "maxmemory": 500,
+      "poolsize": 1
+    }
+  }
+}
+```
+
+- `image` - Runtime image (Python + libraries)
+- `builder` - Builder image (compiles dependencies)
+- Resource limits in millicores (50 = 0.05 CPU) and MB
+
+#### packages
+
+Define how to build your code:
+
+```json
+{
+  "packages": {
+    "myproject": {
+      "buildcmd": "./build.sh",
+      "sourcearchive": "package.zip",
+      "env": "myproject-py"
+    }
+  }
+}
+```
+
+- `buildcmd` - Build script inside builder container
+- `sourcearchive` - Generated by builder from `sourcepath`
+- `env` - Links to environment definition
+
+#### function_common
+
+Default configuration for all functions:
+
+```json
+{
+  "function_common": {
+    "pkg": "myproject",
+    "secrets": ["fission-myproject-env"],
+    "configmaps": ["fission-myproject-config"],
+    "executor": { ... },
+    "mincpu": 50,
+    "maxcpu": 100,
+    "minmemory": 50,
+    "maxmemory": 500
+  }
+}
+```
+
+- `pkg` - Package name to use
+- `secrets` / `configmaps` - K8s resources to mount into functions
+- `executor` - Execution strategy (poolmgr or newdeploy)
+
+#### secrets / configmaps
+
+**Placeholder definitions only**. These inform Fission what secret names to expect, but the actual values go in real K8s secrets:
+
+```json
+{
+  "secrets": {
+    "fission-myproject-env": {
+      "literals": [
+        "PG_HOST=localhost",
+        "PG_PORT=5432"
+      ]
+    }
+  }
+}
+```
+
+Create the actual secret:
+
+```bash
+kubectl create secret generic fission-myproject-env \
+  --from-literal=PG_HOST=prod-db.example.com \
+  --from-literal=PG_PORT=5432 \
+  --from-literal=PG_USER=myuser \
+  --from-literal=PG_PASS=mypassword
+```
+
+## Executors
+
+Fission supports two executor types:
+
+### poolmgr (default)
+
+Good for:
+- High-concurrency HTTP functions
+- Functions that should scale to zero
+- Stateless request/response patterns
+
+Configuration:
+
+```json
+"executor": {
+  "select": "poolmgr",
+  "poolmgr": {
+    "concurrency": 1,        // Requests per pod
+    "requestsperpod": 1,
+    "onceonly": false
+  }
+}
+```
+
+- `concurrency` - How many concurrent requests each pod handles (usually 1 for Python due to GIL)
+- `poolsize` from environment controls number of pods in pool
+
+### newdeploy
+
+Good for:
+- Dedicated function instances
+- Long-running or background jobs
+- Functions needing stable network identity
+
+Configuration:
+
+```json
+"executor": {
+  "select": "newdeploy",
+  "newdeploy": {
+    "minscale": 1,       // Minimum pods (set to 0 for scale-to-zero)
+    "maxscale": 5,       // Maximum pods
+    "targetcpu": 80      // Scale up when CPU > 80%
+  }
+}
+```
+
+- `minscale` - Keep at least N pods running (0 = scale to zero)
+- `maxscale` - Maximum pods for auto-scaling
+- `targetcpu` - CPU threshold for scaling
+
+## Resource Tuning
+
+Resources are defined in millicores (m) and MB:
+
+- `mincpu` / `maxcpu`: 1000 = 1 CPU core
+- `minmemory` / `maxmemory`: in MB
+
+**Example settings**:
+
+| Function Type | mincpu | maxcpu | minmemory | maxmemory |
+|--------------|--------|--------|-----------|-----------|
+| Simple API | 50 | 100 | 128 | 256 |
+| DB-intensive | 200 | 500 | 256 | 512 |
+| ML inference | 1000 | 2000 | 1024 | 2048 |
+
+**Tips**:
+- Start conservatively, monitor, then adjust
+- Function pods are killed if they exceed `maxmemory`
+- CPU limits are enforced by Kubernetes scheduler
+- Use `minmemory` >= 128 to avoid OOM kills
+
+### Checking Current Usage
+
+```bash
+# Get function pods
+kubectl get pods -n fission
+
+# Describe pod for resource usage
+kubectl describe pod <pod-name> -n fission
+
+# See metrics (if metrics-server installed)
+kubectl top pod <pod-name> -n fission
+```
+
+## Environments
+
+You can have multiple deployment environments (dev, staging, prod):
+
+### Using deployment.json variants
+
+- `deployment.json` - Production (default)
+- `dev-deployment.json` - Development (used with `fission deploy --dev`)
+
+Example `dev-deployment.json`:
+
+```json
+{
+  "namespace": "fission-dev",
+  "function_common": {
+    "secrets": ["fission-myproject-dev-env"],
+    "configmaps": ["fission-myproject-dev-config"]
+  }
+}
+```
+
+### Switching Environments
+
+```bash
+# Deploy to dev
+fission deploy --dev
+
+# Deploy to prod (default)
+fission deploy
+
+# Specify namespace
+fission deploy --namespace fission-staging
+```
+
+## Secrets Management
+
+### Creating Secrets
+
+```bash
+# Basic secret from literals
+kubectl create secret generic fission-myproject-env \
+  --from-literal=PG_HOST=localhost \
+  --from-literal=PG_PORT=5432
+
+# From file
+kubectl create secret generic fission-myproject-env \
+  --from-file=secrets.properties
+
+# With multiple namespaces
+kubectl create secret generic fission-myproject-env \
+  --namespace fission-dev \
+  --from-literal=PG_HOST=dev-db.example.com
+```
+
+### Encrypted Secrets (Vault)
+
+To encrypt sensitive values:
+
+```python
+# On your local machine (with PyNaCl installed)
+from vault import encrypt_vault
+
+key = "your-32-byte-hex-key-here..."  # 64 hex chars
+encrypted = encrypt_vault("super-secret-password", key)
+print(encrypted)  # vault:v1:base64...
+```
+
+Store the encrypted string in K8s secret:
+
+```bash
+kubectl create secret generic fission-myproject-env \
+  --from-literal=PG_PASS='vault:v1:base64...'
+```
+
+Set `CRYPTO_KEY` in `helpers.py` to the hex key:
+
+```python
+CRYPTO_KEY = "e24ad6ceed96115520f6e6dc8a0da506ae9a706823d54f30a5b75447ecf477b6"
+```
+
+**Important**: Rotate keys periodically. When changing key, re-encrypt all secrets.
+
+### Updating Secrets
+
+```bash
+# Edit secret
+kubectl edit secret fission-myproject-env
+
+# Update single key
+kubectl set secret secret fission-myproject-env \
+  --from-literal=PG_PASS='new-password'
+
+# Roll function to pick up new secret
+fission function update --name my-function
+```
+
+## Rolling Updates
+
+### Deploy Changes
+
+```bash
+# Build and deploy
+./src/build.sh
+fission deploy
+
+# Or deploy single function
+fission function update --name my-function
+```
+
+### Zero-Downtime Deployments
+
+Fission handles rolling updates automatically:
+1. New package is built
+2. New function pods are created with new code
+3. Old pods continue serving traffic until new pods are ready
+4. Old pods are terminated
+
+**No downtime** by default for HTTP triggers.
+
+### Canary Deployments
+
+For canary deployments:
+1. Deploy new version with different function name: `my-function-v2`
+2. Route some traffic using ingress annotations or service mesh
+3. Gradually shift traffic
+4. Delete old function
+
+## Monitoring & Logging
+
+### Viewing Logs
+
+```bash
+# All function logs in namespace
+kubectl logs -n fission -l fission-function=true --tail=100
+
+# Specific function
+kubectl logs -n fission -l fission-function/name=my-function --tail=100
+
+# Follow logs
+kubectl logs -n fission -l fission-function/name=my-function -f
+
+# Container logs (if multiple containers)
+kubectl logs -n fission -l fission-function/name=my-function -c builder
+```
+
+### Structured Logging
+
+Use `logger` from `helpers.py` (already configured):
+
+```python
+logger.info("Processing request", extra={"user_id": user_id})
+logger.error("Database error", exc_info=True, extra={"query": sql})
+```
+
+Logs are collected by the container runtime and available via `kubectl logs`.
+
+### Metrics
+
+Fission exposes Prometheus metrics:
+
+```bash
+# Get metrics endpoint
+kubectl port-forward -n fission svc/fission-prometheus-server 9090:9090
+
+# Or query via kubectl
+kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/fission/pods/*" | jq .
+```
+
+Metrics include:
+- Request rate
+- Error rate
+- Response latency
+- Pod counts
+
+## Troubleshooting
+
+### Deployment Fails
+
+**Error**: `Error building package`
+
+Check:
+- `build.sh` is executable: `chmod +x src/build.sh`
+- All dependencies in `requirements.txt` are valid
+- Python syntax is correct: `python -m py_compile src/*.py`
+
+**Error**: `Function not found after deploy`
+
+Check:
+- Fission docstring block is properly formatted (must be ````fission` with backticks)
+- No YAML/JSON syntax errors in docstring
+- Function file is in `src/` directory
+
+### Function Not Responding
+
+**Check pod status**:
+```bash
+kubectl get pods -n fission -l fission-function/name=my-function
+```
+
+**Pod stuck in Pending** - Insufficient resources or image pull error
+
+**Pod stuck in ContainerCreating** - Volume mount issue or image pull
+
+**Pod CrashLoopBackOff** - Application error. Check logs:
+```bash
+kubectl logs -n fission <pod-name> --previous
+```
+
+### Configuration Not Loading
+
+**Secrets not available**:
+```bash
+# Check secret exists in correct namespace
+kubectl get secret fission-myproject-env -n fission
+
+# Verify secret is mounted
+kubectl exec -it <pod-name> -n fission -- ls /secrets/default/
+```
+
+**ConfigMaps not available**:
+```bash
+kubectl get configmap fission-myproject-config -n fission
+```
+
+**Profusion parms not reading**:
+- Ensure `SECRET_NAME` in helpers.py matches created secret name
+- Path format: `/secrets/{namespace}/{secret-name}/{key}`
+
+### Slow Performance
+
+1. **Increase resources**: Raise `maxmemory` and `maxcpu`
+2. **Connection pooling**: Use connection pooler like PgBouncer for heavy DB load
+3. **Database queries**: Check slow queries, add indexes
+4. **Cold starts**: Set `minscale: 1` with newdeploy executor to keep warm
+
+### Database Connection Errors
+
+**Error**: `could not connect to server: Connection refused`
+
+- Verify database is reachable from cluster
+- Check security groups/network policies
+- Test connectivity from pod:
+  ```bash
+  kubectl exec -it <pod-name> -n fission -- nc -zv $PG_HOST $PG_PORT
+  ```
+
+**Error**: `password authentication failed`
+
+- Verify credentials in secret
+- Check PG_USER format (with `plaintext:` prefix for vault)
+
+## Advanced Topics
+
+### Custom Runtime Image
+
+If you need system packages:
+
+```dockerfile
+FROM ghcr.io/fission/python-env:latest
+RUN apk add --no-cache gcc libffi-dev
+```
+
+Build and push:
+```bash
+docker build -t myregistry/python-custom:latest .
+docker push myregistry/python-custom:latest
+```
+
+Update `deployment.json`:
+```json
+"environments": {
+  "myproject-py": {
+    "image": "myregistry/python-custom:latest",
+    ...
+  }
+}
+```
+
+### Environment Variables from ConfigMap
+
+```json
+"configmaps": {
+  "fission-myproject-config": {
+    "literals": [
+      "LOG_LEVEL=DEBUG",
+      "FEATURE_FLAG_X=true"
+    ]
+  }
+}
+```
+
+Access in code:
+```python
+import os
+log_level = os.getenv("LOG_LEVEL", "INFO")
+```
+
+### Lifecycle Hooks
+
+Use `function_pre_remove` and `function_post_remove` in deployment hooks:
+
+```json
+"hooks": {
+  "function_pre_remove": [
+    {
+      "type": "http",
+      "url": "http://cleanup-service/cleanup",
+      "timeout": 30000
+    }
+  ]
+}
+```
+
+## Common Commands Reference
+
+```bash
+# List functions
+fission function list
+
+# Test function manually
+fission function test --name my-function
+
+# Update single function
+fission function update --name my-function
+
+# Delete function
+fission function delete --name my-function
+
+# View function pods
+kubectl get pods -n fission -l fission-function/name=my-function
+
+# View logs
+kubectl logs -n fission -l fission-function/name=my-function -f
+
+# Exec into pod
+kubectl exec -it <pod-name> -n fission -- /bin/sh
+
+# Describe function
+fission function describe --name my-function
+
+# Get function YAML
+fission function get --name my-function -o yaml
+
+# Check Fission version
+fission version
+
+# Check Fission status
+kubectl get pods -n fission
+```
+
+## Further Reading
+
+- [Fission Deployment Documentation](https://fission.io/docs/usage/deploy/)
+- [Fission Executors](https://fission.io/docs/architecture/executor/)
+- [Kubernetes Resource Management](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
+- [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/)