claude-gen/.claude/plans/distributed-coalescing-breeze.md

# Plan: Update Fission Python Template Based on Example Projects

## Context

The current Fission Python template (`fission-python/template/`) is essentially a copy of the `py-eom-quota` example project, making it **quota-specific** rather than a **generic starting point** for new Fission Python projects.

Three example projects were analyzed:
- `py-eom-quota` - User quota management API
- `py-eom-storage` - Storage resource management with S3 integration
- `py-ailbl-scheduler` - Background job scheduler with Dagster integration

All examples share common infrastructure patterns but differ in business logic. This plan will make the template **generic, reusable, and production-ready** by extracting shared best practices.

---

## Key Findings from Examples

### 1. Common Infrastructure (All Projects Share)

- **vault.py** - Identical across all three projects (encryption/decryption using PyNaCl)
- **helpers.py** - Nearly identical core utilities:
  - `get_secret()` / `get_config()` (K8s secrets/configmaps with vault support)
  - `init_db_connection()` (PostgreSQL connection)
  - `db_row_to_dict()` / `db_rows_to_array()`
  - `get_user_from_headers()` (extract user for audit logging)
  - `format_error_response()` (standardized error format)
  - `check_port_open()` (DB readiness check)
  - `str_to_bool()` utility
- **Fission Configuration** - Using docstring format in `main()` functions
- **Exception Patterns** - Custom exception hierarchies with:
  - `error_code` (machine-readable)
  - `http_status` (HTTP status)
  - `error_msg` (human-readable)
  - `x_user` (optional user tracking)
  - `details` (optional additional context)
- **Pydantic Models** - Request validation, response schemas, pagination/filtering
- **Project Structure** - Consistent layout:
  ```
  project/
  ├── .fission/deployment.json
  ├── src/
  │   ├── __init__.py
  │   ├── exceptions.py
  │   ├── helpers.py
  │   ├── models.py
  │   ├── vault.py
  │   ├── build.sh
  │   └── <business logic>.py
  ├── test/
  ├── migrates/
  ├── manifests/
  ├── specs/
  ├── requirements.txt
  ├── dev-requirements.txt
  └── README.md
  ```

### 2. Variations Between Projects

**Database Connection:**
- `py-eom-quota`: Advanced `DBConfig` dataclass with `from_remote_config()` support
- `py-eom-storage` & `py-ailbl-scheduler`: Simplified direct connection from secrets

**Additional Dependencies:**
- Storage: `boto3` (S3/MinIO), `botocore`
- Scheduler: `gql` (GraphQL), `cron-descriptor`
- All: `pydantic`, `psycopg2-binary`, `PyNaCl`, `Flask`, `requests`

**Executors:**
- Quota: `poolmgr` (concurrency=1)
- Storage: `poolmgr` (concurrency=3, maxscale=3)
- Scheduler: `newdeploy` (minscale=1, maxscale=1)

### 3. Issues to Fix

- **README outdated** - References `pymake`, `fission.json`, `fission.yaml` (not used)
- **Missing Flask** - `src/requirements.txt` needs Flask (currently only in dev-requirements)
- **Quota-specific code** - Template should be generic (no `QuotaException`, `QuotaResponse`, etc.)
- **No .env.example** - Missing environment variable template
- **Test dependencies minimal** - Should include `pytest`, `pytest-mock`, `requests`, `flake8`, `black`
- **build.sh** - Should handle both alpine (apk) and debian (apt) properly
- **deployment.json** - Should not hardcode `fission-eom-quota-env` secret names
- **Missing Python version** - Should specify Python 3.11+ (scheduler uses 3.11-alpine)

---

## Recommended Changes

### Phase 1: Core Infrastructure (Keep Generic)

**Files to MODIFY:**

1. **`src/vault.py`** - Keep as-is (already perfect, identical in all examples)

2. **`src/helpers.py`** - Use the simplified pattern from `py-eom-storage` but add:
   - Keep: `get_secret()`, `get_config()`, `init_db_connection()`, `db_row_to_dict()`, `db_rows_to_array()`, `get_current_namespace()`, `str_to_bool()`, `check_port_open()`, `get_user_from_headers()`, `format_error_response()`
   - Remove: `DBConfig` class (too specific to quota, keep it simple)
   - Add: `.strip()` when reading files (as in scheduler)
   - Keep CORS_HEADERS and constants but make them configurable

3. **`src/exceptions.py`** - Replace quota-specific with generic patterns:
   ```python
   class ServiceException(Exception):
       """Base exception for service errors."""
       def __init__(self, error_code, http_status, error_msg, x_user=None, details=None):
           ...

   class ValidationError(ServiceException):   # 400
   class NotFoundError(ServiceException):    # 404
   class ConflictError(ServiceException):    # 409
   class DatabaseError(ServiceException):   # 500
   ```
   (Based on `py-eom-storage` pattern - cleaner and more generic)

4. **`src/models.py`** - Replace with generic example patterns:
   - Remove: All quota-specific models
   - Add: Generic `ItemResponse`, `PaginatedResponse`, `ErrorResponse`
   - Include examples of Pydantic models with Field descriptions and json_schema_extra
   - Show patterns for: Enums, nested models, dataclasses for filters

5. **`src/requirements.txt`** - Update to include actual runtime deps:
   ```
   Flask==2.1.1
   pydantic==2.11.7
   psycopg2-binary==2.9.10
   PyNaCl==1.6.0
   requests==2.32.2
   ```
   (Remove commented examples - these go in docs, not requirements.txt)

6. **`dev-requirements.txt`** - Expand with useful dev tools:
   ```
   Flask==2.1.1
   requests==2.32.2
   pytest==8.2.0
   pytest-mock==3.14.0
   flake8==7.0.0
   black==24.1.1
   mypy==1.8.0
   ```

7. **`README.md`** - Complete rewrite:
   - Remove references to pymake, fission.json
   - Explain actual project structure
   - Document Fission configuration in docstrings
   - Show how to use deployment.json
   - Document environment variables (secrets/configmaps)
   - Explain testing approach
   - Add development workflow
   - Include examples from all three projects as inspiration

8. **`.fission/deployment.json`** - Make generic with placeholders:
   - Use `your-service-py` as environment name
   - Use `your-package` as package name
   - Use generic secret/configmap names: `fission-${PROJECT_NAME}-env`, `fission-${PROJECT_NAME}-config`
   - Show both `poolmgr` and `newdeploy` executor examples (commented)
   - Include optional fields like `imagepullsecret`, `runtime_envs`, `configmaps`

9. **`test/requirements.txt`** - Add:
   ```
   pytest==8.2.0
   pytest-mock==3.14.0
   requests==2.32.3
   ```

10. **`build.sh`** - Fix to use `${SRC_PKG}` properly (current version is correct)

### Phase 2: Documentation & Examples

**New Files to ADD:**

1. **`src/__init__.py`** - Already exists, keep as is

2. **`examples/` directory** (new) - Sample function implementations:
   - `example_crud.py` - Basic CRUD with Pydantic validation
   - `example_webhook.py` - Webhook receiver pattern
   - `example_scheduler.py` - Background job pattern (from ailbl-scheduler)
   - Each should have proper Fission docstring config

3. **`.env.example`** - Template showing all environment variables:
   ```
   # PostgreSQL
   PG_HOST=
   PG_PORT=5432
   PG_DB=
   PG_USER=
   PG_PASS=
   PG_DBSCHEMA=

   # Optional: Service-specific config (via ConfigMap)
   # YOUR_SERVICE_CONFIG_ENDPOINT=

   # Optional: Vault encryption key (if using encrypted secrets)
   # CRYPTO_KEY=
   ```

4. **`docs/` directory** (new) - Additional documentation:
   - `STRUCTURE.md` - Detailed file structure explanation
   - `TESTING.md` - How to write and run tests
   - `DEPLOYMENT.md` - Deployment options and tuning
   - `SECRETS.md` - Managing secrets and configmaps
   - `MIGRATIONS.md` - Database migration workflow

5. **`pytest.ini`** - Default pytest configuration:
   ```ini
   [pytest]
   testpaths = test
   python_files = test_*.py
   python_classes = Test*
   python_functions = test_*
   log_cli = true
   log_cli_level = INFO
   ```

6. **`.gitignore`** - Ensure it excludes:
   - `__pycache__/`
   - `*.pyc`
   - `.env`
   - `.venv/`
   - `venv/`
   - `.pytest_cache/`
   - `.mypy_cache/`
   - `.coverage`
   - `coverage.xml`
   - `specs/` (optional - generated files)

7. **`MANIFEST.md`** - Template for Kubernetes manifests (if not using auto-generated)

### Phase 3: Modernization

**Update CI/CD:**

Review `.gitea/workflows/` files:
- Ensure they install dependencies correctly
- Add linting (flake8/black) steps
- Add test execution
- Add deployment steps with proper environment detection
- Consider adding security scanning

**Python Version:**

- Ensure all files are compatible with Python 3.11+
- Update `build.sh` to use Python 3.11 image (like scheduler does) or keep generic
- Consider adding `runtime.txt` or `pyproject.toml` to specify Python version

---

## Files to Modify Summary

**Direct modifications:**
- `src/helpers.py` - Simplify, improve
- `src/exceptions.py` - Make generic
- `src/models.py` - Replace with generic patterns
- `src/requirements.txt` - Add Flask, remove commented section
- `dev-requirements.txt` - Comprehensive dev dependencies
- `test/requirements.txt` - Test dependencies
- `README.md` - Complete rewrite
- `.fission/deployment.json` - Generic placeholders
- `build.sh` - Already good, just ensure compatibility

**New files to add:**
- `.env.example`
- `pytest.ini`
- `.gitignore` (enhance)
- `examples/` directory with sample functions
- `docs/` directory with detailed guides
- `src/example_crud.py` (or in examples/)
- `src/example_webhook.py` (or in examples/)
- `src/example_scheduler.py` (or in examples/)

**New directories:**
- `examples/`
- `docs/`

---

## Implementation Approach

1. **Backup current template** (git branch)
2. **Modify core files** in order: helpers → exceptions → models → requirements → deployment.json → README
3. **Add new files** (examples, docs, configs)
4. **Test the template**:
   - Run `create-project.sh` to generate a new project
   - Verify build.sh works
   - Run tests
   - Check Fission spec generation
5. **Commit with clear message**
6. **Update plugin documentation** if needed

---

## Verification Steps

After implementing the changes:

1. **Create a test project** from the updated template:
   ```bash
   ./create-project.sh test-project ./tmp-test/
   ```
2. **Inspect generated project**:
   - Verify all files are present
   - Check that placeholders are substituted correctly
   - Ensure imports work
3. **Build the package**:
   ```bash
   cd tmp-test
   ./src/build.sh
   ```
4. **Run tests** (if any):
   ```bash
   pip install -r dev-requirements.txt
   pytest
   ```
5. **Check syntax**:
   ```bash
   python -m py_compile src/*.py
   flake8 src/
   black --check src/
   ```
6. **Validate Fission config**:
   ```bash
   fission spec verify --file=.fission/deployment.json
   ```
7. **Review README** - Does it accurately describe the project?

---

## Success Criteria

- Template is **generic**, not domain-specific
- All examples' best practices are incorporated
- Documentation is accurate and complete
- Dependencies are correctly listed (Flask in requirements, not just dev)
- README reflects actual Fission workflow (docstrings, not fission.yaml)
- Multiple example implementations provided (CRUD, webhook, scheduler)
- Secrets/configuration clearly explained
- Testing setup is comprehensive
- Project passes linting and type checks

---

## Risks & Mitigations

| Risk | Mitigation |
|------|------------|
| Breaking existing template users | Keep changes minimal in helpers; preserve backward compatibility where possible |
| Over-engineering | Stick to patterns that appear in at least 2 of 3 examples |
| Missing edge cases | Include optional advanced patterns (like DBConfig) in docs, not in core |
| Documentation drift | Keep docs close to code; add examples that mirror real projects |

---

## Post-Implementation

After the template is updated:
1. Consider creating a **template validation script** to ensure quality
2. Update the **plugin SKILL.md** to reflect template changes
3. Add **templating tests** to the fission-python-skill test suite
4. Document the **update process** for future template modifications
5. Consider **versioning** the template (e.g., `template-v2/`)