claude-gen/.claude/plans/distributed-coalescing-breeze.md

Plan: Update Fission Python Template Based on Example Projects

Context

The current Fission Python template (fission-python/template/) is essentially a copy of the py-eom-quota example project, making it quota-specific rather than a generic starting point for new Fission Python projects.

Three example projects were analyzed:

• py-eom-quota - User quota management API
• py-eom-storage - Storage resource management with S3 integration
• py-ailbl-scheduler - Background job scheduler with Dagster integration

All examples share common infrastructure patterns but differ in business logic. This plan will make the template generic, reusable, and production-ready by extracting shared best practices.

---

Key Findings from Examples

1. Common Infrastructure (All Projects Share)

• vault.py - Identical across all three projects (encryption/decryption using PyNaCl)
• helpers.py - Nearly identical core utilities:
  • get_secret() / get_config() (K8s secrets/configmaps with vault support)
  • init_db_connection() (PostgreSQL connection)
  • db_row_to_dict() / db_rows_to_array()
  • get_user_from_headers() (extract user for audit logging)
  • format_error_response() (standardized error format)
  • check_port_open() (DB readiness check)
  • str_to_bool() utility
• Fission Configuration - Using docstring format in main() functions
• Exception Patterns - Custom exception hierarchies with:
  • error_code (machine-readable)
  • http_status (HTTP status)
  • error_msg (human-readable)
  • x_user (optional user tracking)
  • details (optional additional context)
• Pydantic Models - Request validation, response schemas, pagination/filtering
• Project Structure - Consistent layout:

  project/
  ├── .fission/deployment.json
  ├── src/
  │   ├── __init__.py
  │   ├── exceptions.py
  │   ├── helpers.py
  │   ├── models.py
  │   ├── vault.py
  │   ├── build.sh
  │   └── <business logic>.py
  ├── test/
  ├── migrates/
  ├── manifests/
  ├── specs/
  ├── requirements.txt
  ├── dev-requirements.txt
  └── README.md
  

2. Variations Between Projects

Database Connection:

• py-eom-quota: Advanced DBConfig dataclass with from_remote_config() support
• py-eom-storage & py-ailbl-scheduler: Simplified direct connection from secrets

Additional Dependencies:

• Storage: boto3 (S3/MinIO), botocore
• Scheduler: gql (GraphQL), cron-descriptor
• All: pydantic, psycopg2-binary, PyNaCl, Flask, requests

Executors:

• Quota: poolmgr (concurrency=1)
• Storage: poolmgr (concurrency=3, maxscale=3)
• Scheduler: newdeploy (minscale=1, maxscale=1)

3. Issues to Fix

• README outdated - References pymake, fission.json, fission.yaml (not used)
• Missing Flask - src/requirements.txt needs Flask (currently only in dev-requirements)
• Quota-specific code - Template should be generic (no QuotaException, QuotaResponse, etc.)
• No .env.example - Missing environment variable template
• Test dependencies minimal - Should include pytest, pytest-mock, requests, flake8, black
• build.sh - Should handle both alpine (apk) and debian (apt) properly
• deployment.json - Should not hardcode fission-eom-quota-env secret names
• Missing Python version - Should specify Python 3.11+ (scheduler uses 3.11-alpine)

---

Recommended Changes

Phase 1: Core Infrastructure (Keep Generic)

Files to MODIFY:

1. src/vault.py - Keep as-is (already perfect, identical in all examples)

2. src/helpers.py - Use the simplified pattern from py-eom-storage but add:

   • Keep: get_secret(), get_config(), init_db_connection(), db_row_to_dict(), db_rows_to_array(), get_current_namespace(), str_to_bool(), check_port_open(), get_user_from_headers(), format_error_response()
   • Remove: DBConfig class (too specific to quota, keep it simple)
   • Add: .strip() when reading files (as in scheduler)
   • Keep CORS_HEADERS and constants but make them configurable

3. src/exceptions.py - Replace quota-specific with generic patterns:

   class ServiceException(Exception):
       """Base exception for service errors."""
       def __init__(self, error_code, http_status, error_msg, x_user=None, details=None):
           ...
   
   class ValidationError(ServiceException):   # 400
   class NotFoundError(ServiceException):    # 404
   class ConflictError(ServiceException):    # 409
   class DatabaseError(ServiceException):   # 500
   

   (Based on py-eom-storage pattern - cleaner and more generic)

4. src/models.py - Replace with generic example patterns:

   • Remove: All quota-specific models
   • Add: Generic ItemResponse, PaginatedResponse, ErrorResponse
   • Include examples of Pydantic models with Field descriptions and json_schema_extra
   • Show patterns for: Enums, nested models, dataclasses for filters

5. src/requirements.txt - Update to include actual runtime deps:

   Flask==2.1.1
   pydantic==2.11.7
   psycopg2-binary==2.9.10
   PyNaCl==1.6.0
   requests==2.32.2
   

   (Remove commented examples - these go in docs, not requirements.txt)

6. dev-requirements.txt - Expand with useful dev tools:

   Flask==2.1.1
   requests==2.32.2
   pytest==8.2.0
   pytest-mock==3.14.0
   flake8==7.0.0
   black==24.1.1
   mypy==1.8.0
   

7. README.md - Complete rewrite:

   • Remove references to pymake, fission.json
   • Explain actual project structure
   • Document Fission configuration in docstrings
   • Show how to use deployment.json
   • Document environment variables (secrets/configmaps)
   • Explain testing approach
   • Add development workflow
   • Include examples from all three projects as inspiration

8. .fission/deployment.json - Make generic with placeholders:

   • Use your-service-py as environment name
   • Use your-package as package name
   • Use generic secret/configmap names: fission-${PROJECT_NAME}-env, fission-${PROJECT_NAME}-config
   • Show both poolmgr and newdeploy executor examples (commented)
   • Include optional fields like imagepullsecret, runtime_envs, configmaps

9. test/requirements.txt - Add:

   pytest==8.2.0
   pytest-mock==3.14.0
   requests==2.32.3
   

10. build.sh - Fix to use ${SRC_PKG} properly (current version is correct)

Phase 2: Documentation & Examples

New Files to ADD:

1. src/__init__.py - Already exists, keep as is

2. examples/ directory (new) - Sample function implementations:

   • example_crud.py - Basic CRUD with Pydantic validation
   • example_webhook.py - Webhook receiver pattern
   • example_scheduler.py - Background job pattern (from ailbl-scheduler)
   • Each should have proper Fission docstring config

3. .env.example - Template showing all environment variables:

   # PostgreSQL
   PG_HOST=
   PG_PORT=5432
   PG_DB=
   PG_USER=
   PG_PASS=
   PG_DBSCHEMA=
   
   # Optional: Service-specific config (via ConfigMap)
   # YOUR_SERVICE_CONFIG_ENDPOINT=
   
   # Optional: Vault encryption key (if using encrypted secrets)
   # CRYPTO_KEY=
   

4. docs/ directory (new) - Additional documentation:

   • STRUCTURE.md - Detailed file structure explanation
   • TESTING.md - How to write and run tests
   • DEPLOYMENT.md - Deployment options and tuning
   • SECRETS.md - Managing secrets and configmaps
   • MIGRATIONS.md - Database migration workflow

5. pytest.ini - Default pytest configuration:

   [pytest]
   testpaths = test
   python_files = test_*.py
   python_classes = Test*
   python_functions = test_*
   log_cli = true
   log_cli_level = INFO
   

6. .gitignore - Ensure it excludes:

   • __pycache__/
   • *.pyc
   • .env
   • .venv/
   • venv/
   • .pytest_cache/
   • .mypy_cache/
   • .coverage
   • coverage.xml
   • specs/ (optional - generated files)

7. MANIFEST.md - Template for Kubernetes manifests (if not using auto-generated)

Phase 3: Modernization

Update CI/CD:

Review .gitea/workflows/ files:

• Ensure they install dependencies correctly
• Add linting (flake8/black) steps
• Add test execution
• Add deployment steps with proper environment detection
• Consider adding security scanning

Python Version:

• Ensure all files are compatible with Python 3.11+
• Update build.sh to use Python 3.11 image (like scheduler does) or keep generic
• Consider adding runtime.txt or pyproject.toml to specify Python version

---

Files to Modify Summary

Direct modifications:

• src/helpers.py - Simplify, improve
• src/exceptions.py - Make generic
• src/models.py - Replace with generic patterns
• src/requirements.txt - Add Flask, remove commented section
• dev-requirements.txt - Comprehensive dev dependencies
• test/requirements.txt - Test dependencies
• README.md - Complete rewrite
• .fission/deployment.json - Generic placeholders
• build.sh - Already good, just ensure compatibility

New files to add:

• .env.example
• pytest.ini
• .gitignore (enhance)
• examples/ directory with sample functions
• docs/ directory with detailed guides
• src/example_crud.py (or in examples/)
• src/example_webhook.py (or in examples/)
• src/example_scheduler.py (or in examples/)

New directories:

• examples/
• docs/

---

Implementation Approach

1. Backup current template (git branch)
2. Modify core files in order: helpers → exceptions → models → requirements → deployment.json → README
3. Add new files (examples, docs, configs)
4. Test the template:
   • Run create-project.sh to generate a new project
   • Verify build.sh works
   • Run tests
   • Check Fission spec generation
5. Commit with clear message
6. Update plugin documentation if needed

---

Verification Steps

After implementing the changes:

1. Create a test project from the updated template:

   ./create-project.sh test-project ./tmp-test/
   

2. Inspect generated project:
   • Verify all files are present
   • Check that placeholders are substituted correctly
   • Ensure imports work
3. Build the package:

   cd tmp-test
   ./src/build.sh
   

4. Run tests (if any):

   pip install -r dev-requirements.txt
   pytest
   

5. Check syntax:

   python -m py_compile src/*.py
   flake8 src/
   black --check src/
   

6. Validate Fission config:

   fission spec verify --file=.fission/deployment.json
   

7. Review README - Does it accurately describe the project?

---

Success Criteria

• Template is generic, not domain-specific
• All examples' best practices are incorporated
• Documentation is accurate and complete
• Dependencies are correctly listed (Flask in requirements, not just dev)
• README reflects actual Fission workflow (docstrings, not fission.yaml)
• Multiple example implementations provided (CRUD, webhook, scheduler)
• Secrets/configuration clearly explained
• Testing setup is comprehensive
• Project passes linting and type checks

---

Risks & Mitigations

Risk	Mitigation
Breaking existing template users	Keep changes minimal in helpers; preserve backward compatibility where possible
Over-engineering	Stick to patterns that appear in at least 2 of 3 examples
Missing edge cases	Include optional advanced patterns (like DBConfig) in docs, not in core
Documentation drift	Keep docs close to code; add examples that mirror real projects

---

Post-Implementation

After the template is updated:

1. Consider creating a template validation script to ensure quality
2. Update the plugin SKILL.md to reflect template changes
3. Add templating tests to the fission-python-skill test suite
4. Document the update process for future template modifications
5. Consider versioning the template (e.g., template-v2/)