ref: up
This commit is contained in:
Duc Nguyen
360
.claude/plans/distributed-coalescing-breeze.md
Normal file
360
.claude/plans/distributed-coalescing-breeze.md
Normal file
@@ -0,0 +1,360 @@
|
||||
# Plan: Update Fission Python Template Based on Example Projects
|
||||
|
||||
## Context
|
||||
|
||||
The current Fission Python template (`fission-python/template/`) is essentially a copy of the `py-eom-quota` example project, making it **quota-specific** rather than a **generic starting point** for new Fission Python projects.
|
||||
|
||||
Three example projects were analyzed:
|
||||
- `py-eom-quota` - User quota management API
|
||||
- `py-eom-storage` - Storage resource management with S3 integration
|
||||
- `py-ailbl-scheduler` - Background job scheduler with Dagster integration
|
||||
|
||||
All examples share common infrastructure patterns but differ in business logic. This plan will make the template **generic, reusable, and production-ready** by extracting shared best practices.
|
||||
|
||||
---
|
||||
|
||||
## Key Findings from Examples
|
||||
|
||||
### 1. Common Infrastructure (All Projects Share)
|
||||
|
||||
- **vault.py** - Identical across all three projects (encryption/decryption using PyNaCl)
|
||||
- **helpers.py** - Nearly identical core utilities:
|
||||
- `get_secret()` / `get_config()` (K8s secrets/configmaps with vault support)
|
||||
- `init_db_connection()` (PostgreSQL connection)
|
||||
- `db_row_to_dict()` / `db_rows_to_array()`
|
||||
- `get_user_from_headers()` (extract user for audit logging)
|
||||
- `format_error_response()` (standardized error format)
|
||||
- `check_port_open()` (DB readiness check)
|
||||
- `str_to_bool()` utility
|
||||
- **Fission Configuration** - Using docstring format in `main()` functions
|
||||
- **Exception Patterns** - Custom exception hierarchies with:
|
||||
- `error_code` (machine-readable)
|
||||
- `http_status` (HTTP status)
|
||||
- `error_msg` (human-readable)
|
||||
- `x_user` (optional user tracking)
|
||||
- `details` (optional additional context)
|
||||
- **Pydantic Models** - Request validation, response schemas, pagination/filtering
|
||||
- **Project Structure** - Consistent layout:
|
||||
```
|
||||
project/
|
||||
├── .fission/deployment.json
|
||||
├── src/
|
||||
│ ├── __init__.py
|
||||
│ ├── exceptions.py
|
||||
│ ├── helpers.py
|
||||
│ ├── models.py
|
||||
│ ├── vault.py
|
||||
│ ├── build.sh
|
||||
│ └── <business logic>.py
|
||||
├── test/
|
||||
├── migrates/
|
||||
├── manifests/
|
||||
├── specs/
|
||||
├── requirements.txt
|
||||
├── dev-requirements.txt
|
||||
└── README.md
|
||||
```
|
||||
|
||||
### 2. Variations Between Projects
|
||||
|
||||
**Database Connection:**
|
||||
- `py-eom-quota`: Advanced `DBConfig` dataclass with `from_remote_config()` support
|
||||
- `py-eom-storage` & `py-ailbl-scheduler`: Simplified direct connection from secrets
|
||||
|
||||
**Additional Dependencies:**
|
||||
- Storage: `boto3` (S3/MinIO), `botocore`
|
||||
- Scheduler: `gql` (GraphQL), `cron-descriptor`
|
||||
- All: `pydantic`, `psycopg2-binary`, `PyNaCl`, `Flask`, `requests`
|
||||
|
||||
**Executors:**
|
||||
- Quota: `poolmgr` (concurrency=1)
|
||||
- Storage: `poolmgr` (concurrency=3, maxscale=3)
|
||||
- Scheduler: `newdeploy` (minscale=1, maxscale=1)
|
||||
|
||||
### 3. Issues to Fix
|
||||
|
||||
- **README outdated** - References `pymake`, `fission.json`, `fission.yaml` (not used)
|
||||
- **Missing Flask** - `src/requirements.txt` needs Flask (currently only in dev-requirements)
|
||||
- **Quota-specific code** - Template should be generic (no `QuotaException`, `QuotaResponse`, etc.)
|
||||
- **No .env.example** - Missing environment variable template
|
||||
- **Test dependencies minimal** - Should include `pytest`, `pytest-mock`, `requests`, `flake8`, `black`
|
||||
- **build.sh** - Should handle both alpine (apk) and debian (apt) properly
|
||||
- **deployment.json** - Should not hardcode `fission-eom-quota-env` secret names
|
||||
- **Missing Python version** - Should specify Python 3.11+ (scheduler uses 3.11-alpine)
|
||||
|
||||
---
|
||||
|
||||
## Recommended Changes
|
||||
|
||||
### Phase 1: Core Infrastructure (Keep Generic)
|
||||
|
||||
**Files to MODIFY:**
|
||||
|
||||
1. **`src/vault.py`** - Keep as-is (already perfect, identical in all examples)
|
||||
|
||||
2. **`src/helpers.py`** - Use the simplified pattern from `py-eom-storage` but add:
|
||||
- Keep: `get_secret()`, `get_config()`, `init_db_connection()`, `db_row_to_dict()`, `db_rows_to_array()`, `get_current_namespace()`, `str_to_bool()`, `check_port_open()`, `get_user_from_headers()`, `format_error_response()`
|
||||
- Remove: `DBConfig` class (too specific to quota, keep it simple)
|
||||
- Add: `.strip()` when reading files (as in scheduler)
|
||||
- Keep CORS_HEADERS and constants but make them configurable
|
||||
|
||||
3. **`src/exceptions.py`** - Replace quota-specific with generic patterns:
|
||||
```python
|
||||
class ServiceException(Exception):
|
||||
"""Base exception for service errors."""
|
||||
def __init__(self, error_code, http_status, error_msg, x_user=None, details=None):
|
||||
...
|
||||
|
||||
class ValidationError(ServiceException): # 400
|
||||
class NotFoundError(ServiceException): # 404
|
||||
class ConflictError(ServiceException): # 409
|
||||
class DatabaseError(ServiceException): # 500
|
||||
```
|
||||
(Based on `py-eom-storage` pattern - cleaner and more generic)
|
||||
|
||||
4. **`src/models.py`** - Replace with generic example patterns:
|
||||
- Remove: All quota-specific models
|
||||
- Add: Generic `ItemResponse`, `PaginatedResponse`, `ErrorResponse`
|
||||
- Include examples of Pydantic models with Field descriptions and json_schema_extra
|
||||
- Show patterns for: Enums, nested models, dataclasses for filters
|
||||
|
||||
5. **`src/requirements.txt`** - Update to include actual runtime deps:
|
||||
```
|
||||
Flask==2.1.1
|
||||
pydantic==2.11.7
|
||||
psycopg2-binary==2.9.10
|
||||
PyNaCl==1.6.0
|
||||
requests==2.32.2
|
||||
```
|
||||
(Remove commented examples - these go in docs, not requirements.txt)
|
||||
|
||||
6. **`dev-requirements.txt`** - Expand with useful dev tools:
|
||||
```
|
||||
Flask==2.1.1
|
||||
requests==2.32.2
|
||||
pytest==8.2.0
|
||||
pytest-mock==3.14.0
|
||||
flake8==7.0.0
|
||||
black==24.1.1
|
||||
mypy==1.8.0
|
||||
```
|
||||
|
||||
7. **`README.md`** - Complete rewrite:
|
||||
- Remove references to pymake, fission.json
|
||||
- Explain actual project structure
|
||||
- Document Fission configuration in docstrings
|
||||
- Show how to use deployment.json
|
||||
- Document environment variables (secrets/configmaps)
|
||||
- Explain testing approach
|
||||
- Add development workflow
|
||||
- Include examples from all three projects as inspiration
|
||||
|
||||
8. **`.fission/deployment.json`** - Make generic with placeholders:
|
||||
- Use `your-service-py` as environment name
|
||||
- Use `your-package` as package name
|
||||
- Use generic secret/configmap names: `fission-${PROJECT_NAME}-env`, `fission-${PROJECT_NAME}-config`
|
||||
- Show both `poolmgr` and `newdeploy` executor examples (commented)
|
||||
- Include optional fields like `imagepullsecret`, `runtime_envs`, `configmaps`
|
||||
|
||||
9. **`test/requirements.txt`** - Add:
|
||||
```
|
||||
pytest==8.2.0
|
||||
pytest-mock==3.14.0
|
||||
requests==2.32.3
|
||||
```
|
||||
|
||||
10. **`build.sh`** - Fix to use `${SRC_PKG}` properly (current version is correct)
|
||||
|
||||
### Phase 2: Documentation & Examples
|
||||
|
||||
**New Files to ADD:**
|
||||
|
||||
1. **`src/__init__.py`** - Already exists, keep as is
|
||||
|
||||
2. **`examples/` directory** (new) - Sample function implementations:
|
||||
- `example_crud.py` - Basic CRUD with Pydantic validation
|
||||
- `example_webhook.py` - Webhook receiver pattern
|
||||
- `example_scheduler.py` - Background job pattern (from ailbl-scheduler)
|
||||
- Each should have proper Fission docstring config
|
||||
|
||||
3. **`.env.example`** - Template showing all environment variables:
|
||||
```
|
||||
# PostgreSQL
|
||||
PG_HOST=
|
||||
PG_PORT=5432
|
||||
PG_DB=
|
||||
PG_USER=
|
||||
PG_PASS=
|
||||
PG_DBSCHEMA=
|
||||
|
||||
# Optional: Service-specific config (via ConfigMap)
|
||||
# YOUR_SERVICE_CONFIG_ENDPOINT=
|
||||
|
||||
# Optional: Vault encryption key (if using encrypted secrets)
|
||||
# CRYPTO_KEY=
|
||||
```
|
||||
|
||||
4. **`docs/` directory** (new) - Additional documentation:
|
||||
- `STRUCTURE.md` - Detailed file structure explanation
|
||||
- `TESTING.md` - How to write and run tests
|
||||
- `DEPLOYMENT.md` - Deployment options and tuning
|
||||
- `SECRETS.md` - Managing secrets and configmaps
|
||||
- `MIGRATIONS.md` - Database migration workflow
|
||||
|
||||
5. **`pytest.ini`** - Default pytest configuration:
|
||||
```ini
|
||||
[pytest]
|
||||
testpaths = test
|
||||
python_files = test_*.py
|
||||
python_classes = Test*
|
||||
python_functions = test_*
|
||||
log_cli = true
|
||||
log_cli_level = INFO
|
||||
```
|
||||
|
||||
6. **`.gitignore`** - Ensure it excludes:
|
||||
- `__pycache__/`
|
||||
- `*.pyc`
|
||||
- `.env`
|
||||
- `.venv/`
|
||||
- `venv/`
|
||||
- `.pytest_cache/`
|
||||
- `.mypy_cache/`
|
||||
- `.coverage`
|
||||
- `coverage.xml`
|
||||
- `specs/` (optional - generated files)
|
||||
|
||||
7. **`MANIFEST.md`** - Template for Kubernetes manifests (if not using auto-generated)
|
||||
|
||||
### Phase 3: Modernization
|
||||
|
||||
**Update CI/CD:**
|
||||
|
||||
Review `.gitea/workflows/` files:
|
||||
- Ensure they install dependencies correctly
|
||||
- Add linting (flake8/black) steps
|
||||
- Add test execution
|
||||
- Add deployment steps with proper environment detection
|
||||
- Consider adding security scanning
|
||||
|
||||
**Python Version:**
|
||||
|
||||
- Ensure all files are compatible with Python 3.11+
|
||||
- Update `build.sh` to use Python 3.11 image (like scheduler does) or keep generic
|
||||
- Consider adding `runtime.txt` or `pyproject.toml` to specify Python version
|
||||
|
||||
---
|
||||
|
||||
## Files to Modify Summary
|
||||
|
||||
**Direct modifications:**
|
||||
- `src/helpers.py` - Simplify, improve
|
||||
- `src/exceptions.py` - Make generic
|
||||
- `src/models.py` - Replace with generic patterns
|
||||
- `src/requirements.txt` - Add Flask, remove commented section
|
||||
- `dev-requirements.txt` - Comprehensive dev dependencies
|
||||
- `test/requirements.txt` - Test dependencies
|
||||
- `README.md` - Complete rewrite
|
||||
- `.fission/deployment.json` - Generic placeholders
|
||||
- `build.sh` - Already good, just ensure compatibility
|
||||
|
||||
**New files to add:**
|
||||
- `.env.example`
|
||||
- `pytest.ini`
|
||||
- `.gitignore` (enhance)
|
||||
- `examples/` directory with sample functions
|
||||
- `docs/` directory with detailed guides
|
||||
- `src/example_crud.py` (or in examples/)
|
||||
- `src/example_webhook.py` (or in examples/)
|
||||
- `src/example_scheduler.py` (or in examples/)
|
||||
|
||||
**New directories:**
|
||||
- `examples/`
|
||||
- `docs/`
|
||||
|
||||
---
|
||||
|
||||
## Implementation Approach
|
||||
|
||||
1. **Backup current template** (git branch)
|
||||
2. **Modify core files** in order: helpers → exceptions → models → requirements → deployment.json → README
|
||||
3. **Add new files** (examples, docs, configs)
|
||||
4. **Test the template**:
|
||||
- Run `create-project.sh` to generate a new project
|
||||
- Verify build.sh works
|
||||
- Run tests
|
||||
- Check Fission spec generation
|
||||
5. **Commit with clear message**
|
||||
6. **Update plugin documentation** if needed
|
||||
|
||||
---
|
||||
|
||||
## Verification Steps
|
||||
|
||||
After implementing the changes:
|
||||
|
||||
1. **Create a test project** from the updated template:
|
||||
```bash
|
||||
./create-project.sh test-project ./tmp-test/
|
||||
```
|
||||
2. **Inspect generated project**:
|
||||
- Verify all files are present
|
||||
- Check that placeholders are substituted correctly
|
||||
- Ensure imports work
|
||||
3. **Build the package**:
|
||||
```bash
|
||||
cd tmp-test
|
||||
./src/build.sh
|
||||
```
|
||||
4. **Run tests** (if any):
|
||||
```bash
|
||||
pip install -r dev-requirements.txt
|
||||
pytest
|
||||
```
|
||||
5. **Check syntax**:
|
||||
```bash
|
||||
python -m py_compile src/*.py
|
||||
flake8 src/
|
||||
black --check src/
|
||||
```
|
||||
6. **Validate Fission config**:
|
||||
```bash
|
||||
fission spec verify --file=.fission/deployment.json
|
||||
```
|
||||
7. **Review README** - Does it accurately describe the project?
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- Template is **generic**, not domain-specific
|
||||
- All examples' best practices are incorporated
|
||||
- Documentation is accurate and complete
|
||||
- Dependencies are correctly listed (Flask in requirements, not just dev)
|
||||
- README reflects actual Fission workflow (docstrings, not fission.yaml)
|
||||
- Multiple example implementations provided (CRUD, webhook, scheduler)
|
||||
- Secrets/configuration clearly explained
|
||||
- Testing setup is comprehensive
|
||||
- Project passes linting and type checks
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Breaking existing template users | Keep changes minimal in helpers; preserve backward compatibility where possible |
|
||||
| Over-engineering | Stick to patterns that appear in at least 2 of 3 examples |
|
||||
| Missing edge cases | Include optional advanced patterns (like DBConfig) in docs, not in core |
|
||||
| Documentation drift | Keep docs close to code; add examples that mirror real projects |
|
||||
|
||||
---
|
||||
|
||||
## Post-Implementation
|
||||
|
||||
After the template is updated:
|
||||
1. Consider creating a **template validation script** to ensure quality
|
||||
2. Update the **plugin SKILL.md** to reflect template changes
|
||||
3. Add **templating tests** to the fission-python-skill test suite
|
||||
4. Document the **update process** for future template modifications
|
||||
5. Consider **versioning** the template (e.g., `template-v2/`)
|
||||
Reference in New Issue
Block a user