241 lines
8.2 KiB
Markdown
241 lines
8.2 KiB
Markdown
|
# Project Structure
|
||
|
|
|
||
|
|
This document explains the purpose and contents of each directory and file in a Fission Python project.
|
||
|
|
|
||
|
|
## Directory Layout
|
||
|
|
|
||
|
|
```
|
||
|
|
project/
|
||
|
|
├── .fission/ # Fission configuration
|
||
|
|
│ ├── deployment.json # Main deployment configuration
|
||
|
|
│ ├── dev-deployment.json # Development environment overrides
|
||
|
|
│ └── local-deployment.json # Local development overrides
|
||
|
|
├── src/ # Source code
|
||
|
|
│ ├── __init__.py # Package initialization
|
||
|
|
│ ├── vault.py # Vault encryption utilities
|
||
|
|
│ ├── helpers.py # Shared utility functions
|
||
|
|
│ ├── exceptions.py # Custom exception classes
|
||
|
|
│ ├── models.py # Pydantic models for validation
|
||
|
|
│ ├── build.sh # Build script (executable)
|
||
|
|
│ └── *.py # Your function implementations
|
||
|
|
├── test/ # Unit and integration tests
|
||
|
|
│ ├── __init__.py
|
||
|
|
│ ├── test_*.py # Test files
|
||
|
|
│ └── requirements.txt # Test dependencies
|
||
|
|
├── migrates/ # Database migration scripts
|
||
|
|
│ └── *.sql # SQL migration files
|
||
|
|
├── manifests/ # Kubernetes manifests (optional)
|
||
|
|
│ └── *.yaml # K8s resources
|
||
|
|
├── specs/ # Generated Fission specs
|
||
|
|
│ ├── fission-deployment-config.yaml
|
||
|
|
│ └── ...
|
||
|
|
├── requirements.txt # Runtime dependencies
|
||
|
|
├── dev-requirements.txt # Development dependencies
|
||
|
|
├── .env.example # Environment variable template
|
||
|
|
├── pytest.ini # Pytest configuration
|
||
|
|
├── README.md # Project documentation
|
||
|
|
└── (other project files)
|
||
|
|
```
|
||
|
|
|
||
|
|
## File Purposes
|
||
|
|
|
||
|
|
### .fission/deployment.json
|
||
|
|
|
||
|
|
This is **the most important configuration file** for Fission deployment. It defines:
|
||
|
|
|
||
|
|
- **environments**: Build environment configuration (image, builder, resources)
|
||
|
|
- **archives**: Source code packaging (typically "package.zip" from src/)
|
||
|
|
- **packages**: Package definitions linking source to environment
|
||
|
|
- **function_common**: Default settings applied to all functions
|
||
|
|
- **secrets**: Secret definitions (literal values are placeholders - actual secrets go in K8s)
|
||
|
|
- **configmaps**: ConfigMap definitions (non-sensitive configuration)
|
||
|
|
|
||
|
|
**Important**: The secret and configmap literals are **placeholders only**. In production, you create actual K8s secrets/configmaps with the same names containing real values.
|
||
|
|
|
||
|
|
**Placeholders**:
|
||
|
|
- `${PROJECT_NAME}` - Replaced with your project name by `create-project.sh`
|
||
|
|
- Secret name pattern: `fission-${PROJECT_NAME}-env`
|
||
|
|
- ConfigMap name pattern: `fission-${PROJECT_NAME}-config`
|
||
|
|
|
||
|
|
### src/vault.py
|
||
|
|
|
||
|
|
Provides encryption/decryption utilities using PyNaCl (SecretBox). This is used when you want to store encrypted values in K8s secrets rather than plaintext.
|
||
|
|
|
||
|
|
**Key functions**:
|
||
|
|
- `encrypt_vault(plaintext, key)` - Encrypt and return vault format string
|
||
|
|
- `decrypt_vault(vault, key)` - Decrypt vault format string
|
||
|
|
- `is_valid_vault_format(vault)` - Check if string is vault-encrypted
|
||
|
|
|
||
|
|
**Usage in helpers.py**: The `get_secret()` and `get_config()` functions automatically detect vault format (`vault:v1:...`) and decrypt if a valid `CRYPTO_KEY` is set.
|
||
|
|
|
||
|
|
### src/helpers.py
|
||
|
|
|
||
|
|
Shared utilities used across functions:
|
||
|
|
|
||
|
|
**Database**:
|
||
|
|
- `init_db_connection()` - Creates PostgreSQL connection from secrets
|
||
|
|
- `db_row_to_dict(cursor, row)` - Convert row tuple to dict
|
||
|
|
- `db_rows_to_array(cursor, rows)` - Convert multiple rows to list of dicts
|
||
|
|
|
||
|
|
**Configuration**:
|
||
|
|
- `get_secret(key, default=None)` - Read from K8s secret volume
|
||
|
|
- `get_config(key, default=None)` - Read from K8s config volume
|
||
|
|
- `get_current_namespace()` - Get current K8s namespace
|
||
|
|
|
||
|
|
**Utilities**:
|
||
|
|
- `str_to_bool(input)` - Convert string to boolean
|
||
|
|
- `check_port_open(ip, port, timeout)` - TCP port connectivity check
|
||
|
|
- `get_user_from_headers()` - Extract user ID from request headers
|
||
|
|
- `format_error_response(...)` - Build standardized error dict
|
||
|
|
|
||
|
|
**Logging**:
|
||
|
|
- Helper uses `current_app.logger` (Flask) for error logging
|
||
|
|
|
||
|
|
### src/exceptions.py
|
||
|
|
|
||
|
|
Custom exception hierarchy:
|
||
|
|
|
||
|
|
```
|
||
|
|
ServiceException (base)
|
||
|
|
├── ValidationError (400) - Invalid input
|
||
|
|
├── NotFoundError (404) - Resource not found
|
||
|
|
├── ConflictError (409) - Duplicate/conflict
|
||
|
|
└── DatabaseError (500) - Database failure
|
||
|
|
```
|
||
|
|
|
||
|
|
All exceptions include:
|
||
|
|
- `error_code` - Machine-readable code
|
||
|
|
- `http_status` - HTTP status
|
||
|
|
- `error_msg` - Human-readable message
|
||
|
|
- `x_user` (optional) - User identifier
|
||
|
|
- `details` (optional) - Additional context dict
|
||
|
|
|
||
|
|
When raised in a Fission function, these automatically return proper JSON error responses.
|
||
|
|
|
||
|
|
### src/models.py
|
||
|
|
|
||
|
|
Pydantic models for request/response validation:
|
||
|
|
|
||
|
|
**Patterns included**:
|
||
|
|
- Enums (e.g., `Status`, `DataType`)
|
||
|
|
- Dataclass filters (e.g., `ItemFilter`, `Pagination`)
|
||
|
|
- Request models (`ItemCreateRequest`, `ItemUpdateRequest`)
|
||
|
|
- Response models (`ItemResponse`, `PaginatedResponse`)
|
||
|
|
- ErrorResponse model (used by exceptions)
|
||
|
|
|
||
|
|
**Key concepts**:
|
||
|
|
- Use `Field(...)` with constraints (min_length, max_length, ge, le)
|
||
|
|
- Provide `description` for API documentation
|
||
|
|
- Use `json_schema_extra` for example values
|
||
|
|
- Set `from_attributes = True` for ORM compatibility
|
||
|
|
|
||
|
|
### src/build.sh
|
||
|
|
|
||
|
|
Bash script that builds the dependency package. It:
|
||
|
|
1. Detects OS (Debian vs Alpine)
|
||
|
|
2. Installs build dependencies (gcc, libpq-dev/python3-dev/postgresql-dev)
|
||
|
|
3. Installs Python requirements into `src/` directory
|
||
|
|
4. Copies `src/` to package destination
|
||
|
|
|
||
|
|
**Important**: Must be executable (`chmod +x src/build.sh`)
|
||
|
|
|
||
|
|
The script expects environment variables:
|
||
|
|
- `SRC_PKG` - Source package directory (e.g., `src`)
|
||
|
|
- `DEPLOY_PKG` - Destination package (e.g., `specs/package`)
|
||
|
|
|
||
|
|
Fission builder sets these automatically.
|
||
|
|
|
||
|
|
### test/
|
||
|
|
|
||
|
|
Contains unit and integration tests.
|
||
|
|
|
||
|
|
**Structure**:
|
||
|
|
- `test_*.py` - Test files following pytest conventions
|
||
|
|
- `requirements.txt` - Test dependencies (pytest, pytest-mock, requests)
|
||
|
|
|
||
|
|
**Running tests**:
|
||
|
|
```bash
|
||
|
|
pip install -r dev-requirements.txt
|
||
|
|
pytest
|
||
|
|
```
|
||
|
|
|
||
|
|
## Fission Configuration in Docstrings
|
||
|
|
|
||
|
|
Each Python function that should be exposed as a Fission function **must** include a ````fission` block in its docstring:
|
||
|
|
|
||
|
|
```python
|
||
|
|
def my_function(event, context):
|
||
|
|
"""
|
||
|
|
```fission
|
||
|
|
{
|
||
|
|
"name": "my-function",
|
||
|
|
"http_triggers": {
|
||
|
|
"my-trigger": {
|
||
|
|
"url": "/api/endpoint",
|
||
|
|
"methods": ["GET", "POST"]
|
||
|
|
}
|
||
|
|
}
|
||
|
|
}
|
||
|
|
```
|
||
|
|
Human-readable description here.
|
||
|
|
"""
|
||
|
|
# Implementation
|
||
|
|
```
|
||
|
|
|
||
|
|
The Fission Python builder parses these docstrings and generates the `specs/fission-deployment-config.yaml` and other spec files.
|
||
|
|
|
||
|
|
**Supported trigger types**:
|
||
|
|
- `http_triggers` - HTTP endpoints
|
||
|
|
- `kafka_triggers` - Kafka topics
|
||
|
|
- `timer_triggers` - Scheduled execution
|
||
|
|
- `message_queue_triggers` - MQTT, NATS, etc.
|
||
|
|
|
||
|
|
## Configuration Precedence
|
||
|
|
|
||
|
|
1. **deployment.json** - Base configuration (committed to repo)
|
||
|
|
2. **dev-deployment.json** - Overrides for dev environment (not always committed)
|
||
|
|
3. **local-deployment.json** - Local overrides (typically .gitignored)
|
||
|
|
|
||
|
|
When deploying:
|
||
|
|
- `fission deploy` uses deployment.json
|
||
|
|
- `fission deploy --dev` uses dev-deployment.json if present
|
||
|
|
|
||
|
|
## Secrets and Configuration Flow
|
||
|
|
|
||
|
|
1. **Define placeholders** in `deployment.json`:
|
||
|
|
```json
|
||
|
|
"secrets": {
|
||
|
|
"fission-myproject-env": {
|
||
|
|
"literals": ["PG_HOST=localhost", "PG_PORT=5432"]
|
||
|
|
}
|
||
|
|
}
|
||
|
|
```
|
||
|
|
|
||
|
|
2. **Create actual K8s secret**:
|
||
|
|
```bash
|
||
|
|
kubectl create secret generic fission-myproject-env \
|
||
|
|
--from-literal=PG_HOST=prod-db.example.com \
|
||
|
|
--from-literal=PG_PORT=5432
|
||
|
|
```
|
||
|
|
|
||
|
|
3. **Read in code** via `get_secret()`:
|
||
|
|
```python
|
||
|
|
host = get_secret("PG_HOST")
|
||
|
|
```
|
||
|
|
|
||
|
|
4. **For vault encryption**:
|
||
|
|
- Set `CRYPTO_KEY` in helpers.py or as env override
|
||
|
|
- Store encrypted: `vault:v1:base64data` in K8s secret
|
||
|
|
- `get_secret()` auto-decrypts
|
||
|
|
|
||
|
|
## Summary
|
||
|
|
|
||
|
|
- Keep function code in `src/`
|
||
|
|
- Define Fission metadata in docstring blocks
|
||
|
|
- Use helpers for common operations
|
||
|
|
- Define custom exceptions for error handling
|
||
|
|
- Validate inputs with Pydantic models
|
||
|
|
- Store tests in `test/` with pytest
|
||
|
|
- Manage database migrations in `migrates/`
|
||
|
|
- Do not commit actual secrets to repository
|