ref: up
This commit is contained in:
Duc Nguyen
240
fission-python/template/docs/STRUCTURE.md
Normal file
240
fission-python/template/docs/STRUCTURE.md
Normal file
@@ -0,0 +1,240 @@
|
||||
# Project Structure
|
||||
|
||||
This document explains the purpose and contents of each directory and file in a Fission Python project.
|
||||
|
||||
## Directory Layout
|
||||
|
||||
```
|
||||
project/
|
||||
├── .fission/ # Fission configuration
|
||||
│ ├── deployment.json # Main deployment configuration
|
||||
│ ├── dev-deployment.json # Development environment overrides
|
||||
│ └── local-deployment.json # Local development overrides
|
||||
├── src/ # Source code
|
||||
│ ├── __init__.py # Package initialization
|
||||
│ ├── vault.py # Vault encryption utilities
|
||||
│ ├── helpers.py # Shared utility functions
|
||||
│ ├── exceptions.py # Custom exception classes
|
||||
│ ├── models.py # Pydantic models for validation
|
||||
│ ├── build.sh # Build script (executable)
|
||||
│ └── *.py # Your function implementations
|
||||
├── test/ # Unit and integration tests
|
||||
│ ├── __init__.py
|
||||
│ ├── test_*.py # Test files
|
||||
│ └── requirements.txt # Test dependencies
|
||||
├── migrates/ # Database migration scripts
|
||||
│ └── *.sql # SQL migration files
|
||||
├── manifests/ # Kubernetes manifests (optional)
|
||||
│ └── *.yaml # K8s resources
|
||||
├── specs/ # Generated Fission specs
|
||||
│ ├── fission-deployment-config.yaml
|
||||
│ └── ...
|
||||
├── requirements.txt # Runtime dependencies
|
||||
├── dev-requirements.txt # Development dependencies
|
||||
├── .env.example # Environment variable template
|
||||
├── pytest.ini # Pytest configuration
|
||||
├── README.md # Project documentation
|
||||
└── (other project files)
|
||||
```
|
||||
|
||||
## File Purposes
|
||||
|
||||
### .fission/deployment.json
|
||||
|
||||
This is **the most important configuration file** for Fission deployment. It defines:
|
||||
|
||||
- **environments**: Build environment configuration (image, builder, resources)
|
||||
- **archives**: Source code packaging (typically "package.zip" from src/)
|
||||
- **packages**: Package definitions linking source to environment
|
||||
- **function_common**: Default settings applied to all functions
|
||||
- **secrets**: Secret definitions (literal values are placeholders - actual secrets go in K8s)
|
||||
- **configmaps**: ConfigMap definitions (non-sensitive configuration)
|
||||
|
||||
**Important**: The secret and configmap literals are **placeholders only**. In production, you create actual K8s secrets/configmaps with the same names containing real values.
|
||||
|
||||
**Placeholders**:
|
||||
- `${PROJECT_NAME}` - Replaced with your project name by `create-project.sh`
|
||||
- Secret name pattern: `fission-${PROJECT_NAME}-env`
|
||||
- ConfigMap name pattern: `fission-${PROJECT_NAME}-config`
|
||||
|
||||
### src/vault.py
|
||||
|
||||
Provides encryption/decryption utilities using PyNaCl (SecretBox). This is used when you want to store encrypted values in K8s secrets rather than plaintext.
|
||||
|
||||
**Key functions**:
|
||||
- `encrypt_vault(plaintext, key)` - Encrypt and return vault format string
|
||||
- `decrypt_vault(vault, key)` - Decrypt vault format string
|
||||
- `is_valid_vault_format(vault)` - Check if string is vault-encrypted
|
||||
|
||||
**Usage in helpers.py**: The `get_secret()` and `get_config()` functions automatically detect vault format (`vault:v1:...`) and decrypt if a valid `CRYPTO_KEY` is set.
|
||||
|
||||
### src/helpers.py
|
||||
|
||||
Shared utilities used across functions:
|
||||
|
||||
**Database**:
|
||||
- `init_db_connection()` - Creates PostgreSQL connection from secrets
|
||||
- `db_row_to_dict(cursor, row)` - Convert row tuple to dict
|
||||
- `db_rows_to_array(cursor, rows)` - Convert multiple rows to list of dicts
|
||||
|
||||
**Configuration**:
|
||||
- `get_secret(key, default=None)` - Read from K8s secret volume
|
||||
- `get_config(key, default=None)` - Read from K8s config volume
|
||||
- `get_current_namespace()` - Get current K8s namespace
|
||||
|
||||
**Utilities**:
|
||||
- `str_to_bool(input)` - Convert string to boolean
|
||||
- `check_port_open(ip, port, timeout)` - TCP port connectivity check
|
||||
- `get_user_from_headers()` - Extract user ID from request headers
|
||||
- `format_error_response(...)` - Build standardized error dict
|
||||
|
||||
**Logging**:
|
||||
- Helper uses `current_app.logger` (Flask) for error logging
|
||||
|
||||
### src/exceptions.py
|
||||
|
||||
Custom exception hierarchy:
|
||||
|
||||
```
|
||||
ServiceException (base)
|
||||
├── ValidationError (400) - Invalid input
|
||||
├── NotFoundError (404) - Resource not found
|
||||
├── ConflictError (409) - Duplicate/conflict
|
||||
└── DatabaseError (500) - Database failure
|
||||
```
|
||||
|
||||
All exceptions include:
|
||||
- `error_code` - Machine-readable code
|
||||
- `http_status` - HTTP status
|
||||
- `error_msg` - Human-readable message
|
||||
- `x_user` (optional) - User identifier
|
||||
- `details` (optional) - Additional context dict
|
||||
|
||||
When raised in a Fission function, these automatically return proper JSON error responses.
|
||||
|
||||
### src/models.py
|
||||
|
||||
Pydantic models for request/response validation:
|
||||
|
||||
**Patterns included**:
|
||||
- Enums (e.g., `Status`, `DataType`)
|
||||
- Dataclass filters (e.g., `ItemFilter`, `Pagination`)
|
||||
- Request models (`ItemCreateRequest`, `ItemUpdateRequest`)
|
||||
- Response models (`ItemResponse`, `PaginatedResponse`)
|
||||
- ErrorResponse model (used by exceptions)
|
||||
|
||||
**Key concepts**:
|
||||
- Use `Field(...)` with constraints (min_length, max_length, ge, le)
|
||||
- Provide `description` for API documentation
|
||||
- Use `json_schema_extra` for example values
|
||||
- Set `from_attributes = True` for ORM compatibility
|
||||
|
||||
### src/build.sh
|
||||
|
||||
Bash script that builds the dependency package. It:
|
||||
1. Detects OS (Debian vs Alpine)
|
||||
2. Installs build dependencies (gcc, libpq-dev/python3-dev/postgresql-dev)
|
||||
3. Installs Python requirements into `src/` directory
|
||||
4. Copies `src/` to package destination
|
||||
|
||||
**Important**: Must be executable (`chmod +x src/build.sh`)
|
||||
|
||||
The script expects environment variables:
|
||||
- `SRC_PKG` - Source package directory (e.g., `src`)
|
||||
- `DEPLOY_PKG` - Destination package (e.g., `specs/package`)
|
||||
|
||||
Fission builder sets these automatically.
|
||||
|
||||
### test/
|
||||
|
||||
Contains unit and integration tests.
|
||||
|
||||
**Structure**:
|
||||
- `test_*.py` - Test files following pytest conventions
|
||||
- `requirements.txt` - Test dependencies (pytest, pytest-mock, requests)
|
||||
|
||||
**Running tests**:
|
||||
```bash
|
||||
pip install -r dev-requirements.txt
|
||||
pytest
|
||||
```
|
||||
|
||||
## Fission Configuration in Docstrings
|
||||
|
||||
Each Python function that should be exposed as a Fission function **must** include a ````fission` block in its docstring:
|
||||
|
||||
```python
|
||||
def my_function(event, context):
|
||||
"""
|
||||
```fission
|
||||
{
|
||||
"name": "my-function",
|
||||
"http_triggers": {
|
||||
"my-trigger": {
|
||||
"url": "/api/endpoint",
|
||||
"methods": ["GET", "POST"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
Human-readable description here.
|
||||
"""
|
||||
# Implementation
|
||||
```
|
||||
|
||||
The Fission Python builder parses these docstrings and generates the `specs/fission-deployment-config.yaml` and other spec files.
|
||||
|
||||
**Supported trigger types**:
|
||||
- `http_triggers` - HTTP endpoints
|
||||
- `kafka_triggers` - Kafka topics
|
||||
- `timer_triggers` - Scheduled execution
|
||||
- `message_queue_triggers` - MQTT, NATS, etc.
|
||||
|
||||
## Configuration Precedence
|
||||
|
||||
1. **deployment.json** - Base configuration (committed to repo)
|
||||
2. **dev-deployment.json** - Overrides for dev environment (not always committed)
|
||||
3. **local-deployment.json** - Local overrides (typically .gitignored)
|
||||
|
||||
When deploying:
|
||||
- `fission deploy` uses deployment.json
|
||||
- `fission deploy --dev` uses dev-deployment.json if present
|
||||
|
||||
## Secrets and Configuration Flow
|
||||
|
||||
1. **Define placeholders** in `deployment.json`:
|
||||
```json
|
||||
"secrets": {
|
||||
"fission-myproject-env": {
|
||||
"literals": ["PG_HOST=localhost", "PG_PORT=5432"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. **Create actual K8s secret**:
|
||||
```bash
|
||||
kubectl create secret generic fission-myproject-env \
|
||||
--from-literal=PG_HOST=prod-db.example.com \
|
||||
--from-literal=PG_PORT=5432
|
||||
```
|
||||
|
||||
3. **Read in code** via `get_secret()`:
|
||||
```python
|
||||
host = get_secret("PG_HOST")
|
||||
```
|
||||
|
||||
4. **For vault encryption**:
|
||||
- Set `CRYPTO_KEY` in helpers.py or as env override
|
||||
- Store encrypted: `vault:v1:base64data` in K8s secret
|
||||
- `get_secret()` auto-decrypts
|
||||
|
||||
## Summary
|
||||
|
||||
- Keep function code in `src/`
|
||||
- Define Fission metadata in docstring blocks
|
||||
- Use helpers for common operations
|
||||
- Define custom exceptions for error handling
|
||||
- Validate inputs with Pydantic models
|
||||
- Store tests in `test/` with pytest
|
||||
- Manage database migrations in `migrates/`
|
||||
- Do not commit actual secrets to repository
|
||||
Reference in New Issue
Block a user