ref: up
This commit is contained in:
Duc Nguyen
16
.claude-plugin/marketplace.json
Normal file
16
.claude-plugin/marketplace.json
Normal file
@@ -0,0 +1,16 @@
|
||||
{
|
||||
"name" : "vega-claude-marketplace",
|
||||
"owner" : {"name": "tiendd", "email": "fdm.dev17@gmail.com"},
|
||||
"plugins": [
|
||||
{
|
||||
"id" : "fission-python",
|
||||
"name" : "FissionPython",
|
||||
"description": "Skill for creating, analyzing, and managing Fission Python projects.",
|
||||
"source" : "./",
|
||||
"type" : "skill",
|
||||
"path" : "fission-python",
|
||||
"tools" : ["create-project", "analyze-config", "update-docstring"],
|
||||
"version" : "1.0.1"
|
||||
}
|
||||
]
|
||||
}
|
||||
360
.claude/plans/distributed-coalescing-breeze.md
Normal file
360
.claude/plans/distributed-coalescing-breeze.md
Normal file
@@ -0,0 +1,360 @@
|
||||
# Plan: Update Fission Python Template Based on Example Projects
|
||||
|
||||
## Context
|
||||
|
||||
The current Fission Python template (`fission-python/template/`) is essentially a copy of the `py-eom-quota` example project, making it **quota-specific** rather than a **generic starting point** for new Fission Python projects.
|
||||
|
||||
Three example projects were analyzed:
|
||||
- `py-eom-quota` - User quota management API
|
||||
- `py-eom-storage` - Storage resource management with S3 integration
|
||||
- `py-ailbl-scheduler` - Background job scheduler with Dagster integration
|
||||
|
||||
All examples share common infrastructure patterns but differ in business logic. This plan will make the template **generic, reusable, and production-ready** by extracting shared best practices.
|
||||
|
||||
---
|
||||
|
||||
## Key Findings from Examples
|
||||
|
||||
### 1. Common Infrastructure (All Projects Share)
|
||||
|
||||
- **vault.py** - Identical across all three projects (encryption/decryption using PyNaCl)
|
||||
- **helpers.py** - Nearly identical core utilities:
|
||||
- `get_secret()` / `get_config()` (K8s secrets/configmaps with vault support)
|
||||
- `init_db_connection()` (PostgreSQL connection)
|
||||
- `db_row_to_dict()` / `db_rows_to_array()`
|
||||
- `get_user_from_headers()` (extract user for audit logging)
|
||||
- `format_error_response()` (standardized error format)
|
||||
- `check_port_open()` (DB readiness check)
|
||||
- `str_to_bool()` utility
|
||||
- **Fission Configuration** - Using docstring format in `main()` functions
|
||||
- **Exception Patterns** - Custom exception hierarchies with:
|
||||
- `error_code` (machine-readable)
|
||||
- `http_status` (HTTP status)
|
||||
- `error_msg` (human-readable)
|
||||
- `x_user` (optional user tracking)
|
||||
- `details` (optional additional context)
|
||||
- **Pydantic Models** - Request validation, response schemas, pagination/filtering
|
||||
- **Project Structure** - Consistent layout:
|
||||
```
|
||||
project/
|
||||
├── .fission/deployment.json
|
||||
├── src/
|
||||
│ ├── __init__.py
|
||||
│ ├── exceptions.py
|
||||
│ ├── helpers.py
|
||||
│ ├── models.py
|
||||
│ ├── vault.py
|
||||
│ ├── build.sh
|
||||
│ └── <business logic>.py
|
||||
├── test/
|
||||
├── migrates/
|
||||
├── manifests/
|
||||
├── specs/
|
||||
├── requirements.txt
|
||||
├── dev-requirements.txt
|
||||
└── README.md
|
||||
```
|
||||
|
||||
### 2. Variations Between Projects
|
||||
|
||||
**Database Connection:**
|
||||
- `py-eom-quota`: Advanced `DBConfig` dataclass with `from_remote_config()` support
|
||||
- `py-eom-storage` & `py-ailbl-scheduler`: Simplified direct connection from secrets
|
||||
|
||||
**Additional Dependencies:**
|
||||
- Storage: `boto3` (S3/MinIO), `botocore`
|
||||
- Scheduler: `gql` (GraphQL), `cron-descriptor`
|
||||
- All: `pydantic`, `psycopg2-binary`, `PyNaCl`, `Flask`, `requests`
|
||||
|
||||
**Executors:**
|
||||
- Quota: `poolmgr` (concurrency=1)
|
||||
- Storage: `poolmgr` (concurrency=3, maxscale=3)
|
||||
- Scheduler: `newdeploy` (minscale=1, maxscale=1)
|
||||
|
||||
### 3. Issues to Fix
|
||||
|
||||
- **README outdated** - References `pymake`, `fission.json`, `fission.yaml` (not used)
|
||||
- **Missing Flask** - `src/requirements.txt` needs Flask (currently only in dev-requirements)
|
||||
- **Quota-specific code** - Template should be generic (no `QuotaException`, `QuotaResponse`, etc.)
|
||||
- **No .env.example** - Missing environment variable template
|
||||
- **Test dependencies minimal** - Should include `pytest`, `pytest-mock`, `requests`, `flake8`, `black`
|
||||
- **build.sh** - Should handle both alpine (apk) and debian (apt) properly
|
||||
- **deployment.json** - Should not hardcode `fission-eom-quota-env` secret names
|
||||
- **Missing Python version** - Should specify Python 3.11+ (scheduler uses 3.11-alpine)
|
||||
|
||||
---
|
||||
|
||||
## Recommended Changes
|
||||
|
||||
### Phase 1: Core Infrastructure (Keep Generic)
|
||||
|
||||
**Files to MODIFY:**
|
||||
|
||||
1. **`src/vault.py`** - Keep as-is (already perfect, identical in all examples)
|
||||
|
||||
2. **`src/helpers.py`** - Use the simplified pattern from `py-eom-storage` but add:
|
||||
- Keep: `get_secret()`, `get_config()`, `init_db_connection()`, `db_row_to_dict()`, `db_rows_to_array()`, `get_current_namespace()`, `str_to_bool()`, `check_port_open()`, `get_user_from_headers()`, `format_error_response()`
|
||||
- Remove: `DBConfig` class (too specific to quota, keep it simple)
|
||||
- Add: `.strip()` when reading files (as in scheduler)
|
||||
- Keep CORS_HEADERS and constants but make them configurable
|
||||
|
||||
3. **`src/exceptions.py`** - Replace quota-specific with generic patterns:
|
||||
```python
|
||||
class ServiceException(Exception):
|
||||
"""Base exception for service errors."""
|
||||
def __init__(self, error_code, http_status, error_msg, x_user=None, details=None):
|
||||
...
|
||||
|
||||
class ValidationError(ServiceException): # 400
|
||||
class NotFoundError(ServiceException): # 404
|
||||
class ConflictError(ServiceException): # 409
|
||||
class DatabaseError(ServiceException): # 500
|
||||
```
|
||||
(Based on `py-eom-storage` pattern - cleaner and more generic)
|
||||
|
||||
4. **`src/models.py`** - Replace with generic example patterns:
|
||||
- Remove: All quota-specific models
|
||||
- Add: Generic `ItemResponse`, `PaginatedResponse`, `ErrorResponse`
|
||||
- Include examples of Pydantic models with Field descriptions and json_schema_extra
|
||||
- Show patterns for: Enums, nested models, dataclasses for filters
|
||||
|
||||
5. **`src/requirements.txt`** - Update to include actual runtime deps:
|
||||
```
|
||||
Flask==2.1.1
|
||||
pydantic==2.11.7
|
||||
psycopg2-binary==2.9.10
|
||||
PyNaCl==1.6.0
|
||||
requests==2.32.2
|
||||
```
|
||||
(Remove commented examples - these go in docs, not requirements.txt)
|
||||
|
||||
6. **`dev-requirements.txt`** - Expand with useful dev tools:
|
||||
```
|
||||
Flask==2.1.1
|
||||
requests==2.32.2
|
||||
pytest==8.2.0
|
||||
pytest-mock==3.14.0
|
||||
flake8==7.0.0
|
||||
black==24.1.1
|
||||
mypy==1.8.0
|
||||
```
|
||||
|
||||
7. **`README.md`** - Complete rewrite:
|
||||
- Remove references to pymake, fission.json
|
||||
- Explain actual project structure
|
||||
- Document Fission configuration in docstrings
|
||||
- Show how to use deployment.json
|
||||
- Document environment variables (secrets/configmaps)
|
||||
- Explain testing approach
|
||||
- Add development workflow
|
||||
- Include examples from all three projects as inspiration
|
||||
|
||||
8. **`.fission/deployment.json`** - Make generic with placeholders:
|
||||
- Use `your-service-py` as environment name
|
||||
- Use `your-package` as package name
|
||||
- Use generic secret/configmap names: `fission-${PROJECT_NAME}-env`, `fission-${PROJECT_NAME}-config`
|
||||
- Show both `poolmgr` and `newdeploy` executor examples (commented)
|
||||
- Include optional fields like `imagepullsecret`, `runtime_envs`, `configmaps`
|
||||
|
||||
9. **`test/requirements.txt`** - Add:
|
||||
```
|
||||
pytest==8.2.0
|
||||
pytest-mock==3.14.0
|
||||
requests==2.32.3
|
||||
```
|
||||
|
||||
10. **`build.sh`** - Fix to use `${SRC_PKG}` properly (current version is correct)
|
||||
|
||||
### Phase 2: Documentation & Examples
|
||||
|
||||
**New Files to ADD:**
|
||||
|
||||
1. **`src/__init__.py`** - Already exists, keep as is
|
||||
|
||||
2. **`examples/` directory** (new) - Sample function implementations:
|
||||
- `example_crud.py` - Basic CRUD with Pydantic validation
|
||||
- `example_webhook.py` - Webhook receiver pattern
|
||||
- `example_scheduler.py` - Background job pattern (from ailbl-scheduler)
|
||||
- Each should have proper Fission docstring config
|
||||
|
||||
3. **`.env.example`** - Template showing all environment variables:
|
||||
```
|
||||
# PostgreSQL
|
||||
PG_HOST=
|
||||
PG_PORT=5432
|
||||
PG_DB=
|
||||
PG_USER=
|
||||
PG_PASS=
|
||||
PG_DBSCHEMA=
|
||||
|
||||
# Optional: Service-specific config (via ConfigMap)
|
||||
# YOUR_SERVICE_CONFIG_ENDPOINT=
|
||||
|
||||
# Optional: Vault encryption key (if using encrypted secrets)
|
||||
# CRYPTO_KEY=
|
||||
```
|
||||
|
||||
4. **`docs/` directory** (new) - Additional documentation:
|
||||
- `STRUCTURE.md` - Detailed file structure explanation
|
||||
- `TESTING.md` - How to write and run tests
|
||||
- `DEPLOYMENT.md` - Deployment options and tuning
|
||||
- `SECRETS.md` - Managing secrets and configmaps
|
||||
- `MIGRATIONS.md` - Database migration workflow
|
||||
|
||||
5. **`pytest.ini`** - Default pytest configuration:
|
||||
```ini
|
||||
[pytest]
|
||||
testpaths = test
|
||||
python_files = test_*.py
|
||||
python_classes = Test*
|
||||
python_functions = test_*
|
||||
log_cli = true
|
||||
log_cli_level = INFO
|
||||
```
|
||||
|
||||
6. **`.gitignore`** - Ensure it excludes:
|
||||
- `__pycache__/`
|
||||
- `*.pyc`
|
||||
- `.env`
|
||||
- `.venv/`
|
||||
- `venv/`
|
||||
- `.pytest_cache/`
|
||||
- `.mypy_cache/`
|
||||
- `.coverage`
|
||||
- `coverage.xml`
|
||||
- `specs/` (optional - generated files)
|
||||
|
||||
7. **`MANIFEST.md`** - Template for Kubernetes manifests (if not using auto-generated)
|
||||
|
||||
### Phase 3: Modernization
|
||||
|
||||
**Update CI/CD:**
|
||||
|
||||
Review `.gitea/workflows/` files:
|
||||
- Ensure they install dependencies correctly
|
||||
- Add linting (flake8/black) steps
|
||||
- Add test execution
|
||||
- Add deployment steps with proper environment detection
|
||||
- Consider adding security scanning
|
||||
|
||||
**Python Version:**
|
||||
|
||||
- Ensure all files are compatible with Python 3.11+
|
||||
- Update `build.sh` to use Python 3.11 image (like scheduler does) or keep generic
|
||||
- Consider adding `runtime.txt` or `pyproject.toml` to specify Python version
|
||||
|
||||
---
|
||||
|
||||
## Files to Modify Summary
|
||||
|
||||
**Direct modifications:**
|
||||
- `src/helpers.py` - Simplify, improve
|
||||
- `src/exceptions.py` - Make generic
|
||||
- `src/models.py` - Replace with generic patterns
|
||||
- `src/requirements.txt` - Add Flask, remove commented section
|
||||
- `dev-requirements.txt` - Comprehensive dev dependencies
|
||||
- `test/requirements.txt` - Test dependencies
|
||||
- `README.md` - Complete rewrite
|
||||
- `.fission/deployment.json` - Generic placeholders
|
||||
- `build.sh` - Already good, just ensure compatibility
|
||||
|
||||
**New files to add:**
|
||||
- `.env.example`
|
||||
- `pytest.ini`
|
||||
- `.gitignore` (enhance)
|
||||
- `examples/` directory with sample functions
|
||||
- `docs/` directory with detailed guides
|
||||
- `src/example_crud.py` (or in examples/)
|
||||
- `src/example_webhook.py` (or in examples/)
|
||||
- `src/example_scheduler.py` (or in examples/)
|
||||
|
||||
**New directories:**
|
||||
- `examples/`
|
||||
- `docs/`
|
||||
|
||||
---
|
||||
|
||||
## Implementation Approach
|
||||
|
||||
1. **Backup current template** (git branch)
|
||||
2. **Modify core files** in order: helpers → exceptions → models → requirements → deployment.json → README
|
||||
3. **Add new files** (examples, docs, configs)
|
||||
4. **Test the template**:
|
||||
- Run `create-project.sh` to generate a new project
|
||||
- Verify build.sh works
|
||||
- Run tests
|
||||
- Check Fission spec generation
|
||||
5. **Commit with clear message**
|
||||
6. **Update plugin documentation** if needed
|
||||
|
||||
---
|
||||
|
||||
## Verification Steps
|
||||
|
||||
After implementing the changes:
|
||||
|
||||
1. **Create a test project** from the updated template:
|
||||
```bash
|
||||
./create-project.sh test-project ./tmp-test/
|
||||
```
|
||||
2. **Inspect generated project**:
|
||||
- Verify all files are present
|
||||
- Check that placeholders are substituted correctly
|
||||
- Ensure imports work
|
||||
3. **Build the package**:
|
||||
```bash
|
||||
cd tmp-test
|
||||
./src/build.sh
|
||||
```
|
||||
4. **Run tests** (if any):
|
||||
```bash
|
||||
pip install -r dev-requirements.txt
|
||||
pytest
|
||||
```
|
||||
5. **Check syntax**:
|
||||
```bash
|
||||
python -m py_compile src/*.py
|
||||
flake8 src/
|
||||
black --check src/
|
||||
```
|
||||
6. **Validate Fission config**:
|
||||
```bash
|
||||
fission spec verify --file=.fission/deployment.json
|
||||
```
|
||||
7. **Review README** - Does it accurately describe the project?
|
||||
|
||||
---
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- Template is **generic**, not domain-specific
|
||||
- All examples' best practices are incorporated
|
||||
- Documentation is accurate and complete
|
||||
- Dependencies are correctly listed (Flask in requirements, not just dev)
|
||||
- README reflects actual Fission workflow (docstrings, not fission.yaml)
|
||||
- Multiple example implementations provided (CRUD, webhook, scheduler)
|
||||
- Secrets/configuration clearly explained
|
||||
- Testing setup is comprehensive
|
||||
- Project passes linting and type checks
|
||||
|
||||
---
|
||||
|
||||
## Risks & Mitigations
|
||||
|
||||
| Risk | Mitigation |
|
||||
|------|------------|
|
||||
| Breaking existing template users | Keep changes minimal in helpers; preserve backward compatibility where possible |
|
||||
| Over-engineering | Stick to patterns that appear in at least 2 of 3 examples |
|
||||
| Missing edge cases | Include optional advanced patterns (like DBConfig) in docs, not in core |
|
||||
| Documentation drift | Keep docs close to code; add examples that mirror real projects |
|
||||
|
||||
---
|
||||
|
||||
## Post-Implementation
|
||||
|
||||
After the template is updated:
|
||||
1. Consider creating a **template validation script** to ensure quality
|
||||
2. Update the **plugin SKILL.md** to reflect template changes
|
||||
3. Add **templating tests** to the fission-python-skill test suite
|
||||
4. Document the **update process** for future template modifications
|
||||
5. Consider **versioning** the template (e.g., `template-v2/`)
|
||||
32
.claude/plans/ethereal-giggling-sunrise.md
Normal file
32
.claude/plans/ethereal-giggling-sunrise.md
Normal file
@@ -0,0 +1,32 @@
|
||||
# Plan: Update marketplace.json
|
||||
|
||||
## Context
|
||||
The marketplace.json file currently has an empty plugins array. The goal is to register the existing `fission-python-skill` plugin in the marketplace by adding it to the plugins list. Owner information will remain unchanged.
|
||||
|
||||
## Current State
|
||||
- **File**: `.claude-plugin/marketplace.json`
|
||||
- **Current content**: `{ "name": "vega-claude-marketplace", "owner": {"name": "tiendd", "email": "fdm.dev17@gmail.com"}, "plugins": [] }`
|
||||
- **Plugin to add**: `fission-python-skill/.claude-plugin/plugin.json` contains:
|
||||
- id: "fission-python-skill"
|
||||
- name: "Fission Python Skill"
|
||||
- description: "Skill for creating, analyzing, and managing Fission Python projects."
|
||||
- type: "skill"
|
||||
- path: "fission-python-skill"
|
||||
- tools: ["create-project", "analyze-config", "update-docstring"]
|
||||
- version: "1.0.0"
|
||||
|
||||
## Implementation
|
||||
1. Read the current `plugin.json` from `fission-python-skill/.claude-plugin/` to extract plugin metadata
|
||||
2. Update `.claude-plugin/marketplace.json`:
|
||||
- Keep existing name and owner unchanged
|
||||
- Add a plugin object to the plugins array with the data from plugin.json
|
||||
|
||||
## Critical Files
|
||||
- `.claude-plugin/marketplace.json` (to be modified)
|
||||
- `fission-python-skill/.claude-plugin/plugin.json` (source of plugin data)
|
||||
|
||||
## Verification
|
||||
After modification, verify:
|
||||
1. The file contains valid JSON
|
||||
2. The plugins array contains the fission-python-skill object
|
||||
3. Owner information is unchanged
|
||||
182
.claude/plans/iridescent-meandering-blanket.md
Normal file
182
.claude/plans/iridescent-meandering-blanket.md
Normal file
@@ -0,0 +1,182 @@
|
||||
# README.md Generation Plan
|
||||
|
||||
## Context
|
||||
|
||||
The user requested to "review project and generate README.md file" for the Claude Marketplace repository. This repository contains a plugin ecosystem for Claude Code with two major components:
|
||||
|
||||
1. **Fission Python Skill** - A plugin for creating, analyzing, and managing Fission serverless Python projects
|
||||
2. **SDLC Agent System** - A complete multi-agent Software Development Life Cycle system for automated planning, architecture, coding, and code review
|
||||
|
||||
The repository is missing a root-level README.md file that documents these components, their usage, and how they work together.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The repository needs a comprehensive README.md at the root level that:
|
||||
|
||||
1. Introduces the Claude Marketplace project and its purpose
|
||||
2. Documents the Fission Python Skill plugin (tools, installation, usage)
|
||||
3. Documents the SDLC Agent System (agents, setup, workflow)
|
||||
4. Explains the project structure and key directories
|
||||
5. Provides quick start guides for both components
|
||||
6. Includes reference information (technologies, environment variables, common tasks)
|
||||
7. Links to existing detailed documentation (CLAUDE.md, agent docs, skill docs)
|
||||
|
||||
## Solution Approach
|
||||
|
||||
### What to Include
|
||||
|
||||
Based on repository analysis, the README should cover:
|
||||
|
||||
1. **Project Overview**
|
||||
- What is Claude Marketplace?
|
||||
- Key components (Fission Python Skill, SDLC Agents)
|
||||
- Relationship between components
|
||||
|
||||
2. **Fission Python Skill**
|
||||
- Purpose and use cases
|
||||
- Available tools (create-project, analyze-config, update-docstring)
|
||||
- Installation/setup (chmod +x on scripts)
|
||||
- Usage examples for each tool
|
||||
- Project structure
|
||||
- Links to detailed docs (SKILL.md, reference.md)
|
||||
|
||||
3. **SDLC Agent System**
|
||||
- Overview of the 7 agents (Initializer, Planning, Architect, Coding, Code Review, Curator, Retro)
|
||||
- Agent workflow and handoffs
|
||||
- Setup procedure (setup.sh script)
|
||||
- agent-context directory structure
|
||||
- Skills system (stack detection, patterns, frameworks)
|
||||
- Quality gates and harness scripts
|
||||
- Links to detailed agent docs
|
||||
|
||||
4. **Project Structure**
|
||||
- Directory layout with descriptions
|
||||
- Key configuration files
|
||||
- Template locations
|
||||
|
||||
5. **Quick Start**
|
||||
- Using Fission Python Skill to create a project
|
||||
- Setting up SDLC Agents in an existing project
|
||||
- Development environment (devcontainer)
|
||||
|
||||
6. **Development**
|
||||
- Making changes to skill scripts
|
||||
- Updating plugin metadata
|
||||
- Testing approaches
|
||||
|
||||
7. **Configuration**
|
||||
- Environment variables (for devcontainer)
|
||||
- Claude Code settings
|
||||
|
||||
8. **Related Documentation**
|
||||
- CLAUDE.md (comprehensive project guide)
|
||||
- Agent-specific documentation
|
||||
- Skill documentation
|
||||
|
||||
### Design Decisions
|
||||
|
||||
- **Structure**: Standard GitHub README with clear sections using markdown headings
|
||||
- **Tone**: Professional, concise, informative
|
||||
- **Format**: Single file at repository root
|
||||
- **Links**: Cross-reference existing documentation rather than duplicating content
|
||||
- **Code blocks**: Include practical examples for all commands
|
||||
- **Tables**: Use for quick reference (tools, skills, agents)
|
||||
|
||||
### Reuse Existing Content
|
||||
|
||||
- CLAUDE.md contains excellent detailed information - will summarize and link to it
|
||||
- Individual agent .md files have authoritative content - will link rather than copy
|
||||
- Skill files (SKILL.md) already have user-facing docs - will summarize and link
|
||||
- The template structure is documented in CLADE.md - will extract key info
|
||||
|
||||
## Implementation Steps
|
||||
|
||||
1. Create `/workspaces/claude-marketplace/README.md` with:
|
||||
|
||||
a. **Header section**
|
||||
- Badges (if applicable)
|
||||
- Title and subtitle
|
||||
- One-sentence description
|
||||
|
||||
b. **Table of Contents** (auto-generated with markdown-toc or manual)
|
||||
|
||||
c. **Project Overview**
|
||||
- Purpose of Claude Marketplace
|
||||
- Components summary
|
||||
- Key technologies
|
||||
|
||||
d. **Fission Python Skill section**
|
||||
- Description
|
||||
- Tools table
|
||||
- Installation
|
||||
- Usage examples
|
||||
- Project structure
|
||||
- Links to SKILL.md and reference.md
|
||||
|
||||
e. **SDLC Agent System section**
|
||||
- What are SDLC Agents?
|
||||
- The 7 agents with brief descriptions
|
||||
- Setup instructions
|
||||
- Agent-context structure
|
||||
- Skills system overview
|
||||
- Harness and quality gates
|
||||
- Links to detailed docs
|
||||
|
||||
f. **Project Structure section**
|
||||
- Directory tree visualization
|
||||
- Key files table
|
||||
|
||||
g. **Quick Start section**
|
||||
- Setting up dev environment
|
||||
- Creating Fission project
|
||||
- Initializing SDLC Agents
|
||||
|
||||
h. **Development section**
|
||||
- Modifying skills
|
||||
- Plugin registration
|
||||
- Testing
|
||||
|
||||
i. **Configuration section**
|
||||
- Environment variables table
|
||||
- Settings files
|
||||
|
||||
j. **License** (check if exists)
|
||||
|
||||
2. Ensure the README:
|
||||
- Is comprehensive but concise
|
||||
- Uses consistent formatting (h2 for major sections, h3 for subsections)
|
||||
- Includes practical examples with code blocks
|
||||
- Links to existing detailed documentation
|
||||
- Has a clear call-to-action for both components
|
||||
|
||||
3. Quality checks:
|
||||
- Verify all linked files exist
|
||||
- Ensure markdown renders properly (no broken syntax)
|
||||
- Check for consistency with CLAUDE.md
|
||||
|
||||
## Critical Files
|
||||
|
||||
- `/workspaces/claude-marketplace/README.md` - The file to create
|
||||
- `/workspaces/claude-marketplace/CLAUDE.md` - Source for detailed project information
|
||||
- `/workspaces/claude-marketplace/fission-python-skill/SKILL.md` - Skill documentation source
|
||||
- `/workspaces/claude-marketplace/.sdlc-agents/` - Agent documentation directory
|
||||
- `/workspaces/claude-marketplace/.sdlc-agents/setup.sh` - Agent setup script
|
||||
|
||||
## Verification
|
||||
|
||||
After generating the README.md:
|
||||
|
||||
1. Check markdown syntax (headings, lists, code blocks, tables)
|
||||
2. Verify all internal links point to existing files
|
||||
3. Ensure all referenced tools and scripts actually exist
|
||||
4. Confirm information consistency with source files
|
||||
5. Review for completeness: Does it answer "What is this repo?" and "How do I use it?"
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- README.md exists at repository root
|
||||
- Provides clear overview of both major components
|
||||
- Includes practical usage examples
|
||||
- Links to authoritative detailed documentation
|
||||
- Follows standard GitHub README conventions
|
||||
- New users can understand the project and get started
|
||||
421
.claude/plans/rosy-giggling-flurry.md
Normal file
421
.claude/plans/rosy-giggling-flurry.md
Normal file
@@ -0,0 +1,421 @@
|
||||
# Plan: Enhance Fission Python Projects with Exceptions, Pydantic Models, and Code Quality Improvements
|
||||
|
||||
## Context
|
||||
|
||||
Three Fission Python projects need systematic improvements to enhance error handling, data validation, and code maintainability:
|
||||
|
||||
- **py-eom-storage**: Storage management API (GET/POST /storages, GET/PUT/DELETE /storages/{id})
|
||||
- **py-eom-quota**: Quota management API (GET/POST /quotas, POST/DELETE /users/{userId}/quotas/{quotaId})
|
||||
- **py-ailbl-scheduler**: Background worker system for scheduled tasks
|
||||
|
||||
Currently, all projects use generic `Exception` with simple error messages returned as `{"error": str(err)}` with 500 status. There's no structured error handling, request validation, or consistent response formatting. Some projects have Pydantic models but not comprehensively used.
|
||||
|
||||
## Goals
|
||||
|
||||
1. **Custom Exceptions**: Implement domain-specific exception classes with:
|
||||
- `error_code`: Machine-readable error identifier
|
||||
- `http_status_code`: Appropriate HTTP status (400, 404, 409, 500, etc.)
|
||||
- `error_msg`: Human-readable message
|
||||
- `x_user`: User identifier from request header (X-Fission-Params-UserId or similar)
|
||||
|
||||
2. **Pydantic Models**: Add comprehensive request/response models for all endpoints:
|
||||
- Request body validation (POST/PUT)
|
||||
- Query parameter validation (GET)
|
||||
- Structured response schemas
|
||||
- Consistent error response format
|
||||
|
||||
3. **Code Quality**: Improve maintainability with:
|
||||
- Detailed docstrings for all functions and classes
|
||||
- Refactoring of complex, multi-responsibility functions
|
||||
- Consistent error handling patterns
|
||||
- Fix broken imports and type issues
|
||||
|
||||
## Project-Specific Plans
|
||||
|
||||
### 1. py-eom-storage
|
||||
|
||||
**Current State:**
|
||||
- Has Pydantic models: `S3Resource`, `S3Credential` (unused)
|
||||
- Uses dataclasses: `Page`, `Filter` (should be Pydantic)
|
||||
- Endpoints: `/eom/admin/storages` (filter_or_insert.py), `/eom/admin/storages/{StorageId}` (update_or_delete.py)
|
||||
|
||||
**Changes Needed:**
|
||||
|
||||
**A. Create `src/exceptions.py`:**
|
||||
```python
|
||||
class StorageException(Exception):
|
||||
"""Base exception for storage-related errors."""
|
||||
def __init__(self, error_code: str, http_status: int, error_msg: str, x_user: str = None):
|
||||
self.error_code = error_code
|
||||
self.http_status = http_status
|
||||
self.error_msg = error_msg
|
||||
self.x_user = x_user
|
||||
super().__init__(self.error_msg)
|
||||
|
||||
class ValidationError(StorageException):
|
||||
"""Invalid input data."""
|
||||
class NotFoundError(StorageException):
|
||||
"""Resource not found."""
|
||||
class ConflictError(StorageException):
|
||||
"""Resource conflict (e.g., duplicate name)."""
|
||||
class DatabaseError(StorageException):
|
||||
"""Database operation failed."""
|
||||
class S3ConnectionError(StorageException):
|
||||
"""S3/MinIO connection failed."""
|
||||
```
|
||||
|
||||
**B. Create/Update `src/models.py` (or extend existing):**
|
||||
```python
|
||||
# Request models
|
||||
class StorageCreateRequest(BaseModel):
|
||||
name: str = Field(..., min_length=1, max_length=255)
|
||||
description: typing.Optional[str] = None
|
||||
resource: dict # Should validate S3 structure
|
||||
|
||||
class StorageUpdateRequest(BaseModel):
|
||||
name: typing.Optional[str] = None
|
||||
description: typing.Optional[str] = None
|
||||
resource: typing.Optional[dict] = None
|
||||
active: typing.Optional[bool] = None
|
||||
|
||||
# Query models (convert Page/Filter to Pydantic)
|
||||
class StorageFilter(BaseModel):
|
||||
ids: typing.Optional[typing.List[str]] = None
|
||||
keyword: typing.Optional[str] = None
|
||||
collection_id: typing.Optional[str] = None
|
||||
enable: typing.Optional[bool] = None
|
||||
created_from: typing.Optional[datetime] = None
|
||||
created_to: typing.Optional[datetime] = None
|
||||
# ... other filters
|
||||
|
||||
class StorageQuery(BaseModel):
|
||||
page: int = 0
|
||||
size: int = Field(8, ge=1, le=100)
|
||||
asc: bool = True
|
||||
sortby: typing.Optional[Literal["name", "enable", "created", "modified"]] = None
|
||||
filter: StorageFilter = StorageFilter()
|
||||
|
||||
# Response models
|
||||
class StorageResponse(BaseModel):
|
||||
id: str
|
||||
name: str
|
||||
description: typing.Optional[str]
|
||||
resource: dict
|
||||
enable: bool
|
||||
created: datetime
|
||||
modified: datetime
|
||||
|
||||
class ErrorResponse(BaseModel):
|
||||
error_code: str
|
||||
http_status: int
|
||||
error_msg: str
|
||||
x_user: typing.Optional[str] = None
|
||||
details: typing.Optional[dict] = None
|
||||
```
|
||||
|
||||
**C. Refactor `filter_or_insert.py`:**
|
||||
- Replace try-except to catch custom exceptions
|
||||
- Validate request body using Pydantic in `make_insert_request`
|
||||
- Use Pydantic for query parsing in `make_filter_request`
|
||||
- Add helper function `handle_exception` to format error responses consistently
|
||||
- Extract SQL queries into separate functions for testability
|
||||
- Add comprehensive docstrings explaining each endpoint's behavior
|
||||
|
||||
**D. Refactor `update_or_delete.py`:**
|
||||
- Similar pattern: custom exceptions, Pydantic validation
|
||||
- Refactor `is_depended_on_storage` - this function does too much, split into smaller helpers
|
||||
- Add detailed comments for each database operation
|
||||
- Ensure proper error messages with appropriate HTTP status codes
|
||||
|
||||
**E. Update `helpers.py`:**
|
||||
- Add utility `get_user_from_header(request)` to extract x-user from various headers
|
||||
|
||||
---
|
||||
|
||||
### 2. py-eom-quota
|
||||
|
||||
**Current State:**
|
||||
- Already has extensive Pydantic models in `models.py` (QuotaPage, UserQuotaPage, ScheduleCreate, etc.)
|
||||
- But: `userquota_filter.py` imports from `quota_update_or_delete` which doesn't exist (broken import)
|
||||
- Need to expand models to cover all request/response scenarios
|
||||
- Endpoints: `/eom/admin/quotas` (filter), `/eom/admin/users/{UserId}/quotas` (filter/insert), `/eom/admin/users/{UserId}/quotas/{QuotaId}` (update/delete)
|
||||
|
||||
**Changes Needed:**
|
||||
|
||||
**A. Create `src/exceptions.py`:**
|
||||
```python
|
||||
class QuotaException(Exception):
|
||||
"""Base exception for quota management."""
|
||||
def __init__(self, error_code: str, http_status: int, error_msg: str, x_user: str = None):
|
||||
self.error_code = error_code
|
||||
self.http_status = http_status
|
||||
self.error_msg = error_msg
|
||||
self.x_user = x_user
|
||||
super().__init__(self.error_msg)
|
||||
|
||||
class QuotaNotFoundError(QuotaException):
|
||||
"""Quota does not exist."""
|
||||
class UserQuotaConflictError(QuotaException):
|
||||
"""User already has this type of quota."""
|
||||
class ValidationError(QuotaException):
|
||||
"""Invalid request data."""
|
||||
class DatabaseError(QuotaException):
|
||||
"""Database operation failed."""
|
||||
```
|
||||
|
||||
**B. Extend `src/models.py`:**
|
||||
The existing models mix schedule and quota models. Need to:
|
||||
- Separate or clearly document which are for quotas vs schedules
|
||||
- Add request models:
|
||||
```python
|
||||
class QuotaCreateRequest(BaseModel):
|
||||
name: str
|
||||
description: typing.Optional[str] = None
|
||||
type: QuotaType
|
||||
value: typing.Union[MaxSizeBody, MaxOrderTimesBody]
|
||||
expire: ExpireBody
|
||||
|
||||
class QuotaUpdateRequest(BaseModel):
|
||||
name: typing.Optional[str] = None
|
||||
description: typing.Optional[str] = None
|
||||
enable: typing.Optional[bool] = None
|
||||
type: typing.Optional[QuotaType] = None
|
||||
value: typing.Optional[typing.Union[MaxSizeBody, MaxOrderTimesBody]] = None
|
||||
expire: typing.Optional[ExpireBody] = None
|
||||
|
||||
class UserQuotaAssignRequest(BaseModel):
|
||||
quota_id: str
|
||||
```
|
||||
|
||||
- Ensure response models exist (QuotaResponse, UserQuotaResponse)
|
||||
|
||||
**C. Fix `userquota_filter.py`:**
|
||||
- Fix broken import: `from quota_update_or_delete import __get_by_id` → `from userquota_insert_or_delete import __get_by_id` (or better: move `__get_by_id` to a shared helpers module)
|
||||
- Refactor `make_filter_request`:
|
||||
- Use `UserQuotaPage` Pydantic model properly
|
||||
- Validate user_id header is present using Pydantic
|
||||
- Replace try-except with custom exceptions
|
||||
- Add comprehensive docstring
|
||||
- The function currently manually sets `paging.filter.user_ids = [user_id]` - this should be part of a validation layer
|
||||
|
||||
**D. Refactor `userquota_insert_or_delete.py`:**
|
||||
- Fix the same broken import pattern (it imports nothing but uses `__get_by_id` in filter)
|
||||
- Add proper request validation using Pydantic models
|
||||
- Replace generic exceptions with `UserQuotaConflictError`, `QuotaNotFoundError`, etc.
|
||||
- Refactor `__validate_user_quota_type` - currently SQL query is hardcoded, add comments explaining business logic
|
||||
- The insert SQL has wrong columns: `INSERT INTO eom_user_quota(id, name, description, type, value, expire)` but the table likely only has (id, user_id, quota_id). Need to check database schema but from the code it seems mismatched.
|
||||
|
||||
**E. Improve `helpers.py`:**
|
||||
- Add utility functions for extracting and validating user headers
|
||||
- Add consistent error handling helpers
|
||||
|
||||
---
|
||||
|
||||
### 3. py-ailbl-scheduler
|
||||
|
||||
**Current State:**
|
||||
- No HTTP endpoints (only time-triggered workers)
|
||||
- No Pydantic models needed per user's choice
|
||||
- Needs custom exceptions and code quality improvements
|
||||
- Workers: `worker_session_picker.py`, `worker_session_poller.py`, `worker_scheduler_scan.py`, `worker_schedule_auto_disable.py`
|
||||
- Common utilities in `common.py`, `helpers.py`
|
||||
|
||||
**Changes Needed:**
|
||||
|
||||
**A. Create `src/exceptions.py`:**
|
||||
```python
|
||||
class SchedulerException(Exception):
|
||||
"""Base exception for scheduler operations."""
|
||||
def __init__(self, error_code: str, error_msg: str, details: dict = None):
|
||||
self.error_code = error_code
|
||||
self.error_msg = error_msg
|
||||
self.details = details
|
||||
super().__init__(self.error_msg)
|
||||
|
||||
class ScheduleNotFoundError(SchedulerException):
|
||||
"""Schedule does not exist."""
|
||||
class SessionLockError(SchedulerError):
|
||||
"""Failed to acquire session lock."""
|
||||
class DagsterError(SchedulerError):
|
||||
"""Dagster pipeline execution failed."""
|
||||
class CronParseError(SchedulerError):
|
||||
"""Invalid cron expression."""
|
||||
class ConfigurationError(SchedulerError):
|
||||
"""Missing or invalid configuration."""
|
||||
```
|
||||
|
||||
**B. Refactor `worker_scheduler_scan.py`:**
|
||||
This is the most complex function (446 lines). Goals:
|
||||
- Extract helper functions:
|
||||
- `_normalize_cron_for_cronner` (already exists)
|
||||
- `_as_date`, `_as_time` (already exist)
|
||||
- `_within_active_window` (already exists)
|
||||
- `_is_due_by_cron` (already exists)
|
||||
- `_is_due_by_freq` (already exists)
|
||||
- Extract the schedule creation logic into `_create_session_for_schedule(cur, schedule, now, slot_start)`
|
||||
- Extract the candidate schedule selection into `_fetch_due_schedules(cur, now, slot_start, slot_end, limit=50)`
|
||||
- Add detailed docstrings explaining the overall algorithm: "Scan for schedules that are due in the current time slot and create sessions atomically"
|
||||
- Improve variable names (e.g., `s` → `schedule`, `cur` → `cursor`)
|
||||
- Add comments explaining the advisory lock strategy and why it's needed
|
||||
- Ensure proper exception handling with custom exceptions
|
||||
- The function currently catches generic Exception at the end - wrap specific operations with appropriate custom exceptions
|
||||
|
||||
**C. Refactor `worker_session_picker.py`:**
|
||||
- Similar breakdown: extract `_pick_and_claim_sessions(conn, limit=20)` helper
|
||||
- Extract `_process_kind5_session(session, ctx)` and `_process_kind1_session(session, ctx)` into separate functions
|
||||
- Add detailed docstring explaining the picking strategy (FOR UPDATE SKIP LOCKED)
|
||||
- Replace bare `except Exception` with specific exception types
|
||||
- Add comments explaining the kind handling logic (kind 5 vs kind 1)
|
||||
- The function `_build_run_config_kind5` is specific to that kind - could be moved to a separate module if needed
|
||||
|
||||
**D. Refactor `worker_session_poller.py`:**
|
||||
- Extract `_update_completed_session(cur, session_id, status_info, now)` helper
|
||||
- Extract `_update_started_session(cur, session_id, started_dt)` helper
|
||||
- Add docstring explaining polling strategy
|
||||
- Replace generic exception handling with `DagsterError` when Dagster calls fail
|
||||
- Add type hints for the row unpacking: `for sid, run_id, started, cron_description, created_by in rows:`
|
||||
|
||||
**E. Refactor `worker_schedule_auto_disable.py`:**
|
||||
- This is simple enough already but still add comprehensive docstring
|
||||
- Consider adding custom exception for database errors
|
||||
|
||||
**F. Improve `helpers.py` (in scheduler):**
|
||||
- The `GraphQL` class and related functions are specific to Dagster - add docstrings
|
||||
- `safe_notify` is good, add docstring
|
||||
- Consider creating a `SchedulerHelper` class to group related utilities
|
||||
|
||||
**G. Improve `common.py`:**
|
||||
- Already has good docstrings but could be expanded
|
||||
- Add type hints to function signatures
|
||||
- Break `launch_pipeline_execution` if too complex (handles multiple error cases)
|
||||
|
||||
---
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Exception Hierarchy
|
||||
|
||||
Each project will have:
|
||||
```python
|
||||
class BaseProjectException(Exception):
|
||||
"""Base with error_code, http_status (if applicable), message, metadata."""
|
||||
pass
|
||||
|
||||
# Specific exceptions inherit from base
|
||||
class NotFoundError(BaseProjectException): ...
|
||||
class ValidationError(BaseProjectException): ...
|
||||
class ConflictError(BaseProjectException): ...
|
||||
class DatabaseError(BaseProjectException): ...
|
||||
# Domain-specific: StorageNotFoundError, QuotaConflictError, ScheduleNotFoundError, etc.
|
||||
```
|
||||
|
||||
### Error Response Format
|
||||
|
||||
Standardized JSON response:
|
||||
```json
|
||||
{
|
||||
"error_code": "STORAGE_NOT_FOUND",
|
||||
"http_status": 404,
|
||||
"error_msg": "Storage with id 'xyz' does not exist",
|
||||
"x_user": "user123",
|
||||
"details": { /* optional additional context */ }
|
||||
}
|
||||
```
|
||||
|
||||
### Middleware Pattern
|
||||
|
||||
In each HTTP endpoint function:
|
||||
```python
|
||||
def main():
|
||||
try:
|
||||
# Extract user header
|
||||
x_user = request.headers.get("X-Fission-Params-UserId")
|
||||
# Route to handler
|
||||
return handler()
|
||||
except ValidationError as e:
|
||||
return error_response(e), 400
|
||||
except NotFoundError as e:
|
||||
return error_response(e), 404
|
||||
except ConflictError as e:
|
||||
return error_response(e), 409
|
||||
except StorageException as e:
|
||||
logger.error(f"Storage error: {e.error_code}: {e.error_msg}")
|
||||
return error_response(e), 500
|
||||
except Exception as e:
|
||||
logger.exception("Unexpected error")
|
||||
return {"error": "Internal server error"}, 500
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Implementation Order
|
||||
|
||||
1. **Phase 1**: Create exception modules for all three projects
|
||||
2. **Phase 2**: Add/expand Pydantic models (storage, then complete quota)
|
||||
3. **Phase 3**: Refactor endpoints to use exceptions and models
|
||||
4. **Phase 4**: Refactor complex functions in scheduler
|
||||
5. **Phase 5**: Documentation pass - ensure all functions have docstrings
|
||||
6. **Phase 6**: Test manually by running functions (no automated tests to update)
|
||||
|
||||
---
|
||||
|
||||
## Verification Steps
|
||||
|
||||
1. **Manual Testing**:
|
||||
- Deploy each function to local Fission or use test environment
|
||||
- Test error cases: invalid input, missing resources, database failures
|
||||
- Verify error response format matches specification
|
||||
- Check logs for proper error logging
|
||||
|
||||
2. **Code Review**:
|
||||
- All functions have docstrings with Args, Returns, Raises sections
|
||||
- No function exceeds ~50 lines (extracted helpers where needed)
|
||||
- All exceptions are specific, not generic `Exception`
|
||||
- Request validation happens before business logic
|
||||
|
||||
3. **Import Verification**:
|
||||
- Fix broken imports (especially in py-eom-quota's userquota_filter.py)
|
||||
- Ensure circular dependencies are avoided
|
||||
|
||||
4. **Type Safety**:
|
||||
- Run static type checker if available (mypy/pyright)
|
||||
- Ensure all functions have return type hints
|
||||
|
||||
---
|
||||
|
||||
## Critical Files to Modify
|
||||
|
||||
**py-eom-storage:**
|
||||
- `src/exceptions.py` (new)
|
||||
- `src/models.py` (create/extend)
|
||||
- `src/filter_or_insert.py` (refactor)
|
||||
- `src/update_or_delete.py` (refactor)
|
||||
- `src/helpers.py` (add utilities)
|
||||
- `src/vault.py` (minor: improve docs)
|
||||
|
||||
**py-eom-quota:**
|
||||
- `src/exceptions.py` (new)
|
||||
- `src/models.py` (extend with request models)
|
||||
- `src/userquota_filter.py` (fix imports, refactor)
|
||||
- `src/userquota_insert_or_delete.py` (refactor, fix SQL if needed)
|
||||
- `src/helpers.py` (add utilities)
|
||||
|
||||
**py-ailbl-scheduler:**
|
||||
- `src/exceptions.py` (new)
|
||||
- `src/worker_scheduler_scan.py` (major refactor)
|
||||
- `src/worker_session_picker.py` (refactor)
|
||||
- `src/worker_session_poller.py` (refactor)
|
||||
- `src/worker_schedule_auto_disable.py` (docs)
|
||||
- `src/common.py` (docs, type hints)
|
||||
- `src/helpers.py` (docs, maybe extract class)
|
||||
|
||||
---
|
||||
|
||||
## Notes
|
||||
|
||||
- All changes are in `/workspaces/claude-marketplace/data/examples/`
|
||||
- Preserve existing API contracts (URLs, HTTP methods)
|
||||
- Do not change database schema
|
||||
- Maintain backward compatibility with existing clients
|
||||
- Focus on internal improvements: error handling, validation, documentation
|
||||
- Use consistent patterns across all three projects
|
||||
72
.claude/plans/snappy-launching-pony.md
Normal file
72
.claude/plans/snappy-launching-pony.md
Normal file
@@ -0,0 +1,72 @@
|
||||
# Fission Python Skill Plan
|
||||
|
||||
## Context
|
||||
The user wanted to create a new skill called `fission-python-skill` in the `@fission-plugin/skills` directory. This skill should provide three main capabilities:
|
||||
1. Create a new fission python project with template (based on @data/py-eom-quota)
|
||||
2. Analyze configuration in .fission of each fission-python project
|
||||
3. Parse and update docstring of fission function method
|
||||
|
||||
## Approach
|
||||
Based on my exploration of the codebase:
|
||||
- The example project @data/py-eom-quota shows a standard fission python project structure
|
||||
- Fission configuration is stored in .fission/deployment.json and similar files
|
||||
- Python functions contain fission configuration in their docstrings using a specific format (between ```fission and ``` markers)
|
||||
- The fission-plugin/skills directory currently contained empty SKILL.md and reference.md files
|
||||
|
||||
I implemented all three requested capabilities as shell scripts within the skill directory.
|
||||
|
||||
## Implementation Summary
|
||||
|
||||
### Phase 1: Skill Creation Completed
|
||||
1. Created the skill directory: `/workspaces/claude-marketplace/fission-plugin/skills/fission-python-skill/`
|
||||
2. Created SKILL.md following the skill format from the documentation
|
||||
3. Created reference.md with detailed usage instructions
|
||||
4. Implemented the three core tools as shell scripts:
|
||||
- `create-project.sh`: Creates a new fission python project from template
|
||||
- `analyze-config.sh`: Analyzes .fission configuration in a project
|
||||
- `update-docstring.sh`: Parses and updates docstrings in fission function methods
|
||||
|
||||
### Phase 2: Tool Implementation Details (Completed)
|
||||
|
||||
#### create-project.sh
|
||||
- Takes project name and optional destination directory
|
||||
- Copies template from @data/py-eom-quota (excluding .git, etc.)
|
||||
- Replaces placeholder values in configuration files
|
||||
- Provides usage instructions for next steps
|
||||
|
||||
#### analyze-config.sh
|
||||
- Takes path to a fission project
|
||||
- Reads and parses .fission/deployment.json (and related files)
|
||||
- Outputs structured summary of:
|
||||
- Environments and their resource settings
|
||||
- Packages and build commands
|
||||
- Functions and their triggers
|
||||
- Secrets and configmaps
|
||||
- Archives and source configuration
|
||||
|
||||
#### update-docstring.sh
|
||||
- Takes path to a python file and optionally function name
|
||||
- Parses docstrings to extract embedded fission configuration (between ```fission markers)
|
||||
- Allows updating the fission configuration within docstrings using --set flag
|
||||
- Can retrieve current configuration using --get flag (default)
|
||||
- Preserves existing function code and documentation outside fission blocks
|
||||
- Uses Python script for robust JSON handling and string manipulation
|
||||
|
||||
### Phase 3: Testing (Completed)
|
||||
- Tested create-project.sh by generating a new project and verifying structure
|
||||
- Tested analyze-config.sh on the existing @data/py-eom-quota project
|
||||
- Tested update-docstring.sh by retrieving and modifying fission configuration in function docstrings
|
||||
- All tools have proper help text and error handling
|
||||
|
||||
## Files Created
|
||||
- `/workspaces/claude-marketplace/fission-python-skill/SKILL.md`
|
||||
- `/workspaces/claude-marketplace/fission-python-skill/reference.md`
|
||||
- `/workspaces/claude-marketplace/fission-python-skill/create-project.sh`
|
||||
- `/workspaces/claude-marketplace/fission-python-skill/analyze-config.sh`
|
||||
- `/workspaces/claude-marketplace/fission-python-skill/update-docstring.sh`
|
||||
|
||||
## Verification Results
|
||||
✓ create-project.sh: Successfully creates new fission python projects from template
|
||||
✓ analyze-config.sh: Successfully analyzes .fission configuration showing environments, packages, functions, secrets, etc.
|
||||
✓ update-docstring.sh: Successfully extracts and updates fission configuration in function docstrings
|
||||
All tools are executable and include proper error handling and usage instructions.
|
||||
156
.claude/plans/update-fission-skill.md
Normal file
156
.claude/plans/update-fission-skill.md
Normal file
@@ -0,0 +1,156 @@
|
||||
# Plan: Update FissionPython Skill
|
||||
|
||||
## Context
|
||||
|
||||
The `fission-python-skill` plugin needs to be updated to meet new requirements for Fission Python projects:
|
||||
|
||||
1. **Build script**: `src/build.sh` must exist and be referenced correctly in `.fission/deployment.json`
|
||||
2. **Dependencies**: `src/requirements.txt` must exist and contain necessary packages (pydantic, etc.)
|
||||
3. **CI/CD**: All projects must include `.gitea/workflows/` directory with deployment workflows
|
||||
4. **API Design**: HTTP trigger functions must use Pydantic models for request/response validation
|
||||
5. **Documentation**: All functions must have proper docstrings and code comments
|
||||
6. **Portability**: Remove hardcoded absolute paths - the plugin should work from any location
|
||||
|
||||
Current issues:
|
||||
- `create-project.sh` uses hardcoded path `/workspaces/claude-marketplace/data/py-eom-quota`
|
||||
- Template resides in `data/examples/` which is outside the plugin
|
||||
- No validation of generated projects
|
||||
- Documentation references incorrect paths
|
||||
|
||||
## Approach
|
||||
|
||||
**Step 1: Make Plugin Portable**
|
||||
- Copy the `py-eom-quota` template into `fission-python-skill/template/`
|
||||
- Update `create-project.sh` to find template relative to script location using `dirname "$0"`
|
||||
- Remove all absolute path references
|
||||
|
||||
**Step 2: Add Project Validation**
|
||||
Enhance `create-project.sh` with post-creation validation:
|
||||
- Check `src/build.sh` exists and is executable
|
||||
- Verify `.fission/deployment.json` references the correct build command (`./build.sh`)
|
||||
- Check `src/requirements.txt` exists and contains required dependencies:
|
||||
- `pydantic==2.x`
|
||||
- `flask` (for HTTP handlers)
|
||||
- `psycopg2-binary` or `psycopg2` (if database usage)
|
||||
- Verify `.gitea/workflows/` directory exists with the 4 standard workflow files
|
||||
- Validate that function files contain pydantic models (basic grep check)
|
||||
- Warn if docstrings appear minimal or missing
|
||||
|
||||
**Step 3: Update Documentation**
|
||||
- Fix `SKILL.md` and `reference.md` to reference the correct template path
|
||||
- Document the new validation checks
|
||||
- Update examples to show portable usage
|
||||
|
||||
**Step 4: Potentially Add New Tool**
|
||||
Consider adding a separate validation tool (`validate-project.sh`) that can be run on existing projects to check compliance with standards.
|
||||
|
||||
## Critical Files to Modify
|
||||
|
||||
1. **`fission-python-skill/create-project.sh`**
|
||||
- Change `TEMPLATE_DIR` to use relative path: `$(dirname "$0")/template`
|
||||
- Add validation functions after project creation
|
||||
- Improve error messages
|
||||
- Add warnings for missing documentation
|
||||
|
||||
2. **`fission-python-skill/template/`** (new directory)
|
||||
- Copy entire structure from `data/examples/py-eom-quota/`
|
||||
- Ensure `build.sh` has correct permissions (755)
|
||||
- Verify all configuration files
|
||||
|
||||
3. **`fission-python-skill/SKILL.md`** and **`reference.md`**
|
||||
- Update template path references
|
||||
- Document validation behavior
|
||||
- Update examples
|
||||
|
||||
4. **`.claude-plugin/marketplace.json`**
|
||||
- No changes needed (plugin registration OK)
|
||||
|
||||
## Implementation Details
|
||||
|
||||
### Template Structure
|
||||
```
|
||||
fission-python-skill/
|
||||
└── template/
|
||||
├── .fission/
|
||||
│ ├── deployment.json
|
||||
│ ├── dev-deployment.json
|
||||
│ └── local-deployment.json
|
||||
├── .gitea/
|
||||
│ └── workflows/
|
||||
│ ├── dev-deployment.yaml
|
||||
│ ├── install-dispatch.yaml
|
||||
│ ├── uninstall-dispatch.yaml
|
||||
│ └── analystic-dispatch.yaml
|
||||
├── src/
|
||||
│ ├── build.sh (executable)
|
||||
│ ├── requirements.txt (with pydantic, flask, etc.)
|
||||
│ ├── models.py (with pydantic models)
|
||||
│ ├── exceptions.py
|
||||
│ ├── helpers.py
|
||||
│ ├── vault.py
|
||||
│ └── <example functions>.py (with docstrings and pydantic usage)
|
||||
├── test/
|
||||
├── manifests/
|
||||
├── migrates/
|
||||
├── specs/
|
||||
├── dev-requirements.txt
|
||||
├── README.md
|
||||
├── .gitignore
|
||||
└── .devcontainer/
|
||||
```
|
||||
|
||||
### Validation Checklist in create-project.sh
|
||||
After copying template and doing substitutions:
|
||||
1. `[ -f "$PROJECT_PATH/src/build.sh" ]` || warning
|
||||
2. `[ -x "$PROJECT_PATH/src/build.sh" ]` || chmod +x
|
||||
3. Check `deployment.json` contains `"./build.sh"` in packages.buildcmd
|
||||
4. `[ -f "$PROJECT_PATH/src/requirements.txt" ]` || error
|
||||
5. Check requirements.txt contains `pydantic` (grep -q "pydantic")
|
||||
6. Check requirements.txt contains `flask` (grep -q "flask")
|
||||
7. `[ -d "$PROJECT_PATH/.gitea/workflows" ]` || warning/copy from template
|
||||
8. Count workflow files: should have at least 4 .yaml files
|
||||
9. Optional: Check that Python files have docstrings (grep for triple quotes)
|
||||
10. Optional: Check for pydantic BaseModel usage in models.py
|
||||
|
||||
### Portable Path Resolution
|
||||
```bash
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
TEMPLATE_DIR="$SCRIPT_DIR/template"
|
||||
```
|
||||
|
||||
This ensures the plugin works regardless of where it's invoked from.
|
||||
|
||||
## Verification Steps
|
||||
|
||||
1. **Test create-project**:
|
||||
- Run `./fission-python-skill/create-project.sh test-project ./tmp/`
|
||||
- Verify all expected directories/files exist
|
||||
- Check that validation warnings/errors appear appropriately
|
||||
|
||||
2. **Test portability**:
|
||||
- Move plugin to a different directory
|
||||
- Run create-project from there
|
||||
- Should still work without path adjustments
|
||||
|
||||
3. **Test validation**:
|
||||
- Manually delete `src/requirements.txt` from template and create project → should error
|
||||
- Remove pydantic from requirements.txt → should warn
|
||||
- Remove .gitea/workflows → should warn
|
||||
- Change build.sh buildcmd to something else → should warn
|
||||
|
||||
4. **Test generated project**:
|
||||
- Verify functions have docstrings with fission config blocks
|
||||
- Verify models.py uses pydantic BaseModel
|
||||
- Verify HTTP triggers properly defined in deployment.json
|
||||
|
||||
## Risks and Considerations
|
||||
|
||||
- **Template duplication**: Moving template into plugin duplicates existing examples. That's acceptable - the examples in `data/examples/` are finished projects, while the template is a starter. Keep both.
|
||||
- **Validation strictness**: Start with warnings for most checks, errors only for critical missing files (requirements.txt build.sh). Can tighten later.
|
||||
- **Template maintenance**: When updating the template, only modify `fission-python-skill/template/`. The examples in `data/examples/` are independent and can diverge if needed.
|
||||
|
||||
## Post-Implementation
|
||||
|
||||
- Update any scripts or docs that reference the old template path
|
||||
- Test the skill end-to-end through Claude Code
|
||||
- Consider adding a `validate-project.sh` tool for existing projects
|
||||
6
.claude/settings.json
Normal file
6
.claude/settings.json
Normal file
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"permissions" : {"read": true, "write": true, "execute": true},
|
||||
"enabledPlugins" : {"engineering-skills@claude-code-skills": true, "skill-creator@daymade-skills": true},
|
||||
"plansDirectory" : "/workspaces/claude-marketplace/.claude/plans",
|
||||
"bypassPermissions": true
|
||||
}
|
||||
59
.devcontainer/devcontainer.json
Normal file
59
.devcontainer/devcontainer.json
Normal file
@@ -0,0 +1,59 @@
|
||||
{
|
||||
"customizations" : {
|
||||
"vscode": {
|
||||
"extensions": [
|
||||
// VS Code specific
|
||||
"ms-azuretools.vscode-docker",
|
||||
"dbaeumer.vscode-eslint",
|
||||
"j-brooke.fracturedjsonvsc",
|
||||
// Python specific
|
||||
"ms-python.python",
|
||||
"charliermarsh.ruff",
|
||||
// Markdown specific
|
||||
"yzhang.markdown-all-in-one",
|
||||
// JSON formatter
|
||||
"j-brooke.fracturedjsonvsc",
|
||||
// YAML formatter
|
||||
"kennylong.kubernetes-yaml-formatter",
|
||||
"Continue.continue" // AI
|
||||
],
|
||||
"settings" : {
|
||||
"diffEditor.renderSideBySide": true,
|
||||
"editor.suggestSelection" : "first",
|
||||
"editor.tabSize" : 4,
|
||||
"editor.wordWrap" : "off",
|
||||
"editor.wordWrapColumn" : 200,
|
||||
"explorer.confirmDelete" : false,
|
||||
"explorer.confirmDragAndDrop": false,
|
||||
"files.exclude" : {
|
||||
"**/.classpath" : true,
|
||||
"**/.DS_Store" : true,
|
||||
"**/.factorypath": true,
|
||||
"**/.git" : true,
|
||||
"**/.project" : true,
|
||||
"**/.settings" : true,
|
||||
"**/*.js" : {"when": "$(basename).ts"},
|
||||
"**/*.js.map" : true
|
||||
},
|
||||
"ansible.validation.enabled" : false,
|
||||
"telemetry.telemetryLevel" : "off"
|
||||
}
|
||||
}
|
||||
},
|
||||
"forwardPorts" : [],
|
||||
"dockerComposeFile": ["docker-compose.yaml"],
|
||||
"service" : "devcontainer",
|
||||
"workspaceFolder" : "/workspaces/${localWorkspaceFolderBasename}",
|
||||
"mounts" : [
|
||||
// "source=${localEnv:HOME}/.claude,target=/home/vscode/.claude,type=bind",
|
||||
"source=${localEnv:HOME}/Workspaces/self/sdlc-agents/agents,target=/workspaces/${localWorkspaceFolderBasename}/.sdlc-agents,type=bind"
|
||||
],
|
||||
"containerEnv" : {
|
||||
"ANTHROPIC_API_KEY" : "",
|
||||
"ANTHROPIC_BASE_URL" : "https://openrouter.ai/api",
|
||||
// "ANTHROPIC_AUTH_TOKEN" : "${localEnv:OPENROUTER_API_KEY}",
|
||||
"ANTHROPIC_MODEL" : "stepfun/step-3.5-flash:free",
|
||||
"ANTHROPIC_SMALL_FAST_MODEL": "nvidia/nemotron-3-super-120b-a12b:free"
|
||||
},
|
||||
"postStartCommand" : "/workspaces/${localWorkspaceFolderBasename}/.devcontainer/setup.sh"
|
||||
}
|
||||
8
.devcontainer/docker-compose.yaml
Normal file
8
.devcontainer/docker-compose.yaml
Normal file
@@ -0,0 +1,8 @@
|
||||
services:
|
||||
devcontainer:
|
||||
image: mcr.microsoft.com/vscode/devcontainers/python:3.11-bullseye
|
||||
volumes:
|
||||
- ../..:/workspaces:cached
|
||||
command: sleep infinity
|
||||
env_file:
|
||||
- .env
|
||||
2
.devcontainer/example.env
Normal file
2
.devcontainer/example.env
Normal file
@@ -0,0 +1,2 @@
|
||||
OPENROUTER_API_KEY=
|
||||
ANTHROPIC_AUTH_TOKEN=$OPENROUTER_API_KEY
|
||||
20
.devcontainer/setup.sh
Executable file
20
.devcontainer/setup.sh
Executable file
@@ -0,0 +1,20 @@
|
||||
#!/bin/bash
|
||||
|
||||
# For debugging
|
||||
# set -eux
|
||||
|
||||
|
||||
sudo apt update
|
||||
sudo apt install -y tmux
|
||||
|
||||
|
||||
# install claude code
|
||||
curl -fsSL https://claude.ai/install.sh | bash
|
||||
|
||||
# install claude plugin
|
||||
claude plugin marketplace add https://github.com/daymade/claude-code-skills
|
||||
claude plugin marketplace add https://github.com/alirezarezvani/claude-skills
|
||||
|
||||
# Marketplace name: daymade-skills (from marketplace.json)
|
||||
# claude plugin install skill-creator@daymade-skills
|
||||
# claude plugin install engineering-skills@claude-code-skills
|
||||
5
.gitignore
vendored
Normal file
5
.gitignore
vendored
Normal file
@@ -0,0 +1,5 @@
|
||||
/.sdlc-agents
|
||||
/.devcontainer/.env
|
||||
/.vscode
|
||||
/.claude/settings.local.json
|
||||
/data
|
||||
121
CLAUDE.md
Normal file
121
CLAUDE.md
Normal file
@@ -0,0 +1,121 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Repository Overview
|
||||
|
||||
This is a **Claude Marketplace** repository — a plugin ecosystem for Claude Code. It contains:
|
||||
|
||||
- **Plugin Registry** (`.claude-plugin/marketplace.json`): Defines available plugins/skills
|
||||
- **Fission Python Skill** (`fission-python/`): Plugin providing tools for Fission serverless Python projects
|
||||
- **Example Projects** (`data/examples/`): Example Fission Python projects (py-eom-quota, py-eom-storage, py-ailbl-scheduler)
|
||||
- **Fission Python Template** (`fission-python/template/`): Default template used when creating new Fission Python projects
|
||||
|
||||
> **Note:** `.sdlc-agents/` is **not** part of this repo. It is bind-mounted into the devcontainer from `~/Workspaces/self/sdlc-agents/agents` on the host.
|
||||
|
||||
## Plugin System Architecture
|
||||
|
||||
Claude Code discovers plugins via `.claude-plugin/marketplace.json` at the repository root. Each plugin entry:
|
||||
- Has an `id`, `name`, `description`, `type` (`skill`), `path`, and `tools` array
|
||||
- Points to a directory containing the tool implementations (executable `.sh` scripts)
|
||||
|
||||
The `fission-python` plugin (`fission-python/.claude-plugin/plugin.json`) exposes three tools:
|
||||
- `create-project` — `create-project.sh`
|
||||
- `analyze-config` — `analyze-config.sh`
|
||||
- `update-docstring` — `update-docstring.sh`
|
||||
|
||||
Both `marketplace.json` (registry) and `plugin.json` (plugin definition) must be present for discovery to work.
|
||||
|
||||
## Working with the Fission Python Skill
|
||||
|
||||
```bash
|
||||
# Make scripts executable (required after fresh clone)
|
||||
chmod +x fission-python/*.sh
|
||||
|
||||
# Create a new Fission Python project
|
||||
./fission-python/create-project.sh <project-name> [destination]
|
||||
|
||||
# Analyze .fission configuration in an existing project
|
||||
./fission-python/analyze-config.sh <project-path> # requires jq
|
||||
|
||||
# View or update Fission config embedded in a function docstring
|
||||
./fission-python/update-docstring.sh <file.py> <function-name> --get
|
||||
./fission-python/update-docstring.sh <file.py> <function-name> --set '<json>'
|
||||
```
|
||||
|
||||
### Fission Configuration in Docstrings
|
||||
|
||||
Fission config is embedded between ` ```fission ` and ` ``` ` markers in Python function docstrings:
|
||||
|
||||
```python
|
||||
def main():
|
||||
"""
|
||||
```fission
|
||||
{
|
||||
"name": "function-name",
|
||||
"http_triggers": {
|
||||
"trigger-name": {"url": "/endpoint", "methods": ["GET"]}
|
||||
}
|
||||
}
|
||||
```
|
||||
"""
|
||||
```
|
||||
|
||||
The `update-docstring.sh` tool parses and updates these blocks.
|
||||
|
||||
### Updating the Project Template
|
||||
|
||||
The template lives in `fission-python/template/`. When `create-project.sh` runs, it copies this directory and performs string substitutions (e.g., replacing project name). The plugin locates the template relative to its own position, making it portable.
|
||||
|
||||
Key template areas to modify:
|
||||
- `src/` — default function structure and helpers
|
||||
- `.fission/deployment.json` — default environment/package/function config
|
||||
- `.gitea/workflows/` — CI/CD pipeline workflows
|
||||
- `requirements.txt` / `dev-requirements.txt` — dependencies
|
||||
|
||||
## Testing
|
||||
|
||||
No centralized test runner. Test skill scripts by running them directly with various arguments, verifying argument parsing, file operations, and JSON output. Example projects in `data/examples/` use pytest.
|
||||
|
||||
## Configuration
|
||||
|
||||
- `.claude/settings.json` — Claude Code settings (plans directory)
|
||||
- `.claude-plugin/marketplace.json` — Plugin registry
|
||||
- `fission-python/.claude-plugin/plugin.json` — Plugin definition
|
||||
|
||||
## Development Environment (Devcontainer)
|
||||
|
||||
The `.devcontainer/devcontainer.json` configures a VS Code dev container with:
|
||||
- `postStartCommand`: `.devcontainer/setup.sh`
|
||||
- Bind mount: `~/Workspaces/self/sdlc-agents/agents` → `.sdlc-agents/` (external SDLC system)
|
||||
|
||||
Environment variables set in devcontainer:
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|---|---|---|
|
||||
| `ANTHROPIC_API_KEY` | (empty) | API key |
|
||||
| `ANTHROPIC_BASE_URL` | `https://openrouter.ai/api` | API endpoint |
|
||||
| `ANTHROPIC_MODEL` | `stepfun/step-3.5-flash:free` | Default model |
|
||||
| `ANTHROPIC_SMALL_FAST_MODEL` | `nvidia/nemotron-3-super-120b-a12b:free` | Alternate model |
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- **Plugin tools must be executable**: Run `chmod +x fission-python/*.sh`
|
||||
- **jq dependency**: `analyze-config.sh` requires `jq` for JSON parsing
|
||||
- **sed portability**: Scripts use GNU sed — on macOS use `sed -i ''` instead of `sed -i`
|
||||
- **Template exclusions**: `create-project.sh` excludes `.git`, `__pycache__`, `*.pyc`, `.env` on copy
|
||||
- **SDLC agents**: Only available inside devcontainer (bind-mounted, not in this repo)
|
||||
|
||||
## AI Guidelines
|
||||
|
||||
### Planning Rule
|
||||
Before making code changes for non-trivial tasks:
|
||||
1. Use `EnterPlanMode` to create a detailed implementation plan
|
||||
2. Explore the codebase to understand existing patterns
|
||||
3. Present the plan for approval before implementing
|
||||
4. Use `TaskList` to track progress on multi-step tasks
|
||||
|
||||
### Agent Usage Rule
|
||||
Use agents for complex, multi-step, or parallelizable tasks:
|
||||
- Research/exploration → Explore agent
|
||||
- Implementation planning → Plan agent
|
||||
509
README.md
Normal file
509
README.md
Normal file
@@ -0,0 +1,509 @@
|
||||
# Claude Marketplace
|
||||
|
||||
[](https://fission.io/)
|
||||
[](https://python.org)
|
||||
[](https://claude.ai/code)
|
||||
|
||||
A plugin ecosystem for Claude Code featuring the **Fission Python Skill** for serverless Python projects and a complete **SDLC Agent System** for automated software development workflows.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Fission Python Skill](#fission-python-skill)
|
||||
- [SDLC Agent System](#sdlc-agent-system)
|
||||
- [Project Structure](#project-structure)
|
||||
- [Quick Start](#quick-start)
|
||||
- [Development](#development)
|
||||
- [Configuration](#configuration)
|
||||
- [Related Documentation](#related-documentation)
|
||||
- [Key Technologies](#key-technologies)
|
||||
|
||||
## Overview
|
||||
|
||||
Claude Marketplace is a repository that provides extensible plugins and tools for Claude Code, Anthropic's AI-powered development environment. It contains two major components:
|
||||
|
||||
1. **Fission Python Skill** - A plugin for creating, analyzing, and managing Fission serverless Python projects on Kubernetes
|
||||
2. **SDLC Agent System** - A complete multi-agent Software Development Life Cycle system for automated planning, architecture validation, coding, and code review
|
||||
|
||||
These components work independently but can be used together to create and maintain production-ready software with AI assistance.
|
||||
|
||||
## Fission Python Skill
|
||||
|
||||
The Fission Python Skill provides three essential tools for working with [Fission](https://fission.io/) serverless functions written in Python.
|
||||
|
||||
### When to Use
|
||||
|
||||
- Creating new Fission Python projects from a standardized template
|
||||
- Analyzing existing Fission configuration files
|
||||
- Parsing and updating Fission configuration embedded in Python function docstrings
|
||||
|
||||
### Available Tools
|
||||
|
||||
| Tool | Purpose |
|
||||
|------|---------|
|
||||
| `create-project.sh` | Create new Fission Python project from template |
|
||||
| `analyze-config.sh` | Analyze `.fission` configuration in a project |
|
||||
| `update-docstring.sh` | Parse and update Fission config in function docstrings |
|
||||
|
||||
### Installation
|
||||
|
||||
The skill scripts are ready to use. Make them executable:
|
||||
|
||||
```bash
|
||||
chmod +x fission-python-skill/*.sh
|
||||
```
|
||||
|
||||
### Usage Examples
|
||||
|
||||
**Create a new Fission project:**
|
||||
```bash
|
||||
./fission-python-skill/create-project.sh my-function ./projects/
|
||||
```
|
||||
|
||||
**Analyze project configuration:**
|
||||
```bash
|
||||
./fission-python-skill/analyze-config.sh ./my-fission-project
|
||||
```
|
||||
|
||||
**View function configuration:**
|
||||
```bash
|
||||
./fission-python-skill/update-docstring.sh ./src/func.py main --get
|
||||
```
|
||||
|
||||
**Update function configuration:**
|
||||
```bash
|
||||
./fission-python-skill/update-docstring.sh ./src/func.py main --set '{"http_triggers": {"api": {"url": "/v1/data", "methods": ["GET"]}}}'
|
||||
```
|
||||
|
||||
### Project Structure
|
||||
|
||||
Projects created with `create-project.sh` follow the standard Fission Python layout:
|
||||
|
||||
```
|
||||
project/
|
||||
├── .fission/
|
||||
│ ├── deployment.json # Main configuration
|
||||
│ ├── dev-deployment.json # Development overrides
|
||||
│ └── local-deployment.json # Local overrides
|
||||
├── src/ # Python function source files
|
||||
│ └── function.py # Functions with fission config in docstrings
|
||||
├── specs/ # Generated Fission specs
|
||||
├── test/ # Unit tests
|
||||
├── manifests/ # Kubernetes manifests
|
||||
├── migrates/ # Database migrations
|
||||
├── requirements.txt # Runtime dependencies
|
||||
└── dev-requirements.txt # Development dependencies
|
||||
```
|
||||
|
||||
### Fission Configuration in Docstrings
|
||||
|
||||
Fission configuration is embedded in Python function docstrings between ````fission` and ```` markers:
|
||||
|
||||
```python
|
||||
def main():
|
||||
"""
|
||||
```fission
|
||||
{
|
||||
"name": "function-name",
|
||||
"http_triggers": {
|
||||
"trigger-name": {
|
||||
"url": "/endpoint",
|
||||
"methods": ["GET", "POST"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
"""
|
||||
# implementation
|
||||
```
|
||||
|
||||
The `update-docstring.sh` tool parses and updates these configurations.
|
||||
|
||||
## SDLC Agent System
|
||||
|
||||
The SDLC (Software Development Life Cycle) Agent System is a complete multi-agent framework for automated software development. It orchestrates seven specialized agents to take a feature request from planning through implementation and review.
|
||||
|
||||
### The Seven Agents
|
||||
|
||||
| Agent | Role | Key Responsibilities |
|
||||
|-------|------|---------------------|
|
||||
| **Initializer** | Setup | Initialize `agent-context/`, detect stack, generate harness scripts, populate domain skills |
|
||||
| **Planning** | Planning | Transform requests into structured, architecture-aware plans with isolated tasks |
|
||||
| **Architect** | Review | Evaluate architecture, enforce architectural rules, validate design decisions |
|
||||
| **Coding** | Implementation | Implement one task at a time from self-contained task files |
|
||||
| **Code Review** | Quality | Review code for correctness, architecture adherence, and debt awareness |
|
||||
| **Curator** | Orchestration | Manage task progression, feature completion, and handoffs |
|
||||
| **Retro** | Improvement | Conduct retrospectives and process improvement |
|
||||
|
||||
### Agent Workflow
|
||||
|
||||
```
|
||||
User Request
|
||||
↓
|
||||
Initializer (one-time setup)
|
||||
↓
|
||||
Planning Agent (creates feature + tasks)
|
||||
↓
|
||||
Architect Agent (reviews plan)
|
||||
↓
|
||||
Coding Agent (implements task 1)
|
||||
↓
|
||||
Coding Agent (implements task 2)
|
||||
↓
|
||||
...
|
||||
↓
|
||||
Code Review Agent (reviews implementation)
|
||||
↓
|
||||
Curator (approves/requests changes)
|
||||
↓
|
||||
Feature Complete
|
||||
```
|
||||
|
||||
### Setup
|
||||
|
||||
To use the SDLC Agent System in a project:
|
||||
|
||||
```bash
|
||||
# Copy agent templates to project root
|
||||
./.sdlc-agents/setup.sh /path/to/target-project
|
||||
|
||||
# This creates: /path/to/target-project/agent-context/
|
||||
# with harness/, memory/, features/, and extensions/ directories
|
||||
```
|
||||
|
||||
The setup is **idempotent** - safe to run multiple times. Use `-f` to force overwrite existing files.
|
||||
|
||||
### Agent Context Structure
|
||||
|
||||
After setup, the project will have:
|
||||
|
||||
```
|
||||
agent-context/
|
||||
├── harness/ # Task execution scripts and tracking
|
||||
│ ├── init-project.sh
|
||||
│ ├── run-quality-gates.sh
|
||||
│ ├── run-arch-tests.sh
|
||||
│ ├── run-feature.sh
|
||||
│ ├── start-task.sh
|
||||
│ └── ...
|
||||
├── memory/ # Learning playbook and retrieval
|
||||
│ └── learning-playbook.md
|
||||
├── features/ # Feature specs and task definitions
|
||||
│ ├── FEAT-001/
|
||||
│ │ ├── feature.md # Feature context
|
||||
│ │ ├── progress-log.md # Progress tracking
|
||||
│ │ └── tasks/
|
||||
│ │ ├── T01-setup.md
|
||||
│ │ └── T02-implement.md
|
||||
└── extensions/ # Custom rules and skills
|
||||
├── _all-agents/ # Global constraints
|
||||
├── planning-agent/
|
||||
├── architect-agent/
|
||||
├── coding-agent/
|
||||
├── codereview-agent/
|
||||
└── skills/ # Project-specific skills
|
||||
├── domain/
|
||||
└── ...
|
||||
```
|
||||
|
||||
### Skill System
|
||||
|
||||
Agents can load domain-specific skills that provide patterns, constraints, and guidance:
|
||||
|
||||
**Stack Skills** (auto-detected):
|
||||
- Java/Kotlin, TypeScript/JavaScript, Python, Go, Rust, .NET/C#, Ruby, PHP
|
||||
|
||||
**Pattern Skills** (on-demand):
|
||||
- Hexagonal, Layered, Modular Monolith, Microservices, Spec-Driven
|
||||
|
||||
**Framework Skills**:
|
||||
- Embabel
|
||||
|
||||
Skills are discovered using `stack-detection.md` and can be explicitly requested via skill directives:
|
||||
- `#TDD` - Force-load TDD skill
|
||||
- `#Hexagonal,Clean` - Load multiple skills
|
||||
- `#only:Python,Security` - Use only these skills
|
||||
- `!Kafka` - Exclude a skill
|
||||
|
||||
### Harness and Quality Gates
|
||||
|
||||
The generated harness scripts provide:
|
||||
|
||||
- **`init-project.sh`** - Install dependencies, compile, verify environment
|
||||
- **`run-quality-gates.sh`** - Run all quality checks (tests, lint, arch, coverage, security)
|
||||
- **`run-arch-tests.sh`** - Run architecture validation tests
|
||||
- **`run-feature.sh`** - Run tests for a specific feature
|
||||
- **`start-task.sh`** - Mark task as in_progress
|
||||
- **`collect-metrics.sh`**, **`compare-metrics.sh`**, **`archive-metrics.sh`** - Metrics management
|
||||
|
||||
These scripts are **stack-specific** - generated based on the detected technology stack and configured tools.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
claude-marketplace/
|
||||
├── .claude-plugin/
|
||||
│ └── marketplace.json # Plugin registry for Claude Code
|
||||
├── .sdlc-agents/ # Complete SDLC agent system
|
||||
│ ├── agents/ # Agent configurations (7 agents)
|
||||
│ │ ├── initializer-agent.md
|
||||
│ │ ├── planning-agent.md
|
||||
│ │ ├── architect-agent.md
|
||||
│ │ ├── coding-agent.md
|
||||
│ │ ├── codereview-agent.md
|
||||
│ │ ├── curator-agent.md
|
||||
│ │ └── retro-agent.md
|
||||
│ ├── guardrails/ # Quality guidelines
|
||||
│ ├── skills/ # Stack, pattern, and framework skills
|
||||
│ │ ├── stacks/ # Language-specific (Python, Java, TS, Go, etc.)
|
||||
│ │ ├── patterns/ # Architecture patterns
|
||||
│ │ └── frameworks/ # Framework-specific guidance
|
||||
│ ├── templates/ # Templates for agent-context structure
|
||||
│ ├── tools/ # Utility scripts (discovery, validation, skills)
|
||||
│ └── setup.sh # Initialize agent-context in a project
|
||||
├── fission-python-skill/ # Fission Python plugin
|
||||
│ ├── .claude-plugin/
|
||||
│ │ └── plugin.json # Plugin definition
|
||||
│ ├── template/ # Project template (copied when creating new projects)
|
||||
│ │ ├── .fission/
|
||||
│ │ ├── src/
|
||||
│ │ ├── test/
|
||||
│ │ ├── manifests/
|
||||
│ │ ├── migrates/
|
||||
│ │ ├── specs/
|
||||
│ │ ├── requirements.txt
|
||||
│ │ └── dev-requirements.txt
|
||||
│ ├── create-project.sh # Create new Fission project
|
||||
│ ├── analyze-config.sh # Analyze .fission configuration
|
||||
│ ├── update-docstring.sh # Parse/update function docstrings
|
||||
│ ├── SKILL.md # Quick reference
|
||||
│ └── reference.md # Detailed tool documentation
|
||||
├── data/
|
||||
│ └── examples/ # Example Fission Python projects
|
||||
│ ├── py-eom-quota/
|
||||
│ ├── py-eom-storage/
|
||||
│ └── py-ailbl-scheduler/
|
||||
└── .devcontainer/ # VS Code dev container configuration
|
||||
└── devcontainer.json
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Setting Up Development Environment
|
||||
|
||||
**Recommended:** Open the repository in a devcontainer (VS Code with Dev Containers extension):
|
||||
|
||||
```bash
|
||||
# Open in container from VS Code
|
||||
# The devcontainer will set up dependencies and environment
|
||||
```
|
||||
|
||||
**Manual setup:**
|
||||
```bash
|
||||
# Make scripts executable
|
||||
chmod +x fission-python-skill/*.sh
|
||||
chmod +x .sdlc-agents/setup.sh
|
||||
```
|
||||
|
||||
### Using the Fission Python Skill
|
||||
|
||||
1. **Create a new Fission project:**
|
||||
```bash
|
||||
./fission-python-skill/create-project.sh my-api ./projects/
|
||||
```
|
||||
|
||||
2. **Analyze the configuration:**
|
||||
```bash
|
||||
./fission-python-skill/analyze-config.sh ./projects/my-api
|
||||
```
|
||||
|
||||
3. **Develop your function** in `src/`, update docstrings as needed with `update-docstring.sh`
|
||||
|
||||
4. **Deploy to Fission** following Fission documentation
|
||||
|
||||
### Using the SDLC Agent System
|
||||
|
||||
1. **Initialize an existing project** (new or legacy):
|
||||
|
||||
```bash
|
||||
./.sdlc-agents/setup.sh /path/to/your/project
|
||||
```
|
||||
|
||||
2. The Initializer Agent will:
|
||||
- Detect the technology stack
|
||||
- Generate stack-specific harness scripts
|
||||
- Create initial feature structure
|
||||
- Populate domain skills based on project analysis
|
||||
- For legacy projects: discover and document architecture
|
||||
|
||||
3. **Start a new feature:**
|
||||
- Create a feature request in `agent-context/features/`
|
||||
- The Planning Agent will create detailed task files
|
||||
- The Architect Agent will review and approve the plan
|
||||
- Coding Agents implement each task
|
||||
- Code Review Agents validate each implementation
|
||||
- Curator manages handoffs and completion
|
||||
|
||||
4. **Run quality gates:**
|
||||
```bash
|
||||
cd /path/to/your/project
|
||||
./agent-context/harness/run-quality-gates.sh
|
||||
```
|
||||
|
||||
### Testing Changes
|
||||
|
||||
- **Fission skill scripts:** Run directly with various arguments
|
||||
- **Template projects:** Use pytest in generated projects
|
||||
- **SDLC agents:** Run `setup.sh` and verify `agent-context/` creation
|
||||
|
||||
## Development
|
||||
|
||||
### Modifying Fission Python Skill
|
||||
|
||||
1. Edit scripts in `fission-python-skill/`
|
||||
2. Update `plugin.json` to modify tools or metadata
|
||||
3. Update `marketplace.json` to change plugin registry
|
||||
4. Test by running scripts directly
|
||||
|
||||
### Updating the Project Template
|
||||
|
||||
The template at `fission-python-skill/template/` is used by `create-project.sh`:
|
||||
|
||||
- Modify `src/` to change default function structure
|
||||
- Update `.fission/deployment.json` for environment configuration
|
||||
- Adjust `requirements.txt` for different dependencies
|
||||
- The template `build.sh` is used when building packages
|
||||
- Ensure `.gitea/workflows/` includes CI/CD workflows
|
||||
|
||||
The plugin locates the template relative to its own location, making it portable.
|
||||
|
||||
### Plugin Registration
|
||||
|
||||
Both files must be present:
|
||||
- `.claude-plugin/marketplace.json` (registry - lists all plugins)
|
||||
- `fission-python-skill/.claude-plugin/plugin.json` (plugin definition)
|
||||
|
||||
### Invoking Skills in Claude Code
|
||||
|
||||
Once registered, skills can be invoked through Claude Code's tool system. The tool names correspond to the script basenames:
|
||||
- `create-project`
|
||||
- `analyze-config`
|
||||
- `update-docstring`
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables (Dev Container)
|
||||
|
||||
Set in `.devcontainer/devcontainer.json`:
|
||||
|
||||
| Variable | Default | Purpose |
|
||||
|----------|---------|---------|
|
||||
| `ANTHROPIC_API_KEY` | (empty) | API key (mounted from local) |
|
||||
| `ANTHROPIC_BASE_URL` | `https://openrouter.ai/api` | API endpoint |
|
||||
| `ANTHROPIC_MODEL` | `stepfun/step-3.5-flash:free` | Default model |
|
||||
| `ANTHROPIC_SMALL_FAST_MODEL` | `nvidia/nemotron-3-super-120b-a12b:free` | Alternate model |
|
||||
|
||||
### Claude Code Settings
|
||||
|
||||
- `.claude/settings.json` - Main settings (plans directory, hooks)
|
||||
- `.claude/settings.local.json` - Local overrides (not committed)
|
||||
|
||||
### Dependencies
|
||||
|
||||
The `analyze-config.sh` tool requires `jq` for JSON parsing:
|
||||
```bash
|
||||
# Debian/Ubuntu
|
||||
sudo apt-get install jq
|
||||
|
||||
# macOS
|
||||
brew install jq
|
||||
```
|
||||
|
||||
## Related Documentation
|
||||
|
||||
### Comprehensive Guides
|
||||
- **[CLAUDE.md](CLAUDE.md)** - Complete project guide with detailed architecture and workflows
|
||||
|
||||
### Fission Python Skill
|
||||
- **[fission-python-skill/SKILL.md](fission-python-skill/SKILL.md)** - Quick reference and usage patterns
|
||||
- **[fission-python-skill/reference.md](fission-python-skill/reference.md)** - Detailed tool documentation
|
||||
|
||||
### SDLC Agents
|
||||
- **[.sdlc-agents/skills/README.md](.sdlc-agents/skills/README.md)** - Skill system usage and discovery
|
||||
- **[.sdlc-agents/initializer-agent.md](.sdlc-agents/initializer-agent.md)** - Setup agent documentation
|
||||
- **[.sdlc-agents/planning-agent.md](.sdlc-agents/planning-agent.md)** - Planning agent documentation
|
||||
- **[.sdlc-agents/architect-agent.md](.sdlc-agents/architect-agent.md)** - Architecture review agent
|
||||
- **[.sdlc-agents/coding-agent.md](.sdlc-agents/coding-agent.md)** - Implementation agent
|
||||
- **[.sdlc-agents/codereview-agent.md](.sdlc-agents/codereview-agent.md)** - Code review agent
|
||||
- **[.sdlc-agents/curator-agent.md](.sdlc-agents/curator-agent.md)** - Orchestration agent
|
||||
- **[.sdlc-agents/retro-agent.md](.sdlc-agents/retro-agent.md)** - Retrospective agent
|
||||
- **[.sdlc-agents/skills/harness-spec.md](.sdlc-agents/skills/harness-spec.md)** - Harness specification
|
||||
|
||||
## Key Technologies
|
||||
|
||||
- **Fission** - Serverless framework for Kubernetes
|
||||
- **Python** - Primary language for the Fission skill plugin
|
||||
- **Claude Code** - AI-powered development environment by Anthropic
|
||||
- **Kubernetes** - Container orchestration platform for Fission
|
||||
- **SDLC Agents** - Multi-agent system for automated software development
|
||||
- **Shell scripting** - Tool implementations (Bash with `jq` for JSON)
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
- **Plugin tools must be executable**: Always run `chmod +x` on `.sh` files
|
||||
- **Do not commit secrets**: `.env` files are gitignored; template creates placeholder only
|
||||
- **Plugin registration**: Both `marketplace.json` AND plugin's `plugin.json` must exist
|
||||
- **Template path**: `create-project.sh` uses relative path to `fission-python-skill/template/` (portable)
|
||||
- **jq dependency**: `analyze-config.sh` requires `jq` installed
|
||||
- **sed portability**: Scripts use GNU sed; may need adjustment for macOS (`sed -i ''`)
|
||||
|
||||
## Quick Reference
|
||||
|
||||
### Fission Python Skill
|
||||
|
||||
```bash
|
||||
# Create new project
|
||||
fission-python-skill/create-project.sh my-project ./output/
|
||||
|
||||
# Analyze configuration
|
||||
fission-python-skill/analyze-config.sh ./existing-project
|
||||
|
||||
# View docstring config
|
||||
fission-python-skill/update-docstring.sh ./src/func.py main --get
|
||||
|
||||
# Update docstring config
|
||||
fission-python-skill/update-docstring.sh ./src/func.py main --set '{"http_triggers": {"api": {"url": "/v1", "methods": ["GET"]}}}'
|
||||
```
|
||||
|
||||
### SDLC Agent System
|
||||
|
||||
```bash
|
||||
# Setup agents in a project
|
||||
.sdlc-agents/setup.sh /path/to/project
|
||||
|
||||
# Resolve skill paths for planning agent
|
||||
.sdlc-agents/tools/skills/resolve-skills.sh --agent planning python tdd
|
||||
|
||||
# Parse skill directives from user prompt
|
||||
.sdlc-agents/tools/skills/parse-skill-directives.sh "Use #Hexagonal but !Kafka"
|
||||
```
|
||||
|
||||
### Harness Scripts (after setup)
|
||||
|
||||
```bash
|
||||
# Initialize project (install deps, compile)
|
||||
./agent-context/harness/init-project.sh
|
||||
|
||||
# Run all quality checks
|
||||
./agent-context/harness/run-quality-gates.sh
|
||||
|
||||
# Run architecture tests only
|
||||
./agent-context/harness/run-arch-tests.sh
|
||||
|
||||
# Run feature-specific tests
|
||||
./agent-context/harness/run-feature.sh FEAT-001
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
Built for the Claude Code ecosystem. See [CLAUDE.md](CLAUDE.md) for comprehensive documentation.
|
||||
8
fission-python/.claude-plugin/plugin.json
Normal file
8
fission-python/.claude-plugin/plugin.json
Normal file
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"id" : "fission-python",
|
||||
"name" : "FissionPython",
|
||||
"description": "Skill for creating, analyzing, and managing Fission Python projects.",
|
||||
"type" : "skill",
|
||||
"tools" : ["create-project", "analyze-config", "update-docstring"],
|
||||
"version" : "1.0.1"
|
||||
}
|
||||
83
fission-python/SKILL.md
Normal file
83
fission-python/SKILL.md
Normal file
@@ -0,0 +1,83 @@
|
||||
# Fission Python Skill
|
||||
|
||||
Skill for creating, analyzing, and managing Fission Python projects.
|
||||
|
||||
---
|
||||
|
||||
## When to Use
|
||||
|
||||
- Creating new Fission Python projects from templates
|
||||
- Analyzing Fission configuration (.fission files) in existing projects
|
||||
- Parsing and updating embedded Fission configuration in function docstrings
|
||||
|
||||
## Key Concepts
|
||||
|
||||
- **Fission Project Structure**: Standard layout with src/, .fission/, specs/ directories
|
||||
- **Function Docstring Configuration**: Fission configuration embedded in Python function docstrings between ```fission markers
|
||||
- **Configuration Files**: .fission/deployment.json defines environments, packages, functions, secrets, and configmaps
|
||||
|
||||
## Patterns
|
||||
|
||||
### Project Template
|
||||
Standard Fission Python project includes:
|
||||
- `src/` - Python function source files with build.sh and requirements.txt
|
||||
- `.fission/` - Fission configuration (deployment.json, etc.)
|
||||
- `specs/` - Generated Fission specs
|
||||
- `test/` - Unit tests
|
||||
- `manifests/` - Kubernetes manifests
|
||||
- `migrates/` - Database migrations
|
||||
- `.devcontainer/` - Development container configuration
|
||||
- `.gitea/workflows/` - CI/CD deployment workflows
|
||||
|
||||
All functions should use pydantic models for request/response validation and include comprehensive docstrings.
|
||||
|
||||
### Docstring Format
|
||||
Fission configuration in docstrings follows this pattern:
|
||||
```python
|
||||
def main():
|
||||
"""
|
||||
Function description
|
||||
|
||||
```fission
|
||||
{
|
||||
"name": "function-name",
|
||||
"http_triggers": {
|
||||
"trigger-name": {
|
||||
"url": "/endpoint",
|
||||
"methods": ["GET", "POST"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
"""
|
||||
# function implementation
|
||||
```
|
||||
|
||||
## Tools
|
||||
|
||||
| Tool | Purpose |
|
||||
|------|---------|
|
||||
| `create-project.sh` | Create new Fission Python project from template |
|
||||
| `analyze-config.sh` | Analyze .fission configuration in a project |
|
||||
| `update-docstring.sh` | Parse and update docstrings in fission function methods |
|
||||
|
||||
## Examples
|
||||
|
||||
### Create a new project
|
||||
```bash
|
||||
fission-python-skill create-project my-new-function ./projects/
|
||||
```
|
||||
|
||||
### Analyze project configuration
|
||||
```bash
|
||||
fission-python-skill analyze-config ./my-fission-project
|
||||
```
|
||||
|
||||
### Update function docstring
|
||||
```bash
|
||||
fission-python-skill update-docstring ./src/my_function.py main
|
||||
```
|
||||
|
||||
## Related Skills
|
||||
|
||||
- None - this is a specialized skill for Fission Python projects
|
||||
151
fission-python/analyze-config.sh
Executable file
151
fission-python/analyze-config.sh
Executable file
@@ -0,0 +1,151 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Fission Configuration Analyzer
|
||||
# Analyzes and displays fission configuration from .fission directory
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
usage() {
|
||||
echo "Usage: $0 <project-path>"
|
||||
echo " project-path: Path to the fission project directory (should contain .fission/ subdirectory)"
|
||||
exit 1
|
||||
}
|
||||
|
||||
if [[ $# -ne 1 ]]; then
|
||||
usage
|
||||
fi
|
||||
|
||||
PROJECT_PATH="$1"
|
||||
|
||||
# Validate project path
|
||||
if [[ ! -d "$PROJECT_PATH" ]]; then
|
||||
echo "Error: Project path '$PROJECT_PATH' does not exist"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
FISSION_DIR="$PROJECT_PATH/.fission"
|
||||
|
||||
if [[ ! -d "$FISSION_DIR" ]]; then
|
||||
echo "Error: .fission directory not found in '$PROJECT_PATH'"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "🔍 Analyzing Fission configuration in '$PROJECT_PATH'"
|
||||
echo "=================================================="
|
||||
|
||||
# Analyze deployment.json (main configuration)
|
||||
if [[ -f "$FISSION_DIR/deployment.json" ]]; then
|
||||
echo ""
|
||||
echo "📋 DEPLOYMENT CONFIGURATION"
|
||||
echo "--------------------------"
|
||||
|
||||
# Extract namespaces
|
||||
namespace=$(jq -r '.namespace // "default"' "$FISSION_DIR/deployment.json")
|
||||
echo "Namespace: $namespace"
|
||||
|
||||
# Analyze environments
|
||||
echo ""
|
||||
echo "🌐 ENVIRONMENTS:"
|
||||
if jq -e '.environments' "$FISSION_DIR/deployment.json" >/dev/null; then
|
||||
jq -r '.environments | to_entries[] | " \(.key):" +
|
||||
"\n Image: \(.value.image // "N/A")" +
|
||||
"\n Builder: \(.value.builder // "N/A")" +
|
||||
"\n CPU: \(.value.mincpu // 0)m-\(.value.maxcpu // 0)m" +
|
||||
"\n Memory: \(.value.minmemory // 0)Mi-\(.value.maxmemory // 0)Mi" +
|
||||
"\n Pool Size: \(.value.poolsize // 1)"' "$FISSION_DIR/deployment.json"
|
||||
else
|
||||
echo " No environments defined"
|
||||
fi
|
||||
|
||||
# Analyze packages
|
||||
echo ""
|
||||
echo "📦 PACKAGES:"
|
||||
if jq -e '.packages' "$FISSION_DIR/deployment.json" >/dev/null; then
|
||||
jq -r '.packages | to_entries[] | " \(.key):" +
|
||||
"\n Build Command: \(.value.buildcmd // "N/A")" +
|
||||
"\n Source Archive: \(.value.sourcearchive // "N/A")" +
|
||||
"\n Environment: \(.value.env // "N/A")"' "$FISSION_DIR/deployment.json"
|
||||
else
|
||||
echo " No packages defined"
|
||||
fi
|
||||
|
||||
# Analyze functions (from function_common and individual functions)
|
||||
echo ""
|
||||
echo "⚙️ FUNCTIONS:"
|
||||
if jq -e '.function_common' "$FISSION_DIR/deployment.json" >/dev/null; then
|
||||
echo " Common Configuration:"
|
||||
jq -r '.function_common |
|
||||
" Package: \(.pkg // "N/A")" +
|
||||
"\n Executor Type: \(.executor.select // "N/A")" +
|
||||
"\n CPU: \(.mincpu // 0)m-\(.maxcpu // 0)m" +
|
||||
"\n Memory: \(.minmemory // 0)Mi-\(.maxmemory // 0)Mi"' "$FISSION_DIR/deployment.json"
|
||||
fi
|
||||
|
||||
# Look for individual function definitions
|
||||
if jq -e '.functions' "$FISSION_DIR/deployment.json" >/dev/null; then
|
||||
echo ""
|
||||
echo " Individual Functions:"
|
||||
jq -r '.functions | to_entries[] | " \(.key):" +
|
||||
"\n Executor: \(.value.executor // "N/A")"' "$FISSION_DIR/deployment.json"
|
||||
else
|
||||
echo " No individual functions defined (using function_common)"
|
||||
fi
|
||||
|
||||
# Analyze secrets
|
||||
echo ""
|
||||
echo "🔐 SECRETS:"
|
||||
if jq -e '.secrets' "$FISSION_DIR/deployment.json" >/dev/null; then
|
||||
jq -r '.secrets | to_entries[] | " \(.key):" +
|
||||
"\n Type: \(.value.kind // "literal")" +
|
||||
"\n Literal Count: \(.value.literals | length // 0)"' "$FISSION_DIR/deployment.json"
|
||||
else
|
||||
echo " No secrets defined"
|
||||
fi
|
||||
|
||||
# Analyze configmaps
|
||||
echo ""
|
||||
echo "⚙️ CONFIGMAPS:"
|
||||
if jq -e '.configmaps' "$FISSION_DIR/deployment.json" >/dev/null; then
|
||||
jq -r '.configmaps | to_entries[] | " \(.key):" +
|
||||
"\n Literal Count: \(.value.literals | length // 0)"' "$FISSION_DIR/deployment.json"
|
||||
else
|
||||
echo " No configmaps defined"
|
||||
fi
|
||||
|
||||
# Analyze archives
|
||||
echo ""
|
||||
echo "📦 ARCHIVES:"
|
||||
if jq -e '.archives' "$FISSION_DIR/deployment.json" >/dev/null; then
|
||||
jq -r '.archives | to_entries[] | " \(.key):" +
|
||||
"\n Source Path: \(.value.sourcepath // "N/A")"' "$FISSION_DIR/deployment.json"
|
||||
else
|
||||
echo " No archives defined"
|
||||
fi
|
||||
else
|
||||
echo ""
|
||||
echo "⚠️ deployment.json not found in .fission directory"
|
||||
fi
|
||||
|
||||
# Check for other fission files
|
||||
echo ""
|
||||
echo "📄 OTHER FISSION FILES:"
|
||||
shopt -s nullglob
|
||||
fission_files=("$FISSION_DIR"/*.json)
|
||||
if [[ ${#fission_files[@]} -gt 0 ]]; then
|
||||
for file in "${fission_files[@]}"; do
|
||||
filename=$(basename "$file")
|
||||
if [[ "$filename" != "deployment.json" ]]; then
|
||||
echo " $filename"
|
||||
# Show brief content
|
||||
if [[ -s "$file" ]]; then
|
||||
line_count=$(wc -l < "$file")
|
||||
echo " ($line_count lines)"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
else
|
||||
echo " No other .json files found"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "✅ Analysis complete"
|
||||
225
fission-python/create-project.sh
Executable file
225
fission-python/create-project.sh
Executable file
@@ -0,0 +1,225 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Fission Python Project Creator
|
||||
# Creates a new fission python project from template
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
# Get the directory where this script is located
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
TEMPLATE_DIR="$SCRIPT_DIR/template"
|
||||
|
||||
usage() {
|
||||
echo "Usage: $0 <project-name> [destination-directory]"
|
||||
echo " project-name: Name for the new fission project"
|
||||
echo " destination-directory: Optional directory where project should be created (default: current directory)"
|
||||
exit 1
|
||||
}
|
||||
|
||||
if [[ $# -lt 1 || $# -gt 2 ]]; then
|
||||
usage
|
||||
fi
|
||||
|
||||
PROJECT_NAME="$1"
|
||||
DEST_DIR="${2:-./}"
|
||||
|
||||
# Validate project name
|
||||
if [[ ! "$PROJECT_NAME" =~ ^[a-zA-Z0-9_-]+$ ]]; then
|
||||
echo "Error: Project name can only contain letters, numbers, hyphens, and underscores"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Create destination directory if it doesn't exist
|
||||
mkdir -p "$DEST_DIR"
|
||||
|
||||
PROJECT_PATH="$DEST_DIR/$PROJECT_NAME"
|
||||
|
||||
# Check if project already exists
|
||||
if [[ -d "$PROJECT_PATH" ]]; then
|
||||
echo "Error: Project directory '$PROJECT_PATH' already exists"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check if template exists
|
||||
if [[ ! -d "$TEMPLATE_DIR" ]]; then
|
||||
echo "Error: Template directory not found at '$TEMPLATE_DIR'"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Creating fission python project '$PROJECT_NAME' in '$PROJECT_PATH'..."
|
||||
|
||||
# Copy template excluding unwanted files/directories
|
||||
rsync -av --exclude='.git' --exclude='__pycache__' --exclude='*.pyc' --exclude='.env' \
|
||||
"$TEMPLATE_DIR/" "$PROJECT_PATH/"
|
||||
|
||||
# Replace placeholder values in configuration files
|
||||
|
||||
# 1. deployment.json - replace ${PROJECT_NAME} and old eom-quota references
|
||||
if [[ -f "$PROJECT_PATH/.fission/deployment.json" ]]; then
|
||||
sed -i "s/\${PROJECT_NAME}/$PROJECT_NAME/g" "$PROJECT_PATH/.fission/deployment.json"
|
||||
sed -i "s/eom-quota/$PROJECT_NAME/g" "$PROJECT_PATH/.fission/deployment.json"
|
||||
sed -i "s/fission-eom-quota/fission-$PROJECT_NAME/g" "$PROJECT_PATH/.fission/deployment.json"
|
||||
fi
|
||||
|
||||
# 2. Override files - dev-deployment.json and local-deployment.json
|
||||
for override in dev-deployment.json local-deployment.json; do
|
||||
if [[ -f "$PROJECT_PATH/.fission/$override" ]]; then
|
||||
sed -i "s/\${PROJECT_NAME}/$PROJECT_NAME/g" "$PROJECT_PATH/.fission/$override"
|
||||
sed -i "s/eom-quota/$PROJECT_NAME/g" "$PROJECT_PATH/.fission/$override"
|
||||
sed -i "s/fission-eom-quota/fission-$PROJECT_NAME/g" "$PROJECT_PATH/.fission/$override"
|
||||
fi
|
||||
done
|
||||
|
||||
# 3. helpers.py - update SECRET_NAME and CONFIG_NAME
|
||||
if [[ -f "$PROJECT_PATH/src/helpers.py" ]]; then
|
||||
sed -i "s/\${PROJECT_NAME}/$PROJECT_NAME/g" "$PROJECT_PATH/src/helpers.py"
|
||||
fi
|
||||
|
||||
# 4. README.md - update with project name and clean up
|
||||
if [[ -f "$PROJECT_PATH/README.md" ]]; then
|
||||
sed -i "s/\${PROJECT_NAME}/$PROJECT_NAME/g" "$PROJECT_PATH/README.md"
|
||||
sed -i "s/^# Fission Python Template/# $PROJECT_NAME/g" "$PROJECT_PATH/README.md"
|
||||
sed -i "s/your-service-py/$PROJECT_NAME-py/g" "$PROJECT_PATH/README.md"
|
||||
sed -i "s/your-package/$PROJECT_NAME/g" "$PROJECT_PATH/README.md"
|
||||
fi
|
||||
|
||||
# ========== VALIDATION ==========
|
||||
echo ""
|
||||
echo "🔍 Validating project structure..."
|
||||
|
||||
validation_errors=0
|
||||
validation_warnings=0
|
||||
|
||||
# 1. Check build.sh exists and is executable
|
||||
if [[ -f "$PROJECT_PATH/src/build.sh" ]]; then
|
||||
if [[ -x "$PROJECT_PATH/src/build.sh" ]]; then
|
||||
echo " ✓ build.sh exists and is executable"
|
||||
else
|
||||
echo " ⚠ build.sh exists but is not executable. Fixing with chmod +x..."
|
||||
chmod +x "$PROJECT_PATH/src/build.sh"
|
||||
validation_warnings=$((validation_warnings + 1))
|
||||
fi
|
||||
else
|
||||
echo " ✗ ERROR: src/build.sh is missing (required)"
|
||||
validation_errors=$((validation_errors + 1))
|
||||
fi
|
||||
|
||||
# 2. Check deployment.json references ./build.sh
|
||||
if [[ -f "$PROJECT_PATH/.fission/deployment.json" ]]; then
|
||||
if grep -q '"./build.sh"' "$PROJECT_PATH/.fission/deployment.json" 2>/dev/null; then
|
||||
echo " ✓ deployment.json references ./build.sh"
|
||||
else
|
||||
echo " ⚠ deployment.json does not reference './build.sh' in buildcmd"
|
||||
validation_warnings=$((validation_warnings + 1))
|
||||
fi
|
||||
else
|
||||
echo " ✗ ERROR: .fission/deployment.json is missing"
|
||||
validation_errors=$((validation_errors + 1))
|
||||
fi
|
||||
|
||||
# 3. Check requirements.txt exists
|
||||
if [[ -f "$PROJECT_PATH/src/requirements.txt" ]]; then
|
||||
echo " ✓ requirements.txt exists"
|
||||
|
||||
# Check for essential dependencies
|
||||
missing_deps=()
|
||||
if ! grep -qi 'pydantic' "$PROJECT_PATH/src/requirements.txt"; then
|
||||
missing_deps+=("pydantic")
|
||||
fi
|
||||
if ! grep -qi 'flask' "$PROJECT_PATH/src/requirements.txt"; then
|
||||
missing_deps+=("flask")
|
||||
fi
|
||||
|
||||
if [[ ${#missing_deps[@]} -gt 0 ]]; then
|
||||
echo " ⚠ requirements.txt missing recommended dependencies: ${missing_deps[*]}"
|
||||
validation_warnings=$((validation_warnings + 1))
|
||||
else
|
||||
echo " ✓ Contains essential dependencies (pydantic, flask)"
|
||||
fi
|
||||
else
|
||||
echo " ✗ ERROR: src/requirements.txt is missing (required)"
|
||||
validation_errors=$((validation_errors + 1))
|
||||
fi
|
||||
|
||||
# 4. Check .gitea/workflows directory exists
|
||||
if [[ -d "$PROJECT_PATH/.gitea/workflows" ]]; then
|
||||
echo " ✓ .gitea/workflows directory exists"
|
||||
|
||||
# Count workflow files
|
||||
workflow_count=$(find "$PROJECT_PATH/.gitea/workflows" -type f -name "*.yaml" 2>/dev/null | wc -l)
|
||||
if [[ $workflow_count -ge 4 ]]; then
|
||||
echo " ✓ Found $workflow_count workflow files"
|
||||
else
|
||||
echo " ⚠ Only found $workflow_count workflow files (expected at least 4)"
|
||||
validation_warnings=$((validation_warnings + 1))
|
||||
fi
|
||||
else
|
||||
echo " ⚠ .gitea/workflows directory is missing (recommended for CI/CD)"
|
||||
validation_warnings=$((validation_warnings + 1))
|
||||
fi
|
||||
|
||||
# 5. Check for Python files with docstrings (basic check, excluding __init__.py)
|
||||
python_files=$(find "$PROJECT_PATH/src" -type f -name "*.py" ! -name "__init__.py" 2>/dev/null | wc -l)
|
||||
if [[ $python_files -gt 0 ]]; then
|
||||
files_with_docstrings=$(grep -l '"""' "$PROJECT_PATH/src/"*.py 2>/dev/null | grep -v '__init__.py' | wc -l)
|
||||
if [[ $files_with_docstrings -eq $python_files ]]; then
|
||||
echo " ✓ All Python files contain docstrings"
|
||||
else
|
||||
echo " ⚠ Only $files_with_docstrings/$python_files Python files have docstrings"
|
||||
validation_warnings=$((validation_warnings + 1))
|
||||
fi
|
||||
else
|
||||
echo " ⚠ No Python files found in src/ to check docstrings"
|
||||
fi
|
||||
|
||||
# 6. Check for pydantic BaseModel usage in models.py (if exists)
|
||||
if [[ -f "$PROJECT_PATH/src/models.py" ]]; then
|
||||
if grep -q "pydantic.BaseModel" "$PROJECT_PATH/src/models.py"; then
|
||||
echo " ✓ models.py uses pydantic.BaseModel"
|
||||
else
|
||||
echo " ⚠ models.py does not appear to use pydantic.BaseModel (recommended for HTTP triggers)"
|
||||
validation_warnings=$((validation_warnings + 1))
|
||||
fi
|
||||
fi
|
||||
|
||||
# Summary
|
||||
echo ""
|
||||
if [[ $validation_errors -eq 0 && $validation_warnings -eq 0 ]]; then
|
||||
echo "✅ All validations passed!"
|
||||
elif [[ $validation_errors -eq 0 ]]; then
|
||||
echo "⚠️ $validation_warnings validation warning(s). Project is usable but review above."
|
||||
else
|
||||
echo "❌ $validation_errors validation error(s) and $validation_warnings warning(s)."
|
||||
echo " The project was created but may have issues. Review the messages above."
|
||||
fi
|
||||
|
||||
# Create basic .env file if it doesn't exist
|
||||
if [[ ! -f "$PROJECT_PATH/.env" ]]; then
|
||||
cat > "$PROJECT_PATH/.env" << EOF
|
||||
# Environment variables for $PROJECT_NAME
|
||||
# Copy this to .env.local for local overrides
|
||||
FISSION_ROUTE_SERVICE_ENDPOINT=http://router.fission.svc.cluster.local
|
||||
EOF
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "🎉 Project '$PROJECT_NAME' created!"
|
||||
|
||||
if [[ $validation_errors -eq 0 && $validation_warnings -eq 0 ]]; then
|
||||
echo "✅ All validations passed!"
|
||||
elif [[ $validation_errors -eq 0 ]]; then
|
||||
echo "⚠️ $validation_warnings warning(s) found - review above."
|
||||
else
|
||||
echo "❌ $validation_errors error(s) and $validation_warnings warning(s) - review above."
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "Next steps:"
|
||||
echo "1. cd $PROJECT_PATH"
|
||||
echo "2. Review and update configuration in .fission/deployment.json"
|
||||
echo "3. Install dependencies: pip install --upgrade --force-reinstall -r dev-requirements.txt"
|
||||
echo "4. Customize your functions in the src/ directory (see examples/ for patterns)"
|
||||
echo "5. Ensure HTTP trigger functions have proper fission config in docstrings"
|
||||
echo "6. Write tests in test/ directory"
|
||||
echo "7. Create Kubernetes secrets: kubectl create secret generic fission-$PROJECT_NAME-env --from-literal=... (see docs/SECRETS.md)"
|
||||
echo "8. Build and deploy: ./src/build.sh && fission deploy"
|
||||
185
fission-python/reference.md
Normal file
185
fission-python/reference.md
Normal file
@@ -0,0 +1,185 @@
|
||||
# Fission Python Skill Reference
|
||||
|
||||
Detailed reference for the fission-python-skill tools.
|
||||
|
||||
---
|
||||
|
||||
## create-project.sh
|
||||
|
||||
Create a new Fission Python project from the standard template.
|
||||
|
||||
### Usage
|
||||
```bash
|
||||
fission-python-skill create-project <project-name> [destination-directory]
|
||||
```
|
||||
|
||||
### Arguments
|
||||
- `project-name`: Name for the new fission project (used for directories and configuration)
|
||||
- `destination-directory`: Optional directory where the project should be created (defaults to current directory)
|
||||
|
||||
### Description
|
||||
Creates a new Fission Python project by copying the template from the plugin's `template/` directory. The plugin is portable and works from any location - the template is stored relative to the plugin itself.
|
||||
|
||||
The template includes:
|
||||
- Standard directory structure (src/, .fission/, specs/, test/, manifests/, migrates/)
|
||||
- Example Python functions with fission configuration in docstrings
|
||||
- Configuration files (.fission/deployment.json, etc.)
|
||||
- Development setup (devcontainer, requirements, etc.)
|
||||
- CI/CD workflows (.gitea/workflows/)
|
||||
- build.sh script for packaging
|
||||
|
||||
After project creation, the script validates that:
|
||||
- `src/build.sh` exists and is executable
|
||||
- `.fission/deployment.json` references `./build.sh`
|
||||
- `src/requirements.txt` exists and contains essential dependencies (pydantic, flask)
|
||||
- `.gitea/workflows/` directory exists with at least 4 workflow files
|
||||
- Python files have docstrings (excluding __init__.py)
|
||||
- models.py uses pydantic.BaseModel
|
||||
|
||||
Validation warnings or errors are displayed to ensure the project follows best practices.
|
||||
|
||||
### Examples
|
||||
```bash
|
||||
# Create project in current directory
|
||||
fission-python-skill create-project my-function
|
||||
|
||||
# Create project in specific directory
|
||||
fission-python-skill create-project my-function ./projects/
|
||||
|
||||
# Create project with different name
|
||||
fission-python-skill create-project data-processing-tool ./services/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## analyze-config.sh
|
||||
|
||||
Analyze and display Fission configuration from a project's .fission directory.
|
||||
|
||||
### Usage
|
||||
```bash
|
||||
fission-python-skill analyze-config <project-path>
|
||||
```
|
||||
|
||||
### Arguments
|
||||
- `project-path`: Path to the fission project directory (should contain .fission/ subdirectory)
|
||||
|
||||
### Description
|
||||
Reads and parses the fission configuration files (.fission/deployment.json and related files) to provide a structured summary of:
|
||||
- Environments and their resource allocation (CPU, memory, scaling)
|
||||
- Packages and their build commands
|
||||
- Functions and their HTTP triggers
|
||||
- Secrets and ConfigMaps
|
||||
- Archives and source configuration
|
||||
|
||||
### Output Format
|
||||
The analysis is displayed in a human-readable format showing:
|
||||
- Project overview
|
||||
- Environment configurations
|
||||
- Package details
|
||||
- Function specifications
|
||||
- Security configurations (secrets/configmaps)
|
||||
|
||||
### Examples
|
||||
```bash
|
||||
# Analyze current directory project
|
||||
fission-python-skill analyze-config .
|
||||
|
||||
# Analyze specific project
|
||||
fission-python-skill analyze-config ./my-fission-project
|
||||
|
||||
# Analyze project in parent directory
|
||||
fission-python-skill analyze-config ../data-processing-service
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## update-docstring.sh
|
||||
|
||||
Parse and update the embedded Fission configuration in Python function docstrings.
|
||||
|
||||
### Usage
|
||||
```bash
|
||||
fission-python-skill update-docstring <file-path> [function-name] [--set "<json>"] [--get] [--help]
|
||||
```
|
||||
|
||||
### Arguments
|
||||
- `file-path`: Path to the Python file containing the function
|
||||
- `function-name`: Optional specific function name to target (if not provided, processes all functions with fission configuration)
|
||||
- `--set "<json>"`: Set the fission configuration to the provided JSON string
|
||||
- `--get`: Get/display the current fission configuration (default action if neither --set nor --get provided)
|
||||
- `--help`: Show help message
|
||||
|
||||
### Description
|
||||
This tool extracts, displays, and can modify the Fission configuration embedded in Python function docstrings. The configuration is expected to be between ```fission and ``` markers in the docstring.
|
||||
|
||||
The tool preserves:
|
||||
- All function code outside the docstring
|
||||
- Docstring content outside the fission configuration blocks
|
||||
- Formatting and indentation of the existing code
|
||||
- Only modifies the content between ```fission markers
|
||||
|
||||
### Examples
|
||||
```bash
|
||||
# View current fission configuration in a function
|
||||
fission-python-skill update-docstring ./src/my_function.py main --get
|
||||
|
||||
# Update fission configuration with new JSON
|
||||
fission-python-skill update-docstring ./src/my_function.py main --set '{"name": "updated-function", "http_triggers": {"updated-trigger": {"url": "/new-endpoint", "methods": ["POST"]}}}'
|
||||
|
||||
# Process all functions with fission configuration in a file
|
||||
fission-python-skill update-docstring ./src/functions.py --get
|
||||
|
||||
# View help
|
||||
fission-python-skill update-docstring --help
|
||||
```
|
||||
|
||||
### Configuration Format
|
||||
The fission configuration should be valid JSON representing the function's fission definition, typically including:
|
||||
- `name`: Function name
|
||||
- `environment`: Optional environment override
|
||||
- `http_triggers`: HTTP endpoint configuration
|
||||
- `schedule_triggers`: Cron-based triggers (if applicable)
|
||||
- `message_queue_triggers`: Message queue triggers (if applicable)
|
||||
|
||||
### Error Handling
|
||||
- Returns error if file doesn't exist
|
||||
- Returns error if function not found (when specified)
|
||||
- Returns error if no fission configuration found in function docstring
|
||||
- Returns error if provided JSON is invalid (when using --set)
|
||||
- Preserves original file on error (no partial writes)
|
||||
|
||||
---
|
||||
|
||||
## Installation
|
||||
|
||||
The fission-python-skill is automatically available when the fission-plugin is installed. To use the tools directly:
|
||||
|
||||
1. Navigate to the plugin directory and run tools directly:
|
||||
```bash
|
||||
cd /path/to/fission-python-skill
|
||||
./create-project.sh my-project
|
||||
```
|
||||
|
||||
2. Or add the plugin directory to your PATH:
|
||||
```bash
|
||||
export PATH="$PATH:/path/to/fission-python-skill"
|
||||
fission-python-skill create-project my-project
|
||||
```
|
||||
|
||||
The plugin is portable and does not require any hardcoded paths - it will locate its template relative to its own location.
|
||||
|
||||
---
|
||||
|
||||
## Template Source
|
||||
|
||||
The project template is sourced from: `fission-python-skill/template/` (relative to the plugin location).
|
||||
|
||||
When creating new projects, the following files/directories are excluded from copying:
|
||||
- .git/
|
||||
- __pycache__/
|
||||
- *.pyc
|
||||
- .env
|
||||
- test/ (test files are copied but may need project-specific updates)
|
||||
|
||||
The plugin is fully portable - it uses relative paths based on the script location, so it can be moved to different directories or systems without breaking.
|
||||
20
fission-python/template/.devcontainer/.env.example
Normal file
20
fission-python/template/.devcontainer/.env.example
Normal file
@@ -0,0 +1,20 @@
|
||||
# For download Rake tool
|
||||
PRIVATE_GIT_TOKEN=
|
||||
|
||||
# Rake tool's profile
|
||||
FISSION_PROFILE=local
|
||||
|
||||
# Rancher K3S version (docker-compose)
|
||||
K3S_VERSION=v1.32.4-k3s1
|
||||
K3S_TOKEN=
|
||||
|
||||
FISSION_VER=v1.21.0
|
||||
FISSION_NAMESPACE=fission
|
||||
|
||||
# Nginx ingress
|
||||
NGINX_INGRESS_VER=v1.7.1
|
||||
|
||||
# Metrics
|
||||
METRICS_NAMESPACE=monitoring
|
||||
OPENTELEMETRY_NAMESPACE=opentelemetry-operator-system
|
||||
JAEGER_NAMESPACE=jaeger
|
||||
42
fission-python/template/.devcontainer/devcontainer.json
Normal file
42
fission-python/template/.devcontainer/devcontainer.json
Normal file
@@ -0,0 +1,42 @@
|
||||
// For format details, see https://aka.ms/devcontainer.json. For config options, see the
|
||||
// README at: https://github.com/devcontainers/templates/tree/main/src/rust
|
||||
{
|
||||
// Or use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
|
||||
// "image": "mcr.microsoft.com/devcontainers/rust:0-1-bullseye",
|
||||
// Use docker compose file
|
||||
"dockerComposeFile": ["docker-compose.yaml", "docker-compose-k3s.yaml"],
|
||||
"service": "devcontainer",
|
||||
"workspaceFolder": "/workspaces/${localWorkspaceFolderBasename}",
|
||||
// Features to add to the dev container. More info: https://containers.dev/features.
|
||||
// "features": {},
|
||||
// Configure tool-specific properties.
|
||||
"customizations": {
|
||||
// Configure properties specific to VS Code.
|
||||
"vscode": {
|
||||
"settings": {"terminal.integrated.defaultProfile.linux": "bash"},
|
||||
"extensions": [
|
||||
// VS Code specific
|
||||
"ms-azuretools.vscode-docker",
|
||||
"dbaeumer.vscode-eslint",
|
||||
"j-brooke.fracturedjsonvsc",
|
||||
// Python specific
|
||||
"ms-python.python",
|
||||
"charliermarsh.ruff",
|
||||
// Markdown specific
|
||||
"yzhang.markdown-all-in-one",
|
||||
// YAML formatter
|
||||
"kennylong.kubernetes-yaml-formatter",
|
||||
// hightlight and format `pyproject.toml`
|
||||
"tamasfe.even-better-toml"
|
||||
]
|
||||
}
|
||||
},
|
||||
"mounts": [],
|
||||
// "runArgs": [
|
||||
// "--env-file",
|
||||
// ".devcontainer/.env"
|
||||
// ],
|
||||
"postStartCommand": "/workspaces/${localWorkspaceFolderBasename}/.devcontainer/initscript.sh",
|
||||
// Use 'forwardPorts' to make a list of ports inside the container available locally.
|
||||
"forwardPorts": []
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
services:
|
||||
k3s-server:
|
||||
image: "rancher/k3s:${K3S_VERSION:-latest}"
|
||||
# command: server --disable traefik --disable servicelb
|
||||
command: server --disable traefik
|
||||
hostname: k3s-server
|
||||
dns:
|
||||
- 10.10.20.100
|
||||
tmpfs: [ "/run", "/var/run" ]
|
||||
ulimits:
|
||||
nproc: 65535
|
||||
nofile:
|
||||
soft: 65535
|
||||
hard: 65535
|
||||
privileged: true
|
||||
restart: always
|
||||
environment:
|
||||
- K3S_TOKEN=${K3S_TOKEN:-secret}
|
||||
- K3S_KUBECONFIG_OUTPUT=/output/kubeconfig.yaml
|
||||
- K3S_KUBECONFIG_MODE=666
|
||||
volumes:
|
||||
- k3s-server:/var/lib/rancher/k3s
|
||||
# This is just so that we get the kubeconfig file out
|
||||
- .:/output
|
||||
ports:
|
||||
- 6443 # Kubernetes API Server
|
||||
- 80 # Ingress controller port 80
|
||||
- 443 # Ingress controller port 443
|
||||
|
||||
k3s-agent:
|
||||
image: "rancher/k3s:${K3S_VERSION:-latest}"
|
||||
hostname: k3s-agent
|
||||
dns:
|
||||
- 10.10.20.100
|
||||
tmpfs: [ "/run", "/var/run" ]
|
||||
ulimits:
|
||||
nproc: 65535
|
||||
nofile:
|
||||
soft: 65535
|
||||
hard: 65535
|
||||
privileged: true
|
||||
restart: always
|
||||
environment:
|
||||
- K3S_URL=https://k3s-server:6443
|
||||
- K3S_TOKEN=${K3S_TOKEN:-secret}
|
||||
volumes:
|
||||
- k3s-agent:/var/lib/rancher/k3s
|
||||
profiles: [ "cluster" ] # only start agent if run with profile `cluster`
|
||||
|
||||
volumes:
|
||||
k3s-server: {}
|
||||
k3s-agent: {}
|
||||
13
fission-python/template/.devcontainer/docker-compose.yaml
Normal file
13
fission-python/template/.devcontainer/docker-compose.yaml
Normal file
@@ -0,0 +1,13 @@
|
||||
services:
|
||||
devcontainer:
|
||||
# All tags avaiable at: https://mcr.microsoft.com/v2/devcontainers/rust/tags/list
|
||||
# image: mcr.microsoft.com/vscode/devcontainers/python:3.10-bullseye
|
||||
image: registry.vegastar.vn/vegacloud/fission-python:3.10-bullseye
|
||||
volumes:
|
||||
- ../..:/workspaces:cached
|
||||
command: sleep infinity
|
||||
env_file:
|
||||
- .env
|
||||
# Comment out depend if you only run devcontainer
|
||||
depends_on:
|
||||
- k3s-server
|
||||
166
fission-python/template/.devcontainer/initscript.sh
Executable file
166
fission-python/template/.devcontainer/initscript.sh
Executable file
@@ -0,0 +1,166 @@
|
||||
#!/bin/bash
|
||||
|
||||
## For debugging
|
||||
# set -eux
|
||||
|
||||
|
||||
# wait few seconds to ensure k3s server is ready
|
||||
sleep 60
|
||||
|
||||
|
||||
#############################
|
||||
### DEV PACKAGES
|
||||
#############################
|
||||
export RAKE_VER=0.1.7
|
||||
|
||||
curl -L https://$PRIVATE_GIT_TOKEN@registry.vegastar.vn/vegacloud/make/releases/download/$RAKE_VER/rake-$RAKE_VER-x86_64-unknown-linux-musl.tar.gz | tar xzv -C /tmp/
|
||||
sudo install -o root -g root -m 0755 /tmp/rake-$RAKE_VER-x86_64-unknown-linux-musl/rake /usr/local/bin/rake
|
||||
|
||||
#############################
|
||||
### KUBECTL
|
||||
#############################
|
||||
|
||||
## Config kubectl
|
||||
mkdir -p ~/.kube
|
||||
cp ${PWD}/.devcontainer/kubeconfig.yaml ~/.kube/config
|
||||
sed -i 's/127.0.0.1/k3s-server/g' ~/.kube/config
|
||||
|
||||
## allow insecure connection
|
||||
shopt -s expand_aliases
|
||||
echo 'alias kubectl="kubectl --insecure-skip-tls-verify"' >> ~/.bashrc
|
||||
echo 'alias k="kubectl --insecure-skip-tls-verify"' >> ~/.bashrc
|
||||
|
||||
#############################
|
||||
### K9S
|
||||
#############################
|
||||
|
||||
# install k9s
|
||||
wget https://github.com/derailed/k9s/releases/download/v0.50.6/k9s_linux_amd64.deb -O /tmp/k9s_linux_amd64.deb
|
||||
sudo dpkg -i /tmp/k9s_linux_amd64.deb
|
||||
|
||||
#############################
|
||||
### NGINX INGRESS
|
||||
#############################
|
||||
|
||||
# kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-$NGINX_INGRESS_VER/deploy/static/provider/cloud/deploy.yaml
|
||||
# cat <<EOT >> /tmp/nginx-service.yaml
|
||||
# apiVersion: v1
|
||||
# kind: Service
|
||||
# metadata:
|
||||
# name: ingress-nginx-controller-loadbalancer
|
||||
# namespace: ingress-nginx
|
||||
# spec:
|
||||
# selector:
|
||||
# app.kubernetes.io/component: controller
|
||||
# app.kubernetes.io/instance: ingress-nginx
|
||||
# app.kubernetes.io/name: ingress-nginx
|
||||
# ports:
|
||||
# - name: http
|
||||
# port: 80
|
||||
# protocol: TCP
|
||||
# targetPort: 80
|
||||
# - name: https
|
||||
# port: 443
|
||||
# protocol: TCP
|
||||
# targetPort: 443
|
||||
# type: LoadBalancer
|
||||
# EOT
|
||||
# kubectl apply -f /tmp/nginx-service.yaml
|
||||
# rm -f /tmp/nginx-service.yaml
|
||||
|
||||
#############################
|
||||
### OPEN TELEMETRY
|
||||
#############################
|
||||
# kubectl create namespace $JAEGER_NAMESPACE
|
||||
# kubectl create namespace $OPENTELEMETRY_NAMESPACE
|
||||
|
||||
# ## cert-manager
|
||||
# kubectl apply -f https://github.com/jetstack/cert-manager/releases/latest/download/cert-manager.yaml
|
||||
|
||||
# ## install jaeger
|
||||
# helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
|
||||
# helm install jaeger jaegertracing/jaeger -n $JAEGER_NAMESPACE
|
||||
# kubectl -n $JAEGER_NAMESPACE get po
|
||||
|
||||
# ## open telemetry operator
|
||||
# kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
|
||||
|
||||
# ## create an OpenTelemetry Collector instance
|
||||
# kubectl -n $OPENTELEMETRY_NAMESPACE apply -f .devcontainer/helm/opentelemetry-collector.yaml
|
||||
|
||||
#############################
|
||||
### FISSION PODs
|
||||
#############################
|
||||
kubectl create namespace $FISSION_NAMESPACE
|
||||
|
||||
# ## install with helm
|
||||
# kubectl create -k "github.com/fission/fission/crds/v1?ref=${FISSION_VER}"
|
||||
# helm repo add fission-charts https://fission.github.io/fission-charts/ && helm repo update
|
||||
# kubectl apply -f - <<EOF
|
||||
# apiVersion: v1
|
||||
# kind: Namespace
|
||||
# metadata:
|
||||
# name: fission
|
||||
# ---
|
||||
# apiVersion: v1
|
||||
# kind: Namespace
|
||||
# metadata:
|
||||
# name: gh-eom
|
||||
# EOF
|
||||
# kubectl apply -f - <<EOF
|
||||
# type: kubernetes.io/dockerconfigjson
|
||||
# apiVersion: v1
|
||||
# kind: Secret
|
||||
# metadata:
|
||||
# name: vega-container-registry
|
||||
# namespace: fission
|
||||
# data:
|
||||
# .dockerconfigjson: >-
|
||||
# eyJhdXRocyI6eyJyZWdpc3RyeS52ZWdhc3Rhci52biI6eyJ1c2VybmFtZSI6InRpZW5kZCIsInBhc3N3b3JkIjoiYTBjY2JjMDVjNzMyYzExMjU3OTg1NjMwNjY5ZTFjNjEyNDg0NzU1MyIsImF1dGgiOiJkR2xsYm1Sa09tRXdZMk5pWXpBMVl6Y3pNbU14TVRJMU56azROVFl6TURZMk9XVXhZell4TWpRNE5EYzFOVE09In19fQ==
|
||||
# EOF
|
||||
# helm upgrade --install fission fission-charts/fission-all --namespace $FISSION_NAMESPACE -f - <<EOF
|
||||
# imagePullSecrets:
|
||||
# - name: vega-container-registry
|
||||
# defaultNamespace: default
|
||||
# additionalFissionNamespaces:
|
||||
# - gh-eom
|
||||
# EOF
|
||||
|
||||
## install without helm
|
||||
kubectl create -k "github.com/fission/fission/crds/v1?ref=${FISSION_VER}"
|
||||
kubectl create namespace $FISSION_NAMESPACE
|
||||
kubectl config set-context --current --namespace=$FISSION_NAMESPACE
|
||||
kubectl apply -f https://github.com/fission/fission/releases/download/${FISSION_VER}/fission-all-${FISSION_VER}-minikube.yaml
|
||||
kubectl config set-context --current --namespace=default #to change context to default namespace after installation
|
||||
|
||||
|
||||
#############################
|
||||
### PROMETHEUS AND GRAFANA
|
||||
#############################
|
||||
# kubectl create namespace $METRICS_NAMESPACE
|
||||
|
||||
# helm repo add prometheus-community https://prometheus-community.github.io/helm-charts && helm repo update
|
||||
# helm install prometheus prometheus-community/kube-prometheus-stack -n $METRICS_NAMESPACE
|
||||
|
||||
#############################
|
||||
### UPDATE FISSION
|
||||
#############################
|
||||
|
||||
# helm upgrade fission fission-charts/fission-all --namespace $FISSION_NAMESPACE -f .devcontainer/helm/fission-values.yaml
|
||||
|
||||
#############################
|
||||
### PORT FORWARDING
|
||||
#############################
|
||||
|
||||
## To access jaeger-query, you can use Kubernetes port forwarding
|
||||
# kubectl -n jaeger port-forward svc/jaeger-query 8080:80 --address='0.0.0.0'
|
||||
## To access kabana, you can use Kubernetes port forwarding
|
||||
# kubectl --namespace monitoring port-forward svc/prometheus-grafana 3000:80
|
||||
## For password, you'll need to run the following command:
|
||||
# kubectl get secret --namespace monitoring prometheus-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
|
||||
|
||||
#############################
|
||||
### INSTALLING PYTHON PACKAGES
|
||||
#############################
|
||||
|
||||
pip install -r dev-requirements.txt -r src/requirements.txt
|
||||
24
fission-python/template/.env.example
Normal file
24
fission-python/template/.env.example
Normal file
@@ -0,0 +1,24 @@
|
||||
# PostgreSQL Database Configuration
|
||||
PG_HOST=
|
||||
PG_PORT=5432
|
||||
PG_DB=
|
||||
PG_USER=
|
||||
PG_PASS=
|
||||
PG_DBSCHEMA=public
|
||||
|
||||
# Optional: Service-specific configuration (via ConfigMap)
|
||||
# YOUR_SERVICE_CONFIG_ENDPOINT=
|
||||
|
||||
# Optional: Vault encryption key (32-byte hex string)
|
||||
# Required if using encrypted secrets (vault:v1:...)
|
||||
CRYPTO_KEY=
|
||||
|
||||
# Example: If using MinIO/S3
|
||||
# S3_ENDPOINT=
|
||||
# S3_ACCESS_KEY=
|
||||
# S3_SECRET_KEY=
|
||||
# S3_BUCKET=
|
||||
|
||||
# Example: If using external APIs
|
||||
# API_ENDPOINT=
|
||||
# API_KEY=
|
||||
65
fission-python/template/.fission/deployment.json
Normal file
65
fission-python/template/.fission/deployment.json
Normal file
@@ -0,0 +1,65 @@
|
||||
{
|
||||
"namespace": "default",
|
||||
"environments": {
|
||||
"${PROJECT_NAME}-py": {
|
||||
"image": "ghcr.io/fission/python-env",
|
||||
"builder": "ghcr.io/fission/python-builder",
|
||||
"mincpu": 50,
|
||||
"maxcpu": 100,
|
||||
"minmemory": 50,
|
||||
"maxmemory": 500,
|
||||
"poolsize": 1
|
||||
}
|
||||
},
|
||||
"archives": { "package.zip": {"sourcepath": "src"} },
|
||||
"packages": {
|
||||
"${PROJECT_NAME}": {
|
||||
"buildcmd": "./build.sh",
|
||||
"sourcearchive": "package.zip",
|
||||
"env": "${PROJECT_NAME}-py"
|
||||
}
|
||||
},
|
||||
"function_common": {
|
||||
"pkg": "${PROJECT_NAME}",
|
||||
"secrets": ["fission-${PROJECT_NAME}-env"],
|
||||
"configmaps": ["fission-${PROJECT_NAME}-config"],
|
||||
"executor": {
|
||||
"select": "poolmgr",
|
||||
"poolmgr": {
|
||||
"concurrency": 1,
|
||||
"requestsperpod": 1,
|
||||
"onceonly": false
|
||||
},
|
||||
"newdeploy": {
|
||||
"minscale": 1,
|
||||
"maxscale": 1,
|
||||
"targetcpu": 80
|
||||
}
|
||||
},
|
||||
"mincpu": 50,
|
||||
"maxcpu": 100,
|
||||
"minmemory": 50,
|
||||
"maxmemory": 500
|
||||
},
|
||||
"secrets": {
|
||||
"fission-${PROJECT_NAME}-env": {
|
||||
"literals": [
|
||||
"PG_HOST=YOUR_DB_HOST",
|
||||
"PG_PORT=5432",
|
||||
"PG_DB=YOUR_DB_NAME",
|
||||
"PG_USER=YOUR_DB_USER",
|
||||
"PG_PASS=YOUR_DB_PASSWORD",
|
||||
"PG_DBSCHEMA=public"
|
||||
]
|
||||
}
|
||||
},
|
||||
"configmaps": {
|
||||
"fission-${PROJECT_NAME}-config": {
|
||||
"literals": [
|
||||
"FN_OPTIONAL_CONFIG=http://example.com/config"
|
||||
]
|
||||
}
|
||||
},
|
||||
"imagepullsecret": "",
|
||||
"runtime_envs": {}
|
||||
}
|
||||
22
fission-python/template/.fission/dev-deployment.json
Normal file
22
fission-python/template/.fission/dev-deployment.json
Normal file
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"namespace": "fission-dev",
|
||||
"secrets": {
|
||||
"fission-${PROJECT_NAME}-env": {
|
||||
"literals": [
|
||||
"PG_HOST=dev-db.example.com",
|
||||
"PG_PORT=5432",
|
||||
"PG_DB=devdb",
|
||||
"PG_USER=${PROJECT_NAME}-dev",
|
||||
"PG_PASS=dev-password"
|
||||
]
|
||||
}
|
||||
},
|
||||
"configmaps": {
|
||||
"fission-${PROJECT_NAME}-config": {
|
||||
"literals": [
|
||||
"LOG_LEVEL=DEBUG",
|
||||
"FISSION_ROUTE_SERVICE_ENDPOINT=http://router.fission.svc.cluster.local"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
32
fission-python/template/.fission/local-deployment.json
Normal file
32
fission-python/template/.fission/local-deployment.json
Normal file
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"namespace": "default",
|
||||
"environments": {
|
||||
"${PROJECT_NAME}-py": {
|
||||
"image": "ghcr.io/fission/python-env:3.11",
|
||||
"builder": "ghcr.io/fission/python-builder:3.11",
|
||||
"mincpu": 100,
|
||||
"maxcpu": 200,
|
||||
"minmemory": 128,
|
||||
"maxmemory": 256,
|
||||
"poolsize": 1
|
||||
}
|
||||
},
|
||||
"secrets": {
|
||||
"fission-${PROJECT_NAME}-env": {
|
||||
"literals": [
|
||||
"PG_HOST=localhost",
|
||||
"PG_PORT=5432",
|
||||
"PG_DB=testdb",
|
||||
"PG_USER=postgres",
|
||||
"PG_PASS=test"
|
||||
]
|
||||
}
|
||||
},
|
||||
"configmaps": {
|
||||
"fission-${PROJECT_NAME}-config": {
|
||||
"literals": [
|
||||
"LOG_LEVEL=DEBUG"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
name: "K8S Fission Code Analystics"
|
||||
on:
|
||||
workflow_dispatch:
|
||||
jobs:
|
||||
sonarqube:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
- name: 🔍 SonarQube Scan
|
||||
id: scan
|
||||
uses: sonarsource/sonarqube-scan-action@master
|
||||
env:
|
||||
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
|
||||
SONAR_HOST_URL: ${{ secrets.SONAR_HOST_URL }}
|
||||
with:
|
||||
args: >
|
||||
-Dsonar.projectKey=${{ github.event.repository.name }} -Dsonar.sources=.
|
||||
- name: 🔔 Send notification
|
||||
uses: appleboy/telegram-action@master
|
||||
if: always()
|
||||
with:
|
||||
to: ${{ secrets.TELEGRAM_TO }}
|
||||
token: ${{ secrets.TELEGRAM_TOKEN }}
|
||||
format: markdown
|
||||
socks5: ${{ secrets.TELEGRAM_PROXY_URL != '' && secrets.TELEGRAM_PROXY_URL || '' }}
|
||||
message: |
|
||||
${{ steps.scan.outcome == 'success' && '🟢 (=^ ◡ ^=)' || '🔴 (｡•́︿•̀｡)' }} Scanned ${{ github.event.repository.name }}
|
||||
*Msg*: `${{ github.event.commits[0].message }}`
|
||||
72
fission-python/template/.gitea/workflows/dev-deployment.yaml
Normal file
72
fission-python/template/.gitea/workflows/dev-deployment.yaml
Normal file
@@ -0,0 +1,72 @@
|
||||
name: "Development Deployment"
|
||||
on:
|
||||
push:
|
||||
branches: [ main, develop ]
|
||||
workflow_dispatch:
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
name: Deploy to development
|
||||
runs-on: ubuntu-latest
|
||||
environment: development
|
||||
steps:
|
||||
- name: ☸️ Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: 🐍 Setup Python
|
||||
uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.11'
|
||||
|
||||
- name: 📦 Install dependencies
|
||||
run: |
|
||||
pip install -r dev-requirements.txt
|
||||
|
||||
- name: 🔍 Lint with flake8
|
||||
run: flake8 src/ --max-line-length=88 --extend-ignore=E203,W503
|
||||
|
||||
- name: 🎨 Check formatting with black
|
||||
run: black --check src/
|
||||
|
||||
- name: 🧪 Run tests
|
||||
run: pytest --cov=src --cov-report=xml
|
||||
|
||||
- name: 📤 Upload coverage to Codecov
|
||||
uses: codecov/codecov-action@v4
|
||||
with:
|
||||
token: ${{ secrets.CODECOV_TOKEN }}
|
||||
files: coverage.xml
|
||||
fail_ci_if_error: false
|
||||
|
||||
- name: ☸️ Setup kubectl
|
||||
uses: azure/setup-kubectl@v4
|
||||
with:
|
||||
version: 'v1.28.0'
|
||||
|
||||
- name: 🔐 Configure Kubeconfig
|
||||
uses: azure/k8s-set-context@v4
|
||||
with:
|
||||
method: kubeconfig
|
||||
kubeconfig: ${{ secrets.KUBECONFIG_DEV }}
|
||||
|
||||
- name: 🚀 Install Fission CLI
|
||||
run: |
|
||||
curl -L https://github.com/fission/fission/releases/latest/download/fission-linux-amd64 -o /tmp/fission
|
||||
sudo install /tmp/fission /usr/local/bin/fission
|
||||
fission check
|
||||
|
||||
- name: 📦 Build and Deploy (dev)
|
||||
run: |
|
||||
echo "Deploying to development environment..."
|
||||
fission deploy --dev
|
||||
|
||||
- name: 🔔 Notify success
|
||||
if: always()
|
||||
uses: actions/github-script@v7
|
||||
with:
|
||||
script: |
|
||||
const status = '${{ job.status }}';
|
||||
const emoji = status === 'success' ? '🟢' : '🔴';
|
||||
const message = `${emoji} Dev deployment ${status} for ${{ github.repository }}@${{ github.sha }}\nCommit: ${{ github.event.commits[0].message }}`;
|
||||
// Send to Slack/Telegram/etc - customize as needed
|
||||
console.log(message);
|
||||
@@ -0,0 +1,56 @@
|
||||
name: "Manual Deployment"
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
environment:
|
||||
description: 'Deployment environment (dev, staging, prod)'
|
||||
required: true
|
||||
type: choice
|
||||
options:
|
||||
- dev
|
||||
- staging
|
||||
- prod
|
||||
|
||||
jobs:
|
||||
deploy:
|
||||
name: Deploy to ${{ github.event.inputs.environment }}
|
||||
runs-on: ubuntu-latest
|
||||
environment: ${{ github.event.inputs.environment }}
|
||||
steps:
|
||||
- name: ☸️ Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: ☸️ Setup kubectl
|
||||
uses: azure/setup-kubectl@v4
|
||||
with:
|
||||
version: 'v1.28.0'
|
||||
|
||||
- name: 🔐 Configure Kubeconfig
|
||||
uses: azure/k8s-set-context@v4
|
||||
with:
|
||||
method: kubeconfig
|
||||
kubeconfig: ${{ secrets[format('KUBECONFIG_{0}', github.event.inputs.environment)] }}
|
||||
|
||||
- name: 🚀 Install Fission CLI
|
||||
run: |
|
||||
curl -L https://github.com/fission/fission/releases/latest/download/fission-linux-amd64 -o /tmp/fission
|
||||
sudo install /tmp/fission /usr/local/bin/fission
|
||||
fission check
|
||||
|
||||
- name: 📦 Deploy
|
||||
run: |
|
||||
echo "Deploying to ${{ github.event.inputs.environment }} environment..."
|
||||
if [ "${{ github.event.inputs.environment }}" = "dev" ]; then
|
||||
fission deploy --dev
|
||||
else
|
||||
fission deploy
|
||||
fi
|
||||
|
||||
- name: 🔔 Notify
|
||||
if: always()
|
||||
uses: actions/github-script@v7
|
||||
with:
|
||||
script: |
|
||||
const env = '${{ github.event.inputs.environment }}';
|
||||
const status = '${{ job.status }}';
|
||||
console.log(`Deployment to ${env} completed with status: ${status}`);
|
||||
@@ -0,0 +1,54 @@
|
||||
name: "Manual Uninstall"
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
environment:
|
||||
description: 'Environment to uninstall from (dev, staging, prod)'
|
||||
required: true
|
||||
type: choice
|
||||
options:
|
||||
- dev
|
||||
- staging
|
||||
- prod
|
||||
|
||||
jobs:
|
||||
uninstall:
|
||||
name: Uninstall from ${{ github.event.inputs.environment }}
|
||||
runs-on: ubuntu-latest
|
||||
environment: ${{ github.event.inputs.environment }}
|
||||
steps:
|
||||
- name: ☸️ Checkout
|
||||
uses: actions/checkout@v4
|
||||
|
||||
- name: ☸️ Setup kubectl
|
||||
uses: azure/setup-kubectl@v4
|
||||
with:
|
||||
version: 'v1.28.0'
|
||||
|
||||
- name: 🔐 Configure Kubeconfig
|
||||
uses: azure/k8s-set-context@v4
|
||||
with:
|
||||
method: kubeconfig
|
||||
kubeconfig: ${{ secrets[format('KUBECONFIG_{0}', github.event.inputs.environment)] }}
|
||||
|
||||
- name: 🚀 Install Fission CLI
|
||||
run: |
|
||||
curl -L https://github.com/fission/fission/releases/latest/download/fission-linux-amd64 -o /tmp/fission
|
||||
sudo install /tmp/fission /usr/local/bin/fission
|
||||
fission check
|
||||
|
||||
- name: 🗑️ Uninstall functions
|
||||
run: |
|
||||
echo "Uninstalling from ${{ github.event.inputs.environment }} environment..."
|
||||
# Delete all functions in this repository/package
|
||||
# Note: This will remove functions defined in deployment.json
|
||||
fission function list --all-namespaces | grep "${{ github.event.repository.name }}" | awk '{print $1}' | xargs -r fission function delete --name
|
||||
|
||||
- name: 🔔 Notify
|
||||
if: always()
|
||||
uses: actions/github-script@v7
|
||||
with:
|
||||
script: |
|
||||
const env = '${{ github.event.inputs.environment }}';
|
||||
const status = '${{ job.status }}';
|
||||
console.log(`Uninstall from ${env} completed with status: ${status}`);
|
||||
190
fission-python/template/.gitignore
vendored
Normal file
190
fission-python/template/.gitignore
vendored
Normal file
@@ -0,0 +1,190 @@
|
||||
# Byte-compiled / optimized / DLL files
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
|
||||
# C extensions
|
||||
*.so
|
||||
|
||||
# Distribution / packaging
|
||||
.Python
|
||||
build/
|
||||
develop-eggs/
|
||||
dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
lib/
|
||||
lib64/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
wheels/
|
||||
share/python-wheels/
|
||||
# *.egg-info/
|
||||
.installed.cfg
|
||||
*.egg
|
||||
MANIFEST
|
||||
|
||||
# PyInstaller
|
||||
# Usually these files are written by a python script from a template
|
||||
# before PyInstaller builds the exe, so as to inject date/other infos into it.
|
||||
*.manifest
|
||||
*.spec
|
||||
|
||||
# Installer logs
|
||||
pip-log.txt
|
||||
pip-delete-this-directory.txt
|
||||
|
||||
# Unit test / coverage reports
|
||||
htmlcov/
|
||||
.tox/
|
||||
.nox/
|
||||
.coverage
|
||||
.coverage.*
|
||||
.cache
|
||||
nosetests.xml
|
||||
coverage.xml
|
||||
*.cover
|
||||
*.py,cover
|
||||
.hypothesis/
|
||||
.pytest_cache/
|
||||
cover/
|
||||
|
||||
# Translations
|
||||
*.mo
|
||||
*.pot
|
||||
|
||||
# Django stuff:
|
||||
*.log
|
||||
local_settings.py
|
||||
db.sqlite3
|
||||
db.sqlite3-journal
|
||||
|
||||
# Flask stuff:
|
||||
instance/
|
||||
.webassets-cache
|
||||
|
||||
# Scrapy stuff:
|
||||
.scrapy
|
||||
|
||||
# Sphinx documentation
|
||||
docs/_build/
|
||||
|
||||
# PyBuilder
|
||||
.pybuilder/
|
||||
target/
|
||||
|
||||
# Jupyter Notebook
|
||||
.ipynb_checkpoints
|
||||
|
||||
# IPython
|
||||
profile_default/
|
||||
ipython_config.py
|
||||
|
||||
# pyenv
|
||||
# For a library or package, you might want to ignore these files since the code is
|
||||
# intended to run in multiple environments; otherwise, check them in:
|
||||
# .python-version
|
||||
|
||||
# pipenv
|
||||
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
|
||||
# However, in case of collaboration, if having platform-specific dependencies or dependencies
|
||||
# having no cross-platform support, pipenv may install dependencies that don't work, or not
|
||||
# install all needed dependencies.
|
||||
#Pipfile.lock
|
||||
|
||||
# poetry
|
||||
# Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.
|
||||
# This is especially recommended for binary packages to ensure reproducibility, and is more
|
||||
# commonly ignored for libraries.
|
||||
# https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control
|
||||
#poetry.lock
|
||||
|
||||
# pdm
|
||||
# Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.
|
||||
#pdm.lock
|
||||
# pdm stores project-wide configurations in .pdm.toml, but it is recommended to not include it
|
||||
# in version control.
|
||||
# https://pdm.fming.dev/latest/usage/project/#working-with-version-control
|
||||
.pdm.toml
|
||||
.pdm-python
|
||||
.pdm-build/
|
||||
|
||||
# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm
|
||||
__pypackages__/
|
||||
|
||||
# Celery stuff
|
||||
celerybeat-schedule
|
||||
celerybeat.pid
|
||||
|
||||
# SageMath parsed files
|
||||
*.sage.py
|
||||
|
||||
# Environments
|
||||
.env
|
||||
.venv
|
||||
env/
|
||||
venv/
|
||||
ENV/
|
||||
env.bak/
|
||||
venv.bak/
|
||||
|
||||
# Spyder project settings
|
||||
.spyderproject
|
||||
.spyproject
|
||||
|
||||
# Rope project settings
|
||||
.ropeproject
|
||||
|
||||
# mkdocs documentation
|
||||
/site
|
||||
|
||||
# mypy
|
||||
.mypy_cache/
|
||||
.dmypy.json
|
||||
dmypy.json
|
||||
|
||||
# Pyre type checker
|
||||
.pyre/
|
||||
|
||||
# pytype static type analyzer
|
||||
.pytype/
|
||||
|
||||
# Cython debug symbols
|
||||
cython_debug/
|
||||
|
||||
# PyCharm
|
||||
# JetBrains specific template is maintained in a separate JetBrains.gitignore that can
|
||||
# be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
|
||||
# and can be added to the global gitignore or merged into this file. For a more nuclear
|
||||
# option (not recommended) you can uncomment the following to ignore the entire idea folder.
|
||||
#.idea/
|
||||
|
||||
## Ignore Temporary directory of Dagster
|
||||
/tmp*
|
||||
|
||||
## Devcontainer cache files, that will make devcontainer start faster after first run
|
||||
/.vscache/.vscode-server/*
|
||||
!/.vscache/.vscode-server/.gitkeep
|
||||
/.vscache/.devcontainer/*
|
||||
!/.vscache/.devcontainer/.gitkeep
|
||||
|
||||
## Ignore K3S config file
|
||||
/.devcontainer/kubeconfig.yaml
|
||||
|
||||
## Ignore packaged files
|
||||
/*.zip
|
||||
# !/package.zip
|
||||
/*.bak
|
||||
|
||||
# No Makefile in this template - uses build.sh instead
|
||||
|
||||
## Ignore fission's specs files
|
||||
/specs/*
|
||||
!/specs/fission-deployment-config.yaml
|
||||
!/specs/README
|
||||
|
||||
/manifests/*
|
||||
|
||||
/fission-dumps
|
||||
436
fission-python/template/README.md
Normal file
436
fission-python/template/README.md
Normal file
@@ -0,0 +1,436 @@
|
||||
# Fission Python Template
|
||||
|
||||
A production-ready template for building Fission serverless Python functions with best practices for configuration, database connectivity, error handling, and testing.
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
project/
|
||||
├── .fission/
|
||||
│ ├── deployment.json # Fission function deployment configuration
|
||||
│ ├── dev-deployment.json # Development overrides
|
||||
│ └── local-deployment.json # Local development overrides
|
||||
├── src/
|
||||
│ ├── __init__.py # Package initialization
|
||||
│ ├── vault.py # Vault encryption/decryption utilities
|
||||
│ ├── helpers.py # Shared utilities (DB, secrets, configs)
|
||||
│ ├── exceptions.py # Custom exception hierarchy
|
||||
│ ├── models.py # Pydantic models (request/response schemas)
|
||||
│ ├── build.sh # Package build script
|
||||
│ └── your_function.py # Your function implementations
|
||||
├── test/
|
||||
│ ├── __init__.py
|
||||
│ ├── test_*.py # Unit tests
|
||||
│ └── requirements.txt # Test dependencies
|
||||
├── migrates/
|
||||
│ └── schema.sql # Database migration scripts
|
||||
├── manifests/ # Kubernetes manifests (optional)
|
||||
├── specs/ # Generated Fission specs (created by fission CLI)
|
||||
├── requirements.txt # Runtime dependencies
|
||||
├── dev-requirements.txt # Development dependencies
|
||||
├── .env.example # Environment variable template
|
||||
├── pytest.ini # Pytest configuration
|
||||
└── README.md # Project documentation
|
||||
```
|
||||
|
||||
## Key Components
|
||||
|
||||
### Fission Configuration in Docstrings
|
||||
|
||||
Fission reads function metadata from docstrings using the ````fission` marker:
|
||||
|
||||
```python
|
||||
def my_function(event, context):
|
||||
"""
|
||||
```fission
|
||||
{
|
||||
"name": "my-function",
|
||||
"http_triggers": {
|
||||
"my-trigger": {
|
||||
"url": "/api/my-endpoint",
|
||||
"methods": ["GET", "POST"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
"""
|
||||
# Your implementation
|
||||
return {"message": "Hello World"}
|
||||
```
|
||||
|
||||
**Note:** Do not use `fission.yaml` or `fission.json`. The Fission Python builder reads the docstring annotations directly from your Python source files.
|
||||
|
||||
### Environment Variables & Secrets
|
||||
|
||||
Configuration is managed through Kubernetes Secrets and ConfigMaps:
|
||||
|
||||
- **Secrets**: Database credentials, API keys, encryption keys (sensitive)
|
||||
- **ConfigMaps**: Non-sensitive configuration, endpoints, feature flags
|
||||
|
||||
Access them via helper functions:
|
||||
|
||||
```python
|
||||
from helpers import get_secret, get_config
|
||||
|
||||
# Read secret (with optional default)
|
||||
db_host = get_secret("PG_HOST", "localhost")
|
||||
db_port = int(get_secret("PG_PORT", "5432"))
|
||||
|
||||
# Read config
|
||||
api_endpoint = get_config("EXTERNAL_API_ENDPOINT")
|
||||
```
|
||||
|
||||
**Placeholder variables** in `deployment.json`:
|
||||
- `${PROJECT_NAME}` - Replaced with your actual project name during project creation
|
||||
- Secret/configmap names follow pattern: `fission-${PROJECT_NAME}-env` and `fission-${PROJECT_NAME}-config`
|
||||
|
||||
### Database Connectivity
|
||||
|
||||
Use the provided `init_db_connection()` helper:
|
||||
|
||||
```python
|
||||
from helpers import init_db_connection, db_rows_to_array
|
||||
|
||||
conn = init_db_connection()
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("SELECT * FROM items")
|
||||
rows = db_rows_to_array(cursor, cursor.fetchall())
|
||||
```
|
||||
|
||||
The helper automatically:
|
||||
- Reads connection parameters from secrets (PG_HOST, PG_PORT, PG_DB, PG_USER, PG_PASS, PG_DBSCHEMA)
|
||||
- Checks port connectivity before connecting
|
||||
- Uses LoggingConnection for query logging
|
||||
- Applies schema search path if PG_DBSCHEMA is set
|
||||
|
||||
### Error Handling
|
||||
|
||||
Use the exception hierarchy from `exceptions.py`:
|
||||
|
||||
```python
|
||||
from exceptions import ValidationError, NotFoundError, ConflictError, DatabaseError
|
||||
|
||||
def get_item(item_id: str):
|
||||
item = db.fetch_one(item_id)
|
||||
if not item:
|
||||
raise NotFoundError(f"Item {item_id} not found", x_user=get_user_from_headers())
|
||||
return item
|
||||
```
|
||||
|
||||
All exceptions return standardized error responses:
|
||||
|
||||
```json
|
||||
{
|
||||
"error_code": "NOT_FOUND",
|
||||
"http_status": 404,
|
||||
"error_msg": "Item 123 not found",
|
||||
"x_user": "user-456",
|
||||
"details": {"item_id": "123"}
|
||||
}
|
||||
```
|
||||
|
||||
### Validation with Pydantic
|
||||
|
||||
Validate request payloads using Pydantic models:
|
||||
|
||||
```python
|
||||
from models import ItemCreateRequest
|
||||
from pydantic import ValidationError as PydanticValidationError
|
||||
|
||||
def create_item():
|
||||
try:
|
||||
data = ItemCreateRequest(**request.get_json())
|
||||
except PydanticValidationError as e:
|
||||
raise ValidationError(str(e), details=e.errors())
|
||||
```
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### 1. Install Dependencies
|
||||
|
||||
```bash
|
||||
# Install runtime and development dependencies
|
||||
pip install -r dev-requirements.txt
|
||||
|
||||
# Or just runtime dependencies
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
### 2. Local Testing
|
||||
|
||||
Fission provides `fission spec` to test specs locally:
|
||||
|
||||
```bash
|
||||
# Verify your deployment configuration
|
||||
fission spec verify --file=.fission/deployment.json
|
||||
|
||||
# Build and test locally
|
||||
fission function test --name your-function
|
||||
```
|
||||
|
||||
### 3. Unit Testing
|
||||
|
||||
Run tests with pytest:
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
pytest
|
||||
|
||||
# Run with coverage
|
||||
pytest --cov=src
|
||||
|
||||
# Run specific test file
|
||||
pytest test/test_my_function.py
|
||||
|
||||
# Verbose output
|
||||
pytest -v
|
||||
```
|
||||
|
||||
**Example test structure:**
|
||||
|
||||
```python
|
||||
# test/test_my_function.py
|
||||
import pytest
|
||||
from unittest.mock import patch
|
||||
from src.my_function import main
|
||||
|
||||
def test_my_function_success():
|
||||
event = {"key": "value"}
|
||||
context = {}
|
||||
result = main(event, context)
|
||||
assert result["status"] == "success"
|
||||
|
||||
@patch("helpers.init_db_connection")
|
||||
def test_my_function_with_db(mock_db):
|
||||
# Mock database connection
|
||||
mock_conn = MagicMock()
|
||||
mock_db.return_value = mock_conn
|
||||
# Test function
|
||||
```
|
||||
|
||||
### 4. Building the Package
|
||||
|
||||
The `build.sh` script installs dependencies and packages your code:
|
||||
|
||||
```bash
|
||||
# From project root
|
||||
./src/build.sh
|
||||
|
||||
# This produces a package.zip in the specs directory
|
||||
# Ready for deployment with: fission deploy
|
||||
```
|
||||
|
||||
The build script detects the OS (Debian/Alpine) and installs the correct build dependencies (gcc, libpq-dev, python3-dev).
|
||||
|
||||
### 5. Deployment
|
||||
|
||||
```bash
|
||||
# Deploy to Fission
|
||||
fission deploy
|
||||
|
||||
# Or deploy specific function
|
||||
fission function update --name my-function --env your-env
|
||||
```
|
||||
|
||||
## Deployment Configuration
|
||||
|
||||
### Executors
|
||||
|
||||
Choose between two executor types in `deployment.json`:
|
||||
|
||||
**poolmgr** (default) - Good for high-concurrency HTTP functions:
|
||||
```json
|
||||
"executor": {
|
||||
"select": "poolmgr",
|
||||
"poolmgr": {
|
||||
"concurrency": 1,
|
||||
"requestsperpod": 1,
|
||||
"onceonly": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**newdeploy** - Good for dedicated scaling:
|
||||
```json
|
||||
"executor": {
|
||||
"select": "newdeploy",
|
||||
"newdeploy": {
|
||||
"minscale": 1,
|
||||
"maxscale": 5,
|
||||
"targetcpu": 80
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Resource Limits
|
||||
|
||||
Set resource allocation in `function_common`:
|
||||
- `mincpu` / `maxcpu` - CPU allocation in millicores (50 = 0.05 cores)
|
||||
- `minmemory` / `maxmemory` - Memory in MB
|
||||
- Adjust based on your function's needs
|
||||
|
||||
### Environment-Specific Overrides
|
||||
|
||||
Use `dev-deployment.json` for development environment (different secrets, lower resources). Fission will automatically use it when `--dev` flag is passed.
|
||||
|
||||
## Vault Encryption
|
||||
|
||||
For encrypted secrets, use the vault utility functions:
|
||||
|
||||
```python
|
||||
from vault import encrypt_vault, decrypt_vault, is_valid_vault_format
|
||||
|
||||
# Encrypt a value (run locally to generate vault string)
|
||||
encrypted = encrypt_vault("my-secret", "your-hex-key-here")
|
||||
# Result: "vault:v1:base64-encrypted-data"
|
||||
|
||||
# Store the encrypted string in your K8s secret
|
||||
# The helper will auto-decrypt if is_valid_vault_format() returns True
|
||||
```
|
||||
|
||||
**Important:** Set `CRYPTO_KEY` in your helpers.py (or via environment override) to your actual 32-byte key in hex format.
|
||||
|
||||
## Testing Strategies
|
||||
|
||||
### Unit Tests
|
||||
- Mock external dependencies (database, HTTP calls)
|
||||
- Test business logic isolation
|
||||
- Use `pytest-mock` for convenient mocking
|
||||
|
||||
### Integration Tests
|
||||
- Use a test database
|
||||
- Clean up test data after each run
|
||||
- Consider using `pytest.fixtures` for setup/teardown
|
||||
|
||||
### Local Development
|
||||
- Use `.fission/local-deployment.json` for local Fission setup
|
||||
- Override secrets/configmaps for local environment
|
||||
- Run with: `fission function test --local`
|
||||
|
||||
## Migrations
|
||||
|
||||
Place SQL migration scripts in `migrates/`:
|
||||
|
||||
```sql
|
||||
-- migrates/001_create_items_table.sql
|
||||
CREATE TABLE items (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
name VARCHAR(255) NOT NULL,
|
||||
description TEXT,
|
||||
status VARCHAR(50) DEFAULT 'active',
|
||||
created TIMESTAMP DEFAULT NOW(),
|
||||
modified TIMESTAMP DEFAULT NOW()
|
||||
);
|
||||
```
|
||||
|
||||
Apply migrations manually via psql or using a migration tool like `alembic`.
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Keep functions small** - Single responsibility per function
|
||||
2. **Use Pydantic** - Validate all inputs with request models
|
||||
3. **Standardize errors** - Use the provided exception classes
|
||||
4. **Log appropriately** - Use `logger` from helpers (already configured)
|
||||
5. **Track users** - Use `get_user_from_headers()` for audit trails
|
||||
6. **Write tests** - Aim for high coverage of business logic
|
||||
7. **Document functions** - Add docstrings with fission config block
|
||||
8. **Avoid global state** - Functions should be stateless and idempotent
|
||||
|
||||
## Continuous Integration
|
||||
|
||||
The template includes `.gitea/workflows/` for CI/CD:
|
||||
|
||||
- `install-dispatch.yaml` - Triggered on installation events
|
||||
- `uninstall-dispatch.yaml` - Cleanup on uninstall
|
||||
- `dev-deployment.yaml` - Development environment updates
|
||||
- `analystic-dispatch.yaml` - Analytics processing
|
||||
|
||||
Adapt these workflows for your deployment pipeline (GitHub Actions, GitLab CI, etc.).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Spec Generation Fails
|
||||
- Ensure all function files have proper fission config in docstrings
|
||||
- Run: `python -m py_compile src/*.py` to check syntax
|
||||
- Verify `build.sh` is executable: `chmod +x src/build.sh`
|
||||
|
||||
### Cannot Connect to Database
|
||||
- Check that secrets are mounted correctly: `kubectl exec <pod> -- ls /secrets/default/`
|
||||
- Verify PG_HOST, PG_PORT are correct
|
||||
- Use `check_port_open()` debug output
|
||||
- Test connection manually: `psql -h $PG_HOST -p $PG_PORT -U $PG_USER $PG_DB`
|
||||
|
||||
### Missing Dependencies
|
||||
- Ensure `requirements.txt` includes ALL dependencies (Flask is required!)
|
||||
- Check build logs for pip errors
|
||||
- Rebuild package: `./src/build.sh`
|
||||
|
||||
## Example Implementations
|
||||
|
||||
### CRUD Operation
|
||||
|
||||
```python
|
||||
from flask import request
|
||||
from helpers import init_db_connection, format_error_response
|
||||
from exceptions import ValidationError, NotFoundError, DatabaseError
|
||||
from models import ItemCreateRequest, ItemResponse
|
||||
|
||||
def create_item(event, context):
|
||||
"""Create a new item."""
|
||||
try:
|
||||
# Validate input
|
||||
data = ItemCreateRequest(**request.get_json())
|
||||
except Exception as e:
|
||||
raise ValidationError(str(e))
|
||||
|
||||
conn = init_db_connection()
|
||||
try:
|
||||
cursor = conn.cursor()
|
||||
cursor.execute(
|
||||
"INSERT INTO items (name, description, status) VALUES (%s, %s, %s) RETURNING id, created, modified",
|
||||
(data.name, data.description, data.status.value)
|
||||
)
|
||||
row = cursor.fetchone()
|
||||
conn.commit()
|
||||
item = db_row_to_dict(cursor, row)
|
||||
return item
|
||||
except Exception as e:
|
||||
conn.rollback()
|
||||
raise DatabaseError(str(e))
|
||||
finally:
|
||||
conn.close()
|
||||
```
|
||||
|
||||
### Webhook Receiver
|
||||
|
||||
```python
|
||||
def webhook_handler(event, context):
|
||||
"""Process incoming webhook."""
|
||||
# Webhook data is in event
|
||||
payload = event.get("body", {})
|
||||
signature = request.headers.get("X-Webhook-Signature")
|
||||
|
||||
# Verify signature
|
||||
if not verify_signature(payload, signature):
|
||||
raise ValidationError("Invalid signature")
|
||||
|
||||
# Process webhook
|
||||
process_webhook(payload)
|
||||
|
||||
return {"status": "processed"}
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. Replace placeholder values in `.fission/deployment.json`
|
||||
2. Update `SECRET_NAME` and `CONFIG_NAME` in `helpers.py` (or use create-project.sh)
|
||||
3. Implement your business logic in new function files
|
||||
4. Write tests for your functions
|
||||
5. Deploy to Kubernetes cluster with Fission
|
||||
|
||||
## Resources
|
||||
|
||||
- [Fission Documentation](https://fission.io/docs/)
|
||||
- [Fission Python Builder](https://github.com/fission/fission-python-builder)
|
||||
- [Pydantic Documentation](https://docs.pydantic.dev/)
|
||||
- [Flask Documentation](https://flask.palletsprojects.com/)
|
||||
14
fission-python/template/dev-requirements.txt
Normal file
14
fission-python/template/dev-requirements.txt
Normal file
@@ -0,0 +1,14 @@
|
||||
# Runtime dependencies (include these to match production environment)
|
||||
Flask==2.1.1
|
||||
pydantic==2.11.7
|
||||
psycopg2-binary==2.9.10
|
||||
PyNaCl==1.6.0
|
||||
requests==2.32.2
|
||||
|
||||
# Development and testing tools
|
||||
pytest==8.2.0
|
||||
pytest-mock==3.14.0
|
||||
flake8==7.0.0
|
||||
black==24.1.1
|
||||
mypy==1.8.0
|
||||
pytest-cov==4.1.0
|
||||
594
fission-python/template/docs/DEPLOYMENT.md
Normal file
594
fission-python/template/docs/DEPLOYMENT.md
Normal file
@@ -0,0 +1,594 @@
|
||||
# Deployment Guide
|
||||
|
||||
This guide covers deploying Fission Python functions to Kubernetes, including configuration tuning, troubleshooting, and best practices.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Prerequisites](#prerequisites)
|
||||
2. [Quick Start](#quick-start)
|
||||
3. [Deployment Configuration](#deployment-configuration)
|
||||
4. [Executors](#executors)
|
||||
5. [Resource Tuning](#resource-tuning)
|
||||
6. [Environments](#environments)
|
||||
7. [Secrets Management](#secrets-management)
|
||||
8. [Rolling Updates](#rolling-updates)
|
||||
9. [Monitoring & Logging](#monitoring--logging)
|
||||
10. [Troubleshooting](#troubleshooting)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Kubernetes cluster (v1.19+)
|
||||
- Fission installed (`kubectl apply -f https://github.com/fission/fission/releases/latest/download/fission-all.yaml`)
|
||||
- `fission` CLI installed and configured
|
||||
- `kubectl` configured to access cluster
|
||||
- Docker registry access (for custom images if needed)
|
||||
|
||||
## Quick Start
|
||||
|
||||
Assuming you have a project set up:
|
||||
|
||||
```bash
|
||||
# 1. Build the package (creates specs/ directory)
|
||||
cd /path/to/project
|
||||
./src/build.sh
|
||||
|
||||
# 2. Verify deployment configuration
|
||||
fission spec verify --file=.fission/deployment.json
|
||||
|
||||
# 3. Deploy to Fission
|
||||
fission deploy
|
||||
|
||||
# 4. Test deployed function
|
||||
curl http://$FISSION_ROUTER/api/items
|
||||
```
|
||||
|
||||
**That's it!** Fission will:
|
||||
- Build package.zip from src/
|
||||
- Create environment (if not exists)
|
||||
- Create package
|
||||
- Create functions from docstring metadata
|
||||
- Set up HTTP triggers
|
||||
|
||||
## Deployment Configuration
|
||||
|
||||
### deployment.json vs fission.yaml
|
||||
|
||||
This template uses `deployment.json`, **not** `fission.yaml` or `fission.json`. The Fission Python builder extracts function metadata from Python docstrings directly.
|
||||
|
||||
### Key Sections
|
||||
|
||||
#### environments
|
||||
|
||||
Define build environment:
|
||||
|
||||
```json
|
||||
{
|
||||
"environments": {
|
||||
"myproject-py": {
|
||||
"image": "ghcr.io/fission/python-env",
|
||||
"builder": "ghcr.io/fission/python-builder",
|
||||
"mincpu": 50,
|
||||
"maxcpu": 100,
|
||||
"minmemory": 50,
|
||||
"maxmemory": 500,
|
||||
"poolsize": 1
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `image` - Runtime image (Python + libraries)
|
||||
- `builder` - Builder image (compiles dependencies)
|
||||
- Resource limits in millicores (50 = 0.05 CPU) and MB
|
||||
|
||||
#### packages
|
||||
|
||||
Define how to build your code:
|
||||
|
||||
```json
|
||||
{
|
||||
"packages": {
|
||||
"myproject": {
|
||||
"buildcmd": "./build.sh",
|
||||
"sourcearchive": "package.zip",
|
||||
"env": "myproject-py"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `buildcmd` - Build script inside builder container
|
||||
- `sourcearchive` - Generated by builder from `sourcepath`
|
||||
- `env` - Links to environment definition
|
||||
|
||||
#### function_common
|
||||
|
||||
Default configuration for all functions:
|
||||
|
||||
```json
|
||||
{
|
||||
"function_common": {
|
||||
"pkg": "myproject",
|
||||
"secrets": ["fission-myproject-env"],
|
||||
"configmaps": ["fission-myproject-config"],
|
||||
"executor": { ... },
|
||||
"mincpu": 50,
|
||||
"maxcpu": 100,
|
||||
"minmemory": 50,
|
||||
"maxmemory": 500
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `pkg` - Package name to use
|
||||
- `secrets` / `configmaps` - K8s resources to mount into functions
|
||||
- `executor` - Execution strategy (poolmgr or newdeploy)
|
||||
|
||||
#### secrets / configmaps
|
||||
|
||||
**Placeholder definitions only**. These inform Fission what secret names to expect, but the actual values go in real K8s secrets:
|
||||
|
||||
```json
|
||||
{
|
||||
"secrets": {
|
||||
"fission-myproject-env": {
|
||||
"literals": [
|
||||
"PG_HOST=localhost",
|
||||
"PG_PORT=5432"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Create the actual secret:
|
||||
|
||||
```bash
|
||||
kubectl create secret generic fission-myproject-env \
|
||||
--from-literal=PG_HOST=prod-db.example.com \
|
||||
--from-literal=PG_PORT=5432 \
|
||||
--from-literal=PG_USER=myuser \
|
||||
--from-literal=PG_PASS=mypassword
|
||||
```
|
||||
|
||||
## Executors
|
||||
|
||||
Fission supports two executor types:
|
||||
|
||||
### poolmgr (default)
|
||||
|
||||
Good for:
|
||||
- High-concurrency HTTP functions
|
||||
- Functions that should scale to zero
|
||||
- Stateless request/response patterns
|
||||
|
||||
Configuration:
|
||||
|
||||
```json
|
||||
"executor": {
|
||||
"select": "poolmgr",
|
||||
"poolmgr": {
|
||||
"concurrency": 1, // Requests per pod
|
||||
"requestsperpod": 1,
|
||||
"onceonly": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `concurrency` - How many concurrent requests each pod handles (usually 1 for Python due to GIL)
|
||||
- `poolsize` from environment controls number of pods in pool
|
||||
|
||||
### newdeploy
|
||||
|
||||
Good for:
|
||||
- Dedicated function instances
|
||||
- Long-running or background jobs
|
||||
- Functions needing stable network identity
|
||||
|
||||
Configuration:
|
||||
|
||||
```json
|
||||
"executor": {
|
||||
"select": "newdeploy",
|
||||
"newdeploy": {
|
||||
"minscale": 1, // Minimum pods (set to 0 for scale-to-zero)
|
||||
"maxscale": 5, // Maximum pods
|
||||
"targetcpu": 80 // Scale up when CPU > 80%
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `minscale` - Keep at least N pods running (0 = scale to zero)
|
||||
- `maxscale` - Maximum pods for auto-scaling
|
||||
- `targetcpu` - CPU threshold for scaling
|
||||
|
||||
## Resource Tuning
|
||||
|
||||
Resources are defined in millicores (m) and MB:
|
||||
|
||||
- `mincpu` / `maxcpu`: 1000 = 1 CPU core
|
||||
- `minmemory` / `maxmemory`: in MB
|
||||
|
||||
**Example settings**:
|
||||
|
||||
| Function Type | mincpu | maxcpu | minmemory | maxmemory |
|
||||
|--------------|--------|--------|-----------|-----------|
|
||||
| Simple API | 50 | 100 | 128 | 256 |
|
||||
| DB-intensive | 200 | 500 | 256 | 512 |
|
||||
| ML inference | 1000 | 2000 | 1024 | 2048 |
|
||||
|
||||
**Tips**:
|
||||
- Start conservatively, monitor, then adjust
|
||||
- Function pods are killed if they exceed `maxmemory`
|
||||
- CPU limits are enforced by Kubernetes scheduler
|
||||
- Use `minmemory` >= 128 to avoid OOM kills
|
||||
|
||||
### Checking Current Usage
|
||||
|
||||
```bash
|
||||
# Get function pods
|
||||
kubectl get pods -n fission
|
||||
|
||||
# Describe pod for resource usage
|
||||
kubectl describe pod <pod-name> -n fission
|
||||
|
||||
# See metrics (if metrics-server installed)
|
||||
kubectl top pod <pod-name> -n fission
|
||||
```
|
||||
|
||||
## Environments
|
||||
|
||||
You can have multiple deployment environments (dev, staging, prod):
|
||||
|
||||
### Using deployment.json variants
|
||||
|
||||
- `deployment.json` - Production (default)
|
||||
- `dev-deployment.json` - Development (used with `fission deploy --dev`)
|
||||
|
||||
Example `dev-deployment.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"namespace": "fission-dev",
|
||||
"function_common": {
|
||||
"secrets": ["fission-myproject-dev-env"],
|
||||
"configmaps": ["fission-myproject-dev-config"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Switching Environments
|
||||
|
||||
```bash
|
||||
# Deploy to dev
|
||||
fission deploy --dev
|
||||
|
||||
# Deploy to prod (default)
|
||||
fission deploy
|
||||
|
||||
# Specify namespace
|
||||
fission deploy --namespace fission-staging
|
||||
```
|
||||
|
||||
## Secrets Management
|
||||
|
||||
### Creating Secrets
|
||||
|
||||
```bash
|
||||
# Basic secret from literals
|
||||
kubectl create secret generic fission-myproject-env \
|
||||
--from-literal=PG_HOST=localhost \
|
||||
--from-literal=PG_PORT=5432
|
||||
|
||||
# From file
|
||||
kubectl create secret generic fission-myproject-env \
|
||||
--from-file=secrets.properties
|
||||
|
||||
# With multiple namespaces
|
||||
kubectl create secret generic fission-myproject-env \
|
||||
--namespace fission-dev \
|
||||
--from-literal=PG_HOST=dev-db.example.com
|
||||
```
|
||||
|
||||
### Encrypted Secrets (Vault)
|
||||
|
||||
To encrypt sensitive values:
|
||||
|
||||
```python
|
||||
# On your local machine (with PyNaCl installed)
|
||||
from vault import encrypt_vault
|
||||
|
||||
key = "your-32-byte-hex-key-here..." # 64 hex chars
|
||||
encrypted = encrypt_vault("super-secret-password", key)
|
||||
print(encrypted) # vault:v1:base64...
|
||||
```
|
||||
|
||||
Store the encrypted string in K8s secret:
|
||||
|
||||
```bash
|
||||
kubectl create secret generic fission-myproject-env \
|
||||
--from-literal=PG_PASS='vault:v1:base64...'
|
||||
```
|
||||
|
||||
Set `CRYPTO_KEY` in `helpers.py` to the hex key:
|
||||
|
||||
```python
|
||||
CRYPTO_KEY = "e24ad6ceed96115520f6e6dc8a0da506ae9a706823d54f30a5b75447ecf477b6"
|
||||
```
|
||||
|
||||
**Important**: Rotate keys periodically. When changing key, re-encrypt all secrets.
|
||||
|
||||
### Updating Secrets
|
||||
|
||||
```bash
|
||||
# Edit secret
|
||||
kubectl edit secret fission-myproject-env
|
||||
|
||||
# Update single key
|
||||
kubectl set secret secret fission-myproject-env \
|
||||
--from-literal=PG_PASS='new-password'
|
||||
|
||||
# Roll function to pick up new secret
|
||||
fission function update --name my-function
|
||||
```
|
||||
|
||||
## Rolling Updates
|
||||
|
||||
### Deploy Changes
|
||||
|
||||
```bash
|
||||
# Build and deploy
|
||||
./src/build.sh
|
||||
fission deploy
|
||||
|
||||
# Or deploy single function
|
||||
fission function update --name my-function
|
||||
```
|
||||
|
||||
### Zero-Downtime Deployments
|
||||
|
||||
Fission handles rolling updates automatically:
|
||||
1. New package is built
|
||||
2. New function pods are created with new code
|
||||
3. Old pods continue serving traffic until new pods are ready
|
||||
4. Old pods are terminated
|
||||
|
||||
**No downtime** by default for HTTP triggers.
|
||||
|
||||
### Canary Deployments
|
||||
|
||||
For canary deployments:
|
||||
1. Deploy new version with different function name: `my-function-v2`
|
||||
2. Route some traffic using ingress annotations or service mesh
|
||||
3. Gradually shift traffic
|
||||
4. Delete old function
|
||||
|
||||
## Monitoring & Logging
|
||||
|
||||
### Viewing Logs
|
||||
|
||||
```bash
|
||||
# All function logs in namespace
|
||||
kubectl logs -n fission -l fission-function=true --tail=100
|
||||
|
||||
# Specific function
|
||||
kubectl logs -n fission -l fission-function/name=my-function --tail=100
|
||||
|
||||
# Follow logs
|
||||
kubectl logs -n fission -l fission-function/name=my-function -f
|
||||
|
||||
# Container logs (if multiple containers)
|
||||
kubectl logs -n fission -l fission-function/name=my-function -c builder
|
||||
```
|
||||
|
||||
### Structured Logging
|
||||
|
||||
Use `logger` from `helpers.py` (already configured):
|
||||
|
||||
```python
|
||||
logger.info("Processing request", extra={"user_id": user_id})
|
||||
logger.error("Database error", exc_info=True, extra={"query": sql})
|
||||
```
|
||||
|
||||
Logs are collected by the container runtime and available via `kubectl logs`.
|
||||
|
||||
### Metrics
|
||||
|
||||
Fission exposes Prometheus metrics:
|
||||
|
||||
```bash
|
||||
# Get metrics endpoint
|
||||
kubectl port-forward -n fission svc/fission-prometheus-server 9090:9090
|
||||
|
||||
# Or query via kubectl
|
||||
kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/fission/pods/*" | jq .
|
||||
```
|
||||
|
||||
Metrics include:
|
||||
- Request rate
|
||||
- Error rate
|
||||
- Response latency
|
||||
- Pod counts
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Deployment Fails
|
||||
|
||||
**Error**: `Error building package`
|
||||
|
||||
Check:
|
||||
- `build.sh` is executable: `chmod +x src/build.sh`
|
||||
- All dependencies in `requirements.txt` are valid
|
||||
- Python syntax is correct: `python -m py_compile src/*.py`
|
||||
|
||||
**Error**: `Function not found after deploy`
|
||||
|
||||
Check:
|
||||
- Fission docstring block is properly formatted (must be ````fission` with backticks)
|
||||
- No YAML/JSON syntax errors in docstring
|
||||
- Function file is in `src/` directory
|
||||
|
||||
### Function Not Responding
|
||||
|
||||
**Check pod status**:
|
||||
```bash
|
||||
kubectl get pods -n fission -l fission-function/name=my-function
|
||||
```
|
||||
|
||||
**Pod stuck in Pending** - Insufficient resources or image pull error
|
||||
|
||||
**Pod stuck in ContainerCreating** - Volume mount issue or image pull
|
||||
|
||||
**Pod CrashLoopBackOff** - Application error. Check logs:
|
||||
```bash
|
||||
kubectl logs -n fission <pod-name> --previous
|
||||
```
|
||||
|
||||
### Configuration Not Loading
|
||||
|
||||
**Secrets not available**:
|
||||
```bash
|
||||
# Check secret exists in correct namespace
|
||||
kubectl get secret fission-myproject-env -n fission
|
||||
|
||||
# Verify secret is mounted
|
||||
kubectl exec -it <pod-name> -n fission -- ls /secrets/default/
|
||||
```
|
||||
|
||||
**ConfigMaps not available**:
|
||||
```bash
|
||||
kubectl get configmap fission-myproject-config -n fission
|
||||
```
|
||||
|
||||
**Profusion parms not reading**:
|
||||
- Ensure `SECRET_NAME` in helpers.py matches created secret name
|
||||
- Path format: `/secrets/{namespace}/{secret-name}/{key}`
|
||||
|
||||
### Slow Performance
|
||||
|
||||
1. **Increase resources**: Raise `maxmemory` and `maxcpu`
|
||||
2. **Connection pooling**: Use connection pooler like PgBouncer for heavy DB load
|
||||
3. **Database queries**: Check slow queries, add indexes
|
||||
4. **Cold starts**: Set `minscale: 1` with newdeploy executor to keep warm
|
||||
|
||||
### Database Connection Errors
|
||||
|
||||
**Error**: `could not connect to server: Connection refused`
|
||||
|
||||
- Verify database is reachable from cluster
|
||||
- Check security groups/network policies
|
||||
- Test connectivity from pod:
|
||||
```bash
|
||||
kubectl exec -it <pod-name> -n fission -- nc -zv $PG_HOST $PG_PORT
|
||||
```
|
||||
|
||||
**Error**: `password authentication failed`
|
||||
|
||||
- Verify credentials in secret
|
||||
- Check PG_USER format (with `plaintext:` prefix for vault)
|
||||
|
||||
## Advanced Topics
|
||||
|
||||
### Custom Runtime Image
|
||||
|
||||
If you need system packages:
|
||||
|
||||
```dockerfile
|
||||
FROM ghcr.io/fission/python-env:latest
|
||||
RUN apk add --no-cache gcc libffi-dev
|
||||
```
|
||||
|
||||
Build and push:
|
||||
```bash
|
||||
docker build -t myregistry/python-custom:latest .
|
||||
docker push myregistry/python-custom:latest
|
||||
```
|
||||
|
||||
Update `deployment.json`:
|
||||
```json
|
||||
"environments": {
|
||||
"myproject-py": {
|
||||
"image": "myregistry/python-custom:latest",
|
||||
...
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Environment Variables from ConfigMap
|
||||
|
||||
```json
|
||||
"configmaps": {
|
||||
"fission-myproject-config": {
|
||||
"literals": [
|
||||
"LOG_LEVEL=DEBUG",
|
||||
"FEATURE_FLAG_X=true"
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Access in code:
|
||||
```python
|
||||
import os
|
||||
log_level = os.getenv("LOG_LEVEL", "INFO")
|
||||
```
|
||||
|
||||
### Lifecycle Hooks
|
||||
|
||||
Use `function_pre_remove` and `function_post_remove` in deployment hooks:
|
||||
|
||||
```json
|
||||
"hooks": {
|
||||
"function_pre_remove": [
|
||||
{
|
||||
"type": "http",
|
||||
"url": "http://cleanup-service/cleanup",
|
||||
"timeout": 30000
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
## Common Commands Reference
|
||||
|
||||
```bash
|
||||
# List functions
|
||||
fission function list
|
||||
|
||||
# Test function manually
|
||||
fission function test --name my-function
|
||||
|
||||
# Update single function
|
||||
fission function update --name my-function
|
||||
|
||||
# Delete function
|
||||
fission function delete --name my-function
|
||||
|
||||
# View function pods
|
||||
kubectl get pods -n fission -l fission-function/name=my-function
|
||||
|
||||
# View logs
|
||||
kubectl logs -n fission -l fission-function/name=my-function -f
|
||||
|
||||
# Exec into pod
|
||||
kubectl exec -it <pod-name> -n fission -- /bin/sh
|
||||
|
||||
# Describe function
|
||||
fission function describe --name my-function
|
||||
|
||||
# Get function YAML
|
||||
fission function get --name my-function -o yaml
|
||||
|
||||
# Check Fission version
|
||||
fission version
|
||||
|
||||
# Check Fission status
|
||||
kubectl get pods -n fission
|
||||
```
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Fission Deployment Documentation](https://fission.io/docs/usage/deploy/)
|
||||
- [Fission Executors](https://fission.io/docs/architecture/executor/)
|
||||
- [Kubernetes Resource Management](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/)
|
||||
- [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/)
|
||||
582
fission-python/template/docs/MIGRATIONS.md
Normal file
582
fission-python/template/docs/MIGRATIONS.md
Normal file
@@ -0,0 +1,582 @@
|
||||
# Database Migrations
|
||||
|
||||
This guide covers managing database schema changes in Fission Python projects.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Overview](#overview)
|
||||
2. [Migration Files](#migration-files)
|
||||
3. [Applying Migrations](#applying-migrations)
|
||||
4. [Writing Migrations](#writing-migrations)
|
||||
5. [Best Practices](#best-practices)
|
||||
6. [Rollback Strategies](#rollback-strategies)
|
||||
7. [Automation](#automation)
|
||||
|
||||
## Overview
|
||||
|
||||
Database schema changes should be managed through versioned migration scripts, not manual `CREATE TABLE` statements.
|
||||
|
||||
This template uses **plain SQL migration files** (`.sql`), which provide:
|
||||
- Version control of schema changes
|
||||
- Repeatable application to different environments
|
||||
- Clear upgrade/downgrade paths
|
||||
- Audit trail of schema evolution
|
||||
|
||||
## Migration Files
|
||||
|
||||
Place SQL migration scripts in the `migrates/` directory:
|
||||
|
||||
```
|
||||
migrates/
|
||||
├── 001_initial_schema.sql
|
||||
├── 002_add_user_email.sql
|
||||
├── 003_create_indexes.sql
|
||||
└── ...
|
||||
```
|
||||
|
||||
**Naming convention**:
|
||||
- Prefix with sequential number (zero-padded for sorting)
|
||||
- Descriptive name after underscore
|
||||
- `.sql` extension
|
||||
- Numbers should be unique and monotonically increasing
|
||||
|
||||
### Initial Schema Example
|
||||
|
||||
```sql
|
||||
-- migrates/001_create_items_table.sql
|
||||
-- Create items table
|
||||
CREATE TABLE IF NOT EXISTS items (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
name VARCHAR(255) NOT NULL,
|
||||
description TEXT,
|
||||
status VARCHAR(50) DEFAULT 'active',
|
||||
metadata JSONB,
|
||||
created TIMESTAMPTZ DEFAULT NOW(),
|
||||
modified TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Add indexes
|
||||
CREATE INDEX idx_items_status ON items(status);
|
||||
CREATE INDEX idx_items_created ON items(created);
|
||||
|
||||
-- Add comments
|
||||
COMMENT ON TABLE items IS 'Stores item records';
|
||||
COMMENT ON COLUMN items.status IS 'Item status: active, inactive, pending';
|
||||
```
|
||||
|
||||
## Applying Migrations
|
||||
|
||||
### Manually
|
||||
|
||||
```bash
|
||||
# Connect to database
|
||||
psql -h localhost -U postgres -d mydb
|
||||
|
||||
# Run migration file
|
||||
\i migrates/001_create_items_table.sql
|
||||
|
||||
# Run all migrations in order (bash script)
|
||||
for file in $(ls migrates/*.sql | sort); do
|
||||
echo "Applying $file..."
|
||||
psql -h localhost -U postgres -d mydb -f "$file"
|
||||
done
|
||||
```
|
||||
|
||||
### Automatically from Python
|
||||
|
||||
Create a simple migration runner:
|
||||
|
||||
```python
|
||||
# src/migrate.py (not part of function, standalone script)
|
||||
import os
|
||||
import psycopg2
|
||||
from helpers import init_db_connection
|
||||
|
||||
def run_migrations():
|
||||
conn = init_db_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
# Create migrations tracking table if not exists
|
||||
cursor.execute("""
|
||||
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||
version INTEGER PRIMARY KEY,
|
||||
name VARCHAR(255) NOT NULL,
|
||||
applied_at TIMESTAMPTZ DEFAULT NOW()
|
||||
)
|
||||
""")
|
||||
|
||||
# Get already-applied migrations
|
||||
cursor.execute("SELECT version FROM schema_migrations")
|
||||
applied = {row[0] for row in cursor.fetchall()}
|
||||
|
||||
# Find migration files
|
||||
migrates_dir = os.path.join(os.path.dirname(__file__), "..", "migrates")
|
||||
files = sorted([
|
||||
f for f in os.listdir(migrates_dir)
|
||||
if f.endswith(".sql")
|
||||
])
|
||||
|
||||
# Apply pending migrations
|
||||
for filename in files:
|
||||
# Extract version number
|
||||
version = int(filename.split("_")[0])
|
||||
if version in applied:
|
||||
print(f"Skipping {filename} (already applied)")
|
||||
continue
|
||||
|
||||
path = os.path.join(migrates_dir, filename)
|
||||
print(f"Applying {filename}...")
|
||||
with open(path, 'r') as f:
|
||||
sql = f.read()
|
||||
|
||||
try:
|
||||
cursor.execute(sql)
|
||||
cursor.execute(
|
||||
"INSERT INTO schema_migrations (version, name) VALUES (%s, %s)",
|
||||
(version, filename)
|
||||
)
|
||||
conn.commit()
|
||||
print(f" ✓ Applied {filename}")
|
||||
except Exception as e:
|
||||
conn.rollback()
|
||||
print(f" ✗ Failed: {e}")
|
||||
raise
|
||||
|
||||
conn.close()
|
||||
print("All migrations applied")
|
||||
|
||||
if __name__ == "__main__":
|
||||
run_migrations()
|
||||
```
|
||||
|
||||
Run:
|
||||
```bash
|
||||
python src/migrate.py
|
||||
```
|
||||
|
||||
### Using Migration Tools
|
||||
|
||||
For more advanced features (rollbacks, branching), consider:
|
||||
|
||||
- **[Alembic](https://alembic.sqlalchemy.org/)** - Database migration tool for SQLAlchemy (if using ORM)
|
||||
- **[pg migrator](https://github.com/heroku/pg-migrator)** - Heroku's migration tool
|
||||
- **[goose](https://github.com/pressly/goose)** - Multi-database migration tool (can use from Python)
|
||||
- **[yoyo-migrations](https://github.com/gugulet-h/yoyo-migrations)** - Python-based migrations
|
||||
|
||||
## Writing Migrations
|
||||
|
||||
### Principles
|
||||
|
||||
1. **Idempotent** - Script should succeed if run multiple times
|
||||
2. **Additive first** - Add columns/tables before removing/dropping
|
||||
3. **Backward compatible** - New schema should work with old code
|
||||
4. **Atomic** - One logical change per migration file
|
||||
5. **Test locally** - Apply to test database before production
|
||||
|
||||
### Common Operations
|
||||
|
||||
#### Create Table
|
||||
|
||||
```sql
|
||||
CREATE TABLE IF NOT EXISTS orders (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
user_id UUID NOT NULL,
|
||||
total DECIMAL(10,2) NOT NULL,
|
||||
status VARCHAR(50) NOT NULL DEFAULT 'pending',
|
||||
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Add foreign key
|
||||
ALTER TABLE orders
|
||||
ADD CONSTRAINT fk_orders_user
|
||||
FOREIGN KEY (user_id)
|
||||
REFERENCES users(id)
|
||||
ON DELETE CASCADE;
|
||||
|
||||
-- Index for performance
|
||||
CREATE INDEX idx_orders_user_id ON orders(user_id);
|
||||
CREATE INDEX idx_orders_created_at ON orders(created_at);
|
||||
```
|
||||
|
||||
#### Add Column
|
||||
|
||||
```sql
|
||||
-- Add nullable column (safe, backward compatible)
|
||||
ALTER TABLE orders
|
||||
ADD COLUMN shipping_address JSONB;
|
||||
|
||||
-- Add column with default (be careful with large tables!)
|
||||
-- This rewrites entire table - use cautiously
|
||||
ALTER TABLE orders
|
||||
ADD COLUMN tax_amount DECIMAL(10,2) DEFAULT 0.00;
|
||||
```
|
||||
|
||||
#### Rename Column
|
||||
|
||||
```sql
|
||||
-- PostgreSQL 9.2+ supports RENAME COLUMN
|
||||
ALTER TABLE orders
|
||||
RENAME COLUMN total TO order_total;
|
||||
```
|
||||
|
||||
#### Modify Column Type
|
||||
|
||||
```sql
|
||||
-- Change VARCHAR length
|
||||
ALTER TABLE users
|
||||
ALTER COLUMN email TYPE VARCHAR(320);
|
||||
|
||||
-- Convert to different type (use USING clause)
|
||||
ALTER TABLE orders
|
||||
ALTER COLUMN status TYPE VARCHAR(100)
|
||||
USING status::VARCHAR(100);
|
||||
```
|
||||
|
||||
#### Create Index
|
||||
|
||||
```sql
|
||||
-- Simple index
|
||||
CREATE INDEX idx_users_email ON users(email);
|
||||
|
||||
-- Unique index
|
||||
CREATE UNIQUE INDEX idx_users_email_unique ON users(email);
|
||||
|
||||
-- Partial index (only active users)
|
||||
CREATE INDEX idx_users_active ON users(id)
|
||||
WHERE status = 'active';
|
||||
|
||||
-- Multi-column index
|
||||
CREATE INDEX idx_orders_user_status ON orders(user_id, status);
|
||||
```
|
||||
|
||||
#### Drop Column/Table
|
||||
|
||||
```sql
|
||||
-- First, ensure no one is using it
|
||||
-- Consider using SET DEFAULT then dropping in subsequent migration
|
||||
|
||||
-- Drop column
|
||||
ALTER TABLE orders
|
||||
DROP COLUMN IF EXISTS old_column;
|
||||
|
||||
-- Drop table (dangerous!)
|
||||
DROP TABLE IF EXISTS old_logs;
|
||||
```
|
||||
|
||||
### Data Migrations
|
||||
|
||||
Sometimes you need to transform data:
|
||||
|
||||
```sql
|
||||
-- Backfill new column from existing data
|
||||
UPDATE orders
|
||||
SET shipping_address = jsonb_build_object(
|
||||
'street', address_street,
|
||||
'city', address_city,
|
||||
'zip', address_zip
|
||||
)
|
||||
WHERE shipping_address IS NULL;
|
||||
|
||||
-- Migrate enum values
|
||||
UPDATE products
|
||||
SET status = 'active' WHERE status = 'ACTIVE';
|
||||
|
||||
-- Clean up duplicates
|
||||
WITH duplicates AS (
|
||||
SELECT id, ROW_NUMBER() OVER (PARTITION BY email ORDER BY created_at) AS rn
|
||||
FROM users
|
||||
)
|
||||
DELETE FROM users WHERE id IN (SELECT id FROM duplicates WHERE rn > 1);
|
||||
```
|
||||
|
||||
### Transactional Migrations
|
||||
|
||||
Wrap critical migrations in transactions:
|
||||
|
||||
```sql
|
||||
BEGIN;
|
||||
|
||||
-- Multiple related operations
|
||||
ALTER TABLE orders ADD COLUMN shipping_id UUID;
|
||||
UPDATE orders SET shipping_id = uuid_generate_v4() WHERE shipping_id IS NULL;
|
||||
ALTER TABLE orders ALTER COLUMN shipping_id SET NOT NULL;
|
||||
|
||||
COMMIT;
|
||||
```
|
||||
|
||||
**Note**: DDL statements in PostgreSQL auto-commit, so `BEGIN`/`COMMIT` may not work as expected for schema changes. For complex multi-step changes, consider using advisory locks or deployment coordination.
|
||||
|
||||
## Best Practices
|
||||
|
||||
### ✅ Do's
|
||||
|
||||
1. **Test migrations on copy of production database** before applying to prod
|
||||
2. **Keep migrations small** - One logical change per file
|
||||
3. **Write data migrations as separate files** from schema migrations
|
||||
4. **Use `IF NOT EXISTS` and `IF EXISTS`** to make migrations idempotent
|
||||
5. **Never drop columns/tables in the same migration you add them** - Separate to allow rollback
|
||||
6. **Document why** - Add comments explaining the purpose
|
||||
7. **Consider indexes** - Add indexes for frequently queried columns in same migration as table creation
|
||||
8. **Use UUIDs** for primary keys (`gen_random_uuid()` in PostgreSQL 13+)
|
||||
9. **Add `created_at` and `updated_at` timestamps** to all tables
|
||||
10. **Version numbers must be unique and sequential**
|
||||
|
||||
### ❌ Don'ts
|
||||
|
||||
1. **Don't modify already-applied migrations** - They're part of history
|
||||
2. **Don't skip version numbers** - Creates gaps but not critical
|
||||
3. **Don't use destructive operations without backup** - `DROP COLUMN`, `DROP TABLE`
|
||||
4. **Don't run long-running migrations during peak hours** - Use low-traffic windows
|
||||
5. **Don't add NOT NULL without default** on non-empty tables - Will fail due to existing NULL rows
|
||||
6. **Don't assume order of execution** - Always number sequentially
|
||||
7. **Don't mix unrelated changes** in one migration file
|
||||
|
||||
### Zero-Downtime Migrations
|
||||
|
||||
#### Adding Column
|
||||
|
||||
```sql
|
||||
-- Step 1: Add column as nullable or with default (fast)
|
||||
ALTER TABLE orders ADD COLUMN status VARCHAR(50);
|
||||
|
||||
-- Step 2: Deploy code that writes to new column
|
||||
-- Your application updates to populate status
|
||||
|
||||
-- Step 3: Backfill existing rows (if needed)
|
||||
UPDATE orders SET status = 'completed' WHERE status IS NULL AND shipped_at IS NOT NULL;
|
||||
|
||||
-- Step 4: Make column NOT NULL (if needed) - only after all rows have values
|
||||
ALTER TABLE orders ALTER COLUMN status SET NOT NULL;
|
||||
```
|
||||
|
||||
#### Renaming Column
|
||||
|
||||
```sql
|
||||
-- Step 1: Add new column
|
||||
ALTER TABLE orders ADD COLUMN order_status VARCHAR(50);
|
||||
|
||||
-- Step 2: Deploy code writing to both old and new columns (dual-write)
|
||||
|
||||
-- Step 3: Backfill data
|
||||
UPDATE orders SET order_status = status;
|
||||
|
||||
-- Step 4: Deploy code reading from new column, stop writing to old
|
||||
|
||||
-- Step 5: Drop old column (in separate migration)
|
||||
ALTER TABLE orders DROP COLUMN status;
|
||||
```
|
||||
|
||||
## Rollback Strategies
|
||||
|
||||
### Manual Rollback
|
||||
|
||||
For each migration, you may want to write a corresponding "down" migration:
|
||||
|
||||
```sql
|
||||
-- 002_add_user_email.sql (UP)
|
||||
ALTER TABLE users ADD COLUMN email VARCHAR(320);
|
||||
|
||||
-- 002_add_user_email_rollback.sql (DOWN)
|
||||
ALTER TABLE users DROP COLUMN IF EXISTS email;
|
||||
```
|
||||
|
||||
Store rollback scripts alongside migrations or in separate `rollbacks/` directory.
|
||||
|
||||
### Point-in-Time Recovery
|
||||
|
||||
**Best strategy**: Restore database from backup to point before bad migration, then re-apply good migrations.
|
||||
|
||||
```bash
|
||||
# Restore from PITR backup (if using WAL archiving)
|
||||
pg_restore -h localhost -U postgres -d mydb --point-in-time="2025-03-18 10:30:00"
|
||||
|
||||
# Re-run migrations up to good version
|
||||
python src/migrate.py # But this applies all, so need selective
|
||||
```
|
||||
|
||||
### Selective Rollback Script
|
||||
|
||||
```python
|
||||
# rollback.py
|
||||
import sys
|
||||
from helpers import init_db_connection
|
||||
|
||||
def rollback(to_version: int):
|
||||
conn = init_db_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
# Find migrations after target version
|
||||
cursor.execute("""
|
||||
SELECT version, name
|
||||
FROM schema_migrations
|
||||
WHERE version > %s
|
||||
ORDER BY version DESC
|
||||
""", (to_version,))
|
||||
|
||||
migrations = cursor.fetchall()
|
||||
|
||||
for version, name in migrations:
|
||||
rollback_file = f"rollbacks/{version:03d}_{name.split('_', 1)[1]}.sql"
|
||||
print(f"Rolling back {name} using {rollback_file}...")
|
||||
with open(rollback_file, 'r') as f:
|
||||
sql = f.read()
|
||||
cursor.execute(sql)
|
||||
cursor.execute("DELETE FROM schema_migrations WHERE version = %s", (version,))
|
||||
conn.commit()
|
||||
print(f" Rolled back {name}")
|
||||
|
||||
conn.close()
|
||||
print(f"Rolled back to version {to_version}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
target = int(sys.argv[1])
|
||||
rollback(target)
|
||||
```
|
||||
|
||||
## Automation
|
||||
|
||||
### CI/CD Integration
|
||||
|
||||
In your deployment pipeline:
|
||||
|
||||
```bash
|
||||
# Before deploying new code
|
||||
python src/migrate.py
|
||||
|
||||
# If migrations fail, abort deployment
|
||||
if [ $? -ne 0 ]; then
|
||||
echo "Migrations failed, aborting deployment"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Deploy new code
|
||||
fission deploy
|
||||
```
|
||||
|
||||
### Pre-deployment Hooks
|
||||
|
||||
Use Fission hooks to run migrations automatically:
|
||||
|
||||
```json
|
||||
{
|
||||
"hooks": {
|
||||
"function_pre_deploy": [
|
||||
{
|
||||
"type": "http",
|
||||
"url": "http://migration-service/migrate",
|
||||
"timeout": 300000
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Or simpler: run migration as part of `build.sh`:
|
||||
|
||||
```bash
|
||||
#!/bin/sh
|
||||
# src/build.sh
|
||||
|
||||
# Install dependencies
|
||||
pip3 install -r requirements.txt -t .
|
||||
|
||||
# Run migrations against test DB (or do nothing, migrations are separate)
|
||||
# python ../migrate.py
|
||||
|
||||
# Package up
|
||||
cp -r . ${DEPLOY_PKG}
|
||||
```
|
||||
|
||||
### Database Change Management Tools
|
||||
|
||||
Consider specialized tools for larger teams:
|
||||
- **[Flyway](https://flywaydb.org/)** - Java-based, supports repeatable migrations
|
||||
- **[Liquibase](https://www.liquibase.org/)** - XML/YAML/JSON migrations
|
||||
- **[Prisma Migrate](https://www.prisma.io/docs/concepts/components/prisma-migrate)** - If using Prisma ORM
|
||||
- **[Alembic](https://alembic.sqlalchemy.org/)** - Python, SQLAlchemy-specific
|
||||
|
||||
## Example Workflow
|
||||
|
||||
1. **Create migration**:
|
||||
```bash
|
||||
touch migrates/004_add_orders_table.sql
|
||||
```
|
||||
|
||||
2. **Write SQL**:
|
||||
```sql
|
||||
CREATE TABLE orders (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
user_id UUID NOT NULL REFERENCES users(id),
|
||||
total DECIMAL(10,2) NOT NULL,
|
||||
status VARCHAR(50) DEFAULT 'pending',
|
||||
created_at TIMESTAMPTZ DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX idx_orders_user_id ON orders(user_id);
|
||||
```
|
||||
|
||||
3. **Test locally**:
|
||||
```bash
|
||||
createdb test_migration
|
||||
psql test_migration -f migrates/004_add_orders_table.sql
|
||||
```
|
||||
|
||||
4. **Commit migration file**:
|
||||
```bash
|
||||
git add migrates/004_add_orders_table.sql
|
||||
git commit -m "Add orders table"
|
||||
```
|
||||
|
||||
5. **Apply to staging**:
|
||||
```bash
|
||||
# Update dev-deployment.json if new env vars needed
|
||||
fission deploy --dev
|
||||
python src/migrate.py
|
||||
```
|
||||
|
||||
6. **Apply to production**:
|
||||
```bash
|
||||
# Maintenance window or blue-green deployment
|
||||
fission deploy
|
||||
python src/migrate.py
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Migration Fails
|
||||
|
||||
Check error message:
|
||||
- **syntax error**: Validate SQL with `psql -c "SQL"` manually
|
||||
- **duplicate column**: Migration already applied, check `schema_migrations`
|
||||
- **permission denied**: DB user lacks ALTER/CREATE privileges
|
||||
- **lock timeout**: Another migration running, wait or kill process
|
||||
|
||||
### Migration Already Applied But Failed
|
||||
|
||||
If migration was recorded in `schema_migrations` but failed mid-way:
|
||||
|
||||
1. Manually revert partial changes or fix broken state
|
||||
2. Delete row from `schema_migrations`: `DELETE FROM schema_migrations WHERE version = 4;`
|
||||
3. Re-run migration
|
||||
|
||||
### Long-Running Migration
|
||||
|
||||
Large table alterations can lock rows and cause downtime:
|
||||
|
||||
- Run during low-traffic period
|
||||
- Use `CONCURRENTLY` for index creation (PostgreSQL):
|
||||
```sql
|
||||
CREATE INDEX CONCURRENTLY idx_orders_created ON orders(created_at);
|
||||
```
|
||||
- For adding NOT NULL, populate values first with UPDATE, then add constraint
|
||||
- Consider using `pg_repack` for online table reorganization
|
||||
|
||||
## Summary
|
||||
|
||||
- Store migrations in `migrates/` directory, numbered sequentially
|
||||
- Use `init_db_connection()` to run migrations programmatically
|
||||
- Test migrations on staging database before production
|
||||
- Keep migrations backward compatible when possible
|
||||
- Have a rollback plan (backups, down scripts)
|
||||
- Integrate migrations into CI/CD pipeline
|
||||
438
fission-python/template/docs/SECRETS.md
Normal file
438
fission-python/template/docs/SECRETS.md
Normal file
@@ -0,0 +1,438 @@
|
||||
# Secrets and Configuration Management
|
||||
|
||||
This guide covers best practices for managing secrets and configuration in Fission Python functions.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Overview](#overview)
|
||||
2. [Kubernetes Secrets vs ConfigMaps](#kubernetes-secrets-vs-configmaps)
|
||||
3. [Secrets in Fission](#secrets-in-fission)
|
||||
4. [Vault Encryption](#vault-encryption)
|
||||
5. [Secret Rotation](#secret-rotation)
|
||||
6. [Configuration Precedence](#configuration-precedence)
|
||||
7. [Best Practices](#best-practices)
|
||||
|
||||
## Overview
|
||||
|
||||
Sensitive data (passwords, API keys) should **never** be:
|
||||
- Committed to Git
|
||||
- Hardcoded in source code
|
||||
- Passed as plaintext in deployment files
|
||||
|
||||
Instead, use:
|
||||
- **Kubernetes Secrets** - For sensitive values
|
||||
- **Kubernetes ConfigMaps** - For non-sensitive configuration
|
||||
- **Vault encryption** - For encrypting secrets at rest in K8s
|
||||
|
||||
## Kubernetes Secrets vs ConfigMaps
|
||||
|
||||
| Feature | Secrets | ConfigMaps |
|
||||
|---------|---------|------------|
|
||||
| Purpose | Sensitive data (passwords, tokens, keys) | Non-sensitive config (endpoints, feature flags) |
|
||||
| Storage | Base64 encoded (not encrypted by default) | Plain text |
|
||||
| Mount as | Files in `/secrets/` | Files in `/configs/` |
|
||||
| Access in code | `get_secret(key)` | `get_config(key)` |
|
||||
| Max size | 1MB total | 1MB total |
|
||||
| Can be encrypted | Yes, with K8s encryption at rest | Yes |
|
||||
|
||||
**Rule of thumb**:
|
||||
- Use Secrets for: database passwords, API tokens, encryption keys
|
||||
- Use ConfigMaps for: service URLs, feature flags, log levels, non-sensitive constants
|
||||
|
||||
## Secrets in Fission
|
||||
|
||||
### Defining Secret References in deployment.json
|
||||
|
||||
In `.fission/deployment.json`, declare the secret names your functions expect:
|
||||
|
||||
```json
|
||||
{
|
||||
"function_common": {
|
||||
"secrets": ["fission-myproject-env"],
|
||||
"configmaps": ["fission-myproject-config"]
|
||||
},
|
||||
"secrets": {
|
||||
"fission-myproject-env": {
|
||||
"literals": [
|
||||
"PG_HOST=localhost",
|
||||
"PG_PORT=5432"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Important**: The `literals` array here is **only documentation**. The actual secret values must be created separately in Kubernetes.
|
||||
|
||||
### Creating Actual Kubernetes Secrets
|
||||
|
||||
```bash
|
||||
# Create secret with multiple keys
|
||||
kubectl create secret generic fission-myproject-env \
|
||||
--from-literal=PG_HOST=postgres.example.com \
|
||||
--from-literal=PG_PORT=5432 \
|
||||
--from-literal=PG_DB=mydb \
|
||||
--from-literal=PG_USER=myuser \
|
||||
--from-literal=PG_PASS='my-password'
|
||||
|
||||
# In a specific namespace (Fission namespace)
|
||||
kubectl create secret generic fission-myproject-env \
|
||||
--namespace fission \
|
||||
--from-literal=...
|
||||
|
||||
# From environment file
|
||||
kubectl create secret generic fission-myproject-env \
|
||||
--namespace fission \
|
||||
--from-env-file=.env
|
||||
```
|
||||
|
||||
### How Secrets Are Mounted
|
||||
|
||||
Fission mounts secrets as files in the function pod:
|
||||
|
||||
```
|
||||
/secrets/{namespace}/{secret-name}/{key}
|
||||
```
|
||||
|
||||
Example path: `/secrets/default/fission-myproject-env/PG_HOST`
|
||||
|
||||
The `helpers.py` `get_secret()` function reads from this path:
|
||||
|
||||
```python
|
||||
def get_secret(key: str, default=None):
|
||||
namespace = get_current_namespace()
|
||||
path = f"/secrets/{namespace}/{SECRET_NAME}/{key}"
|
||||
with open(path, "r") as f:
|
||||
return f.read()
|
||||
```
|
||||
|
||||
**Note**: `SECRET_NAME` must match the K8s secret name (`fission-myproject-env`).
|
||||
|
||||
### Reading Secrets in Code
|
||||
|
||||
```python
|
||||
from helpers import get_secret
|
||||
|
||||
# With default fallback
|
||||
db_host = get_secret("PG_HOST", "localhost")
|
||||
db_port = int(get_secret("PG_PORT", "5432"))
|
||||
db_user = get_secret("PG_USER")
|
||||
db_pass = get_secret("PG_PASS")
|
||||
|
||||
# If key missing and no default, returns None
|
||||
maybe_value = get_secret("OPTIONAL_KEY")
|
||||
```
|
||||
|
||||
**Always provide a default** for non-critical configuration to avoid crashes if secret is missing.
|
||||
|
||||
### ConfigMaps
|
||||
|
||||
Same pattern, different mount path: `/configs/{namespace}/{configmap-name}/{key}`
|
||||
|
||||
```python
|
||||
from helpers import get_config
|
||||
|
||||
api_endpoint = get_config("API_ENDPOINT", "http://default.api")
|
||||
feature_flag = get_config("FEATURE_X_ENABLED", "false")
|
||||
```
|
||||
|
||||
Create ConfigMap:
|
||||
|
||||
```bash
|
||||
kubectl create configmap fission-myproject-config \
|
||||
--namespace fission \
|
||||
--from-literal=API_ENDPOINT=https://api.example.com \
|
||||
--from-literal=FEATURE_X_ENABLED=true
|
||||
```
|
||||
|
||||
## Vault Encryption
|
||||
|
||||
To encrypt secrets before storing in K8s:
|
||||
|
||||
### Generate Encryption Key
|
||||
|
||||
```bash
|
||||
# Generate 32-byte (64 hex char) random key
|
||||
openssl rand -hex 32
|
||||
# Example output: e24ad6ceed96115520f6e6dc8a0da506ae9a706823d54f30a5b75447ecf477b6
|
||||
```
|
||||
|
||||
### Encrypt a Value
|
||||
|
||||
```python
|
||||
# Encrypt locally
|
||||
from vault import encrypt_vault
|
||||
|
||||
key = "e24ad6ceed96115520f6e6dc8a0da506ae9a706823d54f30a5b75447ecf477b6"
|
||||
encrypted = encrypt_vault("my-secret-password", key)
|
||||
print(encrypted)
|
||||
# Output: vault:v1:base64-encrypted-data
|
||||
```
|
||||
|
||||
### Store Encrypted Value
|
||||
|
||||
Create K8s secret with encrypted value:
|
||||
|
||||
```bash
|
||||
kubectl create secret generic fission-myproject-env \
|
||||
--from-literal=PG_PASS='vault:v1:base64...'
|
||||
```
|
||||
|
||||
### Configure decryption in helpers.py
|
||||
|
||||
```python
|
||||
CRYPTO_KEY = "e24ad6ceed96115520f6e6dc8a0da506ae9a706823d54f30a5b75447ecf477b6"
|
||||
```
|
||||
|
||||
### Automatic Decryption
|
||||
|
||||
`get_secret()` and `get_config()` automatically:
|
||||
1. Read the file content
|
||||
2. Detect if it starts with `vault:v1:` (using `is_valid_vault_format()`)
|
||||
3. Decrypt using `CRYPTO_KEY` if encrypted
|
||||
4. Return plaintext
|
||||
|
||||
**No code changes needed** - it "just works".
|
||||
|
||||
### Verification
|
||||
|
||||
```bash
|
||||
# Test decryption
|
||||
kubectl get secret fission-myproject-env -o jsonpath='{.data.PG_PASS}' | base64 -d
|
||||
# Should show: vault:v1:...
|
||||
|
||||
# Exec into pod and manually check
|
||||
kubectl exec -it <pod-name> -- python3 -c "from helpers import get_secret; print(get_secret('PG_PASS'))"
|
||||
# Should print decrypted value
|
||||
```
|
||||
|
||||
## Secret Rotation
|
||||
|
||||
### Rotating a Secret
|
||||
|
||||
1. **Generate new value** (new password, new API key)
|
||||
2. **Encrypt** (if using vault)
|
||||
3. **Update K8s secret**:
|
||||
```bash
|
||||
kubectl create secret generic fission-myproject-env \
|
||||
--dry-run=client \
|
||||
--from-literal=PG_PASS='new-password' \
|
||||
-o yaml | kubectl apply -f -
|
||||
```
|
||||
4. **Update actual external system** (database, API provider) with new value
|
||||
5. **Verify applications work** (check logs)
|
||||
6. **Remove old value** (if rotating from old to new, both may need to coexist temporarily)
|
||||
|
||||
### Rotating Vault Encryption Key
|
||||
|
||||
**Warning**: Changing `CRYPTO_KEY` requires re-encrypting all secrets!
|
||||
|
||||
1. Deploy new code with updated `CRYPTO_KEY` **temporarily** pointing to new key
|
||||
2. Create new K8s secrets with values encrypted under new key (or re-encrypt via script)
|
||||
3. Switch `CRYPTO_KEY` back to original (or both keys during transition) - actually this is complex
|
||||
|
||||
**Recommended**: Have two keys during rotation:
|
||||
```python
|
||||
CRYPTO_KEYS = [
|
||||
"old-key-hex...", # Keep for decrypting old secrets
|
||||
"new-key-hex..." # Use for encrypting new/updated secrets
|
||||
]
|
||||
```
|
||||
|
||||
Then update `decrypt_vault()` to try each key until one works. After all secrets migrated, remove old key.
|
||||
|
||||
## Configuration Precedence
|
||||
|
||||
Fission supports multiple deployment configuration files:
|
||||
|
||||
1. **deployment.json** - Base configuration (committed to repo)
|
||||
2. **dev-deployment.json** - Development overrides (usually not committed)
|
||||
3. **local-deployment.json** - Local overrides (gitignored)
|
||||
|
||||
### Override Priority
|
||||
|
||||
When using `fission deploy --dev`, Fission loads:
|
||||
- Base configuration from `deployment.json`
|
||||
- Overlay from `dev-deployment.json`
|
||||
|
||||
Values in the overlay file replace or extend base values.
|
||||
|
||||
**Example**: Override secret name for dev:
|
||||
|
||||
**deployment.json**:
|
||||
```json
|
||||
{
|
||||
"function_common": {
|
||||
"secrets": ["fission-myproject-env"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**dev-deployment.json**:
|
||||
```json
|
||||
{
|
||||
"function_common": {
|
||||
"secrets": ["fission-myproject-dev-env"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Now `fission deploy --dev` uses the dev secret, while `fission deploy` uses prod secret.
|
||||
|
||||
### Local Overrides
|
||||
|
||||
Create `.fission/local-deployment.json` for your workstation:
|
||||
|
||||
```json
|
||||
{
|
||||
"function_common": {
|
||||
"secrets": ["fission-myproject-local-env"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Fission automatically uses this if present (no flag needed). `.gitignore` typically excludes it.
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Do's ✅
|
||||
|
||||
1. **Do use Kubernetes Secrets** - Never hardcode credentials
|
||||
2. **Do encrypt with vault** - Prevents plaintext secrets in K8s
|
||||
3. **Do store vault key securely** - In K8s sealed secret, external vault (HashiCorp Vault, AWS Secrets Manager), or as a separate K8s secret in restricted namespace
|
||||
4. **Do namespace secrets** - Use different secrets for dev/staging/prod
|
||||
5. **Do rotate secrets regularly** - Especially database passwords, API tokens
|
||||
6. **Do use ConfigMaps for non-sensitive config** - Cleaner separation
|
||||
7. **Do provide sensible defaults** - In `get_secret()` calls
|
||||
8. **Do validate required secrets** - Fail fast at startup:
|
||||
```python
|
||||
def init():
|
||||
pg_host = get_secret("PG_HOST")
|
||||
if not pg_host:
|
||||
raise ValueError("PG_HOST secret is required")
|
||||
```
|
||||
|
||||
### Don'ts ❌
|
||||
|
||||
1. **Don't commit secrets** - Even in `deployment.json` literals
|
||||
2. **Don't put plaintext in Git** - Use placeholders or remove before commit
|
||||
3. **Don't embed vault key in code for production** - Use environment-specific override or external secret management
|
||||
4. **Don't share vault key publicly** - It's a symmetric key - anyone with it can decrypt all secrets
|
||||
5. **Don't use same secret across namespaces** - Separate environments should have separate credentials
|
||||
6. **Don't rely on obscurity** - Security through obscurity is not security
|
||||
|
||||
### Supply Chain Security
|
||||
|
||||
For production deployments:
|
||||
|
||||
1. **Store vault key in sealed secrets** (if on K8s):
|
||||
```bash
|
||||
kubectl create secret generic crypto-key \
|
||||
--from-literal=key='your-hex-key'
|
||||
# Then use SealedSecrets controller to encrypt in Git
|
||||
```
|
||||
|
||||
2. **Use external secrets operator**:
|
||||
```yaml
|
||||
apiVersion: external-secrets.io/v1beta1
|
||||
kind: ExternalSecret
|
||||
metadata:
|
||||
name: db-creds
|
||||
spec:
|
||||
refreshInterval: "1h"
|
||||
secretStoreRef:
|
||||
name: vault-backend
|
||||
kind: SecretStore
|
||||
target:
|
||||
name: fission-myproject-env
|
||||
creationPolicy: Owner
|
||||
data:
|
||||
- secretKey: PG_PASS
|
||||
remoteRef:
|
||||
key: /prod/db/password
|
||||
```
|
||||
|
||||
3. **Rotate automatically** with cronjobs or external secret manager
|
||||
|
||||
## Environment Variable Alternative
|
||||
|
||||
While the template uses secret files mounted by Fission, you can also use environment variables:
|
||||
|
||||
```json
|
||||
"function_common": {
|
||||
"environment": {
|
||||
"LOG_LEVEL": "INFO",
|
||||
"FEATURE_FLAG": "true"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Access with `os.getenv()`:
|
||||
|
||||
```python
|
||||
import os
|
||||
log_level = os.getenv("LOG_LEVEL", "INFO")
|
||||
```
|
||||
|
||||
**However**: Environment is less flexible than secrets/configmaps for dynamic updates (requires function restart). Prefer secrets/configmaps for values that may change independently of code deployments.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Secret Not Available
|
||||
|
||||
```bash
|
||||
# Check secret exists in correct namespace
|
||||
kubectl get secret fission-myproject-env -n fission
|
||||
|
||||
# Check secret keys
|
||||
kubectl get secret fission-myproject-env -n fission -o jsonpath='{.data}'
|
||||
|
||||
# Check pod mount
|
||||
kubectl exec -it <pod-name> -n fission -- ls -la /secrets/default/
|
||||
```
|
||||
|
||||
Common issues:
|
||||
- Secret in wrong namespace (use Fission namespace, usually `fission` or as configured)
|
||||
- Secret name typo in helpers.py `SECRET_NAME` variable
|
||||
- Secret not mounted due to missing permission (service account restriction)
|
||||
|
||||
### Vault Decryption Failing
|
||||
|
||||
```python
|
||||
from vault import is_valid_vault_format, decrypt_vault
|
||||
|
||||
vault_str = get_secret("PG_PASS")
|
||||
print(is_valid_vault_format(vault_str)) # Should be True
|
||||
print(decrypt_vault(vault_str, "wrong-key")) # Raises CryptoError
|
||||
```
|
||||
|
||||
Check:
|
||||
- `CRYPTO_KEY` is set correctly in `helpers.py`
|
||||
- Key is 64 hex characters (32 bytes)
|
||||
- Encrypted value format is exactly `vault:v1:base64...`
|
||||
|
||||
### Permission Denied Reading Secret
|
||||
|
||||
Pod may lack permission to read secret. Check service account:
|
||||
|
||||
```bash
|
||||
# Get function pod's service account
|
||||
kubectl get pod <pod-name> -n fission -o jsonpath='{.spec.serviceAccountName}'
|
||||
|
||||
# Check role bindings
|
||||
kubectl get rolebinding -n fission
|
||||
kubectl get clusterrolebinding -n fission
|
||||
|
||||
# Add permission if needed (requires cluster admin)
|
||||
kubectl create clusterrolebinding fission-secret-reader \
|
||||
--clusterrole=view \
|
||||
--serviceaccount=fission:default
|
||||
```
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/)
|
||||
- [Kubernetes ConfigMaps](https://kubernetes.io/docs/concepts/configuration/configmap/)
|
||||
- [Fission Environment and Config](https://fission.io/docs/usage/env/)
|
||||
- [PyNaCl Documentation](https://pynacl.readthedocs.io/)
|
||||
- [SealedSecrets](https://github.com/bitnami-labs/sealed-secrets) - Store encrypted secrets in Git
|
||||
240
fission-python/template/docs/STRUCTURE.md
Normal file
240
fission-python/template/docs/STRUCTURE.md
Normal file
@@ -0,0 +1,240 @@
|
||||
# Project Structure
|
||||
|
||||
This document explains the purpose and contents of each directory and file in a Fission Python project.
|
||||
|
||||
## Directory Layout
|
||||
|
||||
```
|
||||
project/
|
||||
├── .fission/ # Fission configuration
|
||||
│ ├── deployment.json # Main deployment configuration
|
||||
│ ├── dev-deployment.json # Development environment overrides
|
||||
│ └── local-deployment.json # Local development overrides
|
||||
├── src/ # Source code
|
||||
│ ├── __init__.py # Package initialization
|
||||
│ ├── vault.py # Vault encryption utilities
|
||||
│ ├── helpers.py # Shared utility functions
|
||||
│ ├── exceptions.py # Custom exception classes
|
||||
│ ├── models.py # Pydantic models for validation
|
||||
│ ├── build.sh # Build script (executable)
|
||||
│ └── *.py # Your function implementations
|
||||
├── test/ # Unit and integration tests
|
||||
│ ├── __init__.py
|
||||
│ ├── test_*.py # Test files
|
||||
│ └── requirements.txt # Test dependencies
|
||||
├── migrates/ # Database migration scripts
|
||||
│ └── *.sql # SQL migration files
|
||||
├── manifests/ # Kubernetes manifests (optional)
|
||||
│ └── *.yaml # K8s resources
|
||||
├── specs/ # Generated Fission specs
|
||||
│ ├── fission-deployment-config.yaml
|
||||
│ └── ...
|
||||
├── requirements.txt # Runtime dependencies
|
||||
├── dev-requirements.txt # Development dependencies
|
||||
├── .env.example # Environment variable template
|
||||
├── pytest.ini # Pytest configuration
|
||||
├── README.md # Project documentation
|
||||
└── (other project files)
|
||||
```
|
||||
|
||||
## File Purposes
|
||||
|
||||
### .fission/deployment.json
|
||||
|
||||
This is **the most important configuration file** for Fission deployment. It defines:
|
||||
|
||||
- **environments**: Build environment configuration (image, builder, resources)
|
||||
- **archives**: Source code packaging (typically "package.zip" from src/)
|
||||
- **packages**: Package definitions linking source to environment
|
||||
- **function_common**: Default settings applied to all functions
|
||||
- **secrets**: Secret definitions (literal values are placeholders - actual secrets go in K8s)
|
||||
- **configmaps**: ConfigMap definitions (non-sensitive configuration)
|
||||
|
||||
**Important**: The secret and configmap literals are **placeholders only**. In production, you create actual K8s secrets/configmaps with the same names containing real values.
|
||||
|
||||
**Placeholders**:
|
||||
- `${PROJECT_NAME}` - Replaced with your project name by `create-project.sh`
|
||||
- Secret name pattern: `fission-${PROJECT_NAME}-env`
|
||||
- ConfigMap name pattern: `fission-${PROJECT_NAME}-config`
|
||||
|
||||
### src/vault.py
|
||||
|
||||
Provides encryption/decryption utilities using PyNaCl (SecretBox). This is used when you want to store encrypted values in K8s secrets rather than plaintext.
|
||||
|
||||
**Key functions**:
|
||||
- `encrypt_vault(plaintext, key)` - Encrypt and return vault format string
|
||||
- `decrypt_vault(vault, key)` - Decrypt vault format string
|
||||
- `is_valid_vault_format(vault)` - Check if string is vault-encrypted
|
||||
|
||||
**Usage in helpers.py**: The `get_secret()` and `get_config()` functions automatically detect vault format (`vault:v1:...`) and decrypt if a valid `CRYPTO_KEY` is set.
|
||||
|
||||
### src/helpers.py
|
||||
|
||||
Shared utilities used across functions:
|
||||
|
||||
**Database**:
|
||||
- `init_db_connection()` - Creates PostgreSQL connection from secrets
|
||||
- `db_row_to_dict(cursor, row)` - Convert row tuple to dict
|
||||
- `db_rows_to_array(cursor, rows)` - Convert multiple rows to list of dicts
|
||||
|
||||
**Configuration**:
|
||||
- `get_secret(key, default=None)` - Read from K8s secret volume
|
||||
- `get_config(key, default=None)` - Read from K8s config volume
|
||||
- `get_current_namespace()` - Get current K8s namespace
|
||||
|
||||
**Utilities**:
|
||||
- `str_to_bool(input)` - Convert string to boolean
|
||||
- `check_port_open(ip, port, timeout)` - TCP port connectivity check
|
||||
- `get_user_from_headers()` - Extract user ID from request headers
|
||||
- `format_error_response(...)` - Build standardized error dict
|
||||
|
||||
**Logging**:
|
||||
- Helper uses `current_app.logger` (Flask) for error logging
|
||||
|
||||
### src/exceptions.py
|
||||
|
||||
Custom exception hierarchy:
|
||||
|
||||
```
|
||||
ServiceException (base)
|
||||
├── ValidationError (400) - Invalid input
|
||||
├── NotFoundError (404) - Resource not found
|
||||
├── ConflictError (409) - Duplicate/conflict
|
||||
└── DatabaseError (500) - Database failure
|
||||
```
|
||||
|
||||
All exceptions include:
|
||||
- `error_code` - Machine-readable code
|
||||
- `http_status` - HTTP status
|
||||
- `error_msg` - Human-readable message
|
||||
- `x_user` (optional) - User identifier
|
||||
- `details` (optional) - Additional context dict
|
||||
|
||||
When raised in a Fission function, these automatically return proper JSON error responses.
|
||||
|
||||
### src/models.py
|
||||
|
||||
Pydantic models for request/response validation:
|
||||
|
||||
**Patterns included**:
|
||||
- Enums (e.g., `Status`, `DataType`)
|
||||
- Dataclass filters (e.g., `ItemFilter`, `Pagination`)
|
||||
- Request models (`ItemCreateRequest`, `ItemUpdateRequest`)
|
||||
- Response models (`ItemResponse`, `PaginatedResponse`)
|
||||
- ErrorResponse model (used by exceptions)
|
||||
|
||||
**Key concepts**:
|
||||
- Use `Field(...)` with constraints (min_length, max_length, ge, le)
|
||||
- Provide `description` for API documentation
|
||||
- Use `json_schema_extra` for example values
|
||||
- Set `from_attributes = True` for ORM compatibility
|
||||
|
||||
### src/build.sh
|
||||
|
||||
Bash script that builds the dependency package. It:
|
||||
1. Detects OS (Debian vs Alpine)
|
||||
2. Installs build dependencies (gcc, libpq-dev/python3-dev/postgresql-dev)
|
||||
3. Installs Python requirements into `src/` directory
|
||||
4. Copies `src/` to package destination
|
||||
|
||||
**Important**: Must be executable (`chmod +x src/build.sh`)
|
||||
|
||||
The script expects environment variables:
|
||||
- `SRC_PKG` - Source package directory (e.g., `src`)
|
||||
- `DEPLOY_PKG` - Destination package (e.g., `specs/package`)
|
||||
|
||||
Fission builder sets these automatically.
|
||||
|
||||
### test/
|
||||
|
||||
Contains unit and integration tests.
|
||||
|
||||
**Structure**:
|
||||
- `test_*.py` - Test files following pytest conventions
|
||||
- `requirements.txt` - Test dependencies (pytest, pytest-mock, requests)
|
||||
|
||||
**Running tests**:
|
||||
```bash
|
||||
pip install -r dev-requirements.txt
|
||||
pytest
|
||||
```
|
||||
|
||||
## Fission Configuration in Docstrings
|
||||
|
||||
Each Python function that should be exposed as a Fission function **must** include a ````fission` block in its docstring:
|
||||
|
||||
```python
|
||||
def my_function(event, context):
|
||||
"""
|
||||
```fission
|
||||
{
|
||||
"name": "my-function",
|
||||
"http_triggers": {
|
||||
"my-trigger": {
|
||||
"url": "/api/endpoint",
|
||||
"methods": ["GET", "POST"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
Human-readable description here.
|
||||
"""
|
||||
# Implementation
|
||||
```
|
||||
|
||||
The Fission Python builder parses these docstrings and generates the `specs/fission-deployment-config.yaml` and other spec files.
|
||||
|
||||
**Supported trigger types**:
|
||||
- `http_triggers` - HTTP endpoints
|
||||
- `kafka_triggers` - Kafka topics
|
||||
- `timer_triggers` - Scheduled execution
|
||||
- `message_queue_triggers` - MQTT, NATS, etc.
|
||||
|
||||
## Configuration Precedence
|
||||
|
||||
1. **deployment.json** - Base configuration (committed to repo)
|
||||
2. **dev-deployment.json** - Overrides for dev environment (not always committed)
|
||||
3. **local-deployment.json** - Local overrides (typically .gitignored)
|
||||
|
||||
When deploying:
|
||||
- `fission deploy` uses deployment.json
|
||||
- `fission deploy --dev` uses dev-deployment.json if present
|
||||
|
||||
## Secrets and Configuration Flow
|
||||
|
||||
1. **Define placeholders** in `deployment.json`:
|
||||
```json
|
||||
"secrets": {
|
||||
"fission-myproject-env": {
|
||||
"literals": ["PG_HOST=localhost", "PG_PORT=5432"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. **Create actual K8s secret**:
|
||||
```bash
|
||||
kubectl create secret generic fission-myproject-env \
|
||||
--from-literal=PG_HOST=prod-db.example.com \
|
||||
--from-literal=PG_PORT=5432
|
||||
```
|
||||
|
||||
3. **Read in code** via `get_secret()`:
|
||||
```python
|
||||
host = get_secret("PG_HOST")
|
||||
```
|
||||
|
||||
4. **For vault encryption**:
|
||||
- Set `CRYPTO_KEY` in helpers.py or as env override
|
||||
- Store encrypted: `vault:v1:base64data` in K8s secret
|
||||
- `get_secret()` auto-decrypts
|
||||
|
||||
## Summary
|
||||
|
||||
- Keep function code in `src/`
|
||||
- Define Fission metadata in docstring blocks
|
||||
- Use helpers for common operations
|
||||
- Define custom exceptions for error handling
|
||||
- Validate inputs with Pydantic models
|
||||
- Store tests in `test/` with pytest
|
||||
- Manage database migrations in `migrates/`
|
||||
- Do not commit actual secrets to repository
|
||||
567
fission-python/template/docs/TESTING.md
Normal file
567
fission-python/template/docs/TESTING.md
Normal file
@@ -0,0 +1,567 @@
|
||||
# Testing Guide
|
||||
|
||||
This document covers testing strategies and best practices for Fission Python functions.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Test Types](#test-types)
|
||||
2. [Dependencies](#dependencies)
|
||||
3. [Unit Testing](#unit-testing)
|
||||
4. [Integration Testing](#integration-testing)
|
||||
5. [Test Database](#test-database)
|
||||
6. [Mocking](#mocking)
|
||||
7. [Fixtures](#fixtures)
|
||||
8. [Coverage](#coverage)
|
||||
9. [Running Tests](#running-tests)
|
||||
10. [CI/CD Integration](#cicd-integration)
|
||||
|
||||
## Test Types
|
||||
|
||||
### Unit Tests
|
||||
|
||||
Test individual functions in isolation, mocking external dependencies:
|
||||
- Database calls
|
||||
- HTTP requests
|
||||
- File I/O
|
||||
- External services
|
||||
|
||||
**Goal**: Verify business logic correctness without infrastructure.
|
||||
|
||||
### Integration Tests
|
||||
|
||||
Test the function with real (or test) dependencies:
|
||||
- Actual database queries
|
||||
- End-to-end request/response flow
|
||||
- Real configuration loading
|
||||
|
||||
**Goal**: Verify integration points work correctly.
|
||||
|
||||
## Dependencies
|
||||
|
||||
Install test dependencies:
|
||||
|
||||
```bash
|
||||
pip install -r test/requirements.txt
|
||||
# Or for dev (includes both runtime and test deps):
|
||||
pip install -r dev-requirements.txt
|
||||
```
|
||||
|
||||
Required packages:
|
||||
- `pytest` - Test framework
|
||||
- `pytest-mock` - Mocking utilities (provides `mocker` fixture)
|
||||
- `requests` - For integration tests making HTTP calls
|
||||
|
||||
## Unit Testing
|
||||
|
||||
### Example Test Structure
|
||||
|
||||
```python
|
||||
# test/test_my_function.py
|
||||
import pytest
|
||||
from unittest.mock import patch, MagicMock
|
||||
from src.my_function import create_item
|
||||
from exceptions import ValidationError
|
||||
|
||||
def test_create_item_success():
|
||||
"""Test successful item creation."""
|
||||
# Arrange
|
||||
mock_conn = MagicMock()
|
||||
mock_cursor = MagicMock()
|
||||
mock_conn.cursor.return_value = mock_cursor
|
||||
mock_cursor.fetchone.return_value = ("item-id", "Item Name", "active")
|
||||
|
||||
# Mock init_db_connection to return our mock
|
||||
with patch("src.my_function.init_db_connection", return_value=mock_conn):
|
||||
# Create a mock Flask request
|
||||
with patch("src.my_function.request") as mock_request:
|
||||
mock_request.get_json.return_value = {
|
||||
"name": "Test Item",
|
||||
"status": "active"
|
||||
}
|
||||
mock_request.view_args = {}
|
||||
|
||||
# Act
|
||||
result = create_item({}, {})
|
||||
|
||||
# Assert
|
||||
assert result["id"] == "item-id"
|
||||
assert result["name"] == "Test Item"
|
||||
mock_cursor.execute.assert_called_once()
|
||||
mock_conn.commit.assert_called_once()
|
||||
|
||||
def test_create_item_validation_error():
|
||||
"""Test validation of missing required fields."""
|
||||
with patch("src.my_function.request") as mock_request:
|
||||
mock_request.get_json.return_value = {"name": ""} # Empty name
|
||||
|
||||
with pytest.raises(ValidationError) as exc_info:
|
||||
create_item({}, {})
|
||||
|
||||
assert "validation" in str(exc_info.value.error_msg).lower()
|
||||
```
|
||||
|
||||
### Mocking Helpers
|
||||
|
||||
Use `patch` to replace dependencies:
|
||||
|
||||
```python
|
||||
# Mock helpers.get_secret
|
||||
@patch("src.my_function.helpers.get_secret")
|
||||
def test_with_mocked_secret(mock_get_secret):
|
||||
mock_get_secret.return_value = "localhost"
|
||||
# Test code...
|
||||
|
||||
# Mock entire module
|
||||
@patch("src.my_function.helpers.init_db_connection")
|
||||
def test_with_mocked_db(mock_init_db):
|
||||
mock_conn = MagicMock()
|
||||
mock_init_db.return_value = mock_conn
|
||||
# Test code...
|
||||
```
|
||||
|
||||
### Mocking Flask Request
|
||||
|
||||
```python
|
||||
from flask import Request
|
||||
|
||||
def test_with_flask_request():
|
||||
with patch("src.my_function.request") as mock_request:
|
||||
mock_request.get_json.return_value = {"key": "value"}
|
||||
mock_request.args.getlist.return_value = []
|
||||
mock_request.headers.get.return_value = "user-123"
|
||||
# Test code...
|
||||
```
|
||||
|
||||
## Integration Testing
|
||||
|
||||
### Test Database Setup
|
||||
|
||||
Use a separate test database:
|
||||
|
||||
```bash
|
||||
# Create test database
|
||||
createdb fission_test
|
||||
|
||||
# Or with Docker:
|
||||
docker run -d -p 5433:5432 -e POSTGRES_PASSWORD=test postgres:15
|
||||
```
|
||||
|
||||
Set environment variables for test database:
|
||||
```bash
|
||||
export PG_HOST=localhost
|
||||
export PG_PORT=5433
|
||||
export PG_DB=fission_test
|
||||
export PG_USER=postgres
|
||||
export PG_PASS=test
|
||||
```
|
||||
|
||||
### pytest Fixtures for Database
|
||||
|
||||
```python
|
||||
# conftest.py (placed in test/ directory)
|
||||
import pytest
|
||||
import psycopg2
|
||||
from helpers import init_db_connection
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def db_connection():
|
||||
"""Create a database connection for the entire test session."""
|
||||
conn = init_db_connection()
|
||||
yield conn
|
||||
conn.close()
|
||||
|
||||
@pytest.fixture(scope="function")
|
||||
def db_cursor(db_connection):
|
||||
"""Create a cursor for each test, with transaction rollback."""
|
||||
conn = db_connection
|
||||
cursor = conn.cursor()
|
||||
# Start a transaction that will be rolled back
|
||||
conn.rollback()
|
||||
yield cursor
|
||||
# Rollback after each test to keep DB clean
|
||||
conn.rollback()
|
||||
```
|
||||
|
||||
### Example Integration Test
|
||||
|
||||
```python
|
||||
# test/test_integration.py
|
||||
def test_create_and_retrieve_item_integration(db_connection):
|
||||
"""Test full CRUD cycle with real database."""
|
||||
from src.models import ItemCreateRequest
|
||||
from src.functions import create_item, get_item
|
||||
|
||||
# Insert test data
|
||||
cursor = db_connection.cursor()
|
||||
cursor.execute("DELETE FROM items WHERE name = 'Integration Test'")
|
||||
db_connection.commit()
|
||||
|
||||
# Create item via function
|
||||
with patch("src.functions.request") as mock_request:
|
||||
mock_request.get_json.return_value = {
|
||||
"name": "Integration Test",
|
||||
"description": "Test item"
|
||||
}
|
||||
mock_request.view_args = {}
|
||||
result = create_item({}, {})
|
||||
|
||||
item_id = result["id"]
|
||||
assert result["name"] == "Integration Test"
|
||||
|
||||
# Retrieve same item
|
||||
with patch("src.functions.request") as mock_request:
|
||||
with patch("src.functions.request.view_args", {"id": item_id}):
|
||||
result = get_item({"path": f"/items/{item_id}"}, {})
|
||||
assert result["id"] == item_id
|
||||
|
||||
# Cleanup
|
||||
cursor.execute("DELETE FROM items WHERE id = %s", (item_id,))
|
||||
db_connection.commit()
|
||||
```
|
||||
|
||||
## Test Database Migrations
|
||||
|
||||
Apply migrations before integration tests:
|
||||
|
||||
```python
|
||||
# conftest.py
|
||||
import subprocess
|
||||
|
||||
def apply_migrations():
|
||||
"""Apply all SQL migrations to test database."""
|
||||
import os
|
||||
migrates_dir = os.path.join(os.path.dirname(__file__), "..", "migrates")
|
||||
for file in sorted(os.listdir(migrates_dir)):
|
||||
if file.endswith(".sql"):
|
||||
path = os.path.join(migrates_dir, file)
|
||||
subprocess.run(
|
||||
["psql", "-d", "fission_test", "-f", path],
|
||||
check=True
|
||||
)
|
||||
|
||||
@pytest.fixture(scope="session", autouse=True)
|
||||
def setup_database():
|
||||
"""Run migrations before any tests."""
|
||||
apply_migrations()
|
||||
yield
|
||||
# Optionally drop and recreate after tests
|
||||
```
|
||||
|
||||
## Mocking
|
||||
|
||||
### Built-in unittest.mock
|
||||
|
||||
```python
|
||||
from unittest.mock import patch, MagicMock, mock_open
|
||||
|
||||
# Simple patch
|
||||
with patch("module.function") as mock_func:
|
||||
mock_func.return_value = "mocked"
|
||||
# call code that uses module.function
|
||||
|
||||
# Assert called with specific args
|
||||
mock_func.assert_called_once_with("arg1", "arg2")
|
||||
|
||||
# Mock context manager
|
||||
with patch("builtins.open", mock_open(read_data="file content")) as mock_file:
|
||||
# code that opens file
|
||||
mock_file.assert_called_with("path/to/file", "r")
|
||||
```
|
||||
|
||||
### pytest-mock Fixture
|
||||
|
||||
Simpler syntax using `mocker` fixture:
|
||||
|
||||
```python
|
||||
def test_with_mocker(mocker):
|
||||
mock_func = mocker.patch("src.function.helper")
|
||||
mock_func.return_value = {"key": "value"}
|
||||
# test code...
|
||||
```
|
||||
|
||||
## Fixtures
|
||||
|
||||
Create reusable fixtures in `conftest.py`:
|
||||
|
||||
```python
|
||||
# test/conftest.py
|
||||
import pytest
|
||||
|
||||
@pytest.fixture
|
||||
def sample_item_data():
|
||||
"""Provide sample item data for tests."""
|
||||
return {
|
||||
"name": "Test Item",
|
||||
"description": "A test item",
|
||||
"status": "active"
|
||||
}
|
||||
|
||||
@pytest.fixture
|
||||
def mock_db_connection(mocker):
|
||||
"""Provide a mocked database connection."""
|
||||
mock_conn = mocker.MagicMock()
|
||||
mock_cursor = mocker.MagicMock()
|
||||
mock_conn.cursor.return_value = mock_cursor
|
||||
mock_cursor.fetchone.return_value = None
|
||||
return mock_conn
|
||||
```
|
||||
|
||||
Fixtures are automatically available to all tests in the directory.
|
||||
|
||||
## Coverage
|
||||
|
||||
Measure test coverage with pytest-cov:
|
||||
|
||||
```bash
|
||||
# Install
|
||||
pip install pytest-cov
|
||||
|
||||
# Run with coverage
|
||||
pytest --cov=src
|
||||
|
||||
# HTML report
|
||||
pytest --cov=src --cov-report=html
|
||||
open htmlcov/index.html
|
||||
|
||||
# Show missing lines
|
||||
pytest --cov=src --cov-report=term-missing
|
||||
```
|
||||
|
||||
Aim for high coverage of business logic (80%+). Don't worry about 100% coverage of trivial getters/setters.
|
||||
|
||||
### Excluding Files
|
||||
|
||||
Add to `pytest.ini`:
|
||||
```ini
|
||||
[pytest]
|
||||
addopts = --cov=src --cov-exclude=src/vault.py
|
||||
```
|
||||
|
||||
Or use `.coveragerc`:
|
||||
```ini
|
||||
[run]
|
||||
omit = src/vault.py
|
||||
```
|
||||
|
||||
## Running Tests
|
||||
|
||||
### Basic Commands
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
pytest
|
||||
|
||||
# Verbose
|
||||
pytest -v
|
||||
|
||||
# Run specific test file
|
||||
pytest test/test_my_function.py
|
||||
|
||||
# Run specific test function
|
||||
pytest test/test_my_function.py::test_create_item_success
|
||||
|
||||
# Run with markers
|
||||
pytest -m "integration" # if using @pytest.mark.integration
|
||||
|
||||
# Stop on first failure
|
||||
pytest -x
|
||||
|
||||
# Show print statements
|
||||
pytest -s
|
||||
```
|
||||
|
||||
### Environment Setup
|
||||
|
||||
Create `test/.env` or set environment variables before tests:
|
||||
|
||||
```bash
|
||||
# For integration tests
|
||||
export PG_HOST=localhost
|
||||
export PG_PORT=5432
|
||||
export PG_DB=fission_test
|
||||
```
|
||||
|
||||
Or use a pytest fixture to load from `.env`:
|
||||
|
||||
```python
|
||||
# conftest.py
|
||||
import os
|
||||
from dotenv import load_dotenv
|
||||
|
||||
@pytest.fixture(scope="session", autouse=True)
|
||||
def load_env():
|
||||
env_path = os.path.join(os.path.dirname(__file__), ".env")
|
||||
load_dotenv(env_path)
|
||||
```
|
||||
|
||||
### Markers
|
||||
|
||||
Mark tests as unit/integration/slow:
|
||||
|
||||
```python
|
||||
import pytest
|
||||
|
||||
@pytest.mark.unit
|
||||
def test_quick_unit():
|
||||
pass
|
||||
|
||||
@pytest.mark.integration
|
||||
def test_full_workflow():
|
||||
pass
|
||||
|
||||
@pytest.mark.slow
|
||||
def test_long_running():
|
||||
pass
|
||||
```
|
||||
|
||||
Run only unit tests:
|
||||
```bash
|
||||
pytest -m "unit"
|
||||
```
|
||||
|
||||
Skip tests:
|
||||
```bash
|
||||
pytest -m "not slow"
|
||||
```
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
### GitHub Actions Example
|
||||
|
||||
```yaml
|
||||
# .github/workflows/test.yaml
|
||||
name: Tests
|
||||
|
||||
on: [push, pull_request]
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:15
|
||||
env:
|
||||
POSTGRES_PASSWORD: test
|
||||
options: >-
|
||||
--health-cmd pg_isready
|
||||
--health-interval 10s
|
||||
--health-timeout 5s
|
||||
--health-retries 5
|
||||
ports:
|
||||
- 5432:5432
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: '3.11'
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
pip install -r dev-requirements.txt
|
||||
- name: Setup database
|
||||
run: |
|
||||
createdb -h localhost -U postgres fission_test
|
||||
psql -h localhost -U postgres fission_test -f migrates/001_schema.sql
|
||||
env:
|
||||
PGPASSWORD: test
|
||||
- name: Run tests
|
||||
run: |
|
||||
pytest --cov=src --cov-report=xml
|
||||
env:
|
||||
PG_HOST: localhost
|
||||
PG_PORT: 5432
|
||||
PG_DB: fission_test
|
||||
PG_USER: postgres
|
||||
PG_PASS: test
|
||||
- name: Upload coverage
|
||||
uses: codecov/codecov-action@v3
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **One assertion per test** - Keep tests focused
|
||||
2. **Use descriptive names** - `test_create_item_validation_error_for_missing_name`
|
||||
3. **Arrange-Act-Assert** - Structure tests clearly
|
||||
4. **Mock external dependencies** - Don't rely on network or external services
|
||||
5. **Test error cases** - Don't just test happy paths
|
||||
6. **Use fixtures** - Reuse setup/teardown code
|
||||
7. **Keep tests independent** - No shared state between tests
|
||||
8. **Test edge cases** - Empty inputs, null values, boundary conditions
|
||||
9. **Don't test libraries** - Don't write tests for Flask/Pydantic themselves
|
||||
10. **Clean up resources** - Use fixtures to ensure cleanup
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Testing Exceptions
|
||||
|
||||
```python
|
||||
def test_raises_not_found():
|
||||
with pytest.raises(NotFoundError) as exc:
|
||||
get_item("nonexistent-id")
|
||||
assert exc.value.http_status == 404
|
||||
```
|
||||
|
||||
### Parametrized Tests
|
||||
|
||||
```python
|
||||
import pytest
|
||||
|
||||
@pytest.mark.parametrize("input,expected", [
|
||||
("true", True),
|
||||
("false", False),
|
||||
("", None),
|
||||
(None, None),
|
||||
])
|
||||
def test_str_to_bool(input, expected):
|
||||
from helpers import str_to_bool
|
||||
assert str_to_bool(input) == expected
|
||||
```
|
||||
|
||||
### Temporary Files/Directories
|
||||
|
||||
```python
|
||||
def test_with_temp_file(tmp_path):
|
||||
# tmp_path is a pathlib.Path to a temporary directory
|
||||
file = tmp_path / "test.txt"
|
||||
file.write_text("content")
|
||||
assert file.read_text() == "content"
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Tests Fail with Database Errors
|
||||
|
||||
- Check test database is running: `pg_isready -h localhost -p 5432`
|
||||
- Verify migrations applied: `psql -l | grep fission_test`
|
||||
- Check environment variables: `echo $PG_HOST`
|
||||
|
||||
### Mock Not Working
|
||||
|
||||
- Ensure you're patching the **correct import location** (where it's used, not where it's defined)
|
||||
```python
|
||||
# Wrong: patching where it's defined
|
||||
@patch("helpers.get_secret")
|
||||
# Right: patching where it's used in your function module
|
||||
@patch("src.my_function.helpers.get_secret")
|
||||
```
|
||||
|
||||
### Import Errors
|
||||
|
||||
Ensure PYTHONPATH includes project root:
|
||||
```bash
|
||||
export PYTHONPATH=/path/to/project:$PYTHONPATH
|
||||
```
|
||||
|
||||
Or use pytest's `pythonpath` option in pytest.ini:
|
||||
```ini
|
||||
[pytest]
|
||||
pythonpath = .
|
||||
```
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [pytest documentation](https://docs.pytest.org/)
|
||||
- [pytest-mock documentation](https://github.com/pytest-dev/pytest-mock)
|
||||
- [Python unittest.mock](https://docs.python.org/3/library/unittest.mock.html)
|
||||
- [Testing Flask Applications](https://flask.palletsprojects.com/en/2.1.x/testing/)
|
||||
433
fission-python/template/examples/example_crud.py
Normal file
433
fission-python/template/examples/example_crud.py
Normal file
@@ -0,0 +1,433 @@
|
||||
"""
|
||||
Example: Basic CRUD operations for a resource.
|
||||
|
||||
This demonstrates:
|
||||
- Pydantic request validation
|
||||
- Database operations with helpers
|
||||
- Standard error handling
|
||||
- Proper Fission docstring configuration
|
||||
"""
|
||||
|
||||
from flask import request
|
||||
from helpers import (
|
||||
init_db_connection,
|
||||
db_row_to_dict,
|
||||
db_rows_to_array,
|
||||
get_user_from_headers,
|
||||
format_error_response,
|
||||
)
|
||||
from exceptions import ValidationError, NotFoundError, ConflictError, DatabaseError
|
||||
from models import ItemResponse, ItemCreateRequest, ItemUpdateRequest
|
||||
|
||||
# Pool manager executor, one request at a time
|
||||
def create(event, context):
|
||||
"""
|
||||
```fission
|
||||
{
|
||||
"name": "create-item",
|
||||
"http_triggers": {
|
||||
"create": {
|
||||
"url": "/api/items",
|
||||
"methods": ["POST"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
Create a new item.
|
||||
|
||||
**Request Body:**
|
||||
```json
|
||||
{
|
||||
"name": "string (required, 1-255 chars)",
|
||||
"description": "string (optional)",
|
||||
"status": "active|inactive|pending",
|
||||
"metadata": {}
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
- 200: Item created successfully
|
||||
- 400: Validation error
|
||||
- 409: Conflict (e.g., duplicate name)
|
||||
- 500: Database error
|
||||
"""
|
||||
# Get user for audit trail
|
||||
x_user = get_user_from_headers()
|
||||
|
||||
# Validate request payload
|
||||
try:
|
||||
data = ItemCreateRequest(**request.get_json())
|
||||
except Exception as e:
|
||||
raise ValidationError(f"Invalid request: {str(e)}", x_user=x_user)
|
||||
|
||||
conn = None
|
||||
try:
|
||||
conn = init_db_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
# Check for conflicts (example)
|
||||
cursor.execute(
|
||||
"SELECT id FROM items WHERE name = %s",
|
||||
(data.name,)
|
||||
)
|
||||
if cursor.fetchone():
|
||||
raise ConflictError(
|
||||
f"Item with name '{data.name}' already exists",
|
||||
x_user=x_user,
|
||||
details={"name": data.name}
|
||||
)
|
||||
|
||||
# Insert new item
|
||||
cursor.execute(
|
||||
"""
|
||||
INSERT INTO items (name, description, status, metadata)
|
||||
VALUES (%s, %s, %s, %s)
|
||||
RETURNING id, name, description, status, metadata, created, modified
|
||||
""",
|
||||
(data.name, data.description, data.status.value, data.metadata)
|
||||
)
|
||||
row = cursor.fetchone()
|
||||
conn.commit()
|
||||
|
||||
# Build response
|
||||
item = db_row_to_dict(cursor, row)
|
||||
return item
|
||||
|
||||
except (ValidationError, NotFoundError, ConflictError, DatabaseError):
|
||||
# Re-raise our own exceptions
|
||||
raise
|
||||
except Exception as e:
|
||||
if conn:
|
||||
conn.rollback()
|
||||
raise DatabaseError(f"Database error: {str(e)}", x_user=x_user)
|
||||
finally:
|
||||
if conn:
|
||||
conn.close()
|
||||
|
||||
|
||||
def list_items(event, context):
|
||||
"""
|
||||
```fission
|
||||
{
|
||||
"name": "list-items",
|
||||
"http_triggers": {
|
||||
"list": {
|
||||
"url": "/api/items",
|
||||
"methods": ["GET"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
List items with optional filtering and pagination.
|
||||
|
||||
**Query Parameters:**
|
||||
- `page` (int): Page number, zero-based (default: 0)
|
||||
- `size` (int): Items per page (default: 10, max: 100)
|
||||
- `asc` (bool): Sort ascending (default: true)
|
||||
- `filter[ids]` (string[]): Filter by specific IDs
|
||||
- `filter[keyword]` (string): Search in name/description
|
||||
- `filter[status]` (string[]): Filter by status values
|
||||
- `filter[created_from]` (datetime): Filter created after
|
||||
- `filter[created_to]` (datetime): Filter created before
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"data": [...],
|
||||
"page": 0,
|
||||
"size": 10,
|
||||
"total": 42
|
||||
}
|
||||
```
|
||||
"""
|
||||
from helpers import str_to_bool
|
||||
|
||||
# Parse pagination
|
||||
page = int(request.args.get("page", 0))
|
||||
size = int(request.args.get("size", 10))
|
||||
asc = str_to_bool(request.args.get("asc", "true"))
|
||||
|
||||
# Parse filters
|
||||
ids = request.args.getlist("filter[ids]")
|
||||
keyword = request.args.get("filter[keyword]")
|
||||
statuses = request.args.getlist("filter[status]")
|
||||
|
||||
# Build query
|
||||
conditions = []
|
||||
params = []
|
||||
|
||||
if ids:
|
||||
conditions.append(f"id IN ({', '.join(['%s'] * len(ids))})")
|
||||
params.extend(ids)
|
||||
if keyword:
|
||||
conditions.append("(name ILIKE %s OR description ILIKE %s)")
|
||||
params.extend([f"%{keyword}%", f"%{keyword}%"])
|
||||
if statuses:
|
||||
conditions.append(f"status IN ({', '.join(['%s'] * len(statuses))})")
|
||||
params.extend(statuses)
|
||||
|
||||
where_clause = "WHERE " + " AND ".join(conditions) if conditions else ""
|
||||
|
||||
conn = None
|
||||
try:
|
||||
conn = init_db_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
# Get total count
|
||||
count_sql = f"SELECT COUNT(*) FROM items {where_clause}"
|
||||
cursor.execute(count_sql, params)
|
||||
total = cursor.fetchone()[0]
|
||||
|
||||
# Get paginated data
|
||||
offset = page * size
|
||||
data_sql = f"""
|
||||
SELECT id, name, description, status, metadata, created, modified
|
||||
FROM items
|
||||
{where_clause}
|
||||
ORDER BY created {'ASC' if asc else 'DESC'}
|
||||
LIMIT %s OFFSET %s
|
||||
"""
|
||||
cursor.execute(data_sql, params + [size, offset])
|
||||
rows = cursor.fetchall()
|
||||
items = [db_row_to_dict(cursor, row) for row in rows]
|
||||
|
||||
return {
|
||||
"data": items,
|
||||
"page": page,
|
||||
"size": size,
|
||||
"total": total
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
raise DatabaseError(f"Failed to list items: {str(e)}")
|
||||
finally:
|
||||
if conn:
|
||||
conn.close()
|
||||
|
||||
|
||||
def get_item(event, context):
|
||||
"""
|
||||
```fission
|
||||
{
|
||||
"name": "get-item",
|
||||
"http_triggers": {
|
||||
"get": {
|
||||
"url": "/api/items/:id",
|
||||
"methods": ["GET"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
Get a specific item by ID.
|
||||
|
||||
**URL Parameters:**
|
||||
- `id` (string): Item UUID
|
||||
|
||||
**Response:**
|
||||
- 200: Item found
|
||||
- 404: Item not found
|
||||
- 500: Database error
|
||||
"""
|
||||
# Extract item ID from path (Fission passes path params differently depending on trigger)
|
||||
# For HTTP triggers, the ID would come from the URL path
|
||||
item_id = request.view_args.get('id') if hasattr(request, 'view_args') else None
|
||||
if not item_id:
|
||||
# Fallback: parse from query or request path
|
||||
item_id = request.path.rstrip('/').split('/')[-1]
|
||||
|
||||
if not item_id:
|
||||
raise ValidationError("Item ID is required")
|
||||
|
||||
conn = None
|
||||
try:
|
||||
conn = init_db_connection()
|
||||
cursor = conn.cursor()
|
||||
cursor.execute(
|
||||
"""
|
||||
SELECT id, name, description, status, metadata, created, modified
|
||||
FROM items WHERE id = %s
|
||||
""",
|
||||
(item_id,)
|
||||
)
|
||||
row = cursor.fetchone()
|
||||
if not row:
|
||||
raise NotFoundError(f"Item {item_id} not found")
|
||||
|
||||
return db_row_to_dict(cursor, row)
|
||||
|
||||
except NotFoundError:
|
||||
raise
|
||||
except Exception as e:
|
||||
raise DatabaseError(f"Failed to fetch item: {str(e)}")
|
||||
finally:
|
||||
if conn:
|
||||
conn.close()
|
||||
|
||||
|
||||
def update_item(event, context):
|
||||
"""
|
||||
```fission
|
||||
{
|
||||
"name": "update-item",
|
||||
"http_triggers": {
|
||||
"update": {
|
||||
"url": "/api/items/:id",
|
||||
"methods": ["PUT", "PATCH"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
Update an existing item.
|
||||
|
||||
**URL Parameters:**
|
||||
- `id` (string): Item UUID
|
||||
|
||||
**Request Body:**
|
||||
```json
|
||||
{
|
||||
"name": "string (optional)",
|
||||
"description": "string (optional)",
|
||||
"status": "active|inactive|pending (optional)"
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
- 200: Item updated successfully
|
||||
- 404: Item not found
|
||||
- 409: Conflict (duplicate name)
|
||||
- 400: Validation error
|
||||
- 500: Database error
|
||||
"""
|
||||
x_user = get_user_from_headers()
|
||||
|
||||
# Extract item ID
|
||||
item_id = request.view_args.get('id') if hasattr(request, 'view_args') else None
|
||||
if not item_id:
|
||||
item_id = request.path.rstrip('/').split('/')[-1]
|
||||
|
||||
if not item_id:
|
||||
raise ValidationError("Item ID is required")
|
||||
|
||||
# Validate request
|
||||
try:
|
||||
data = ItemUpdateRequest(**request.get_json())
|
||||
except Exception as e:
|
||||
raise ValidationError(f"Invalid request: {str(e)}", x_user=x_user)
|
||||
|
||||
# Build update statement dynamically
|
||||
updates = []
|
||||
params = []
|
||||
|
||||
if data.name is not None:
|
||||
updates.append("name = %s")
|
||||
params.append(data.name)
|
||||
if data.description is not None:
|
||||
updates.append("description = %s")
|
||||
params.append(data.description)
|
||||
if data.status is not None:
|
||||
updates.append("status = %s")
|
||||
params.append(data.status.value)
|
||||
if data.metadata is not None:
|
||||
updates.append("metadata = %s")
|
||||
params.append(data.metadata)
|
||||
|
||||
if not updates:
|
||||
raise ValidationError("No update fields provided", x_user=x_user)
|
||||
|
||||
updates.append("modified = NOW()")
|
||||
params.append(item_id)
|
||||
|
||||
conn = None
|
||||
try:
|
||||
conn = init_db_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
# Check for name conflict if name is being updated
|
||||
if data.name:
|
||||
cursor.execute(
|
||||
"SELECT id FROM items WHERE name = %s AND id != %s",
|
||||
(data.name, item_id)
|
||||
)
|
||||
if cursor.fetchone():
|
||||
raise ConflictError(
|
||||
f"Another item with name '{data.name}' already exists",
|
||||
x_user=x_user,
|
||||
details={"name": data.name}
|
||||
)
|
||||
|
||||
# Execute update
|
||||
sql = f"UPDATE items SET {', '.join(updates)} WHERE id = %s RETURNING *"
|
||||
cursor.execute(sql, params)
|
||||
row = cursor.fetchone()
|
||||
conn.commit()
|
||||
|
||||
if not row:
|
||||
raise NotFoundError(f"Item {item_id} not found", x_user=x_user)
|
||||
|
||||
return db_row_to_dict(cursor, row)
|
||||
|
||||
except (ValidationError, NotFoundError, ConflictError, DatabaseError):
|
||||
raise
|
||||
except Exception as e:
|
||||
if conn:
|
||||
conn.rollback()
|
||||
raise DatabaseError(f"Failed to update item: {str(e)}", x_user=x_user)
|
||||
finally:
|
||||
if conn:
|
||||
conn.close()
|
||||
|
||||
|
||||
def delete_item(event, context):
|
||||
"""
|
||||
```fission
|
||||
{
|
||||
"name": "delete-item",
|
||||
"http_triggers": {
|
||||
"delete": {
|
||||
"url": "/api/items/:id",
|
||||
"methods": ["DELETE"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
Delete an item.
|
||||
|
||||
**URL Parameters:**
|
||||
- `id` (string): Item UUID
|
||||
|
||||
**Response:**
|
||||
- 204: Item deleted successfully
|
||||
- 404: Item not found
|
||||
- 500: Database error
|
||||
"""
|
||||
x_user = get_user_from_headers()
|
||||
|
||||
# Extract item ID
|
||||
item_id = request.view_args.get('id') if hasattr(request, 'view_args') else None
|
||||
if not item_id:
|
||||
item_id = request.path.rstrip('/').split('/')[-1]
|
||||
|
||||
if not item_id:
|
||||
raise ValidationError("Item ID is required", x_user=x_user)
|
||||
|
||||
conn = None
|
||||
try:
|
||||
conn = init_db_connection()
|
||||
cursor = conn.cursor()
|
||||
cursor.execute("DELETE FROM items WHERE id = %s", (item_id,))
|
||||
conn.commit()
|
||||
|
||||
if cursor.rowcount == 0:
|
||||
raise NotFoundError(f"Item {item_id} not found", x_user=x_user)
|
||||
|
||||
return None # 204 No Content
|
||||
|
||||
except NotFoundError:
|
||||
raise
|
||||
except Exception as e:
|
||||
if conn:
|
||||
conn.rollback()
|
||||
raise DatabaseError(f"Failed to delete item: {str(e)}", x_user=x_user)
|
||||
finally:
|
||||
if conn:
|
||||
conn.close()
|
||||
311
fission-python/template/examples/example_scheduler.py
Normal file
311
fission-python/template/examples/example_scheduler.py
Normal file
@@ -0,0 +1,311 @@
|
||||
"""
|
||||
Example: Background job / scheduled task pattern.
|
||||
|
||||
This demonstrates:
|
||||
- Long-running job execution
|
||||
- Job status tracking
|
||||
- Error handling and retries
|
||||
- Periodic task scheduling
|
||||
- Worker session management
|
||||
|
||||
Use cases: report generation, batch processing, cleanup jobs, etc.
|
||||
"""
|
||||
|
||||
import datetime
|
||||
import time
|
||||
import uuid
|
||||
from helpers import init_db_connection, db_row_to_dict, db_rows_to_array
|
||||
from exceptions import DatabaseError
|
||||
|
||||
|
||||
def scheduled_job(event, context):
|
||||
"""
|
||||
```fission
|
||||
{
|
||||
"name": "scheduled-job",
|
||||
"http_triggers": {
|
||||
"run": {
|
||||
"url": "/jobs/run",
|
||||
"methods": ["POST"]
|
||||
}
|
||||
},
|
||||
"kafka_triggers": {
|
||||
"job-queue": {
|
||||
"topic": "job-queue",
|
||||
"consumer_group": "scheduler-workers"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
Execute a scheduled or queued background job.
|
||||
|
||||
This function can be triggered:
|
||||
- Manually via HTTP POST /jobs/run
|
||||
- Automatically by message queue (Kafka)
|
||||
- By cron schedule (via Fission timer trigger)
|
||||
|
||||
**Request Body (HTTP trigger):**
|
||||
```json
|
||||
{
|
||||
"job_type": "report_generation",
|
||||
"parameters": {
|
||||
"report_type": "daily",
|
||||
"date": "2025-03-18"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
- 200: Job completed successfully
|
||||
- 202: Job accepted for async processing
|
||||
- 400: Invalid request
|
||||
- 500: Job failed
|
||||
"""
|
||||
# Parse input
|
||||
job_type = event.get("job_type") or event.get("type", "default")
|
||||
parameters = event.get("parameters", {})
|
||||
|
||||
# Generate job ID for tracking
|
||||
job_id = str(uuid.uuid4())
|
||||
started_at = datetime.datetime.utcnow()
|
||||
|
||||
conn = None
|
||||
try:
|
||||
conn = init_db_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
# Record job start
|
||||
cursor.execute(
|
||||
"""
|
||||
INSERT INTO jobs (id, type, parameters, status, started_at)
|
||||
VALUES (%s, %s, %s, 'running', %s)
|
||||
""",
|
||||
(job_id, job_type, parameters, started_at)
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
# Execute job based on type
|
||||
if job_type == "report_generation":
|
||||
result = generate_report(cursor, job_id, parameters)
|
||||
elif job_type == "data_cleanup":
|
||||
result = cleanup_old_data(cursor, job_id, parameters)
|
||||
elif job_type == "sync_external":
|
||||
result = sync_external_system(cursor, job_id, parameters)
|
||||
else:
|
||||
result = run_default_job(cursor, job_id, parameters)
|
||||
|
||||
# Mark job as completed
|
||||
completed_at = datetime.datetime.utcnow()
|
||||
cursor.execute(
|
||||
"""
|
||||
UPDATE jobs
|
||||
SET status = 'completed',
|
||||
result = %s,
|
||||
completed_at = %s,
|
||||
duration = EXTRACT(EPOCH FROM (%s - started_at))
|
||||
WHERE id = %s
|
||||
""",
|
||||
(result, completed_at, completed_at, job_id)
|
||||
)
|
||||
conn.commit()
|
||||
|
||||
return {
|
||||
"job_id": job_id,
|
||||
"status": "completed",
|
||||
"result": result,
|
||||
"duration_seconds": (completed_at - started_at).total_seconds()
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
# Mark job as failed
|
||||
if conn:
|
||||
try:
|
||||
cursor = conn.cursor()
|
||||
cursor.execute(
|
||||
"""
|
||||
UPDATE jobs
|
||||
SET status = 'failed',
|
||||
error = %s,
|
||||
completed_at = NOW()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(str(e), job_id)
|
||||
)
|
||||
conn.commit()
|
||||
except:
|
||||
pass
|
||||
|
||||
raise DatabaseError(f"Job {job_type} failed: {str(e)}")
|
||||
finally:
|
||||
if conn:
|
||||
conn.close()
|
||||
|
||||
|
||||
def generate_report(cursor, job_id: str, parameters: dict):
|
||||
"""
|
||||
Generate a report based on parameters.
|
||||
|
||||
Args:
|
||||
cursor: Database cursor
|
||||
job_id: Job tracking ID
|
||||
parameters: Report configuration (report_type, date, filters, etc.)
|
||||
|
||||
Returns:
|
||||
Dictionary with report metadata and summary
|
||||
"""
|
||||
report_type = parameters.get("report_type", "daily")
|
||||
report_date = parameters.get("date", datetime.datetime.utcnow().strftime("%Y-%m-%d"))
|
||||
|
||||
# Simulate report generation (could be complex aggregation queries)
|
||||
time.sleep(1) # Simulate work
|
||||
|
||||
# Example: Get statistics for the date
|
||||
cursor.execute(
|
||||
"""
|
||||
SELECT
|
||||
COUNT(*) as total_orders,
|
||||
SUM(total) as revenue,
|
||||
COUNT(DISTINCT user_id) as unique_customers
|
||||
FROM orders
|
||||
WHERE DATE(created_at) = %s
|
||||
""",
|
||||
(report_date,)
|
||||
)
|
||||
stats = db_row_to_dict(cursor, cursor.fetchone())
|
||||
|
||||
return {
|
||||
"report_type": report_type,
|
||||
"date": report_date,
|
||||
"statistics": stats,
|
||||
"generated_at": datetime.datetime.utcnow().isoformat()
|
||||
}
|
||||
|
||||
|
||||
def cleanup_old_data(cursor, job_id: str, parameters: dict):
|
||||
"""
|
||||
Clean up old records based on retention policy.
|
||||
|
||||
Args:
|
||||
cursor: Database cursor
|
||||
job_id: Job tracking ID
|
||||
parameters: Cleanup configuration (table, days_to_retain, etc.)
|
||||
|
||||
Returns:
|
||||
Dictionary with cleanup summary
|
||||
"""
|
||||
table = parameters.get("table", "jobs") # Table to clean
|
||||
days_to_retain = int(parameters.get("days_to_retain", 90))
|
||||
cutoff_date = datetime.datetime.utcnow() - datetime.timedelta(days=days_to_retain)
|
||||
|
||||
# Safety: prevent dropping tables
|
||||
if table not in ["jobs", "webhook_events", "logs", "sessions"]:
|
||||
raise ValueError(f"Cannot clean table: {table}")
|
||||
|
||||
# Count records to be deleted
|
||||
cursor.execute(
|
||||
f"SELECT COUNT(*) FROM {table} WHERE created_at < %s",
|
||||
(cutoff_date,)
|
||||
)
|
||||
count = cursor.fetchone()[0]
|
||||
|
||||
# Delete old records
|
||||
cursor.execute(
|
||||
f"DELETE FROM {table} WHERE created_at < %s",
|
||||
(cutoff_date,)
|
||||
)
|
||||
|
||||
return {
|
||||
"table": table,
|
||||
"cutoff_date": cutoff_date.isoformat(),
|
||||
"records_deleted": count
|
||||
}
|
||||
|
||||
|
||||
def sync_external_system(cursor, job_id: str, parameters: dict):
|
||||
"""
|
||||
Synchronize data with external system.
|
||||
|
||||
Args:
|
||||
cursor: Database cursor
|
||||
job_id: Job tracking ID
|
||||
parameters: Sync configuration (system, endpoint, filters, etc.)
|
||||
|
||||
Returns:
|
||||
Dictionary with sync summary
|
||||
"""
|
||||
system = parameters.get("system")
|
||||
endpoint = parameters.get("endpoint")
|
||||
|
||||
# This would typically make HTTP requests to external API
|
||||
# using requests library
|
||||
import requests
|
||||
|
||||
# Fetch last sync timestamp
|
||||
cursor.execute(
|
||||
"SELECT last_sync_at FROM sync_state WHERE system = %s",
|
||||
(system,)
|
||||
)
|
||||
row = cursor.fetchone()
|
||||
last_sync = row[0] if row else None
|
||||
|
||||
# Build query parameters
|
||||
params = {"since": last_sync.isoformat() if last_sync else ""}
|
||||
|
||||
# Make request to external API
|
||||
try:
|
||||
resp = requests.get(endpoint, params=params, timeout=30)
|
||||
resp.raise_for_status()
|
||||
data = resp.json()
|
||||
except Exception as e:
|
||||
raise DatabaseError(f"Failed to fetch from {system}: {str(e)}")
|
||||
|
||||
# Process and store data
|
||||
records_processed = 0
|
||||
for item in data.get("items", []):
|
||||
cursor.execute(
|
||||
"""
|
||||
INSERT INTO external_data (system, external_id, data, synced_at)
|
||||
VALUES (%s, %s, %s, NOW())
|
||||
ON CONFLICT (system, external_id) DO UPDATE SET
|
||||
data = EXCLUDED.data,
|
||||
synced_at = EXCLUDED.synced_at
|
||||
""",
|
||||
(system, item["id"], item)
|
||||
)
|
||||
records_processed += 1
|
||||
|
||||
# Update sync state
|
||||
cursor.execute(
|
||||
"""
|
||||
INSERT INTO sync_state (system, last_sync_at)
|
||||
VALUES (%s, NOW())
|
||||
ON CONFLICT (system) DO UPDATE SET
|
||||
last_sync_at = NOW()
|
||||
""",
|
||||
(system,)
|
||||
)
|
||||
|
||||
return {
|
||||
"system": system,
|
||||
"records_processed": records_processed,
|
||||
"sync_timestamp": datetime.datetime.utcnow().isoformat()
|
||||
}
|
||||
|
||||
|
||||
def run_default_job(cursor, job_id: str, parameters: dict):
|
||||
"""
|
||||
Default no-op job for testing.
|
||||
|
||||
Args:
|
||||
cursor: Database cursor
|
||||
job_id: Job tracking ID
|
||||
parameters: Job parameters
|
||||
|
||||
Returns:
|
||||
Simple acknowledgment
|
||||
"""
|
||||
time.sleep(0.5) # Simulate some work
|
||||
return {
|
||||
"message": "Default job executed",
|
||||
"parameters_received": parameters
|
||||
}
|
||||
296
fission-python/template/examples/example_webhook.py
Normal file
296
fission-python/template/examples/example_webhook.py
Normal file
@@ -0,0 +1,296 @@
|
||||
"""
|
||||
Example: Webhook receiver pattern.
|
||||
|
||||
This demonstrates:
|
||||
- Processing external service callbacks
|
||||
- Signature verification
|
||||
- Event type handling
|
||||
- Idempotency checks
|
||||
- Async processing patterns
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import hmac
|
||||
from flask import request
|
||||
from helpers import init_db_connection, get_secret
|
||||
from exceptions import ValidationError, DatabaseError
|
||||
|
||||
# For signed webhooks, you'll need a secret
|
||||
WEBHOOK_SECRET = get_secret("WEBHOOK_SECRET", "")
|
||||
|
||||
|
||||
def verify_signature(payload: bytes, signature: str) -> bool:
|
||||
"""
|
||||
Verify HMAC-SHA256 webhook signature.
|
||||
|
||||
Args:
|
||||
payload: Raw request body bytes
|
||||
signature: Signature header value (format: "sha256=<hex>")
|
||||
|
||||
Returns:
|
||||
True if signature is valid, False otherwise
|
||||
"""
|
||||
if not WEBHOOK_SECRET:
|
||||
return True # Skip verification if no secret configured (for dev)
|
||||
|
||||
expected = hmac.new(
|
||||
WEBHOOK_SECRET.encode(),
|
||||
payload,
|
||||
hashlib.sha256
|
||||
).hexdigest()
|
||||
|
||||
# Signature header format: "sha256=abcdef..."
|
||||
received = signature.split("=", 1)[1] if "=" in signature else signature
|
||||
return hmac.compare_digest(expected, received)
|
||||
|
||||
|
||||
def webhook_receiver(event, context):
|
||||
"""
|
||||
```fission
|
||||
{
|
||||
"name": "webhook-receiver",
|
||||
"http_triggers": {
|
||||
"webhook": {
|
||||
"url": "/webhooks/external-service",
|
||||
"methods": ["POST"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
Receive and process webhook from external service.
|
||||
|
||||
**Request:**
|
||||
- Raw JSON payload in body
|
||||
- Signature header: `X-Webhook-Signature: sha256=<hmac>`
|
||||
|
||||
**Response:**
|
||||
- 200: Webhook accepted for processing
|
||||
- 400: Invalid signature or payload
|
||||
- 500: Processing error
|
||||
|
||||
**Idempotency:**
|
||||
This function is idempotent - duplicate webhooks with same
|
||||
event ID will not be processed twice.
|
||||
"""
|
||||
# Get raw body for signature verification
|
||||
payload = request.get_data()
|
||||
signature = request.headers.get("X-Webhook-Signature", "")
|
||||
|
||||
# Verify signature
|
||||
if not verify_signature(payload, signature):
|
||||
raise ValidationError("Invalid webhook signature")
|
||||
|
||||
# Parse payload
|
||||
try:
|
||||
data = request.get_json()
|
||||
except Exception as e:
|
||||
raise ValidationError(f"Invalid JSON payload: {str(e)}")
|
||||
|
||||
# Validate required fields
|
||||
event_id = data.get("event_id") or data.get("id")
|
||||
event_type = data.get("event_type") or data.get("type")
|
||||
|
||||
if not event_id:
|
||||
raise ValidationError("Missing event_id in webhook payload")
|
||||
|
||||
if not event_type:
|
||||
raise ValidationError("Missing event_type in webhook payload")
|
||||
|
||||
# Idempotency check: have we already processed this event?
|
||||
conn = None
|
||||
try:
|
||||
conn = init_db_connection()
|
||||
cursor = conn.cursor()
|
||||
|
||||
# Check if event already processed
|
||||
cursor.execute(
|
||||
"SELECT id FROM webhook_events WHERE event_id = %s",
|
||||
(event_id,)
|
||||
)
|
||||
if cursor.fetchone():
|
||||
# Already processed - return success (idempotent)
|
||||
return {"status": "already_processed", "event_id": event_id}
|
||||
|
||||
# Record webhook event (for idempotency)
|
||||
cursor.execute(
|
||||
"""
|
||||
INSERT INTO webhook_events (event_id, event_type, payload, received_at)
|
||||
VALUES (%s, %s, %s, NOW())
|
||||
""",
|
||||
(event_id, event_type, payload.decode('utf-8'))
|
||||
)
|
||||
|
||||
# Process based on event type
|
||||
result = process_event(cursor, event_type, data)
|
||||
|
||||
conn.commit()
|
||||
return {"status": "processed", "event_id": event_id, "result": result}
|
||||
|
||||
except Exception as e:
|
||||
if conn:
|
||||
conn.rollback()
|
||||
raise DatabaseError(f"Failed to process webhook: {str(e)}")
|
||||
finally:
|
||||
if conn:
|
||||
conn.close()
|
||||
|
||||
|
||||
def process_event(cursor, event_type: str, data: dict):
|
||||
"""
|
||||
Route event to appropriate handler.
|
||||
|
||||
Args:
|
||||
cursor: Database cursor
|
||||
event_type: Type of event (e.g., "user.created", "order.updated")
|
||||
data: Event payload
|
||||
|
||||
Returns:
|
||||
Handler result
|
||||
"""
|
||||
handlers = {
|
||||
"user.created": handle_user_created,
|
||||
"user.updated": handle_user_updated,
|
||||
"user.deleted": handle_user_deleted,
|
||||
"order.created": handle_order_created,
|
||||
"order.paid": handle_order_paid,
|
||||
"order.shipped": handle_order_shipped,
|
||||
}
|
||||
|
||||
handler = handlers.get(event_type)
|
||||
if not handler:
|
||||
# Log unknown event type but don't fail
|
||||
logger = get_logger()
|
||||
logger.warning(f"Unhandled webhook event type: {event_type}")
|
||||
return {"skipped": True, "reason": "unknown_event_type"}
|
||||
|
||||
return handler(cursor, data)
|
||||
|
||||
|
||||
def handle_user_created(cursor, data: dict):
|
||||
"""Handle user creation event."""
|
||||
user_id = data.get("user_id") or data.get("id")
|
||||
email = data.get("email")
|
||||
name = data.get("name")
|
||||
|
||||
# Create user record
|
||||
cursor.execute(
|
||||
"""
|
||||
INSERT INTO users (id, email, name, created_at)
|
||||
VALUES (%s, %s, %s, NOW())
|
||||
ON CONFLICT (id) DO UPDATE SET
|
||||
email = EXCLUDED.email,
|
||||
name = EXCLUDED.name,
|
||||
updated_at = NOW()
|
||||
""",
|
||||
(user_id, email, name)
|
||||
)
|
||||
|
||||
# Send welcome email (async via message queue, etc.)
|
||||
# enqueue_welcome_email(user_id, email)
|
||||
|
||||
return {"action": "user_created", "user_id": user_id}
|
||||
|
||||
|
||||
def handle_user_updated(cursor, data: dict):
|
||||
"""Handle user update event."""
|
||||
user_id = data.get("user_id") or data.get("id")
|
||||
updates = data.get("updates", {})
|
||||
|
||||
# Build dynamic update
|
||||
set_clauses = []
|
||||
params = []
|
||||
for key, value in updates.items():
|
||||
set_clauses.append(f"{key} = %s")
|
||||
params.append(value)
|
||||
params.append(user_id)
|
||||
|
||||
cursor.execute(
|
||||
f"UPDATE users SET {', '.join(set_clauses)}, updated_at = NOW() WHERE id = %s",
|
||||
params
|
||||
)
|
||||
|
||||
return {"action": "user_updated", "user_id": user_id}
|
||||
|
||||
|
||||
def handle_user_deleted(cursor, data: dict):
|
||||
"""Handle user deletion event."""
|
||||
user_id = data.get("user_id") or data.get("id")
|
||||
|
||||
# Soft delete (mark as inactive)
|
||||
cursor.execute(
|
||||
"UPDATE users SET status = 'deleted', deleted_at = NOW() WHERE id = %s",
|
||||
(user_id,)
|
||||
)
|
||||
|
||||
return {"action": "user_deleted", "user_id": user_id}
|
||||
|
||||
|
||||
def handle_order_created(cursor, data: dict):
|
||||
"""Handle order creation event."""
|
||||
order_id = data.get("order_id") or data.get("id")
|
||||
user_id = data.get("user_id")
|
||||
total = data.get("total")
|
||||
|
||||
cursor.execute(
|
||||
"""
|
||||
INSERT INTO orders (id, user_id, total, status, created_at)
|
||||
VALUES (%s, %s, %s, 'pending', NOW())
|
||||
""",
|
||||
(order_id, user_id, total)
|
||||
)
|
||||
|
||||
return {"action": "order_created", "order_id": order_id}
|
||||
|
||||
|
||||
def handle_order_paid(cursor, data: dict):
|
||||
"""Handle order payment event."""
|
||||
order_id = data.get("order_id") or data.get("id")
|
||||
payment_id = data.get("payment_id")
|
||||
amount = data.get("amount")
|
||||
|
||||
cursor.execute(
|
||||
"""
|
||||
UPDATE orders
|
||||
SET status = 'paid',
|
||||
paid_amount = %s,
|
||||
payment_id = %s,
|
||||
paid_at = NOW()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(amount, payment_id, order_id)
|
||||
)
|
||||
|
||||
# Trigger fulfillment
|
||||
# enqueue_fulfillment(order_id)
|
||||
|
||||
return {"action": "order_paid", "order_id": order_id}
|
||||
|
||||
|
||||
def handle_order_shipped(cursor, data: dict):
|
||||
"""Handle order shipment event."""
|
||||
order_id = data.get("order_id") or data.get("id")
|
||||
tracking_number = data.get("tracking_number")
|
||||
carrier = data.get("carrier")
|
||||
|
||||
cursor.execute(
|
||||
"""
|
||||
UPDATE orders
|
||||
SET status = 'shipped',
|
||||
tracking_number = %s,
|
||||
carrier = %s,
|
||||
shipped_at = NOW()
|
||||
WHERE id = %s
|
||||
""",
|
||||
(tracking_number, carrier, order_id)
|
||||
)
|
||||
|
||||
# Send shipping notification
|
||||
# send_shipping_email(order_id)
|
||||
|
||||
return {"action": "order_shipped", "order_id": order_id}
|
||||
|
||||
|
||||
def get_logger():
|
||||
"""Get logger instance."""
|
||||
import logging
|
||||
return logging.getLogger(__name__)
|
||||
37
fission-python/template/migrates/schema.sql
Normal file
37
fission-python/template/migrates/schema.sql
Normal file
@@ -0,0 +1,37 @@
|
||||
-- Migration: 001_initial_schema.sql
|
||||
-- Description: Initial database schema with example items table
|
||||
-- To customize: Rename tables/columns and add your own migrations
|
||||
|
||||
-- Create items table (example)
|
||||
CREATE TABLE IF NOT EXISTS items (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
name VARCHAR(255) NOT NULL,
|
||||
description TEXT,
|
||||
status VARCHAR(50) NOT NULL DEFAULT 'active',
|
||||
metadata JSONB,
|
||||
created TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
modified TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
-- Create index on status for faster filtering
|
||||
CREATE INDEX IF NOT EXISTS idx_items_status ON items(status);
|
||||
|
||||
-- Create index on created for sorting
|
||||
CREATE INDEX IF NOT EXISTS idx_items_created ON items(created);
|
||||
|
||||
-- Optional: Trigger to auto-update modified timestamp
|
||||
CREATE OR REPLACE FUNCTION update_modified_column()
|
||||
RETURNS TRIGGER AS $$
|
||||
BEGIN
|
||||
NEW.modified = NOW();
|
||||
RETURN NEW;
|
||||
END;
|
||||
$$ language 'plpgsql';
|
||||
|
||||
CREATE OR REPLACE TRIGGER update_items_modtime
|
||||
BEFORE UPDATE ON items
|
||||
FOR EACH ROW
|
||||
EXECUTE FUNCTION update_modified_column();
|
||||
|
||||
-- Add table comment
|
||||
COMMENT ON TABLE items IS 'Example items table - replace with your own schema';
|
||||
8
fission-python/template/pytest.ini
Normal file
8
fission-python/template/pytest.ini
Normal file
@@ -0,0 +1,8 @@
|
||||
[pytest]
|
||||
testpaths = test
|
||||
python_files = test_*.py
|
||||
python_classes = Test*
|
||||
python_functions = test_*
|
||||
log_cli = true
|
||||
log_cli_level = INFO
|
||||
addopts = -v --tb=short
|
||||
42
fission-python/template/specs/README
Normal file
42
fission-python/template/specs/README
Normal file
@@ -0,0 +1,42 @@
|
||||
|
||||
Fission Specs
|
||||
=============
|
||||
|
||||
This is a set of specifications for a Fission app. This includes functions,
|
||||
environments, and triggers; we collectively call these things "resources".
|
||||
|
||||
How to use these specs
|
||||
----------------------
|
||||
|
||||
These specs are handled with the 'fission spec' command. See 'fission spec --help'.
|
||||
|
||||
'fission spec apply' will "apply" all resources specified in this directory to your
|
||||
cluster. That means it checks what resources exist on your cluster, what resources are
|
||||
specified in the specs directory, and reconciles the difference by creating, updating or
|
||||
deleting resources on the cluster.
|
||||
|
||||
'fission spec apply' will also package up your source code (or compiled binaries) and
|
||||
upload the archives to the cluster if needed. It uses 'ArchiveUploadSpec' resources in
|
||||
this directory to figure out which files to archive.
|
||||
|
||||
You can use 'fission spec apply --watch' to watch for file changes and continuously keep
|
||||
the cluster updated.
|
||||
|
||||
You can add YAMLs to this directory by writing them manually, but it's easier to generate
|
||||
them. Use 'fission function create --spec' to generate a function spec,
|
||||
'fission environment create --spec' to generate an environment spec, and so on.
|
||||
|
||||
You can edit any of the files in this directory, except 'fission-deployment-config.yaml',
|
||||
which contains a UID that you should never change. To apply your changes simply use
|
||||
'fission spec apply'.
|
||||
|
||||
fission-deployment-config.yaml
|
||||
------------------------------
|
||||
|
||||
fission-deployment-config.yaml contains a UID. This UID is what fission uses to correlate
|
||||
resources on the cluster to resources in this directory.
|
||||
|
||||
All resources created by 'fission spec apply' are annotated with this UID. Resources on
|
||||
the cluster that are _not_ annotated with this UID are never modified or deleted by
|
||||
fission.
|
||||
|
||||
0
fission-python/template/src/__init__.py
Normal file
0
fission-python/template/src/__init__.py
Normal file
15
fission-python/template/src/build.sh
Executable file
15
fission-python/template/src/build.sh
Executable file
@@ -0,0 +1,15 @@
|
||||
#!/bin/sh
|
||||
ID=$( grep "^ID=" /etc/os-release | awk -F= '{print $2}' )
|
||||
|
||||
if [ "${ID}" = "debian" ]
|
||||
then
|
||||
apt-get update && apt-get install -y gcc libpq-dev python3-dev
|
||||
else
|
||||
apk update && apk add gcc postgresql-dev python3-dev
|
||||
fi
|
||||
|
||||
if [ -f ${SRC_PKG}/requirements.txt ]
|
||||
then
|
||||
pip3 install -r ${SRC_PKG}/requirements.txt -t ${SRC_PKG}
|
||||
fi
|
||||
cp -r ${SRC_PKG} ${DEPLOY_PKG}
|
||||
103
fission-python/template/src/exceptions.py
Normal file
103
fission-python/template/src/exceptions.py
Normal file
@@ -0,0 +1,103 @@
|
||||
"""
|
||||
Custom exceptions for Fission Python functions.
|
||||
|
||||
All exceptions include:
|
||||
- error_code: Machine-readable error identifier
|
||||
- http_status: Appropriate HTTP status code
|
||||
- error_msg: Human-readable message
|
||||
- x_user: Optional user identifier from request headers
|
||||
- details: Optional additional error context
|
||||
"""
|
||||
|
||||
from typing import Optional
|
||||
|
||||
|
||||
class ServiceException(Exception):
|
||||
"""Base exception for service errors."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
error_code: str,
|
||||
http_status: int,
|
||||
error_msg: str,
|
||||
x_user: Optional[str] = None,
|
||||
details: Optional[dict] = None,
|
||||
):
|
||||
self.error_code = error_code
|
||||
self.http_status = http_status
|
||||
self.error_msg = error_msg
|
||||
self.x_user = x_user
|
||||
self.details = details
|
||||
super().__init__(self.error_msg)
|
||||
|
||||
|
||||
class ValidationError(ServiceException):
|
||||
"""Invalid request data."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
error_msg: str,
|
||||
x_user: Optional[str] = None,
|
||||
details: Optional[dict] = None,
|
||||
):
|
||||
super().__init__(
|
||||
error_code="VALIDATION_ERROR",
|
||||
http_status=400,
|
||||
error_msg=error_msg,
|
||||
x_user=x_user,
|
||||
details=details,
|
||||
)
|
||||
|
||||
|
||||
class NotFoundError(ServiceException):
|
||||
"""Resource not found."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
error_msg: str,
|
||||
x_user: Optional[str] = None,
|
||||
details: Optional[dict] = None,
|
||||
):
|
||||
super().__init__(
|
||||
error_code="NOT_FOUND",
|
||||
http_status=404,
|
||||
error_msg=error_msg,
|
||||
x_user=x_user,
|
||||
details=details,
|
||||
)
|
||||
|
||||
|
||||
class ConflictError(ServiceException):
|
||||
"""Resource conflict (e.g., duplicate name)."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
error_msg: str,
|
||||
x_user: Optional[str] = None,
|
||||
details: Optional[dict] = None,
|
||||
):
|
||||
super().__init__(
|
||||
error_code="CONFLICT",
|
||||
http_status=409,
|
||||
error_msg=error_msg,
|
||||
x_user=x_user,
|
||||
details=details,
|
||||
)
|
||||
|
||||
|
||||
class DatabaseError(ServiceException):
|
||||
"""Database operation failed."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
error_msg: str,
|
||||
x_user: Optional[str] = None,
|
||||
details: Optional[dict] = None,
|
||||
):
|
||||
super().__init__(
|
||||
error_code="DB_ERROR",
|
||||
http_status=500,
|
||||
error_msg=error_msg,
|
||||
x_user=x_user,
|
||||
details=details,
|
||||
)
|
||||
251
fission-python/template/src/helpers.py
Normal file
251
fission-python/template/src/helpers.py
Normal file
@@ -0,0 +1,251 @@
|
||||
"""
|
||||
Helper utilities for Fission Python functions.
|
||||
|
||||
Provides database connectivity, configuration/secrets access, and basic data utilities.
|
||||
"""
|
||||
|
||||
import datetime
|
||||
import logging
|
||||
import socket
|
||||
from typing import Any, Dict, Optional
|
||||
|
||||
import psycopg2
|
||||
from flask import current_app
|
||||
from psycopg2.extras import LoggingConnection
|
||||
from vault import decrypt_vault, is_valid_vault_format
|
||||
|
||||
# Configuration - these will be overridden by environment-specific values
|
||||
CORS_HEADERS = {
|
||||
"Content-Type": "application/json",
|
||||
}
|
||||
# These placeholders will be replaced by create-project.sh with actual project names
|
||||
SECRET_NAME = "${PROJECT_NAME}-env"
|
||||
CONFIG_NAME = "${PROJECT_NAME}-config"
|
||||
K8S_NAMESPACE = "default"
|
||||
CRYPTO_KEY = "" # Set this in your deployment environment
|
||||
|
||||
# Logging setup
|
||||
logging.basicConfig(level=logging.DEBUG)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def init_db_connection():
|
||||
"""
|
||||
Initialize PostgreSQL database connection.
|
||||
|
||||
Configuration is loaded from Kubernetes secrets or defaults.
|
||||
|
||||
Returns:
|
||||
psycopg2 connection object
|
||||
|
||||
Raises:
|
||||
Exception: If database connection fails or port check fails
|
||||
"""
|
||||
db_host = get_secret("PG_HOST", "127.0.0.1")
|
||||
db_port = int(get_secret("PG_PORT", 5432))
|
||||
|
||||
if not check_port_open(ip=db_host, port=db_port):
|
||||
raise Exception(f"Failed to connect to database at {db_host}:{db_port}")
|
||||
|
||||
options = get_secret("PG_DBSCHEMA")
|
||||
if options:
|
||||
options = f"-c search_path={options}" # if specific db schema
|
||||
conn = psycopg2.connect(
|
||||
database=get_secret("PG_DB", "postgres"),
|
||||
user=get_secret("PG_USER", "postgres"),
|
||||
password=get_secret("PG_PASS", "secret"),
|
||||
host=db_host,
|
||||
port=db_port,
|
||||
options=options,
|
||||
connection_factory=LoggingConnection,
|
||||
)
|
||||
conn.initialize(logger)
|
||||
return conn
|
||||
|
||||
|
||||
def db_row_to_dict(cursor, row) -> Dict[str, Any]:
|
||||
"""
|
||||
Convert a database row to a dictionary.
|
||||
|
||||
Args:
|
||||
cursor: Database cursor (with description attribute)
|
||||
row: Database row tuple
|
||||
|
||||
Returns:
|
||||
Dictionary mapping column names to values (datetime converted to isoformat)
|
||||
"""
|
||||
record = {}
|
||||
for i, column in enumerate(cursor.description):
|
||||
data = row[i]
|
||||
if isinstance(data, datetime.datetime):
|
||||
data = data.isoformat()
|
||||
record[column.name] = data
|
||||
return record
|
||||
|
||||
|
||||
def db_rows_to_array(cursor, rows) -> list:
|
||||
"""
|
||||
Convert multiple database rows to list of dictionaries.
|
||||
|
||||
Args:
|
||||
cursor: Database cursor
|
||||
rows: List of row tuples
|
||||
|
||||
Returns:
|
||||
List of dictionaries
|
||||
"""
|
||||
return [db_row_to_dict(cursor, row) for row in rows]
|
||||
|
||||
|
||||
def get_current_namespace() -> str:
|
||||
"""
|
||||
Get current Kubernetes namespace from service account secret.
|
||||
|
||||
Returns:
|
||||
Namespace string or default K8S_NAMESPACE if not available
|
||||
"""
|
||||
try:
|
||||
with open("/var/run/secrets/kubernetes.io/serviceaccount/namespace", "r") as f:
|
||||
namespace = f.read().strip()
|
||||
except Exception:
|
||||
namespace = K8S_NAMESPACE
|
||||
return str(namespace)
|
||||
|
||||
|
||||
def get_secret(key: str, default=None) -> str:
|
||||
"""
|
||||
Read a secret from Kubernetes secrets volume.
|
||||
|
||||
Args:
|
||||
key: Secret key name
|
||||
default: Default value if secret not found
|
||||
|
||||
Returns:
|
||||
Secret value (decrypted if vault-encrypted) or default
|
||||
"""
|
||||
namespace = get_current_namespace()
|
||||
path = f"/secrets/{namespace}/{SECRET_NAME}/{key}"
|
||||
try:
|
||||
with open(path, "r") as f:
|
||||
value = f.read().strip()
|
||||
if value:
|
||||
if is_valid_vault_format(value):
|
||||
return decrypt_vault(value, CRYPTO_KEY)
|
||||
else:
|
||||
return value
|
||||
except Exception as err:
|
||||
current_app.logger.error(f"Failed to read secret {path}: {err}")
|
||||
return default
|
||||
|
||||
|
||||
def get_config(key: str, default=None) -> str:
|
||||
"""
|
||||
Read configuration from Kubernetes config volume.
|
||||
|
||||
Args:
|
||||
key: Config key name
|
||||
default: Default value if config not found
|
||||
|
||||
Returns:
|
||||
Config value (decrypted if vault-encrypted) or default
|
||||
"""
|
||||
namespace = get_current_namespace()
|
||||
path = f"/configs/{namespace}/{CONFIG_NAME}/{key}"
|
||||
try:
|
||||
with open(path, "r") as f:
|
||||
value = f.read().strip()
|
||||
if value:
|
||||
if is_valid_vault_format(value):
|
||||
return decrypt_vault(value, CRYPTO_KEY)
|
||||
else:
|
||||
return value
|
||||
except Exception as err:
|
||||
current_app.logger.error(f"Failed to read config {path}: {err}")
|
||||
return default
|
||||
|
||||
|
||||
def str_to_bool(input: Optional[str]) -> Optional[bool]:
|
||||
"""
|
||||
Convert string representation to boolean.
|
||||
|
||||
Args:
|
||||
input: String value ('true', 'false', or None)
|
||||
|
||||
Returns:
|
||||
True, False, or None if not recognized
|
||||
"""
|
||||
input = input or ""
|
||||
BOOL_MAP = {"true": True, "false": False}
|
||||
return BOOL_MAP.get(input.strip().lower(), None)
|
||||
|
||||
|
||||
def check_port_open(ip: str, port: int, timeout: int = 30) -> bool:
|
||||
"""
|
||||
Check if a TCP port is open on the given IP address.
|
||||
|
||||
Args:
|
||||
ip: IP address or hostname
|
||||
port: Port number
|
||||
timeout: Connection timeout in seconds
|
||||
|
||||
Returns:
|
||||
True if port is open, False otherwise
|
||||
"""
|
||||
try:
|
||||
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
|
||||
s.settimeout(timeout)
|
||||
result = s.connect_ex((ip, port))
|
||||
return result == 0
|
||||
except Exception as err:
|
||||
logger.error(f"Check port open error: {err}")
|
||||
return False
|
||||
|
||||
|
||||
def get_user_from_headers() -> Optional[str]:
|
||||
"""
|
||||
Extract user identifier from request headers.
|
||||
|
||||
Returns:
|
||||
User ID from X-Fission-Params-UserId or similar header, or None if not present.
|
||||
"""
|
||||
from flask import request
|
||||
|
||||
# Try common header names
|
||||
user_id = (
|
||||
request.headers.get("X-Fission-Params-UserId")
|
||||
or request.headers.get("X-User-Id")
|
||||
or request.headers.get("User-Id")
|
||||
)
|
||||
return user_id
|
||||
|
||||
|
||||
def format_error_response(
|
||||
error_code: str,
|
||||
error_msg: str,
|
||||
http_status: int,
|
||||
x_user: Optional[str] = None,
|
||||
details: Optional[dict] = None,
|
||||
) -> dict:
|
||||
"""
|
||||
Create a standardized error response dictionary.
|
||||
|
||||
Args:
|
||||
error_code: Machine-readable error identifier
|
||||
error_msg: Human-readable error message
|
||||
http_status: HTTP status code
|
||||
x_user: Optional user identifier
|
||||
details: Optional additional error context
|
||||
|
||||
Returns:
|
||||
Dictionary formatted as ErrorResponse schema
|
||||
"""
|
||||
response = {
|
||||
"error_code": error_code,
|
||||
"http_status": http_status,
|
||||
"error_msg": error_msg,
|
||||
}
|
||||
if x_user:
|
||||
response["x_user"] = x_user
|
||||
if details:
|
||||
response["details"] = details
|
||||
return response
|
||||
153
fission-python/template/src/models.py
Normal file
153
fission-python/template/src/models.py
Normal file
@@ -0,0 +1,153 @@
|
||||
"""
|
||||
Pydantic models for request/response validation and data schemas.
|
||||
|
||||
This file provides example patterns that you can adapt for your service:
|
||||
- Enums for controlled vocabularies
|
||||
- Request models with validation
|
||||
- Response models with serialization config
|
||||
- Pagination and filtering patterns
|
||||
- Nested model relationships
|
||||
"""
|
||||
|
||||
import datetime
|
||||
import enum
|
||||
import typing
|
||||
|
||||
import pydantic
|
||||
from flask import request
|
||||
|
||||
|
||||
# ========== Example Enums ==========
|
||||
class Status(str, enum.Enum):
|
||||
"""Example status enum."""
|
||||
ACTIVE = "active"
|
||||
INACTIVE = "inactive"
|
||||
PENDING = "pending"
|
||||
|
||||
|
||||
class DataType(str, enum.Enum):
|
||||
"""Example data type enum."""
|
||||
ITEM = "ITEM"
|
||||
COLLECTION = "COLLECTION"
|
||||
|
||||
|
||||
# ========== Filter Models (for query parameters) ==========
|
||||
@typing.dataclass
|
||||
class ItemFilter:
|
||||
"""
|
||||
Example filter using dataclass.
|
||||
|
||||
Filters are often built from request query parameters.
|
||||
"""
|
||||
ids: typing.Optional[typing.List[str]] = None
|
||||
keyword: typing.Optional[str] = None
|
||||
status: typing.Optional[typing.List[str]] = None
|
||||
created_from: typing.Optional[datetime.datetime] = None
|
||||
created_to: typing.Optional[datetime.datetime] = None
|
||||
|
||||
@classmethod
|
||||
def from_request_queries(cls) -> "ItemFilter":
|
||||
"""Build filter from Flask request query parameters."""
|
||||
filter = ItemFilter()
|
||||
filter.ids = request.args.getlist("filter[ids]")
|
||||
filter.keyword = request.args.get("filter[keyword]")
|
||||
filter.status = request.args.getlist("filter[status]")
|
||||
filter.created_from = request.args.get("filter[created_from]")
|
||||
filter.created_to = request.args.get("filter[created_to]")
|
||||
return filter
|
||||
|
||||
|
||||
@typing.dataclass
|
||||
class Pagination:
|
||||
"""Pagination parameters."""
|
||||
page: int = 0
|
||||
size: int = 10
|
||||
asc: bool = True
|
||||
|
||||
@classmethod
|
||||
def from_request_queries(cls) -> "Pagination":
|
||||
"""Build pagination from request query parameters."""
|
||||
p = Pagination()
|
||||
p.page = int(request.args.get("page", 0))
|
||||
p.size = int(request.args.get("size", 10))
|
||||
p.asc = bool(request.args.get("asc", True))
|
||||
return p
|
||||
|
||||
|
||||
# ========== Request Models ==========
|
||||
class ItemCreateRequest(pydantic.BaseModel):
|
||||
"""Request model for creating a new item."""
|
||||
name: str = pydantic.Field(..., min_length=1, max_length=255, description="Item name")
|
||||
description: typing.Optional[str] = pydantic.Field(
|
||||
default=None, description="Item description"
|
||||
)
|
||||
status: Status = pydantic.Field(default=Status.ACTIVE, description="Item status")
|
||||
metadata: typing.Optional[dict] = pydantic.Field(
|
||||
default=None, description="Additional metadata as JSON"
|
||||
)
|
||||
|
||||
class Config:
|
||||
json_schema_extra = {
|
||||
"example": {
|
||||
"name": "Example Item",
|
||||
"description": "A sample item",
|
||||
"status": "active",
|
||||
"metadata": {"key": "value"},
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
class ItemUpdateRequest(pydantic.BaseModel):
|
||||
"""Request model for updating an existing item."""
|
||||
name: typing.Optional[str] = pydantic.Field(
|
||||
default=None, min_length=1, max_length=255, description="Item name"
|
||||
)
|
||||
description: typing.Optional[str] = pydantic.Field(
|
||||
default=None, description="Item description"
|
||||
)
|
||||
status: typing.Optional[Status] = pydantic.Field(
|
||||
default=None, description="Item status"
|
||||
)
|
||||
metadata: typing.Optional[dict] = pydantic.Field(
|
||||
default=None, description="Additional metadata"
|
||||
)
|
||||
|
||||
class Config:
|
||||
json_schema_extra = {
|
||||
"example": {
|
||||
"name": "Updated Item Name",
|
||||
"status": "inactive",
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
# ========== Response Models ==========
|
||||
class ItemResponse(pydantic.BaseModel):
|
||||
"""Standard item response."""
|
||||
id: str = pydantic.Field(..., description="Item unique identifier")
|
||||
name: str = pydantic.Field(..., description="Item name")
|
||||
description: typing.Optional[str] = pydantic.Field(default=None, description="Item description")
|
||||
status: Status = pydantic.Field(..., description="Item status")
|
||||
metadata: typing.Optional[dict] = pydantic.Field(default=None, description="Additional metadata")
|
||||
created: datetime.datetime = pydantic.Field(..., description="Creation timestamp")
|
||||
modified: datetime.datetime = pydantic.Field(..., description="Last modification timestamp")
|
||||
|
||||
class Config:
|
||||
from_attributes = True # Enable ORM mode for SQLAlchemy/psycopg2 compatibility
|
||||
|
||||
|
||||
class PaginatedResponse(pydantic.BaseModel):
|
||||
"""Paginated listing response."""
|
||||
data: typing.List[ItemResponse] = pydantic.Field(..., description="List of items")
|
||||
page: int = pydantic.Field(..., description="Current page number (0-indexed)")
|
||||
size: int = pydantic.Field(..., description="Page size")
|
||||
total: typing.Optional[int] = pydantic.Field(default=None, description="Total count of items")
|
||||
|
||||
|
||||
class ErrorResponse(pydantic.BaseModel):
|
||||
"""Standardized error response format (used by exceptions)."""
|
||||
error_code: str = pydantic.Field(..., description="Machine-readable error identifier")
|
||||
http_status: int = pydantic.Field(..., description="HTTP status code")
|
||||
error_msg: str = pydantic.Field(..., description="Human-readable error message")
|
||||
x_user: typing.Optional[str] = pydantic.Field(default=None, description="User identifier")
|
||||
details: typing.Optional[dict] = pydantic.Field(default=None, description="Additional error context")
|
||||
5
fission-python/template/src/requirements.txt
Normal file
5
fission-python/template/src/requirements.txt
Normal file
@@ -0,0 +1,5 @@
|
||||
Flask==2.1.1
|
||||
pydantic==2.11.7
|
||||
psycopg2-binary==2.9.10
|
||||
PyNaCl==1.6.0
|
||||
requests==2.32.2
|
||||
142
fission-python/template/src/vault.py
Normal file
142
fission-python/template/src/vault.py
Normal file
@@ -0,0 +1,142 @@
|
||||
import base64
|
||||
|
||||
import nacl.secret
|
||||
|
||||
|
||||
def string_to_hex(text: str) -> str:
|
||||
"""
|
||||
Convert a string to hexadecimal representation.
|
||||
|
||||
Args:
|
||||
text: Input string to convert
|
||||
|
||||
Returns:
|
||||
Hexadecimal string representation
|
||||
"""
|
||||
return text.encode("utf-8").hex()
|
||||
|
||||
|
||||
def hex_to_string(hex_string: str) -> str | None:
|
||||
"""
|
||||
Convert a hexadecimal string back to regular string.
|
||||
|
||||
Args:
|
||||
hex_string: Hexadecimal string to convert
|
||||
|
||||
Returns:
|
||||
Decoded string
|
||||
|
||||
Raises:
|
||||
ValueError: If hex_string is not valid hexadecimal
|
||||
"""
|
||||
return bytes.fromhex(hex_string).decode("utf-8")
|
||||
|
||||
|
||||
def decrypt_vault(vault: str, key: str) -> str:
|
||||
"""
|
||||
Decrypt a vault string encrypted with PyNaCl SecretBox.
|
||||
|
||||
Vault format: "vault:v1:<base64_encrypted_data>"
|
||||
|
||||
Args:
|
||||
vault: Vault-formatted string (e.g., "vault:v1:eW91cl9lbmNyeXB0ZWRfZGF0YQ==")
|
||||
key: Hex string representation of 32-byte encryption key
|
||||
|
||||
Returns:
|
||||
Decrypted string
|
||||
|
||||
Raises:
|
||||
ValueError: If vault format is invalid or key is not valid hex
|
||||
nacl.exceptions.CryptoError: If decryption fails (wrong key or corrupted data)
|
||||
"""
|
||||
# Parse vault format
|
||||
parts = vault.split(":", 2)
|
||||
if len(parts) != 3 or parts[0] != "vault" or parts[1] != "v1":
|
||||
raise ValueError("Invalid vault format. Expected 'vault:v1:<encrypted_data>'")
|
||||
|
||||
encrypted_string = parts[2]
|
||||
# Convert hex string key to bytes
|
||||
key_bytes = bytes.fromhex(key)
|
||||
|
||||
# Create a SecretBox instance with the key
|
||||
box = nacl.secret.SecretBox(key_bytes)
|
||||
|
||||
# Decode the base64-encoded encrypted string
|
||||
encrypted_data = base64.b64decode(encrypted_string)
|
||||
|
||||
# Decrypt the data
|
||||
decrypted_bytes = box.decrypt(encrypted_data)
|
||||
|
||||
# Convert bytes to string
|
||||
return decrypted_bytes.decode("utf-8")
|
||||
|
||||
|
||||
def encrypt_vault(plaintext: str, key: str) -> str:
|
||||
"""
|
||||
Encrypt a string and return it in vault format.
|
||||
|
||||
Args:
|
||||
plaintext: String to encrypt
|
||||
key: Hex string representation of 32-byte encryption key
|
||||
|
||||
Returns:
|
||||
Vault-formatted encrypted string (e.g., "vault:v1:<encrypted_data>")
|
||||
|
||||
Raises:
|
||||
ValueError: If key is not valid hex string
|
||||
"""
|
||||
# Convert hex string key to bytes
|
||||
key_bytes = bytes.fromhex(key)
|
||||
|
||||
# Create a SecretBox instance with the key
|
||||
box = nacl.secret.SecretBox(key_bytes)
|
||||
|
||||
# Encrypt the data
|
||||
encrypted = box.encrypt(plaintext.encode("utf-8"))
|
||||
|
||||
# Encode to base64
|
||||
encrypted_string = base64.b64encode(encrypted).decode("utf-8")
|
||||
|
||||
# Return in vault format
|
||||
return f"vault:v1:{encrypted_string}"
|
||||
|
||||
|
||||
def is_valid_vault_format(vault: str) -> bool:
|
||||
"""
|
||||
Check if a string is in valid vault format.
|
||||
|
||||
Vault format: "vault:v1:<base64_encrypted_data>"
|
||||
|
||||
Args:
|
||||
vault: String to validate
|
||||
|
||||
Returns:
|
||||
True if the string matches vault format structure, False otherwise
|
||||
|
||||
Note:
|
||||
This only checks the format structure, not whether the data can be decrypted
|
||||
"""
|
||||
# Parse vault format
|
||||
parts = vault.split(":", 2)
|
||||
|
||||
# Check basic structure: vault:v1:<data>
|
||||
if len(parts) != 3 or parts[0] != "vault" or parts[1] != "v1":
|
||||
return False
|
||||
|
||||
encrypted_data = parts[2]
|
||||
|
||||
# Check if data part is not empty
|
||||
if not encrypted_data:
|
||||
return False
|
||||
|
||||
# Check if data is valid base64
|
||||
try:
|
||||
decoded = base64.b64decode(encrypted_data)
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
# Check if decoded data has at least nonce bytes (24 bytes for NaCl)
|
||||
if len(decoded) < nacl.secret.SecretBox.NONCE_SIZE:
|
||||
return False
|
||||
|
||||
return True
|
||||
0
fission-python/template/test/__init__.py
Normal file
0
fission-python/template/test/__init__.py
Normal file
3
fission-python/template/test/requirements.txt
Normal file
3
fission-python/template/test/requirements.txt
Normal file
@@ -0,0 +1,3 @@
|
||||
pytest==8.2.0
|
||||
pytest-mock==3.14.0
|
||||
requests==2.32.3
|
||||
40
fission-python/template/test/test_example.py
Normal file
40
fission-python/template/test/test_example.py
Normal file
@@ -0,0 +1,40 @@
|
||||
"""
|
||||
Example test file for Fission Python functions.
|
||||
|
||||
This demonstrates basic testing patterns.
|
||||
"""
|
||||
|
||||
import pytest
|
||||
from unittest.mock import patch, MagicMock
|
||||
|
||||
|
||||
def test_placeholder():
|
||||
"""Placeholder test - replace with your actual tests."""
|
||||
assert True
|
||||
|
||||
|
||||
# Example: Testing a function with mocked dependencies
|
||||
@patch("helpers.init_db_connection")
|
||||
def test_example_with_mock(mock_db):
|
||||
"""Example test showing how to mock database."""
|
||||
from examples.example_crud import create_item
|
||||
|
||||
# Setup mock
|
||||
mock_conn = MagicMock()
|
||||
mock_cursor = MagicMock()
|
||||
mock_db.return_value = mock_conn
|
||||
mock_conn.cursor.return_value = mock_cursor
|
||||
mock_cursor.fetchone.return_value = ("id-123", "Test Item", "active")
|
||||
|
||||
# Mock Flask request
|
||||
with patch("examples.example_crud.request") as mock_request:
|
||||
mock_request.get_json.return_value = {"name": "Test Item", "status": "active"}
|
||||
mock_request.view_args = {}
|
||||
|
||||
# Call function
|
||||
result = create_item({}, {})
|
||||
|
||||
# Assert
|
||||
assert result["name"] == "Test Item"
|
||||
mock_cursor.execute.assert_called_once()
|
||||
mock_conn.commit.assert_called_once()
|
||||
290
fission-python/update-docstring.sh
Executable file
290
fission-python/update-docstring.sh
Executable file
@@ -0,0 +1,290 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Fission Docstring Updater
|
||||
# Parses and updates embedded fission configuration in Python function docstrings
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
usage() {
|
||||
echo "Usage: $0 <file-path> [function-name] [--set \"<json>\"] [--get] [--help]"
|
||||
echo ""
|
||||
echo "Arguments:"
|
||||
echo " file-path: Path to the Python file containing the function"
|
||||
echo " function-name: Optional specific function name to target (if not provided, processes all functions with fission configuration)"
|
||||
echo ""
|
||||
echo "Options:"
|
||||
echo " --set <json>: Set the fission configuration to the provided JSON string"
|
||||
echo " --get: Get/display the current fission configuration (default action)"
|
||||
echo " --help: Show this help message"
|
||||
echo ""
|
||||
echo "Examples:"
|
||||
echo " $0 ./src/my_function.py main --get"
|
||||
echo " $0 ./src/my_function.py main --set '{\"name\": \"updated-function\"}'"
|
||||
echo " $0 ./src/functions.py --get"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Check dependencies
|
||||
if ! command -v python3 &> /dev/null; then
|
||||
echo "Error: python3 is required but not found"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Parse arguments
|
||||
if [[ $# -lt 1 ]]; then
|
||||
usage
|
||||
fi
|
||||
|
||||
FILE_PATH="$1"
|
||||
shift
|
||||
|
||||
FUNCTION_NAME=""
|
||||
ACTION="get"
|
||||
SET_VALUE=""
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case $1 in
|
||||
--set)
|
||||
SET_VALUE="$2"
|
||||
ACTION="set"
|
||||
shift 2
|
||||
;;
|
||||
--get)
|
||||
ACTION="get"
|
||||
shift
|
||||
;;
|
||||
--help)
|
||||
usage
|
||||
;;
|
||||
*)
|
||||
if [[ -z "$FUNCTION_NAME" ]]; then
|
||||
FUNCTION_NAME="$1"
|
||||
else
|
||||
echo "Error: Unexpected argument '$1'"
|
||||
usage
|
||||
fi
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# Validate file exists
|
||||
if [[ ! -f "$FILE_PATH" ]]; then
|
||||
echo "Error: File '$FILE_PATH' does not exist"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Validate JSON if setting
|
||||
if [[ "$ACTION" == "set" && -z "$SET_VALUE" ]]; then
|
||||
echo "Error: --set requires a JSON value"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "$ACTION" == "set" ]]; then
|
||||
# Validate JSON format
|
||||
if ! echo "$SET_VALUE" | python3 -m json.tool >/dev/null 2>&1; then
|
||||
echo "Error: Invalid JSON provided for --set"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
# Python script to handle the docstring parsing and updating
|
||||
PYTHON_SCRIPT=$(cat << 'EOF'
|
||||
import re
|
||||
import sys
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
def extract_functions_with_fission(content):
|
||||
"""Extract all functions that have fission configuration in their docstrings."""
|
||||
# Pattern to match Python functions with docstrings containing fission configuration
|
||||
# This looks for def function_name(): followed by a docstring that contains ```fission
|
||||
pattern = r'(\s*def\s+(\w+)\s*\([^)]*\):\s*(?:\n\s*)?\"\"\"[\s\S]*?```fission[\s\S]*?```[\s\S]*?\"\"\"[\s\S]*?)(?=\n\s*def|\n\s*class|\Z)'
|
||||
|
||||
matches = re.finditer(pattern, content, re.MULTILINE)
|
||||
functions = []
|
||||
|
||||
for match in matches:
|
||||
full_match = match.group(1)
|
||||
# Extract function name from the match
|
||||
func_name_match = re.search(r'def\s+(\w+)\s*\(', full_match)
|
||||
if func_name_match:
|
||||
func_name = func_name_match.group(1)
|
||||
functions.append({
|
||||
'name': func_name,
|
||||
'full_text': full_match,
|
||||
'start_pos': match.start(),
|
||||
'end_pos': match.end()
|
||||
})
|
||||
|
||||
return functions
|
||||
|
||||
def extract_fission_config(docstring):
|
||||
"""Extract fission configuration from a docstring."""
|
||||
# Look for ```fission ... ``` blocks
|
||||
pattern = r'```fission\s*([\s\S]*?)\s*```'
|
||||
match = re.search(pattern, docstring)
|
||||
if match:
|
||||
config_text = match.group(1).strip()
|
||||
try:
|
||||
return json.loads(config_text)
|
||||
except json.JSONDecodeError as e:
|
||||
return None
|
||||
return None
|
||||
|
||||
def replace_fission_config_in_docstring(docstring, new_config):
|
||||
"""Replace fission configuration in a docstring with new config."""
|
||||
# Format the new config as JSON with indentation
|
||||
formatted_config = json.dumps(new_config, indent=4)
|
||||
# Replace the ```fission ... ``` block
|
||||
pattern = r'(```fission\s*)[\s\S]*?(\s*```)'
|
||||
replacement = r'\1' + formatted_config + r'\2'
|
||||
return re.sub(pattern, replacement, docstring, flags=re.DOTALL)
|
||||
|
||||
def process_file(file_path, target_function=None, action='get', set_value=None):
|
||||
"""Process the Python file to get or set fission configuration."""
|
||||
try:
|
||||
with open(file_path, 'r') as f:
|
||||
content = f.read()
|
||||
except IOError as e:
|
||||
print(f"Error: Cannot read file '{file_path}': {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
functions = extract_functions_with_fission(content)
|
||||
|
||||
if not functions:
|
||||
print("No functions with fission configuration found in file.", file=sys.stderr)
|
||||
if action == 'get':
|
||||
sys.exit(0)
|
||||
else:
|
||||
sys.exit(1)
|
||||
|
||||
# Filter by function name if specified
|
||||
if target_function:
|
||||
functions = [f for f in functions if f['name'] == target_function]
|
||||
if not functions:
|
||||
print(f"Error: Function '{target_function}' with fission configuration not found.", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
if action == 'get':
|
||||
# Display current configuration for each function
|
||||
for func in functions:
|
||||
# Extract docstring from the function text
|
||||
docstring_match = re.search(r'\"\"\"[\s\S]*?\"\"\"', func['full_text'])
|
||||
if docstring_match:
|
||||
docstring = docstring_match.group(0)
|
||||
config = extract_fission_config(docstring)
|
||||
if config is not None:
|
||||
if len(functions) == 1:
|
||||
print(json.dumps(config, indent=2))
|
||||
else:
|
||||
print(f"Function '{func['name']}':")
|
||||
print(json.dumps(config, indent=2))
|
||||
print()
|
||||
else:
|
||||
if len(functions) == 1:
|
||||
print("No fission configuration found in function docstring.", file=sys.stderr)
|
||||
else:
|
||||
print(f"Function '{func['name']}': No fission configuration found in docstring.", file=sys.stderr)
|
||||
else:
|
||||
if len(functions) == 1:
|
||||
print("Could not extract docstring from function.", file=sys.stderr)
|
||||
else:
|
||||
print(f"Function '{func['name']}': Could not extract docstring.", file=sys.stderr)
|
||||
|
||||
elif action == 'set':
|
||||
if set_value is None:
|
||||
print("Error: No value provided for --set", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
try:
|
||||
new_config = json.loads(set_value)
|
||||
except json.JSONDecodeError as e:
|
||||
print(f"Error: Invalid JSON provided for --set: {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
# Update each function
|
||||
updated_content = content
|
||||
offset = 0 # Track position changes due to replacements
|
||||
|
||||
for func in functions:
|
||||
# Extract docstring from the function text
|
||||
docstring_match = re.search(r'\"\"\"[\s\S]*?\"\"\"', func['full_text'])
|
||||
if docstring_match:
|
||||
docstring = docstring_match.group(0)
|
||||
# Check if fission configuration exists
|
||||
if extract_fission_config(docstring) is not None:
|
||||
# Replace fission configuration in docstring
|
||||
new_docstring = replace_fission_config_in_docstring(docstring, new_config)
|
||||
# Replace the docstring in the function text
|
||||
new_func_text = func['full_text'].replace(docstring, new_docstring, 1)
|
||||
# Replace in the overall content (adjusting for previous changes)
|
||||
start_pos = func['start_pos'] + offset
|
||||
end_pos = func['end_pos'] + offset
|
||||
# Update content with the change
|
||||
before = updated_content[:start_pos]
|
||||
after = updated_content[end_pos:]
|
||||
updated_content = before + new_func_text + after
|
||||
# Update offset for next replacements
|
||||
offset += len(new_func_text) - len(func['full_text'])
|
||||
else:
|
||||
print(f"Warning: No fission configuration found in function '{func['name']}' to update.", file=sys.stderr)
|
||||
else:
|
||||
print(f"Warning: Could not extract docstring from function '{func['name']}'.", file=sys.stderr)
|
||||
|
||||
# Write back to file
|
||||
try:
|
||||
with open(file_path, 'w') as f:
|
||||
f.write(updated_content)
|
||||
if len(functions) == 1:
|
||||
print(f"Updated fission configuration in function '{functions[0]['name']}'.")
|
||||
else:
|
||||
print(f"Updated fission configuration in {len(functions)} function(s).")
|
||||
except IOError as e:
|
||||
print(f"Error: Cannot write to file '{file_path}': {e}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
if __name__ == '__main__':
|
||||
if len(sys.argv) < 2:
|
||||
print("Usage: fission-docstring-updater <file-path> [function-name] [--set \"<json>\"] [--get]", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
file_path = sys.argv[1]
|
||||
target_function = sys.argv[2] if len(sys.argv) > 2 and not sys.argv[2].startswith('--') else None
|
||||
|
||||
# Parse arguments
|
||||
action = 'get'
|
||||
set_value = None
|
||||
|
||||
i = 3 if target_function else 2
|
||||
while i < len(sys.argv):
|
||||
if sys.argv[i] == '--set' and i + 1 < len(sys.argv):
|
||||
action = 'set'
|
||||
set_value = sys.argv[i + 1]
|
||||
i += 2
|
||||
elif sys.argv[i] == '--get':
|
||||
action = 'get'
|
||||
i += 1
|
||||
else:
|
||||
print(f"Error: Unknown argument '{sys.argv[i]}'", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
process_file(file_path, target_function, action, set_value)
|
||||
EOF
|
||||
)
|
||||
|
||||
# Build arguments for Python script
|
||||
PYTHON_ARGS=("$FILE_PATH")
|
||||
if [[ -n "$FUNCTION_NAME" ]]; then
|
||||
PYTHON_ARGS+=("$FUNCTION_NAME")
|
||||
fi
|
||||
|
||||
if [[ "$ACTION" == "set" ]]; then
|
||||
PYTHON_ARGS+=("--set" "$SET_VALUE")
|
||||
else
|
||||
PYTHON_ARGS+=("--get")
|
||||
fi
|
||||
|
||||
# Execute the Python script
|
||||
python3 -c "$PYTHON_SCRIPT" "${PYTHON_ARGS[@]}"
|
||||
Reference in New Issue
Block a user