Created: December 9, 2025 Purpose: Smoke test suite deployment and operations guide Audience: DevOps, Release Engineers, and Development Team Status: ✅ Active - Smoke test suite production deployment
The smoke test suite is a critical-path quality gate that runs on every push to identify failures in the core system quickly (2-3 seconds execution time).
Primary Purpose: Fail fast - prevent wasting time on longer test suites when basic functionality is broken.
Smoke tests run automatically on:
.github/workflows/test.ymlsmoke-tests (first job in sequence)Run smoke tests locally during development:
# Quick smoke check (2-3 seconds)
python -m pytest -m smoke -q
# Verbose output
python -m pytest -m smoke -v
# With specific test matching
python -m pytest -m smoke -k "test_name_pattern" -v
# Single file
python -m pytest tests/unit/test_file.py -m smoke -v
Smoke tests run automatically in GitHub Actions:
# This is what runs in CI/CD (in .github/workflows/test.yml)
python -m pytest -m smoke -q --tb=short
# 1. Check which tests failed
python -m pytest -m smoke -v
# 2. Run failed test in isolation
python -m pytest tests/path/to/test_file.py::test_name -xvs
# 3. Check recent commits
git log --oneline -10
# 4. Check test timing (might be >500ms now)
python -m pytest -m smoke --durations=10 -q
| Cause | Action |
|---|---|
| Recent code change broke test | Fix the code, push again |
| Test execution time >500ms | Remove @pytest.mark.smoke, add to full suite |
| Infrastructure issue (DB down) | Fix infrastructure, retry |
| Flaky test (intermittent fail) | Investigate flakiness, may need fix |
Check if smoke tests are slowing down:
# Measure execution time
time python -m pytest -m smoke -q
# Get detailed breakdown
python -m pytest -m smoke --durations=10 -q
If execution time creeps above 5 seconds:
Only add tests that meet these criteria:
# 1. Add decorator to test function
@pytest.mark.smoke
def test_my_feature():
# test code
# 2. Verify it runs
python -m pytest tests/path/to/test_file.py::test_my_feature -m smoke -v
# 3. Commit
git add tests/path/to/test_file.py
git commit -m "feat: Add test_my_feature to smoke suite"
If a test is slowing down or becoming unreliable:
# 1. Remove decorator
# @pytest.mark.smoke ← Delete or comment out
def test_my_feature():
# test code
# 2. Verify removal
python -m pytest -m smoke -q | grep test_my_feature # Should not appear
# 3. Commit
git commit -m "chore: Remove test_my_feature from smoke suite"
In .github/workflows/test.yml:
smoke-tests:
name: Smoke Tests
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Set up Python 3.11
uses: actions/setup-python@v4
with:
python-version: "3.11"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Run smoke tests (critical path)
run: |
echo "🔥 Running smoke test suite (616 tests, <5s target)"
python -m pytest -m smoke -q --tb=short
echo "✅ Smoke tests passed"
In GitHub repository settings:
main branchAutomatic via GitHub Actions:
To notify Slack on failures:
Recommended: Use native GitHub notifications first, add Slack if needed.
Automatic GitHub email (default):
Symptom: Tests not collected Solution:
pytest -m smoke --collect-only | grep "smoke"
# Should show 616 collected tests
Symptom: Smoke execution >5 seconds Solution:
python -m pytest -m smoke --durations=10 -q
# Find slow tests, check if they still <500ms
# If >500ms, remove @pytest.mark.smoke decorator
Symptom: Test passes locally, fails in CI or vice versa Solution:
for i in {1..10}; do pytest test_name.py; doneSymptom: pytest -m smoke works locally but fails in GitHub Actions
Possible Causes:
Solution:
# Check Python version in CI
python --version # Should be 3.11
# Run tests sequentially (like CI does)
python -m pytest -m smoke -n0 -v
# Check for environment variables
env | grep -i piper
These are the established performance baselines for smoke tests:
| Metric | Value | Status |
|---|---|---|
| Total execution time | 2-3 seconds | ✅ 40-60% under 5s target |
| Average per-test | 5-10ms | ✅ Well under 500ms threshold |
| Slowest test | ~400-500ms | ✅ At threshold but acceptable |
| Pass rate | 100% | ✅ Zero flakes |
| Test count | 616 tests | ✅ 87.5% of unit tests |
Decision: Deploy smoke test suite as first CI/CD quality gate on every push
Rationale:
Approval: Automatic for all branches (not just main)
Notification: GitHub native checks (red X if fail, prevents merge to main)
Last Updated: December 9, 2025 Maintained By: DevOps / Development Team Version: 1.0 - Initial deployment