Python Backends Configuration — JSON, Defaults, and Overrides

Lesson 05 Configuration Management

“Every program carries a silent brain — its configuration.”
— Teslaverse University, Book of Clarity

The command room dimmed.
A constellation of key–value pairs hovered above the console like a neural map.

Nova: “If code is behavior, config is personality.”
Teslanic: “And personality should be deliberate — not hard-coded.”

Lesson 05 is where your backend begins to remember who it is.

↓ Jump to notes

1 · What JSON Is (and Why Backends Love It)

JSON (JavaScript Object Notation) is a lightweight, structured text format — the common language of data interchange.

It is:

• Human-readable
• Language-agnostic
• Ideal for configs, APIs, logs, and backend state
• Git-friendly (easy to diff and version)
• Governed by strict but simple syntax (double quotes, no trailing commas)

Backends love JSON because it moves between systems without friction — it’s data diplomacy in its purest form.

2 · The Config Ladder: How Real Systems Decide Who They Are

Every serious backend defines its identity through layers — a chain of overrides, each closer to the point of execution:

1. Defaults — safe starting values
2. settings.json — local developer overrides
3. Environment variables — deployment-level overrides
4. CLI flags — last-mile, runtime overrides

Golden rule: later wins.

This hierarchy keeps your configuration flexible but predictable, giving you control without chaos.

3 · Reading JSON

from pathlib import Path
import json

def read_json(path: str | Path) -> dict:
    p = Path(path)
    with p.open("r", encoding="utf-8") as f:
        return json.load(f)

Readable, reliable, and Pythonic.

4 · Writing JSON

from pathlib import Path
import json

def write_json(path: str | Path, data: dict) -> None:
    p = Path(path)
    p.parent.mkdir(parents=True, exist_ok=True)
    with p.open("w", encoding="utf-8") as f:
        json.dump(data, f, indent=2, ensure_ascii=False)

Readable indentation will save your sanity down the road. Future you will thank you.

5 · Default Fallbacks and Merging Configs

A proper config-driven backend should:

• Run even if external files are missing
• Fall back to defaults
• Allow targeted overrides (not wholesale copies)
• Merge nested structures cleanly

Deep merging grants flexibility without fragility — it’s composable configuration.

6 · Real Backend Example — Load → Override → Run

Below is a production-shaped configuration loader that gracefully handles all layers:

from __future__ import annotations
from pathlib import Path
import json, os, argparse
from copy import deepcopy

# --------------------------------------
# Deep merge
# --------------------------------------

def deep_merge(base: dict, override: dict) -> dict:
    out = deepcopy(base)
    for k, v in override.items():
        if isinstance(v, dict) and isinstance(out.get(k), dict):
            out[k] = deep_merge(out[k], v)
        else:
            out[k] = v
    return out

# --------------------------------------
# Layers
# --------------------------------------

def load_defaults() -> dict:
    return {
        "app": {"name": "TaskCLI", "env": "dev"},
        "logging": {"level": "INFO"},
        "db": {"engine": "sqlite", "path": "taskcli.db"},
        "features": {"async_tasks": False, "streamlit": False},
    }

def load_json_file(path: Path) -> dict:
    if not path.exists():
        return {}
    with path.open("r", encoding="utf-8") as f:
        return json.load(f)

def infer_type(s: str):
    if s.lower() in {"true", "false"}:
        return s.lower() == "true"
    try:
        if "." in s:
            return float(s)
        return int(s)
    except ValueError:
        return s

def env_overrides(prefix: str = "APP_") -> dict:
    out = {}
    for key, value in os.environ.items():
        if not key.startswith(prefix):
            continue
        _, raw = key.split(prefix, 1)
        path = raw.split("__")   # e.g. logging__level
        cursor = out
        for part in path[:-1]:
            cursor = cursor.setdefault(part, {})
        cursor[path[-1]] = infer_type(value)
    return out

def parse_cli(argv=None) -> tuple[dict, Path]:
    parser = argparse.ArgumentParser()
    parser.add_argument("--env", choices=["dev","test","prod"])
    parser.add_argument("--log-level", choices=["DEBUG","INFO","WARNING","ERROR"])
    parser.add_argument("--db-path")
    parser.add_argument("--enable-async", action="store_true")
    parser.add_argument("--enable-streamlit", action="store_true")
    parser.add_argument("--settings", default="settings.json")
    
    args = parser.parse_args(argv)
    cli = {}

    if args.env:
        cli = deep_merge(cli, {"app": {"env": args.env}})
    if args.log_level:
        cli = deep_merge(cli, {"logging": {"level": args.log_level}})
    if args.db_path:
        cli = deep_merge(cli, {"db": {"path": args.db_path}})
    if args.enable_async:
        cli = deep_merge(cli, {"features": {"async_tasks": True}})
    if args.enable_streamlit:
        cli = deep_merge(cli, {"features": {"streamlit": True}})

    return cli, Path(args.settings)

# --------------------------------------
# Build Final Config
# --------------------------------------

def build_config(argv=None) -> dict:
    defaults = load_defaults()
    cli_overrides, settings_path = parse_cli(argv)
    settings = load_json_file(settings_path)
    env = env_overrides()

    cfg = deep_merge(defaults, settings)
    cfg = deep_merge(cfg, env)
    cfg = deep_merge(cfg, cli_overrides)
    return cfg

# --------------------------------------
# Application Entrypoint
# --------------------------------------

def run_app(cfg: dict) -> None:
    print(f"Starting {cfg['app']['name']} in {cfg['app']['env']} mode…")
    print(f"Log level: {cfg['logging']['level']}")
    print(f"DB: {cfg['db']['engine']} @ {cfg['db']['path']}")
    print(f"Features: async={cfg['features']['async_tasks']}, streamlit={cfg['features']['streamlit']}")

if __name__ == "__main__":
    config = build_config()
    run_app(config)

Clean, layered, and production-minded.

↓ Jump to notes

7 · Example Settings and Overrides

settings.json

{
  "logging": { "level": "DEBUG" },
  "db": { "path": "dev_tasks.db" }
}

Environment override

export APP_logging__level=WARNING
export APP_features__async_tasks=true

CLI override

python config_loader.py --env prod --db-path /srv/tasks.db --enable-streamlit

Notes

• Default fallbacks keep your app runnable with zero external files.
• Deep merge allows clean, minimal overrides.
• Environment variables are deployment-friendly and secure.
• TaskCLI can adopt this structure today for DB path, log level, and feature flags.
• Future Streamlit dashboards can read from the same config file.
• Async tasks should always be feature-flagged, not hard-wired.

↩ Back to code

-* This is a configuration management system for a Python command-line application called TaskCLI.
Let’s break it down piece by piece

Purpose

This script dynamically builds a runtime configuration for a CLI app by merging settings from multiple layers:

1. Built-in defaults
2. A JSON file (e.g. settings.json)
3. Environment variables
4. Command-line arguments

Each subsequent layer overrides the previous one.

1. The Deep Merge Function

def deep_merge(base: dict, override: dict) -> dict:
    out = deepcopy(base)
    for k, v in override.items():
        if isinstance(v, dict) and isinstance(out.get(k), dict):
            out[k] = deep_merge(out[k], v)
        else:
            out[k] = v
    return out

What it does:

It recursively merges dictionaries — useful when merging nested configurations (e.g., updating only one field in "logging" while keeping others intact).

Example:

deep_merge({"logging": {"level": "INFO"}}, {"logging": {"level": "DEBUG"}})
# → {'logging': {'level': 'DEBUG'}}

2. The Layers

Default Configuration

load_defaults()
Provides a base config:

{
  "app": {"name": "TaskCLI", "env": "dev"},
  "logging": {"level": "INFO"},
  "db": {"engine": "sqlite", "path": "taskcli.db"},
  "features": {"async_tasks": False, "streamlit": False},
}

JSON Settings File

load_json_file(path)
If settings.json exists, it loads it and returns a dictionary (otherwise {}).
So users can define app settings in a file like:
{
  "logging": {"level": "DEBUG"},
  "features": {"async_tasks": true}
}

Environment Variable Overrides

env_overrides(prefix="APP_")

Scans all environment variables that start with APP_ and maps them into nested dictionaries using double underscores __.

Example:

export APP_LOGGING__LEVEL=ERROR
export APP_FEATURES__STREAMLIT=true
Becomes:

{
  "logging": {"level": "ERROR"},
  "features": {"streamlit": True}
}

Command-Line Overrides

parse_cli(argv)
Supported flags:

--env [dev|test|prod]
--log-level [DEBUG|INFO|WARNING|ERROR]
--db-path <path>
--enable-async
--enable-streamlit
--settings <path to json file>

Each CLI flag merges into the configuration.
Example:

python app.py --env prod --log-level DEBUG --enable-async

Overrides:

{
  "app": {"env": "prod"},
  "logging": {"level": "DEBUG"},
  "features": {"async_tasks": True}
}

🧩 3. Final Configuration Assembly

def build_config(argv=None):
    defaults = load_defaults()
    cli_overrides, settings_path = parse_cli(argv)
    settings = load_json_file(settings_path)
    env = env_overrides()

    cfg = deep_merge(defaults, settings)
    cfg = deep_merge(cfg, env)
    cfg = deep_merge(cfg, cli_overrides)
    return cfg

It merges everything in the correct order of precedence:

🚀 4. The Entrypoint

if __name__ == "__main__":
    config = build_config()
    run_app(config)

run_app() just prints out the resolved config for now — a placeholder for real startup logic.

Example Run:

$ export APP_DB__PATH=/tmp/test.db
$ python main.py --env prod --enable-streamlit

Output:

Starting TaskCLI in prod mode…
Log level: INFO
DB: sqlite @ /tmp/test.db
Features: async=False, streamlit=True

In Summary

✅ Combines multiple configuration sources
✅ Supports nested overrides
✅ Converts env vars like APP_LOGGING__LEVEL → cfg["logging"]["level"]
✅ Clean separation of concerns
✅ Ideal for CLI or microservice apps with flexible config layers

In essence:

This is a lightweight, extensible configuration loader that supports hierarchical overrides from defaults, JSON files, environment variables, and CLI arguments.

↩ Back to code

Summary

Configuration is your program’s memory of who it should be today.
Keep it layered, structured, and override-friendly — and your backend will stay flexible without becoming fragile.