Semver + PEP 440 specifiers + lock files + dependency groups (PEP 735)

Каждая зависимость имеет version. Каждый dep declaration имеет specifier (constraint) — какие versions acceptable. Resolver выбирает конкретный version для каждого pkg удовлетворяющий все constraints transitively. Lock file фиксирует выбор для reproducibility.

В этом уроке:

1. Semver concept — MAJOR.MINOR.PATCH semantics + pre-release.
2. PEP 440 version specifiers — ~=, >=, <, ==, !=, ranges.
3. Lock files — poetry.lock / uv.lock / requirements.txt frozen comparison.
4. PEP 735 [dependency-groups] — modern dev-deps grouping (replaces [project.optional-dependencies] for dev tools).
5. Resolver conflict resolution — transitive dependency conflicts.
6. Recipe — pyproject.toml dependencies block end-to-end.
7. Cross-course bridges — Spark CI/CD + Airflow Python deps mgmt.

Semver concept (MAJOR.MINOR.PATCH)

Semantic Versioning (semver, semver.org) — convention для interpreting version numbers:

1.2.3
│ │ └── PATCH: bug fixes (backward-compat); 1.2.3 → 1.2.4
│ └──── MINOR: new features (backward-compat); 1.2.3 → 1.3.0
└────── MAJOR: breaking changes (NOT backward-compat); 1.2.3 → 2.0.0

Mental model: semver — это promise автора library о backward compatibility. Когда вы pin pkg>=1.2,<2.0, вы рассчитываете что любой 1.x.y остаётся compatible. MAJOR bump — autor явно заявляет breaking change.

Pre-release versions — 1.0.0-rc1, 1.0.0-beta, 1.0.0a3, 1.0.0.dev1 — NOT для production; pip по умолчанию исключает их (pip install pkg пропустит pre-releases без --pre flag).

Critical caveat: semver — это convention, не enforcement. Authors могут случайно ввести breaking change в MINOR bump — поэтому lock files (см. ниже) необходимы для production.

PEP 440 version specifiers

PEP 440 — Python-specific version scheme + specifier grammar. Расширяет semver — добавляет нюансы (~= compatible release, epoch versions, post-releases).

Most common patterns в pyproject.toml:

[project]
dependencies = [
  "httpx>=0.25,<1.0",         # Range — semver MINOR window
  "pydantic~=2.5",             # Compatible release — MINOR window (aka >=2.5,<3.0)
  "click>=8.0,!=8.1.0",        # Range with exclude (8.1.0 buggy)
  "tomli; python_version<'3.11'",   # Conditional (env marker, see PEP 508)
]

Recommend:

• Libraries (publishable к PyPI) — loose specifiers (>=1.0); даём пользователям flexibility.
• Applications (deployable) — tight specifiers (==1.2.3) или lock file для exact reproducibility.

Cite peps.python.org/pep-0440 + peps.python.org/pep-0508 (env markers).

Lock files — три flavors

Lock file — precise dependency snapshot, пинит exact versions для reproducible installs across machines/CI. Три варианта:

poetry.lock — Poetry projects

# poetry.lock (truncated)
[[package]]
name = "httpx"
version = "0.25.2"
files = [
  { file = "httpx-0.25.2-py3-none-any.whl", hash = "sha256:abc..." },
  { file = "httpx-0.25.2.tar.gz", hash = "sha256:def..." },
]
dependencies = ["anyio>=3.0", "certifi", "httpcore>=0.18,<0.19"]

Что content: complete dependency tree (включая transitive); precise versions; SHA256 hashes для verification; per-package metadata. Generated poetry lock; consumed poetry install.

uv.lock — uv projects

# uv.lock (similar shape, slightly different schema)
version = 1
requires-python = ">=3.11"

[[package]]
name = "httpx"
version = "0.25.2"
source = { registry = "https://pypi.org/simple" }
sdist = { url = "...", hash = "sha256:def..." }
wheels = [
  { url = "...", hash = "sha256:abc..." },
]

Generated uv lock; consumed uv sync (installs ровно lock state). Astral promote uv.lock как cross-tool standard candidate.

requirements.txt frozen — pip + pip-tools

# requirements.txt — `pip freeze` output
anyio==4.0.0
certifi==2023.11.17
httpcore==0.18.0
httpx==0.25.2
pydantic==2.5.0

Generated pip freeze > requirements.txt. Flat list — нет dep tree information. Менее informative; pip install -r requirements.txt тупо ставит каждую line.

pip-compile (pip-tools) — generates rich frozen requirements.txt с comments показывающими transitive paths:

# Generated by pip-compile from pyproject.toml
httpx==0.25.2
    # via -r pyproject.toml
anyio==4.0.0
    # via httpx
certifi==2023.11.17
    # via httpx, requests

Decision rule:

• Poetry project → poetry.lock (built-in).
• Modern speed-critical → uv.lock (uv).
• Pip-only legacy → requirements.txt (pip-tools опционально).

ВСЕГДА commit lock files в git для applications. Lock file — main reproducibility mechanism.

Cite docs.astral.sh/uv/concepts/lockfile + python-poetry.org/docs/cli/#lock.

PEP 735 [dependency-groups] — modern dev-deps grouping

До PEP 735 (2024 — Python Packaging Authority) dev dependencies жили в [project.optional-dependencies]:

[project.optional-dependencies]
dev = ["pytest>=7", "ruff", "mypy"]
docs = ["sphinx", "myst-parser"]

Проблема — [project.optional-dependencies] публикуется в wheel metadata (pkg[dev] доступен после pip install). Это semantic confusion: dev tools — разработчик-only артефакт; не часть public API. PEP 735 решил это новым [dependency-groups] table:

[dependency-groups]
dev = ["pytest>=7", "ruff", "mypy"]
docs = ["sphinx", "myst-parser"]
test = ["pytest>=7", "pytest-cov", { include-group = "dev" }]

Differences:

1. Не публикуется в wheel metadata — pure project-internal artifact.
2. include-group — composition; test группа наследует dev группу.
3. Cross-tool support — pip ≥24.1, uv, Poetry 2.0+, PDM все понимают.

Использование:

# pip ≥24.1
pip install --group dev .

# uv
uv sync --group dev

# Poetry 2.0+
poetry install --with dev

Recommend: новые projects — [dependency-groups] для всех dev tools. [project.optional-dependencies] оставить только для публичных extras (e.g., pkg[s3] — optional cloud storage support).

Cite peps.python.org/pep-0735.

Resolver conflict resolution

Real-world issue — transitive dependency conflict:

Your project:
  ├── pkg-A>=1.0       requires        X<2.0
  └── pkg-B>=2.0       requires        X>=2.0
                                       ^^^
                       Не существует X удовлетворяющего обоим constraints → conflict.

pip backtracking (Dec 2020+):

ERROR: Cannot install pkg-A==1.5 and pkg-B==2.5 because these package versions have conflicting dependencies.

The conflict is caused by:
  pkg-A 1.5 depends on X<2.0
  pkg-B 2.5 depends on X>=2.0

To fix this, try the following:
  1. Loosen the range of package versions you've specified
  2. Remove package versions to allow pip to attempt to solve the dependency conflict

pip пробует более старые versions pkg-A или pkg-B пытаясь найти combination compatible. Backtracking может занять минуты на complex graphs.

uv pubgrub (см. урок 02 — formal SAT-style):

× No solution found when resolving dependencies:
  ╰─▶ Because pkg-A>=1.0 depends on X<2.0
      and pkg-B>=2.0 depends on X>=2.0,
      we conclude that pkg-A>=1.0 and pkg-B>=2.0 are incompatible.
      And because your project depends on pkg-A>=1.0 and pkg-B>=2.0,
      we conclude that your project's requirements are unsatisfiable.

Resolution strategies:

1. Loosen constraints — pkg-A>=0.9 (older version compatible с X<3.0?).
2. Skip one — заменить pkg-A на альтернативу (часто dev tools имеют альтернативы).
3. Vendor — clone library, fix constraints вручную (last resort).
4. Wait для upstream fix — open issue к pkg author.

Recipe — pyproject.toml dependencies block end-to-end

[build-system]
requires = ["hatchling>=1.18"]
build-backend = "hatchling.build"

[project]
name = "myapp"
version = "1.0.0"
requires-python = ">=3.11"
dependencies = [
  "httpx>=0.25,<1.0",
  "pydantic~=2.5",
  "click>=8.0",
  "tomli; python_version<'3.11'",   # PEP 508 env marker
]

[project.optional-dependencies]
s3 = ["boto3>=1.30"]                # Public extra — pkg[s3]
gcs = ["google-cloud-storage>=2.10"]

[dependency-groups]                  # PEP 735 — private to project
dev = ["pytest>=7", "ruff", "mypy"]
test = [
  "pytest>=7",
  "pytest-cov",
  { include-group = "dev" },
]
docs = ["sphinx", "myst-parser"]

[tool.hatch.build.targets.wheel]
packages = ["src/myapp"]

Workflow:

# Install runtime deps только
uv sync                             # или pip install .

# Install runtime + dev tools
uv sync --group dev                 # PEP 735

# Install runtime + s3 extra (deployment with cloud storage)
uv pip install '.[s3]'              # public extra

Cross-course → Spark CI/CD lifecycle + Airflow

Production CI requires deterministic dependency resolution — каждый CI run должен дать identical dependency tree. Без lock file pip install -r requirements.txt может пустить разные versions в разных runs (если specifier loose; PyPI получает new release между builds). Lock file solves это.

Cross-course: Spark course Phase 12 — production operations — CI dependency pinning practices (Maven <dependencyManagement>, Docker layer caching, JAR version coordination). Python parallel — pin через requirements.txt frozen или uv.lock или poetry.lock. Same idea: deterministic CI requires deterministic dependency resolution. Cross-course → Spark course Phase 12 — Airflow orchestration — Airflow Python deps mgmt patterns (PythonVirtualenvOperator + per-task venv + lock file in DAG repo).

Synthesis (cross-link M08 урок 07 CI bridge — carrying Phase 67 ASMT-08): M08 урок 07 представил CI dependency pinning conceptually (testing pytest in CI requires deterministic deps); M13 урок 04 даёт mechanism — PEP 440 specifiers + lock files. Both lessons same point: testing reflects production environment iff dependency resolution deterministic.

Что в следующем уроке

Урок 05 — distribution. sdist (.tar.gz source) vs wheel (.whl pre-built binary); manylinux ABI compatibility; wheel filename anatomy (pkg-1.0-cp312-cp312-manylinux_2_17_x86_64.whl); Pitfall 48 (wheel filename mismatch); python -m build (replaces legacy setup.py sdist bdist_wheel); twine upload к PyPI / TestPyPI conceptual; optional Run-on-Your-Machine #3 — python -m build host demo.