tracemalloc — memory profiling + snapshots

cProfile отвечает «время где». Но в production вторая big-three метрика — память: «почему RSS моего процесса вырос с 200MB до 2GB за 3 часа?». Stdlib tracemalloc (Python 3.4+, PEP 454) instrum ents memory allocations + traceback origin для каждой allocation site — позволяет отвечать «откуда allocations».

В этом уроке:

1. tracemalloc.start() / take_snapshot() / compare_to() / statistics() — API tour.
2. Snapshot diff — leak detection workflow.
3. Overhead caveats (Pitfall 39) — почему tracemalloc.stop() обязателен.
4. sys.getsizeof recap — М02 урок 02 baseline (cross-link).
5. gc.get_objects() alternative — type-based introspection.
6. Bounded examples в Pyodide — small dict/list snapshots OK; production scale → Run-on-Your-Machine.
7. Run-on-Your-Machine #3 — production-scale memory profiling.

tracemalloc API tour

Базовые операции:

import tracemalloc

tracemalloc.start()                          # включить tracking (overhead!)

# ... код ...
data = [str(i) for i in range(100_000)]      # ~10 MB allocation

snapshot = tracemalloc.take_snapshot()       # capture текущего allocation state
top_stats = snapshot.statistics('lineno')    # aggregate by source line

for stat in top_stats[:5]:
    print(stat)
# 100000 strings allocated at script.py:5: 7.6 MiB

tracemalloc.stop()                           # выключить tracking — ВАЖНО (Pitfall 39)

Что хранится в snapshot:

• Per-allocation: pointer, size, traceback (sequence of frames до allocation site).
• Aggregable: statistics(key_type) — group by 'lineno' (source line), 'filename' (source file), 'traceback' (full traceback).

statistics('lineno') — most common — даёт «top-N lines that allocate most memory».

compare_to(other_snapshot, key_type='lineno') — diff двух snapshots, возвращает Statistic objects с size_diff + count_diff. Это и есть leak detection:

import tracemalloc
tracemalloc.start()

snapshot1 = tracemalloc.take_snapshot()       # baseline
run_workload()                                # potential leak
snapshot2 = tracemalloc.take_snapshot()       # after

diff = snapshot2.compare_to(snapshot1, 'lineno')
for stat in diff[:10]:
    print(stat)
# script.py:12: size=+15.2 MiB (+15.2 MiB), count=+50000 (+50000), avg=320 B
# ↑ growth между snapshots — likely leak source

tracemalloc.stop()

Cite docs.python.org/3/library/tracemalloc.html + PEP 454.

Snapshot diff — leak detection workflow

Production pattern:

import tracemalloc

def find_leak():
    tracemalloc.start(25)                     # 25 frames per traceback (default 1)

    snap_baseline = tracemalloc.take_snapshot()

    # Подозрительный workload — repeat N times
    for i in range(100):
        suspect_function()

    snap_after = tracemalloc.take_snapshot()

    # Top-10 growers
    diff = snap_after.compare_to(snap_baseline, 'lineno')
    for stat in diff[:10]:
        print(stat)

    tracemalloc.stop()

tracemalloc.start(N) — N controls frames per traceback. Default 1 (just lineno); 25 — full call stack. Trade-off: больше frames = более точное origin, но overhead grower per allocation.

Decision rule:

• start(1) — quick diagnosis, lineno only — minimal overhead.
• start(25) — full traceback — для глубоко-nested calls (e.g., в frameworks).

Что считается leak: count_diff > 0 после repeat workload который должен быть idempotent. Если 100 invocations добавляют +50,000 objects — каждая invocation leak’ит 500 objects.

Cross-course → Spark 01/07 memory-management — distributed equivalent: Spark UI Storage tab показывает RDD/DataFrame caching memory growth across stages — analogous к tracemalloc.compare_to.

Cross-course → DataFusion 02/07 memory-management — Rust’s memory tracking primitives (MemoryConsumer trait, MemoryReservation) — explicit tracking analogous к Python tracemalloc; Rust no-GC requires explicit reservation patterns.

Overhead caveats (Pitfall 39)

tracemalloc adds per-allocation traceback capture — не дешёво:

Production rule: always tracemalloc.stop() after measurement — не оставляйте включённым на production process.

Recommended pattern (Python 3.10+):

import tracemalloc

with tracemalloc.taking_snapshot() as snapshot:
    ...                                        # WAIT: this isn't actually how it works

Здесь нюанс: tracemalloc API НЕ имеет built-in context manager в stdlib (на момент написания). Production-pattern — manual try/finally:

import tracemalloc

tracemalloc.start()
try:
    snapshot1 = tracemalloc.take_snapshot()
    run_workload()
    snapshot2 = tracemalloc.take_snapshot()
    diff = snapshot2.compare_to(snapshot1, 'lineno')
finally:
    tracemalloc.stop()

Custom context manager (DIY) — короткий wrapper:

from contextlib import contextmanager
import tracemalloc

@contextmanager
def tracing(frames=1):
    tracemalloc.start(frames)
    try:
        yield
    finally:
        tracemalloc.stop()

with tracing():
    snap = tracemalloc.take_snapshot()
    # ...

Cross-link M06 урок 04 — context-manager protocol (__enter__/__exit__) гарантирует cleanup даже при exception. tracemalloc.start()/stop() — exact same invariant requirement.

sys.getsizeof recap (cross-link M02 урок 02)

Альтернативный coarse-grained measurement — sys.getsizeof(obj):

import sys

x = [1, 2, 3]
print(sys.getsizeof(x))                       # ~88 bytes (PyListObject header + 3 pointers)
print(sys.getsizeof([]))                       # ~56 bytes (empty list header)

Cross-link M02 урок 02 (PyListObject baseline): sys.getsizeof returns size of object’s own memory — для list это PyListObject struct (header + ob_item pointer + capacity int) без referenced elements. Recursive size = sum(sys.getsizeof(x) for x in lst) + sys.getsizeof(lst). М02 урок 02 показал precisely эту структуру — 56 bytes header + 8 bytes per pointer.

Production rule: sys.getsizeof для quick sanity checks; tracemalloc для leak hunting; gc.get_objects() для type-based introspection.

gc.get_objects() — type-based introspection

gc модуль предоставляет alternative view — list of all currently-alive Python objects:

import gc

# Count instances per type
from collections import Counter
counter = Counter(type(obj).__name__ for obj in gc.get_objects())
print(counter.most_common(10))
# [('dict', 12453), ('function', 3421), ('tuple', 2891), ...]

Use case — «сколько MyClass instances right now?»:

import gc

class Connection:
    pass

conns = [Connection() for _ in range(10)]
del conns

# Force GC
gc.collect()

# Count surviving Connection instances
count = sum(1 for obj in gc.get_objects() if isinstance(obj, Connection))
print(count)                                    # 0 — no leak; >0 — possibly retained somewhere

Trade-off vs tracemalloc: gc.get_objects() дает type-based snapshot (что сейчас в memory); tracemalloc дает allocation-site origin (где аллоцировано). Together — full picture: «leak’ит что + откуда».

Bounded examples в Pyodide OK

tracemalloc runs в Pyodide (Python implementation, портируемо), но production-scale snapshots (1GB+ heap) — too slow для browser. Bounded examples OK:

import tracemalloc

tracemalloc.start()
snap1 = tracemalloc.take_snapshot()

data = {i: str(i) for i in range(1_000)}      # small dict — bounded

snap2 = tracemalloc.take_snapshot()
diff = snap2.compare_to(snap1, 'lineno')
print(len(diff))                                # number of growth lines

tracemalloc.stop()

Это OK в browser (~10ms total). Production-scale (100k+ items, real services) → Run-on-Your-Machine.

Pragmatic-DEEP принцип: challenges в Phase 69 НЕ запускают heavy tracemalloc loops в browser (T-69-23 disposition — accept; bounded examples только в lesson MDX prose). Memory profiling — conceptual через challenges; real via Run-on-Your-Machine.

Run-on-Your-Machine #3 — production-scale memory profiling

python --version  # >=3.11

import tracemalloc

def workload():
    """Simulate worker that allocates 50_000 dicts in tight loop."""
    cache = []
    for i in range(50_000):
        cache.append({'id': i, 'data': 'x' * 100})
    return cache

def main():
    tracemalloc.start(5)                     # 5 frames per traceback

    snap_baseline = tracemalloc.take_snapshot()

    result = workload()

    snap_after = tracemalloc.take_snapshot()

    print('--- Top 5 memory consumers (lineno key) ---')
    for stat in snap_after.statistics('lineno')[:5]:
        print(stat)

    print('\n--- Top 5 growth between snapshots ---')
    diff = snap_after.compare_to(snap_baseline, 'lineno')
    for stat in diff[:5]:
        print(stat)

    tracemalloc.stop()
    return len(result)

if __name__ == '__main__':
    main()

python mem_demo.py

--- Top 5 memory consumers (lineno key) ---
mem_demo.py:7: size=8.6 MiB, count=50001, average=180 B
mem_demo.py:6: size=4.5 MiB, count=50000, average=95 B
...

--- Top 5 growth between snapshots ---
mem_demo.py:7: size=+8.6 MiB (+8.6 MiB), count=+50000 (+50000), avg=+180 B
mem_demo.py:6: size=+4.5 MiB (+4.5 MiB), count=+50000 (+50000), avg=+95 B
...

Что в следующем уроке

Урок 04 — dis — bytecode introspection. Generators vs list comprehensions на bytecode level (LIST_APPEND vs YIELD_VALUE) — cross-link M05 урок 02 PyGenObject. Pattern 4 challenge py-m12-04-code-1 (RECOMMENDED) — opcode counting через dis.get_instructions + Counter.

Pragmatic-DEEP принцип: tracemalloc showed где аллокации. dis показывает что компилируется в bytecode — иногда [x for x in lst] (LIST_APPEND) vs (x for x in lst) (YIELD_VALUE) — невидимая memory difference visible через bytecode.

Cite docs.python.org/3/library/tracemalloc.html + PEP 454 — tracemalloc + docs.python.org/3/library/sys.html#sys.getsizeof + docs.python.org/3/library/gc.html#gc.get_objects.