Config API: config.get / config.require / config.set

В materializations часто нужно читать model configuration: какой unique_key, какой strategy, какие custom configs. Для этого dbt предоставляет config namespace с get, require, set методами.

Это последний урок модуля 03 — после него вы будете уверенно писать custom materializations, понимая как читать model config и interacting с warehouse через statements.

Что такое config в materialization context

Когда model вызывает {{ config(materialized='table', unique_key='id') }}, dbt записывает эти значения в Manifest. На runtime, в materialization Jinja-коде, config object доступен для чтения этих значений.

{% materialization my_mat, default %}
  
  {%- set materialized = config.get('materialized') -%}
  {%- set unique_key = config.require('unique_key') -%}
  {%- set incremental_strategy = config.get('incremental_strategy', 'merge') -%}
  
  -- materialization body using these
  
{% endmaterialization %}

config.get — optional с default. config.require — required, fails если не set.

config.get(key, default=None)

Reads config value. Если не set — returns default.

{% set mat = config.get('materialized') %}           -- 'table' / 'view' / 'incremental' / custom
{% set on_schema_change = config.get('on_schema_change', 'ignore') %}  -- with default
{% set tags = config.get('tags', []) %}
{% set my_custom = config.get('my_custom_config', 'default_value') %}

Default:

• Если key set in model -> returns that value
• Если not set -> returns default
• Если no default specified -> returns None

Используется когда config optional и есть sensible default.

config.require(key)

Reads config value, fails если не set:

{% set unique_key = config.require('unique_key') %}

require без default. Если model не имеет unique_key в config, materialization падает с:

CompilationError: Required config 'unique_key' not set for model my_model

Используется для required configs в materialization. Например, incremental materialization без unique_key — semantically wrong, лучше fail loud.

config.set(key, value)

Sets config value programmatically в materialization. Reads back through subsequent config.get:

{% set strategy = config.get('incremental_strategy') %}
{% if not strategy %}
  {% if target.type == 'snowflake' %}
    {{ config.set('incremental_strategy', 'merge') }}
  {% else %}
    {{ config.set('incremental_strategy', 'delete+insert') }}
  {% endif %}
{% endif %}

Use cases:

• Adapter-specific defaults (default strategy depends on adapter).
• Computed configs (e.g., generate full_refresh based on date).
• Override invalid combos (with warning).

Caveat: config.set мутирует config только в текущей materialization invocation. Не записывается обратно в Manifest. Subsequent operations на той же модели в other macros не увидят изменение.

Built-in vs custom configs

dbt-core defines стандартные configs (materialized, schema, alias, tags, full_refresh, etc.). User может определять custom configs:

-- model.sql
{{ config(
    materialized='incremental',
    my_custom_setting='value',
    refresh_frequency='hourly'
) }}

-- custom materialization
{% set freq = config.get('refresh_frequency', 'daily') %}
{% if freq == 'hourly' %}
  -- handle hourly refresh
{% endif %}

Custom configs:

• Просто dict keys в ModelConfig.config
• Не validated dbt-core (никаких schema checks)
• Available через config.get(key) в materialization
• Доступны в model.config[key] в model body

Полный pattern: incremental materialization (упрощённо)

Реальный код из dbt-core для incremental materialization (упрощённый):

{% materialization incremental, default %}
  
  -- Read configs
  {%- set unique_key = config.get('unique_key') -%}
  {%- set incremental_strategy = config.get('incremental_strategy') or 'append' -%}
  {%- set on_schema_change = config.get('on_schema_change') or 'ignore' -%}
  {%- set full_refresh_mode = (flags.FULL_REFRESH) or (config.get('full_refresh') == True) -%}
  
  -- Validate
  {% if incremental_strategy == 'merge' and not unique_key %}
    {{ exceptions.raise_compiler_error("merge strategy requires unique_key") }}
  {% endif %}
  
  -- Target relation
  {%- set target_relation = this.incorporate(type='table') -%}
  {%- set existing_relation = load_cached_relation(this) -%}
  
  -- Full refresh detection
  {% set should_full_refresh = (
    full_refresh_mode 
    or existing_relation is none 
    or existing_relation.is_view
  ) %}
  
  {% if should_full_refresh %}
    -- DROP and CREATE
    {% call statement('main') %}
      {{ create_table_as(false, target_relation, sql) }}
    {% endcall %}
  {% else %}
    -- Incremental path
    {% set tmp_relation = make_temp_relation(target_relation) %}
    {% call statement('create_tmp') %}
      {{ create_table_as(true, tmp_relation, sql) }}
    {% endcall %}
    
    {% if incremental_strategy == 'merge' %}
      {% call statement('main') %}
        {{ get_merge_sql(target_relation, tmp_relation, unique_key, ...) }}
      {% endcall %}
    {% elif incremental_strategy == 'delete+insert' %}
      {% call statement('main') %}
        {{ get_delete_insert_merge_sql(target_relation, tmp_relation, unique_key, ...) }}
      {% endcall %}
    {% endif %}
    
    {% call statement('drop_tmp') %}
      DROP TABLE IF EXISTS {{ tmp_relation }}
    {% endcall %}
  {% endif %}
  
  {{ adapter.commit() }}
  {{ return({'relations': [target_relation]}) }}

{% endmaterialization %}

Здесь видно как config.get используется для разных aspects of materialization: strategy choice, unique_key validation, schema change handling, full_refresh detection.

Senior pattern: dispatching config

Combine config + dispatch для adapter-specific behavior:

{% materialization incremental, default %}
  
  {%- set strategy = adapter.dispatch('get_incremental_strategy')() -%}
  
  -- ... use strategy
  
{% endmaterialization %}

{% macro get_incremental_strategy() %}
  {{ return(adapter.dispatch('get_incremental_strategy')()) }}
{% endmacro %}

{% macro default__get_incremental_strategy() %}
  {{ return(config.get('incremental_strategy') or 'append') }}
{% endmacro %}

{% macro snowflake__get_incremental_strategy() %}
  {{ return(config.get('incremental_strategy') or 'merge') }}
{% endmacro %}

{% macro postgres__get_incremental_strategy() %}
  {{ return(config.get('incremental_strategy') or 'delete+insert') }}
{% endmacro %}

Каждый adapter имеет sensible default. Если user override через config.get('incremental_strategy') — это берётся priority.

Default configs из dbt_project.yml

dbt_project.yml:

models:
  my_project:
    +materialized: view
    +tags: ['layer:source']
    
    marts:
      +materialized: table
      +tags: ['layer:mart']
      finance:
        +materialized: incremental
        +unique_key: 'id'

При rendering модели my_project.marts.finance.fct_orders:

• config.get('materialized') -> 'incremental' (most specific)
• config.get('tags') -> ['layer:source', 'layer:mart'] (merged from all levels)
• config.get('unique_key') -> 'id'

Precedence (highest to lowest):

1. In-model {{ config(...) }} call
2. In _models.yml config section
3. In dbt_project.yml (most specific path wins)
4. Default in config.get(key, default)

Merging vs overriding

Some configs merge, some override:

Это уровень config. config.get returns final value после merge.

Senior gotcha: config visibility outside materialization

В model body вы можете читать config через config.get:

{# в model.sql #}
{{ config(
    materialized='incremental',
    unique_key='id'
) }}

{% set strategy = config.get('incremental_strategy', 'merge') %}

SELECT * FROM ...

Это работает в model body. Но в macros вызываемых из model, config.get тоже работает (inherits model’s config).

В non-model contexts (hooks, run-operation, generate_*_name) — config.get undefined или partial. Используйте model.config[key] если есть model object.

config.persist_relation_docs / persist_column_docs

Special config keys checked для persisting docs to warehouse:

{% if config.persist_relation_docs() %}
  COMMENT ON TABLE {{ this }} IS '{{ model.description | replace("'", "''") }}'
{% endif %}

{% if config.persist_column_docs() %}
  {% for column_name, column_info in model.columns.items() %}
    COMMENT ON COLUMN {{ this }}.{{ column_name }} IS '{{ column_info.description | replace("'", "''") }}'
  {% endfor %}
{% endif %}

config.persist_relation_docs() — method, не property. Returns True если model has persist_docs: relation: true. Adapter-specific (Snowflake/Postgres support COMMENT, DuckDB partial).

flags object

В materializations доступен flags — CLI flags:

{% if flags.FULL_REFRESH %}
  -- --full-refresh passed
{% endif %}

{% if flags.WHICH == 'build' %}
  -- dbt build (not just run)
{% endif %}

flags — это Flags namespace. Главные:

• flags.FULL_REFRESH — --full-refresh passed
• flags.WHICH — command name
• flags.DEBUG — debug mode
• flags.FAIL_FAST — fail-fast mode
• flags.STORE_FAILURES — store_failures override

Попробуй сам

1. Create model with custom config:

   -- models/test_config.sql
   {{ config(
       materialized='table',
       my_custom='hello',
       partition_by='date'
   ) }}
   SELECT 1 as x, CURRENT_DATE as date

2. Inspect through macro:

   -- macros/inspect_config.sql
   `{% macro inspect_config() %}`
     `{{ log("materialized: " ~ config.get('materialized'), info=True) }}`
     `{{ log("my_custom: " ~ config.get('my_custom', 'NOT SET'), info=True) }}`
     `{{ log("tags: " ~ (config.get('tags', []) | join(', ')), info=True) }}`
   `{% endmacro %}`

3. Call from model:

   -- models/test_inspect.sql  
   `{{ config(materialized='table', my_custom='hello', tags=['a', 'b']) }}`
   `{{ inspect_config() }}`
   SELECT 1 as x

   dbt run --select test_inspect 2>&1 | grep -E "materialized|my_custom|tags"

4. Read source:

   # In cloned dbt-core
   grep -rn "def get(self" core/dbt/context/

   Find ContextConfig class. Read methods.

5. Look at production materializations:

   • include/global_project/macros/materializations/models/incremental/incremental.sql
   • include/global_project/macros/materializations/models/table/table.sql

   Search for config.get, config.require — see real usage patterns.