dbt table coverage metric - data-warehouse

Question

Given that I have a data warehouse with various tables being created from various sources many of them by dbt, I want to measure a concept like 'dbt table coverage' which I define as:
dtc = count(tables and views that exist) / count(non ephemeral models and sources)
This would be really useful in order to maintain a sense of quality/completeness, especially during transition projects. Is there a dbt command like:
dbt report table-coverage --schemas=['reporting','example']
>>> 96% coverage, 48/50 tables in the schemas provided are captured in dbt.
If not, how can we add this to the project?!
What alternate approaches could I take to solving the problem

Answer 1

To do this I would probably create a model (view) that queried the information_schema and made some assumptions about a 1-to-1 mapping of {sourceTableName} to stg_{sourceTableName} (Assuming this means coverage for you).
Additionally I would look into using the graph.sources.values() JINJA function in order to iterate through all of the documented sources in your project, and then compare that with each of the models in {target.schema}
https://docs.getdbt.com/reference/dbt-jinja-functions/graph#accessing-sources
If you're comparing the existence of source.schema.yml with the source.information_schema. I would alter the approach to consider mapping each of the items in the graph against the total count of items in the information_schema on the source database.

Answer 2

A couple thoughts here since this is pretty intriguing to my current case as well:
dbt doesn't give outputs of queries or return a result to the command line. (That I know of!) So that if 1 inherently unsupported feature at this time. i.e. dbt report or dbt query doesn't exist yet. If that's desired, I'm recommend build out a feature request here:
https://github.com/fishtown-analytics/dbt/issues
If you're ok with making a model in dbt and then just executing that via your client of choice, let's give that a shot. (I'm using postgres so convert accordingly)
WITH schema_map as
(select schemaname as schema,
tablename as name,
'Table' as Type,
CASE WHEN schemaname like '%dbt%' THEN 1
ELSE 0 END as dbt_created
from pg_tables
WHERE NOT schemaname = ANY('{information_schema,pg_catalog}')
UNION
select schemaname as schema,
viewname as name,
'View' as Type,
CASE WHEN schemaname like '%dbt%' THEN 1
ELSE 0 END as dbt_created
from pg_views
WHERE NOT schemaname = ANY('{information_schema,pg_catalog}') )
SELECT count(name) as total_tables_and_views,
sum(dbt_created) as dbt_created,
to_char((sum(dbt_created)::dec/count(name)::dec)*100,'999D99%') as dbt_coverage
FROM schema_map
Gives the result:
total_tables_and_views | dbt_created | dbt_coverage
391 |292 | 74.68%

Answer 3

Just to feed back to the community and thanks to Jordan and Gscott for the inspiration. The solution I executed for SQL Server/ Synapse was:
A Daily execution of Counts of models in INFORMATION_SCHEMA.TABLES and in the dbt graph as one table.
An incremental table built on 1 that selects schemas of interest and aggregates. In my case below I filter out staging and testing.
DbtModelCounts:
{% set models = [] -%}
{% if execute %}
{% for node in graph.nodes.values()
| selectattr("resource_type", "equalto", "model")
%}
{%- do models.append(node.name) -%}
{% endfor %}
{% endif %}
with tables AS
(
SELECT table_catalog [db], table_schema [schema_name], table_name [name], table_type [type]
FROM INFORMATION_SCHEMA.TABLES
),
dbt_tables AS
(
SELECT *
FROM tables
WHERE name in (
{%- for model in models %}
('{{ model}}')
{% if not loop.last %},
{% endif %}
{% endfor %}
)
)
SELECT
tables.db,
tables.schema_name,
tables.type,
COUNT(tables.name) ModelCount,
COUNT(dbt_tables.name) DbtModelCount
FROM tables
LEFT JOIN dbt_tables ON
tables.name=dbt_tables.name AND
tables.schema_name = dbt_tables.schema_name AND
tables.db = dbt_tables.db AND
tables.type = dbt_tables.type
GROUP BY
tables.db,
tables.schema_name,
tables.type
Dbt Coverage:
{{
config(
materialized='incremental',
unique_key='DateCreated'
)
}}
SELECT
CAST(GETDATE() AS DATE) AS DateCreated,
GETDATE() AS DateTimeCreatedUTC,
SUM(DbtModelCount) AS DbtModelCount,
SUM(ModelCount) AS TotalModels,
SUM(DbtModelCount)*100.0/SUM(ModelCount) as DbtCoveragePercentage
FROM {{ref('DbtModelCounts')}}
WHERE schema_name NOT LIKE 'testing%' AND schema_name NOT LIKE 'staging%'
To do, add logic for defined sources to also compute percentage of sources that map to my staging or raw schema tables .