Issue when parsing dbt models

[lewisosborne]

Hey folks!

I've just run 'dbt2looker' in my local dbt repo folder, and I receive the following error:

❯ dbt2looker
12:11:54 ERROR  Cannot parse model with id: "model.smallpdf.brz_exchange_rates" - is the model file empty?
Failed

The model file itself (pictured below) is not empty, therefore I am not sure what the issue with parsing this model dbt2looker appears to have. It is not materialised as a table or view, it is utilised by dbt as ephemeral - is that of importance when parsing files in the project? I've also tried running dbt2looker on a limited subset of dbt models via a tag, the same error appears. Any help is greatly appreciated!

Other details:

• on dbt version dbt 1.0.0
• using dbt-redshift adapter [email protected]
• let me know if anything else is of importance!

[boludo00]

Hey I am getting the same issue. Maybe it has to do with the model being ephemeral, since that's what I am experiencing as well.

[lewisosborne]

Perhaps @owlas or @jaypeedevlin may know if this is intentional? Sorry to tag directly, but the issue is getting stale and I would like to be able to generate lkml from my dbt project. Thanks in advance 🙏

[jaypeedevlin]

Ephemeral models don’t actually exist, they get imported into downstream models as CTEs.

Because of this they can’t be queried directly by dbt2looker (nor looker itself, for that matter). My guess is that dbt2looker needs some special handling of ephemeral models that doesn’t currently exist.

[lewisosborne]

Hey @jaypeedevlin - thanks for the swift response.

I'm not looking to have the ephemeral model generated to lkml. I will only make lkml files out of the estuary "final" dbt models. But it appears that the very existence of an ephemeral model in my dbt project is acting as a blocker for dbt2looker to parse the models - even if that ephemeral model is not tagged with the tag used in the "dbt2looker run".

I will have a try now by tagging only those models that I want to expose as lkml.

[lewisosborne]

Nope, unfortunately that did not have positive affect.

I tagged a single dbt model that I would like to generate lkml for (in model config, tags=["generate_lkml"])
I then ran dbt docs generate
I then ran dbt2looker --tag generate_lkml
Error:
12:15:56 ERROR Cannot parse model with id: "model.smallpdf.brz_exchange_rates" - is the model file empty?

Sigh 😓

[dduran28]

@lewisosborne I was running into the same issue, I bypassed it by deleting that ephemeral tag at the top and then re-running dbt docs generate and then the subsequent dbt2looker command. That gets me passed the initial error highlighted here.

I am running into a different error afterwards where I am seeing not supported for conversion from redshift to looker for all of my data types but maybe we can tackle that one separately.

Let me know if my initial suggestion gets you past the issue with the ephemeral model.

[boludo00]

So yesterday I was modding the code and was able to get past the - is the model file empty? error.

Since the parser uses the manifest.json file, a node has its materialization in the config object:

"config": {
    "enabled": true,
    "alias": null,
    "schema": "staging",
    "database": null,
    "tags": [],
    "meta": {},
    "materialized": "ephemeral",
    "persist_docs": {},
    "quoting": {},
    "column_types": {},
    "full_refresh": null,
    "on_schema_change": "ignore",
    "bind": false,
    "post-hook": [],
    "pre-hook": []
},

If the DbtNode changes to:

class DbtNode(BaseModel):
    unique_id: str
    resource_type: str
    config: Dict[str, Any]

then the parse_models() method could be modified to:

def parse_models(raw_manifest: dict, tag=None) -> List[models.DbtModel]:
    manifest = models.DbtManifest(**raw_manifest)
    all_models: List[models.DbtModel] = [
        node
        for node in manifest.nodes.values()
        if node.resource_type == 'model' and node.config['materialized'] != 'ephemeral'
    ]

    # Empty model files have many missing parameters
    for model in all_models:
        if not hasattr(model, 'name'):
            logging.error('Cannot parse model with id: "%s" - is the model file empty?', model.unique_id)
            raise SystemExit('Failed')

    if tag is None:
        return all_models
    return [model for model in all_models if tags_match(tag, model)]

which should ensure the list of models only contains DbtModel instances. This might be overkill as a solution and maybe even not the intended effect but it solve those issues for me.
Also make sure to dbt build any models that are resulting in that error before you run dbt docs generate.

[fredmny]

Hey, is there a definitive solution for this? Would the solution proposed by @boludo00 be viable to be included in a release?

[mariana-s-fernandes]

I also suggest the tag filtering to be done before the loop on all_models

[mariana-s-fernandes]

Suggestion for parse_models(), based on @boludo00 's code:

def parse_models(raw_manifest: dict, tag=None) -> List[models.DbtModel]:
    manifest = models.DbtManifest(**raw_manifest)
    all_models: List[models.DbtModel] = [
        node
        for node in manifest.nodes.values()
        if node.resource_type == 'model' and node.config['materialized'] != 'ephemeral'
    ]

    if tag is not None:
        all_models = [model for model in all_models if tags_match(tag, model)]

    # Empty model files have many missing parameters
    for model in all_models:
        if not hasattr(model, 'name'):
            logging.error('Cannot parse model with id: "%s" - is the model file empty?', model.unique_id)
            raise SystemExit('Failed')

    return all_models

[andrespapa]

Is it a general rule currently that dbt2Looker does not work when you have ephemeral models?

I also get the error message Cannot parse model with id for every ephemeral model I have in my repo, even if not involved in the models I want to LookMLize

[PaddyAlton]

Hi folks, I installed the tool locally after running into this issue with ephemeral models. I was able to get the solution developed by commenters above to work for me. As you can see, I have opened a PR to incorporate this into dbt2looker.

I don't think there's a test suite to run this against, is there? I can tell you that dbt2looker worked as expected for me after I made this change.