How to load a specific catalog dataset instance in kedro 0.17.0?

Question

We were using kedro version 0.15.8 and we were loading one specific item from the catalog this way:

from kedro.context import load_context
get_context().catalog.datasets.__dict__[key]

Now, we are changing to kedro 0.17.0 and trying to load the catalogs datasets the same way(using the framework context):

from kedro.framework.context import load_context
get_context().catalog.datasets.__dict__[key]

And now we get the error:

kedro.framework.context.context.KedroContextError: Expected an instance of ConfigLoader, got NoneType instead.

It's because the hook register_config_loader from the project it's not being used by the hook_manager that calls the function.

The project hooks are the defined the following way:

class ProjectHooks:

    @hook_impl

    def register_pipelines(self) -> Dict[str, Pipeline]:

        """Register the project's pipeline.

        Returns:

            A mapping from a pipeline name to a ``Pipeline`` object.

        """

        pm = pre_master.create_pipeline()

        return {

            "pre_master": pm,

            "__default__": pm

        }

    @hook_impl

    def register_config_loader(self, conf_paths: Iterable[str]) -> ConfigLoader:

        return ConfigLoader(conf_paths)

    @hook_impl

    def register_catalog(

        self,

        catalog: Optional[Dict[str, Dict[str, Any]]],

        credentials: Dict[str, Dict[str, Any]],

        load_versions: Dict[str, str],

        save_version: str,

        journal: Journal,

    ) -> DataCatalog:

        return DataCatalog.from_config(

            catalog, credentials, load_versions, save_version, journal

        )

project_hooks = ProjectHooks()

And the settings are called the following way: """Project settings."""

from price_based_trading.hooks import ProjectHooks


HOOKS = (ProjectHooks(),)

How can we configure that in a way that the hooks are used calling the method load_context(_working_dir).catalog.datasets ?

I posted the same question in the kedro community: https://discourse.kedro.community/t/how-to-load-a-specific-catalog-item-in-kedro-0-17-0/310

Hi, where are you running this code? Try using the session: https://kedro.readthedocs.io/en/latest/04_kedro_project_setup/03_session.html -- at the bottom of the page there is an example to get the current session and load a specific dataset. — Lim H., Feb 02 '21 at 22:39
Another hunch I have is that maybe you are trying to do this in an IPython notebook and maybe your IPython init script isn't compatible with Kedro 0.17 somehow. Try generating a new project with `kedro new` and overwrite the content of your `.ipython/profile_default/startup/00-kedro-init.py` with content from the same file in the new project. — Lim H., Feb 02 '21 at 22:53
Hello, you pointed into the right direction, I wasn't creating the KedroSession that's reason for the hooks not being used — Javi Hernandez, Feb 04 '21 at 15:31

Javi Hernandez · Accepted Answer · 2021-03-26T10:53:39.583

It was a silly mistake because I was not creating the Kedro session. To load an item of the catalog it can be done with the following code:

from kedro.framework.session import get_current_session
from kedro.framework.session import KedroSession

KedroSession.create("name_of_proyect") as session:
    key = "item_of_catalog"
    session = get_current_session()
    context = session.load_context()
    kedro_connector = context.catalog.datasets.__dict__[key] 
    // or kedro_connector = context.catalog._get_datasets(key)

score 0 · Answer 2 · answered Jul 26 '23 at 08:09

According to Kedro doc something like this would work:

from kedro.framework.session import KedroSession
from kedro.framework.startup import bootstrap_project
project_path=""
bootstrap_project(Path(project_path))
with KedroSession.create(project_path=project_path) as session:
    key = "item_of_catalog"
    context = session.load_context()
    kedro_connector = context.catalog.datasets.__dict__[key]

How to load a specific catalog dataset instance in kedro 0.17.0?

2 Answers2