Create a New Python based KNIME Extension

This is a quickstart guide that will walk you through the essential steps of writing and running your first Python node extension containing a single node. We will use tutorial_extension as the basis. The steps of the tutorial requiring modification of the Python code in extension.py have corresponding comments in the file, for convenience.

For an extensive overview of the full API, please refer to the Defining a KNIME Node in Python: Full API section, as well as our Read the Docs page.

As you develop your Python extension, you are able to run and debug it locally by setting the knime.python.extension.config system property in your KNIME Analytics Platform’s knime.ini to point to the config.yml, or in the launch configuration’s VM arguments in Eclipse. See the Registering Python extensions during development and Customizing the Python executable sections at the end of this guide for more information.

In order to share your Python extension with others, please refer to the Bundling your Python Extension Nodes section.

Besides tables, a node can also consume or produce other port objects and it is possible to define custom port objects for your extension. You can do so by extending knext.PortObject and knext.PortObjectSpec with your custom implementation. In order to use these objects in your node, you need to define a custom port type via the knext.port_type function that takes your PortObject and PortObjectSpec classes as well as a human-readable name for your port type and an optional id. Here is an example:

Let’s start with the PortObjectSpec:

The serialize and deserialize methods are used by the framework to store and load the spec.

Note: The deserialize method must be a classmethod.

The spec_data property is just an example for custom code and you can add arbitrary methods to your spec class as you see fit.

Next we implement the PortObject:

The PortObject class must have a serialize and deserialize method that are called by the framework to persist and restore the object. Again note that deserialize has to be a classmethod.

The predict property is again just an example for custom code that your port object class may contain.

Finally, we create a custom port type to be used as input or output of a node:

The knext.port_type method ties the PortObject and PortObjectSpec together and provides a human-readable name to refer to the custom port type.

It is also possible to specify a custom ID for the port type via the id argument. Note that the ID must be unique. If you don’t provide a custom ID, then the framework generates one of the format your_extension_id.your_module_name.your_port_object_class_name. For example if your extension has the ID org.company.extension and you implement a MyPortObject in the module extension, then the generated ID is org.company.extension.extension.MyPortObject.

Note that there are also connection port objects that can hold non-serializable objects. You can find information about that in the API documentation for knime.extension.ConnectionPortObject.

Check out the next section to learn how to declare your custom port type as input or output of your node.

The input and output ports of a node can be configured by decorating the node class with @knext.input_table, @knext.input_port, and respectively @knext.output_table and @knext.output_port. An image output port can be added with the @knext.output_image decorator. Additionally, a node producing a view should be decorated with the @knext.output_view decorator.

These port decorators have the following properties:

The @knext.input_table and @knext.output_table decorators configure the port to consume and respectively produce a KNIME table. The @knext.output_image decorator configures the port to produce a PNG or SVG image.

If you want to receive or send other data, e.g. a trained machine learning model, use @knext.input_port and @knext.output_port. These decorators have an additional argument, port_type, used to identify the type of port objects going along this port connection. Only ports with equal port_type can be connected. See the previous section to learn how to specify your own port type.

The port configuration determines the expected signature of the configure and execute methods:

Examples how to use knext.Schema and knext.Column` (see the API):

Here is an example with two input ports and one output port. See the previous session for the definitions of MyPortObject, MyPortObjectSpec and my_model_port_type.

Example with two image output ports.

Alternatively, you can populate the input_ports and output_ports attributes of your node class (on class or instance level) for more fine grained control.

Each node in your Python node extension is assigned a category via the category parameter of the @knext.node decorator, which dictates where the node will be located in the node repository of KNIME Analytics Platform. Without an explicit category, the node will be placed in the root of the node repository, thus you should always specify a category for each node.

In order to define a custom category for the nodes of your extension, you can use the knext.category helper function. If autocompletion is enabled in your IDE, you should be able to see the list of the expected parameters of the function, together with their detailed description.

If you are a community developer, you should use the Community Nodes category as the parent category of your Python node extensions. This is done by specifying the path="/community" parameter of the knext.category function:

Note that it is possible to further split your custom category into subcategories. This is useful if, for instance, nodes of your extension can be grouped based on their functionality. By first defining a parent category for the node extension, you can then use it as the path parameter when defining the subcategories:

In order to add parameterization to your node’s functionality, we can define and customize its configuration dialog. The user-configurable parameters that will be displayed there, and whose values can be accessed inside the execute method of the node via self.param_name, are set up using the following parameter classes available in knext:

All of the above have arguments label and description, which are displayed in the node description in KNIME Analytics Platform, as well as in the configuration dialog itself. Additionally, all parameter classes have an optional argument since_version, which can be used to specify the version of the extension that the parameter was introduced in. Please refer to the Versioning your extension section below for a more detailed overview.

Parameters are defined in the form of class attributes inside the node class definition (similar to Python descriptors):

While each parameter type listed above has default type validation, they also support custom validation via a property-like decorator notation. By wrapping a function that receives a tentative parameter value, and raises an exception should some condition be violated, with the @some_param.validator decorator, you are able to add an additional layer of validation to the parameter some_param. This should be done below the definition of the parameter for which you are adding a validator, and above the configure and execute methods:

You can use the @knext.output_view(name="", description="") decorator to specify that a node returns a view. In that case, the execute method should return a tuple of port outputs and the view (of type knime.api.views.NodeView).

You can access the flow variables available to the node in both the configure and execute methods, via the config_context.flow_variables and exec_context.flow_variables attributes respectively. The flow variables are provided as a dictionary of flow_variable_name : flow_variable_value mappings, and support the following types:

By mutating the flow_variables dictionary, you can access, modify, and delete existing flow variables, as well as create new ones to be propagated to downstream nodes.

As you continue to develop your extension after the initial release, you might extend the functionality of your nodes by adding or removing certain parameters. With the versioning capabilities of Python-based node extensions for KNIME Analytics Platform, you can ensure backward compatibility for your users.

As seen in the Python Node Extension Setup section, the knime.yml configuration file contains a version field. This allows you to assign a version to each iteration of your extension. How closely you want to follow the semantic versioning scheme is completely up to you, but we do require adherence to the following formatting-related rule: versions must be composed of three non-negative numeric parts separated by dots (e.g. 1.0.0, 0.2.1, etc.).

When adding a new parameter to a node, you should associate it with the corresponding version of your extension. This is done using the since_version argument that is now available for all parameter types via the appropriate constructor (e.g. knext.IntParameter), as well as parameter groups via the @knext.parameter_group decorator. If not specified, the since_version argument of a parameter or parameter group defaults to 0.0.0, which indicates that the parameter was available from the first iteration of the extension.

A common use-case of extension versioning is to facilitate backward compatibility when opening workflows that were created/saved with an older version of the extension installed on the machine. What KNIME Analytics Platform will try to achieve by default in this case, is to combine the values of the previously configured node settings that are still available in the current version of the extension with the newly added node settings, if any. The latter are then automatically set to their default values, and the node remains configured.

Here is a minimal functional example of a Python-based extension containing a single node with a single parameter. Since the parameter is available from the initial release of the extension, we can forgo setting the since_version argument:

During the next few releases of the extension, MyNode is modified with an addition of several new parameters:

Now, if a user whose version of My Extension is 0.5.0 opens a workflow containing MyNode that was configured/saved on a machine where the version of My Extension was, for instance, 0.2.0, the node settings will automatically be adapted to contain the previously configured value for my_param, and the default values for double_param and string_param. If the user were to execute the node without first reconfiguring it, the execute method would use those default values for the corresponding parameters.

Note how the default value of double_param depends on the version in order to ensure that the node’s output does not change if the workflow is of an older version.

If the behaviour/functionality of the node has changed throughout the various releases of the extension, and you would like to require users to reconfigure the node if certain conditions are met, you can use the config_context.set_warning() or exec_context.set_warning() methods in the configure and execute methods of your node respectively to display a yellow "warning" sign in the node status. Additionally, you can raise an exception to further direct the user to reconfigure the node. For example:

Sometimes it is not possible to change a node and stay backwards compatible e.g. if an input or output port is added. If you find yourself in this scenario do the following:

The description of your node, which is displayed in the Description area of KNIME Analytics Platform when a node is selected, is composed of multiple components. These components come from the descriptions you, as the developer, provide when defining the building blocks of the node, such as the input ports or the configuration parameters.

By including the markdown Python package in the python environment associated with your node extension, you can make use of Markdown syntax when writing these descriptions to improve readability and the overall look of your nodes' documentation.

Below you can find a list of which Markdown syntax is supported for each node description element.

Here is a functional example of using Markdown when writing a Python node:

Below is the resulting node description as seen in KNIME Analytics Platform:

The descriptions of individual node parameters can additionally be accessed from within the configuration dialog of the node:

To ensure that the users you have shared your extension with are able to utilize its functionality fully and error-free, we bundle the source files together with the required packages using pixi-pack.

The knime.yml file (refer to the Python Node Extension Setup section for an example of this configuration file) contains the information required to bundle your extension, including:

The template provided in knime-python-extension-template already contains a pixi.toml file with a base python environment. You can use this file as a starting point for your own extension. You can add additional packages to the pixi.toml file with the command:

This will automatically add the package to the pixi.toml file and make sure that the environment works on all platforms. Packages from PyPI (pip) can be added as well by using the command pixi add --pypi <package_name>. However, it should be noted, that pip-source packages are not supported in the bundling pipline. Make sure to only include conda packages (preferred) or pypi wheel packages. If you can’t avoid using a pip-source packages, you can create your own conda package and host it on anaconda.org or a private conda channel. Before including a package as pypi dependency, make sure to check if it is available on conda-forge or other conda channels. If it is, prefer using the conda package.

If you already have a conda environment with the required packages, you can use the command:

to generate an env.yml file in which the packages installed in the environment are listed. This will reduce the list of dependencies down to the packages that you have manually installed in the environment. Note that this option does not preserve the list of manually specified channels when installing packages (e.g., conda-forge), so you might have to add them yourself. You can then use this file as a starting point for your pixi.toml file. By using:

you can generate a pixi.toml file that contains the packages listed in the env.yml file. Note however, that this file will not contain the base python environment required for your extension unless you manually add it. However, you can merge the dependencies of the two pixi.toml files.

Once you have finished implementing your Python extension, you can bundle it, together with the appropriate python environment, into a local update site. This allows other users to install your extension in the KNIME Analytics Platform.

Follow the steps of extension setup. Once you have prepared the environment used by your extension (meaning a correct and lockable pixi.toml), and have set up the knime.yml file, you can proceed to generating the local update site.

In the pixi.toml we have prepared a dedicated build command for bundling your extension. This command will automatically create a python environment with all the dependencies needed for bundling. Simply run the command:

where dest is the destination directory where the update site will be created. to bundle your extension, where <my_destination> is the path to the directory where the bundled extension update site will be created. The command will create a environment with the name build and install all the dependencies listed in the pixi.toml file, including the knime-extension-bundling package. This package contains the necessary tools to automatically build your extension.

Under the hood, this command will install the build environment (if not already) and run the command build_python_extension . <my_destination> in the pixi shell. The . indicates the current directory, which contains the knime.yml file. The <my_destination> is the path to the directory where the update site will be created. build_python_extension is a python script that is included in the conda package knime-extension-bundling. For most use cases, it suffices to run this command to bundle your extension. However, the build_python_extension.py script can also be run directly, allowing for additional options to be passed. All available options can be found by running the command python build_python_extension.py --help in the terminal (with the build environment activated).

Add the generated repository folder to KNIME Analytics Platform as a Software Site in File → Preferences → Install/Update → Available Software Sites

Finally, install it via File → Install KNIME Extensions

The generated repository can now be shared with and installed by other users.

Once you have finished implementing your Python extension, you can share it, together with the appropriate python environment, on KNIME Community Hub. This allows other users to easily discover, install, and use your extension.

You can use the logging Python module to send warnings and errors to the KNIME Analytics Platform console. By going to File → Preferences → KNIME → KNIME GUI, you can choose the Console View Log Level. Each consecutive level includes the previous levels (i.e. DEBUG will also allow message from INFO, WARN, and ERROR to come through in the console, whereas WARN will only allow WARN and ERROR levels of messages).

In your Python script, you can initiate the logger, and use it to send out messages to the KNIME Analytics Platform console as follows:

In order to allow for a smooth user experience, the Analytics Platform caches the gateways used for non-execution tasks (such as the spec propagation or settings validation) of the last used Python extensions. This cache can be configured via two system properties:

The debug_mode: true propertly of config.yml discussed before effectively disables caching for individual extensions. By default, all extensions use caching.

Resourceful information helps in understanding issues. Relevant information can be obtained in the following ways.

In step 4 of the tutorial an environment is created which you use for your extension.

Three modules are specified for the installation:

You can create an environment with a more recent Python version as follows:

conda create -n my_python_env python=3.11 knime-python-base knime-extension -c knime -c conda-forge

or simply by updating your pixi.toml file which should contain a line like:

Then run pixi lock to update the pixi.lock file. This will update the Python version in the environment to the desired version.

If you want to develop and test multiple extensions simultaneously in your KNIME Analytics Platform, you can alter the config.yml (see step 5 of the tutorial)to contain the necessary information of additional extensions like this:

If during development you get an error similar to

then this is probably because you freshly introduced the parameter. Re-executing the node should solve this. Alternatively, drag and drop the node again from the node repository.

Due to inconsistencies of the different Operating Systems, integer columns in the output table can be of type long. To prevent that, follow this example:

On Windows, the following two errors can happen if you have two KNIME Analytics Platform versions open and both use the Columnar table backend. Close both and start only one.

The following error can appear if the extension was built with build_python_extension for a newer KNIME Analytics Platform (KAP) version. Run the build_python_extension script with the parameter for the specific KAP version or an older KAP version, e.g. build_python_extension.py --knime-version 5.5.

If you encounter an SSL error during the execution of a node, this might be due to the use of a self-signed certificate. If other nodes such as the GET Request node work, but the Python node does not, you can configure the Python nodes to trust the same certificates as the KNIME Analytics Platform. To do this, add the following line to your knime.ini file:

This will point the CA_CERTS and REQUESTS_CA_BUNDLE environment variables to a newly created CA bundle that contains the certicate authorities that the KNIME Analytics Platform trusts. The Python nodes will then trust the same certificates as the KNIME Analytics Platform.

Since KNIME Analytics Platform 5.5.0 Pixi is used to bundle the python environment needed for the extension. Pixi resolves Conda and pip dependencies at the same time and can be more strict about incompatibilities between conda and pip packages compared to the previous approach with conda-lock. It is generally recommended to use Pixi for new extensions. Existing extensions can be migrated to Pixi by following the migration tutorial. pixi.toml files can be created with the pixi init -i env.yml command, where env.yml is an existing environment file that you used before. The pixi.lock file is created by running pixi lock and contains the resolved dependencies. It is generally recommended to use conda packages for the dependencies and only use pip packages if they are not available as conda packages. It should be noted, that for the bundling process pip source packages are not supported, so you need to use pip wheels or (better) conda packages for all dependencies.

If your extension uses pip source packages you need to migrate them to pip wheels or conda packages and host them on a conda channel or PyPI. For packages that require compilation of native code, these must be built in advance to prevent potential build failures or compatibility issues on user systems. The compilation of native code is not described here. If it’s a pure Python package this can often be done in a few steps. However, the exact steps depend on the package and its dependencies. Here we provide an example of how to migrate a pip source package to a conda package, using grayskull.

Make sure that the package will be built as a pure Python package. If not already added by grayskull, you can add the following lines to the meta.yaml file in the created conda package folder:

Furthermore, make sure that all names are matching. Especially _ and - are often used interchangeably in pip package names. If everything is fine, you can build and upload the conda package to a conda channel. If you don’t have a conda channel yet, you can create one on https://anaconda.org/ or use the anaconda-client to create one.

After the package is uploaded, you can check if it is available on anaconda.org and add it to your pixi.toml file as a conda dependency:

If your package is a dependency of another package, you need to create a mapping from conda to pip, so that pip and conda packages know that this is the same package as the original pip source package. Otherwise, you will see two versions of the same package in the pixi list output, one as a conda package (the one you just created) and one as a pip package (the original pip source package). To do this, you need to create a file called conda-pypi-map.json, ideally in the root of your extension folder. The file should contain a mapping from the conda package name to the pip package name, e.g.:

And then you need to tell Pixi about this file by adding the following line to your pixi.toml file:

More information about Pixi and the mapping can be found in the Pixi documentation. After these steps the extension should bundle without issues and the pip source package should be replaced by the conda package.

For performance, we no longer bundle Python packages in Python-based extensions. Therefore, if you wish to install Python-based extensions in an offline ("air-gapped") environment, please follow these steps in addition to adding an offline update site:

Python-based extensions install a dedicated conda environment containing the Python packages required for this extension. By default, KNIME will create these conda environments at the location: '<KNIME-HOME>/bundling/envs/<EXTENSION-NAME>' for Linux and Mac and '<KNIME-HOME>\bundling\envs\<EXTENSION-NAME>' for Windows. However, it is possible to change the directory where the conda environments are created by setting the environment variable KNIME_PYTHON_BUNDLING_PATH to the desired directory. This can be useful to mitigate installation problems due to the limitation of path lengths in Windows.

If you are behind a proxy, you might encounter issues when trying to install Python-based KNIME extensions.

As Python-based KNIME extensions are installed via conda, you need to set the proxy settings in the conda configuration. The conda configuration file is located in the ~/.condarc file. More information about this file can be found in the following tutorial: How to use CondaRC.

To make sure the Python-based extension installation respects your proxy settings, we recommend adding the following lines to the ~/.condarc file:

These will ensure that the proxy settings are correctly set for conda. Proxy settings from KNIME will unfortunately not be propagated for the installation, as this could overwrite the existing proxy settings for conda.

If you run into the following error:

try adding ssl_no_revoke: true to your ~/.condarc file. The error means that your proxy is not configured to forward SSL certificate revocation checks.

If none of these tips work, you can also perform an offline installation of the extension as described in: Offline Installation.