Create a New Python based KNIME Extension

This is a quickstart guide that will walk you through the essential steps of writing and running your first Python node extension containing a single node. We will use tutorial_extension as the basis. The steps of the tutorial requiring modification of the Python code in my_extension.py have corresponding comments in the file, for convenience.

For an extensive overview of the full API, please refer to the Defining a KNIME Node in Python: Full API section, as well as our Read the Docs page.

As you develop your Python extension, you are able to run and debug it locally by setting the knime.python.extension.config system property in your KNIME Analytics Platform’s knime.ini to point to the config.yml, or in the launch configuration’s VM arguments in Eclipse. See the Registering Python extensions during development and Customizing the Python executable sections at the end of this guide for more information.

In order to share your Python extension with others, please refer to the Bundling your Python Extension Nodes section.

Besides tables, a node can also consume or produce other port objects and it is possible to define custom port objects for your extension. You can do so by extending knext.PortObject and knext.PortObjectSpec with your custom implementation. In order to use these objects in your node, you need to define a custom port type via the knext.port_type function that takes your PortObject and PortObjectSpec classes as well as a human-readable name for your port type and an optional id. Here is an example:

Let’s start with the PortObjectSpec:

The serialize and deserialize methods are used by the framework to store and load the spec.

Note: The deserialize method must be a classmethod.

The spec_data property is just an example for custom code and you can add arbitrary methods to your spec class as you see fit.

Next we implement the PortObject:

The PortObject class must have a serialize and deserialize method that are called by the framework to persist and restore the object. Again note that deserialize has to be a classmethod.

The predict property is again just an example for custom code that your port object class may contain.

Finally, we create a custom port type to be used as input or output of a node:

The knext.port_type method ties the PortObject and PortObjectSpec together and provides a human-readable name to refer to the custom port type.

It is also possible to specify a custom ID for the port type via the id argument. Note that the id must be unique! If you don’t provide a custom ID, then the framework generates one of the format your_extension_id.your_module_name.your_port_object_class_name. For example if your extension has the id org.company.extension and you implement a MyPortObject in the module my_extension, then the generated id is org.company.extension.my_extension.MyPortObject.

Note that there are also connection port objects that can hold non-serializable objects. You can find information about that in the API documentation for knime.extension.ConnectionPortObject.

Check out the next section to learn how to declare your custom port type as input or output of your node.

The input and output ports of a node can be configured by decorating the node class with @knext.input_table, @knext.input_port, and respectively @knext.output_table and @knext.output_port. An image output port can be added with the @knext.output_image decorator. Additionally, a node producing a view should be decorated with the @knext.output_view decorator.

These port decorators have the following properties:

The @knext.input_table and @knext.output_table decorators configure the port to consume and respectively produce a KNIME table. The @knext.output_image decorator configures the port to produce a PNG or SVG image.

If you want to receive or send other data, e.g. a trained machine learning model, use @knext.input_port and @knext.output_port. These decorators have an additional argument, port_type, used to identify the type of port objects going along this port connection. Only ports with equal port_type can be connected. See the previous section to learn how to specify your own port type.

The port configuration determines the expected signature of the configure and execute methods:

Examples how to use knext.Schema and knext.Column` (see the API):

Here is an example with two input ports and one output port. See the previous session for the definitions of MyPortObject, MyPortObjectSpec and my_model_port_type.

Example with two image output ports.

Alternatively, you can populate the input_ports and output_ports attributes of your node class (on class or instance level) for more fine grained control.

Each node in your Python node extension is assigned a category via the category parameter of the @knext.node decorator, which dictates where the node will be located in the node repository of KNIME Analytics Platform. Without an explicit category, the node will be placed in the root of the node repository, thus you should always specify a category for each node.

In order to define a custom category for the nodes of your extension, you can use the knext.category helper function. If autocompletion is enabled in your IDE, you should be able to see the list of the expected parameters of the function, together with their detailed description.

If you are a community developer, you should use the Community Nodes category as the parent category of your Python node extensions. This is done by specifying the path="/community" parameter of the knext.category function:

Note that it is possible to further split your custom category into subcategories. This is useful if, for instance, nodes of your extension can be grouped based on their functionality. By first defining a parent category for the node extension, you can then use it as the path parameter when defining the subcategories:

In order to add parameterization to your node’s functionality, we can define and customize its configuration dialog. The user-configurable parameters that will be displayed there, and whose values can be accessed inside the execute method of the node via self.param_name, are set up using the following parameter classes available in knext:

All of the above have arguments label and description, which are displayed in the node description in KNIME Analytics Platform, as well as in the configuration dialog itself. Additionally, all parameter classes have an optional argument since_version, which can be used to specify the version of the extension that the parameter was introduced in. Please refer to the Versioning your extension section below for a more detailed overview.

Parameters are defined in the form of class attributes inside the node class definition (similar to Python descriptors):

While each parameter type listed above has default type validation, they also support custom validation via a property-like decorator notation. By wrapping a function that receives a tentative parameter value, and raises an exception should some condition be violated, with the @some_param.validator decorator, you are able to add an additional layer of validation to the parameter some_param. This should be done below the definition of the parameter for which you are adding a validator, and above the configure and execute methods:

You can use the @knext.output_view(name="", description="") decorator to specify that a node returns a view. In that case, the execute method should return a tuple of port outputs and the view (of type knime.api.views.NodeView).

You can access the flow variables available to the node in both the configure and execute methods, via the config_context.flow_variables and exec_context.flow_variables attributes respectively. The flow variables are provided as a dictionary of flow_variable_name : flow_variable_value mappings, and support the following types:

By mutating the flow_variables dictionary, you can access, modify, and delete existing flow variables, as well as create new ones to be propagated to downstream nodes.

As you continue to develop your extension after the initial release, you might extend the functionality of your nodes by adding or removing certain parameters. With the versioning capabilities of Python-based node extensions for KNIME Analytics Platform, you can ensure backward compatibility for your users.

As seen in the Python Node Extension Setup section, the knime.yml configuration file contains a version field. This allows you to assign a version to each iteration of your extension. How closely you want to follow the semantic versioning scheme is completely up to you, but we do require adherence to the following formatting-related rule: versions must be composed of three non-negative numeric parts separated by dots (e.g. 1.0.0, 0.2.1, etc.).

When adding a new parameter to a node, you should associate it with the corresponding version of your extension. This is done using the since_version argument that is now available for all parameter types via the appropriate constructor (e.g. knext.IntParameter), as well as parameter groups via the @knext.parameter_group decorator. If not specified, the since_version argument of a parameter or parameter group defaults to 0.0.0, which indicates that the parameter was available from the first iteration of the extension.

A common use-case of extension versioning is to facilitate backward compatibility when opening workflows that were created/saved with an older version of the extension installed on the machine. What KNIME Analytics Platform will try to achieve by default in this case, is to combine the values of the previously configured node settings that are still available in the current version of the extension with the newly added node settings, if any. The latter are then automatically set to their default values, and the node remains configured.

Here is a minimal functional example of a Python-based extension containing a single node with a single parameter. Since the parameter is available from the initial release of the extension, we can forgo setting the since_version argument:

During the next few releases of the extension, MyNode is modified with an addition of several new parameters:

Now, if a user whose version of My Extension is 0.5.0 opens a workflow containing MyNode that was configured/saved on a machine where the version of My Extension was, for instance, 0.2.0, the node settings will automatically be adapted to contain the previously configured value for my_param, and the default values for double_param and string_param. If the user were to execute the node without first reconfiguring it, the execute method would use those default values for the corresponding parameters.

Note how the default value of double_param depends on the version in order to ensure that the node’s output does not change if the workflow is of an older version.

If the behaviour/functionality of the node has changed throughout the various releases of the extension, and you would like to require users to reconfigure the node if certain conditions are met, you can use the config_context.set_warning() or exec_context.set_warning() methods in the configure and execute methods of your node respectively to display a yellow "warning" sign in the node status. Additionally, you can raise an exception to further direct the user to reconfigure the node. For example:

Sometimes it is not possible to change a node and stay backwards compatible e.g. if an input or output port is added. If you find yourself in this scenario do the following:

The description of your node, which is displayed in the Description area of KNIME Analytics Platform when a node is selected, is composed of multiple components. These components come from the descriptions you, as the developer, provide when defining the building blocks of the node, such as the input ports or the configuration parameters.

By including the markdown Python package in the conda environment associated with your node extension, you can make use of Markdown syntax when writing these descriptions to improve readability and the overall look of your nodes' documentation.

Below you can find a list of which Markdown syntax is supported for each node description element.

Here is a functional example of using Markdown when writing a Python node:

Below is the resulting node description as seen in KNIME Analytics Platform:

The descriptions of individual node parameters can additionally be accessed from within the configuration dialog of the node:

To ensure that the users you have shared your extension with are able to utilise its functionality fully and error-free, we bundle the source files together with the required packages using conda as the bundling channel.

The knime.yml file (refer to the Python Node Extension Setup section for an example of this configuration file) contains the information required to bundle your extension, including:

These YAML files can be automatically generated by activating the desired environment, and running one of the following commands, which will result in configuration files of various strictness:

Note that, in addition to packages installed with conda, you are also able to install packages from PyPI via pip. When done from within your activated conda environment, such packages are also automatically included in the YAML configuration file generated with the above commands.

Once you have finished implementing your Python extension, you can bundle it, together with the appropriate conda environment, into a local update site. This allows other users to install your extension in the KNIME Analytics Platform.

Follow the steps of extension setup. Once you have prepared the YAML configuration file for the environment used by your extension, and have set up the knime.yml file, you can proceed to generating the local update site.

We provide a special conda package, knime-extension-bundling, which contains the necessary tools to automatically build your extension. Run the following commands in your terminal (Linux/macOS) or Anaconda Prompt (Windows). They will setup a conda environment, which gives the tools to bundle extensions. Then the extension will be bundled.

The generated repository can now be shared with and installed by other users.

Once you have finished implementing your Python extension, you can share it, together with the appropriate conda environment, to KNIME Community Hub.

You can use the logging Python module to send warnings and errors to the KNIME Analytics Platform console. By going to File → Preferences → KNIME → KNIME GUI, you can choose the Console View Log Level. Each consecutive level includes the previous levels (i.e. DEBUG will also allow message from INFO, WARN, and ERROR to come through in the console, whereas WARN will only allow WARN and ERROR levels of messages).

In your Python script, you can initiate the logger, and use it to send out messages to the KNIME Analytics Platform console as follows:

In order to allow for a smooth user experience, the Analytics Platform caches the gateways used for non-execution tasks (such as the spec propagation or settings validation) of the last used Python extensions. This cache can be configured via two system properties:

The debug_mode: true propertly of config.yml discussed before effectively disables caching for individual extensions. By default, all extensions use caching.

Resourceful information helps in understanding issues. Relevant information can be obtained in the following ways.

In step 4 of the tutorial an environment is created which you use for your extension.

Three modules are specified for the installation:

You can create an environment with a more recent Python version as follows:

conda create -n my_python_env python=3.11 knime-python-base knime-extension -c knime -c conda-forge

If you want to develop and test multiple extensions simultaneously in your KNIME Analytics Platform, you can alter the config.yml (see step 5 of the tutorial)to contain the necessary information of additional extensions like this:

If during development you get an error similar to

then this is probably because you freshly introduced the parameter. Re-executing the node should solve this. Alternatively, drag and drop the node again from the node repository.

Due to inconsistencies of the different Operating Systems, integer columns in the output table can be of type long. To prevent that, follow this example:

On Windows, the following two errors can happen if you have two KNIME Analytics Platform versions open and both use the Columnar table backend. Close both and start only one.

The following error can appear if the extension was built with build_python_extension for a newer KNIME Analytics Platform (KAP) version. Run the build_python_extension script with the parameter for the specific KAP version or an older KAP version, e.g. build_python_extension.py --knime-version 5.2.

For performance, we no longer bundle Python packages in Python-based extensions. Therefore, if you wish to install Python-based extensions in an offline ("air-gapped") environment, please follow these steps in addition to adding an offline update site:

Python-based extensions install a dedicated conda environment containing the Python packages required for this extension. By default, KNIME will create these conda environments at the location: '<KNIME-HOME>/bundling/envs/<EXTENSION-NAME>' for Linux and Mac and '<KNIME-HOME>\bundling\envs\<EXTENSION-NAME>' for Windows. However, it is possible to change the directory where the conda environments are created by setting the environment variable KNIME_PYTHON_BUNDLING_PATH to the desired directory. This can be useful to mitigate installation problems due to the limitation of path lengths in Windows.