KNIME Best Practices Guide

KNIME Analytics Platform is an open source software with a visual interface, which is shown below. It allows users to build data science analyses. Users can access, blend, analyze, and visualize their data, and integrate third-party tools and libraries.

If you haven’t downloaded KNIME Analytics Platform yet, you can do so on the download page.

KNIME Hub is the commercial software for collaborating and scaling data science, offered online as KNIME Community Hub or installed into a company’s private infrastructure as KNIME Business Hub.

Workflows created in your local KNIME Analytics Platform installation are just like any other file on your computer. You can uniquely identify them by their file name and sort them into folders. Before you dive into the data, consider the following when saving your workflows:

KNIME workflows are made to operate on data files. If such a file is saved on your local computer, its absolute path uniquely identifies the location of your data file within your file system, for example, C:\Users\username\Desktop on Windows or /home/user on Linux. However, if you share your workflow with others, e.g. using the KNIME Hub as a repository they will execute it in their local KNIME Analytics Platform installation.

Thus, the link with which KNIME Analytics Platform accesses the data file you provide along with your workflow should be a relative path. This can mean that your path is relative to one of the following:

Alternatively, you can provide a Custom URL along with your data for everyone that you share your workflow with. For example,

Find out more about the KNIME file handling infrastructure in the KNIME File Handling Guide.

Find out more about storing and sharing items on the KNIME Hub in the KNIME Community Hub User Guide.

To make your workflow reusable, you and your team need to be able to quickly comprehend what each part and each node are doing. KNIME node names are self-explanatory. However, to describe how each node operates on your specific data, you can change the node label. Select a node in the workflow editor and double-click below the node to change its label. Here, you can describe what operation it performs on your data.

To capture an operation performed by multiple nodes, use a workflow annotation. Right-click anywhere in the workflow editor and select New workflow annotation. You can observe a workflow with both workflow annotations, nodes with node labels, and one node which is selected and does not carry a label yet, in Figure 4.

Workflow annotations subsume smaller sub-groups of nodes within a workflow. The more nodes you add to your workflow, the less comprehensible it gets. In larger workflows, you may want to compress nodes into groups represented by a single node. Those can either be components or metanodes.

Metanodes are used solely to structure your workflows more clearly by subsuming many nodes into one. You can take parts of a larger workflow and collapse them into a metanode, hiding that part of the workflow’s functionality.

In addition to the functions of a metanode, a component can be reused on different data that require the same operations performed on them. For example, if you often connect to a particular database, you can create a connector component that contains all the settings options.

To create a component or metanode select at least two nodes, right-click anywhere in the workflow editor, and select either Create metanode or Create component, as explained in this section of the KNIME Components Guide.

To make a component comprehensible to others, document it by adding a description for the expected input and the different setting options (similar to a KNIME node), as explained in this section of the KNIME Components Guide.

To properly exhaust the flexibility of a component, it should:

You can have your component adjust to different data by adding custom configuration dialogs to access the flow variables inside the component. This allows you and other users of your component to easily customize a component to take variable names from different datasets as inputs. In this respect, components work just like regular KNIME nodes: You provide them with data to which you adjust the node settings. Ultimately, components make frequent data operations faster and easier.

As node sub-groups, components and metanodes should be treated like small workflows with respect to their scope. You should assign one single, self-contained and coherent function to one component or metanode.

Learn more about components and metanodes in the KNIME Components Guide.

After you create a component in a workflow, if you want to reuse it in another workflow, you can copy and paste it into the new workflow. However, in this way, changes to the component in one workflow are not picked up by the others. To make full use of a component’s reusability potential, you can share it on the KNIME Hub. To share a component, right-click it, select Component → Share. By doing so, you can let your collaborators access your changes to a component and also see theirs.

Additionally, by linking a shared component, you can incorporate changes made by others in your local installation. Whenever your collaborators update the component on the KNIME Hub, your KNIME Analytics Platform installation prompts you to update the component. To adapt the update policy, navigate to Preferences → KNIME. Under Component updates, you can set the default update policy to Never, Always, or Prompt.

Find out more about sharing components in the KNIME Components Guide.

Find out how to handle workflows that involve confidential data, in the security section of this guide.

Some components allow Streaming Execution. Read more about how this feature makes your workflows more efficient in the Streaming Execution section of this guide.

You can use components and their composite views to define pages in web application workflows. Once uploaded to KNIME Hub, they can be deployed as Data Apps.

To directly interact with the Data App UI in the browser, you need to implement components with widgets. Widgets allow consumers of your Data App to navigate between its pages. Each page corresponds to a component at the root level of the workflow, containing Widget nodes and View nodes. This means that when you inspect the composite view in KNIME Analytics Platform, you can adjust the parameters in the window that opens.

Find out more about data apps in the respective section of the KNIME Components Guide, or learn how to create a data app step-by-step in the KNIME Data Apps Beginners Guide.

To speed-up your workflow, keep the following guidelines in mind:

When designing a workflow, there are only a few mistakes to absolutely avoid, but many convenient practices that facilitate its usage. Both will be introduced in the following.

An efficient workflow never contains:

To make your workflow run faster, keep the following in mind:

Like any good project, a good workflow is scalable.

To measure which nodes take the longest to execute in your workflow you can use the Timer Info node. It reports individual and aggregate timing and execution information for all nodes. This allows you to identify weak points in your workflow and track your progress over time.

The Timer Info node is included in the KNIME base installation. Simply drag it anywhere into the workflow that you want to evaluate, and execute it.

In KNIME Analytics Platform, the GroupBy and Pivot nodes offer the "Process in memory" option. While it does use significant memory, it also brings a notable performance improvement by reducing disk swaps when these nodes are executed. To activate the "Process in memory" option, open the configuration dialog of the GroupBy or Pivot node. Under the tabs Settings > Groups navigate to Advanced settings and select the checkbox Process in memory.

Some KNIME nodes and components allow streaming execution. You can implement it with the KNIME Streaming Execution (Beta) extension on KNIME Hub.

While the default execution operates node-by-node, in streaming execution nodes are executed concurrently. Each node passes data to the next as soon as it is available, i.e., before the node is fully executed. In particular, all data are divided into a number of batches which are streamed one by one. Especially, when reading or preprocessing data, streaming execution accelerates runtime.

To switch from default to streaming execution, create a component and open its configuration dialog. Open the Job Manager Selection tab in the component configuration dialog. If the streaming execution option is not available for the node you will only see the default option in the menu. In components that allow streaming execution, select the "Simple Streaming" option. The streamable component is now marked with an arrow in its lower left corner.

For further reading on Streaming Execution, please consult the respective section in the KNIME Components Guide.

Find out more about how to structure your workflow with components in the section on reusability of this guide.

You can use the KNIME Columnar Table Backend extension to accelerate workflow execution. It optimizes the use of main memory in KNIME Analytics Platform by reviewing the underlying data representation, where cell elements in a table are represented by Java objects. Its now different underlying data layer is backed by Apache Arrow which is based on a columnar representation.

The advantages are:

You can activate the Columnar Backend by navigating to Preferences > Table Backend.

To adjust the table backend for one specific workflow, e.g., a workflow created before changing your default table backend, navigate to the node description tab in the side panel navigation. Click on the icon. In the window that opens, in the Table Backend tab, select Columnar Backend from the dropdown menu. The parameters relative to memory usage of the Columnar Backend can be configured under Preferences > Table Backend > Columnar Backend.

For more details, please refer to the Preferences section of the KNIME Analytics Platform User Guide.

Another way to speed up workflow execution is parallel computing. You can implement this with loops. They iterate over a certain part of the workflow, e.g. over a workflow branch. Each iteration, a repeated procedure, has different input. Among others, this input can consist of chunks of the data you want to process in KNIME Analytics Platform. These chunks will each be processed in parallel on separate workflow branches.

To get started, you need to have the KNIME Parallel Chunk Loop Nodes Extension installed.

After reading the data, and before your first data manipulation, you can add the Chunk Loop Start node. It splits data into equally sized chunks for each iteration. Then it executes the manipulations in the loop body on each chunk in parallel. You can configure either the number of chunks or the number of rows per chunk. At the end of a parallel chunked block, add the Parallel Chunk End node. It collects and concatenates the results from the parallel workflow branches. If the data segment in the loop generates tables with repeated RowIDs, the Parallel Chunk End node adds a chunk index to the RowID.

For more details about loops, please refer to the Loops section of the KNIME Flow Control Guide.

KNIME Analytics Platform allows you to access, manipulate and share large amounts of data. The most secure and effective way to store this data is in a (relational) database.

With the KNIME Database Extension, which is already part of every KNIME Analytics Platform installation, you can use the full functionality of any JDBC-compliant database within KNIME. This includes the following database operations:

To connect to the database of your choice, KNIME provides Connector nodes that can be found in different extensions. There are three ways to connect to a database in KNIME:

Once connected, you can select relations with the DB Table Selector node and read from the database using the DB Reader node.

In addition to reading from the database, you can perform in-database processing, i.e. data manipulation directly on the database, using the KNIME Database nodes, e.g. DB Joiner or DB GroupBy and formulating your query.

Their configuration is similar to the KNIME manipulation nodes. The DB Query Nodes require no SQL coding, since the nodes themselves construct and output a SQL query.

Finally, Database Writing Nodes let you create, update and delete data on the database of your choice. The DB Connection Table Writer node is useful for efficiency and data lineage maintenance, because it creates a new database table based on the input on the SQL query. As a result, there is no need to actually read the data into KNIME Analytics Platform itself. Additionally, the DB Loader node lets you insert rows into an existing DB table, although this is not supported by all databases.

KNIME Analytics Platform also supports NoSQL and non-relational databases like MongoDB(document-oriented), DynamoDB (document-oriented and key-value-based) and Neo4j (graph-based).

To speed-up your workflow with respect to databases, keep the following guidelines in mind:

The KNIME Database Extension includes nodes for simulating database transactions, such as DB Transaction Start and DB Transaction End. These nodes help group multiple database operations into a single unit, ensuring that either all operations execute or none do. If the transaction is successful, it ends with a commit to the database. If it is not successful, it is rolled back, returning the database to its state at the beginning of the transaction.

This enhances efficiency by:

Find out more about database capabilities with KNIME and what to consider for different database providers in the KNIME Database Extension Guide.

When working with confidential data, it’s critical to have the best security measures in place to protect it from breaches and misuse. Protecting sensitive information is essential for regulatory compliance and maintaining the trust of customers and stakeholders.

KNIME Analytics Platform helps by providing various security features. It includes tools to encrypt data, manage user access, and ensure data integrity so you can handle sensitive information.

When you share a workflow, you can choose to save data that was processed along with it. Intermediate results of data manipulation operations are then available even after the workflow is closed. Particularly, when processing confidential data inside your workflows, you should make sure not to include any information that you do not want to share. When working with sensitive data, keep the following in mind:

Sensitive data needs to be handled with caution. GDPR (General Data Protection Regulation) stipulates that only anonymized data may be used extensively. As a consequence, you may want to anonymize your confidential data while later being able to re-identify the individual data points in the dataset. This is referred to as pseudo-anonymization. To additionally assess the risks of de-anonymization, you can use the Redfield Privacy Nodes extension for KNIME which uses different algorithms for anonymization and the assessment of re-identification risks.

For basic anonymization (pseudo-anonymization) you can use the Anonymization node. In the configuration dialog, simply choose the identifying attributes, i.e. the columns in your table that contain sensitive information. The Anonymization node hashes the values in the columns using SHA-1. You can also choose between different salting strategies. An advantage of this rather uncomplicated approach is that you can go back to the original data, since the translation table is available at the second output port.

For advanced anonymization, you should rely on hierarchies. You can build them by defining complex binning rules, which convert the unmodified data to completely suppressed data. You can create different types of hierarchies with the Create Hierarchy node. Multiple hierarchies may be applied to the data subsequently. To read more about the hierarchy types, please refer to the Building hierarchies section of the following article: Remain Anonymous with Redfield’s Privacy Extension.

To now apply the rules set to the original data, you need to combine them in the Hierarchical Anonymization node.

To finally assess which risks are posed by your anonymization techniques, add the Anonymity Assessment node. To find out more about the types of risks that the node identifies, please refer to the node page of the Anonymity Assessment node on the KNIME Community Hub.

A box with colored borders and optional text that can be placed in a workflow to highlight an area and provide additional information. Right-click on an empty spot in a workflow and select “New workflow annotation” to create a new workflow annotation at that position.

The text underneath a node. Select the node and click on the text. Then you can edit it and add a description of what the node does.

The description of a workflow that can be edited by the user. Click on the at the top-right to edit it. The workflow description is also shown in the KNIME Hub when a user opens the workflow’s page.

The place on the KNIME Hub where workflows, components, and data files are stored. Find out more in the Community Hub User Guide.

Every time a workflow is executed ad hoc or a deployment is executed a job is created on KNIME Hub. More information about jobs can be found here.

A workflow snippet wrapped into a single node. A component can have its own configuration dialog and can also be assigned a view. Components enable you to bundle functionality for a particular part of the workflow and allow sharing and reusing. For more information refer to KNIME Components Guide.

A workflow comprising one or more components that contain view and widget nodes a user can interact with in the browser. A Data App is deployed on KNIME Business Hub. A guide on building Data Apps can be found here.

A schedule allows you to set up a workflow to run automatically at selected times. More information is available:

A service deployment allows you to use the workflow as an API endpoint. For more information refer to KNIME Business Hub User Guide.

A workflow with Workflow Service Input and Workflow Service Output nodes that can be called from other workflows using the Call Workflow Service node.