KNIME Server Release Notes and Update Guide


Release Notes

KNIME Server 4.11 is a feature release of the 4.x release line. All clients that have worked with KNIME Server 4.10 will continue to work with KNIME Server 4.11 without restrictions.

To find out which version of KNIME Server you are currently running, you can check the Administration pages on the WebPortal.

New Features

For a list that includes the new Analytics Platform 4.2 features see here.

Highlighted new functionality is:

A detailed changelog for KNIME Server 4.11 is also available.

Switch to Apache Tomcat

In previous releases, the application server component of KNIME Server was based on Apache TomEE - the Java Enterprise Edition of Apache Tomcat. Now, in the new KNIME Server 4.11, TomEE is replaced by the standard Apache Tomcat.

In line with the switch to the standard Apache Tomcat, the use of the old EJB mountpoints to connect to KNIME Server is now discouraged in favor of the newer REST implementation. Migrating to a REST mountpoint has been made very simple. As soon as the user logs in with an existing EJB mountpoint, there is a prompt from which they can switch to REST with a single click. REST provides many benefits in comparison to EJB - particularly advantages in terms of performance and stability.

The Tomcat version used is 9.0.36. Note that for existing customers who are hesitant to switch to Tomcat, we still provide a KNIME Server installer based on Apache TomEE. This has been updated to the latest version, TomEE 8.0.3.

Execution via Qpid

Previous releases of KNIME Server used RMI to establish a connection between an application server and KNIME Executor. RMI has now been replaced by an embedded message queue based on Apache Qpid. Rather than having direct communication between application Server and Executor, events such as requests for job execution are passed on via the message queue.

Qpid technology comes bundled with the KNIME Server installer, so no additional setup is necessary. All new KNIME Server installations are configured to use Qpid as default. In addition, the KNIME Server installer makes various adjustments to the Executor’s knime.ini file to facilitate the connection to Qpid.

Note that Qpid only supports a single KNIME Executor running on the same host as the application server. In case you want to run multiple distributed KNIME Executors, it is still necessary to set up RabbitMQ. Unlike with previous releases the Executor is not automatically started together with the Server, but instead has to be started individually. The necessary steps are described in the KNIME Server Installation Guide

Switch to REST mountpoints

Starting with the release of KNIME Server 4.11, the use of the old EJB mountpoints to connect to KNIME Server is now discouraged in favor of the newer REST implementation. In case of a Tomcat-based KNIME Server, this will be the only valid connection method. I.e., EJB is not supported anymore for KNIME Servers running on Apache Tomcat.

We have made migrating to a REST mountpoint very simple. Just log in with an existing EJB mountpoint and there will be a prompt from which you can switch to REST with a single click. REST provides many benefits in comparison to EJB - particularly advantages in terms of performance and stability.

After you switch to REST, you will notice some slight differences in behavior. Most notably, we do not show a popup dialog anymore when you start a workflow that uses either Workflow Variables or Workflow Credentials. These options have now been made accessible directly from the workflow execution dialog. To get there, right-click a workflow in the server repository → Execute. Then open the Configuration options tab. This allows you to enter Workflow Variables and Workflow Credentials.

In addition, the Configuration options tab also allows you to set values for all Configuration nodes inside a particular workflow that are located on the top-level of the workflow (i.e., that are not inside a component or metanode). This is available for all nodes with no incoming connection (including flow variables) included in the Workflow Abstraction → Configuration category of the KNIME Analytics Platform node repository

Local file system access by KNIME workflows

A growing number of KNIME nodes (see below) are being revised to use a new shared framework for file access. More nodes will follow over the next releases, so that eventually the file access of all KNIME nodes uses the shared framework.

When executing on KNIME Server, a preference controls whether those nodes may access the local file system of the KNIME Server Executor or not. With this release, local file system access is disallowed by default (previously it was allowed).

To allow local file system access (not recommended), you can add the following line to the preferences.epf or customization profile used by your KNIME Server Executor(s):

/instance/org.knime.filehandling.core/allow_local_fs_access_on_server=true

This preference currently affects the following KNIME nodes: Excel Reader (XLS), Excel Writer (XLS), Excel Sheet Appender (XLS), Line Reader. In addition, this affects all nodes within the new File Handling (Labs) category.

Update

This document guides you through the steps that are needed to update an existing KNIME Server installation from version 4.10 to 4.11 or apply a bugfix for version 4.11. You will find a complete guide to installing KNIME Server in the KNIME Server Installation Guide, and a complete description of all configuration options in the KNIME Server Administration Guide. If you have any questions or need assistance with the update process, please contact your dedicated KNIME support specialist.

New installations of KNIME Server are based on Apache Tomcat, rather than Apache TomEE which was used in previous releases. It is not possible to change your installation from TomEE to Tomcat. Instead, if you plan to use Tomcat, please perform a fresh installation of KNIME Server. In addition, we continue to offer a TomEE based installer for the time being. The TomEE version for this has been updated to version 8.0.3 (from TomEE 7.0.5). If you want to update to the newest TomEE version, it is also necessary to perform a fresh installation of KNIME Server as upgrades are not supported. I.e., the remainder of this document assumes that you update your KNIME Server to version 4.11, while maintaining TomEE 7.0.5 as application server. KNIME Executors are not affected by these changes, and can be upgraded following the standard procedure.
Due to some updated libraries, the KNIME Server REST API documentation available at https://<your-server>/knime/rest/doc/index.html does not work anymore in installations based on TomEE 7.0.5. If you need this functionality, please install a fresh KNIME Server, which is based on TomEE 8.0.3 (or move directly to Tomcat). This does not affect the per-workflow Swagger pages (right-click in KNIME Explorer → Show API Definition), which continue to work on all TomEE versions.

Prerequisites

Server

  • This guide assumes you already have either KNIME Server 4.10.x or KNIME Server 4.11.x running. If you wish to update from a previous version you must study the intermediate update guides to identify configuration changes that may apply, and you may wish to consider a fresh installation in order to update the underlying Tomcat application server.

Client & Executor

  • KNIME Analytics Platform clients need at least KNIME ServerSpace 4.0 otherwise it will not be able to talk to the server.

  • KNIME Server 4.11 requires at least KNIME Analytics Platform 4.2 as executor with the corresponding KNIME Executor Connector version 4.11 installed on the server side.

  • New KNIME WebPortal features are available with any supported web browser.

Backup

Even though all the data on the server should be preserved during the update it is highly recommended to create a backup of all important data (see KNIME Server Administration Guide for backup information).

Application server

Backup existing configuration

Before performing an update, make a backup of <tomcat-folder>/conf/Catalina/localhost/knime.xml. The name of the file might be different if the .war file had a different name (e.g. com.knime.enterprise.server.xml or ROOT.xml).

Failure to backup this file will result in scheduled jobs failing to run.

Also make sure to create a backup of <tomcat-folder>/conf/server.xml.

  1. Download the updated war file war-4.11.x.y.war from the download page.

  2. Stop Tomcat.

  3. Delete the existing folder <tomcat-folder>/webapps/knime (or similar if the .war file had a different name).

  4. Copy the new war-4.11.x.y.war over the existing war file (usually <tomcat-folder>/webapps/knime.war). Please note that it needs to be the same file name as this is what is referred to as context in the application.

  5. Check if the knime.xml file that you backed up before still exists. If not copy the backup to the previous location.

  6. Copy the downloaded file knime-tomcat.jar to <tomcat-folder>/lib

  7. Follow the steps below on updating the KNIME Server Executor, specifically watch out to merge changes to the knime.ini in the workflow repository (<server repository>/config/knime.ini), if any.

  8. Start Tomcat.

Note that it is not possible to run multiple KNIME Server instances in one Tomcat installation.

KNIME Executor

Installing a new KNIME Executor

KNIME Server 4.11 requires KNIME Analytics Platform 4.2 or later. Since an update from KNIME Analytics Platform 4.1 to 4.2 isn’t possible you have to create a new installation of KNIME Analytics Platform 4.2 and install the KNIME Executor connector version 4.11 into it. The reason for this are an updated Eclipse version as well as an upgrade of the bundled JRE.

To install KNIME Analytics Platform 4.2 from scratch you will need to run it as the user that owns the Installation.

Please refer to the KNIME Server Administration Guide for more details about installing a KNIME Executor.

knime.ini file

In case you want to use custom settings from you old KNIME Executor’s knime.ini file, please copy only the lines following -vmargs to the new knime.ini. The only exception to this are the four lines related to Server Managed Customizations (-profileLocation…​), in case those are set. The remaining lines above -vmargs contain paths to resources that have been changed in the new release, so they are not compatible.

server.xml file

Please note that the server.xml file (<tomcat-folder>/conf/server.xml) also cannot be copied over from an existing installation due to some larger changes. Any custom changes applied to your existing server.xml need to be manually copied over to the new server.xml

KNIME Server license

The KNIME Server license continues working with the new Server. If you haven’t received a license file or if it is not working correctly, please contact KNIME or your dedicated KNIME account manager.

Enabling execution via Apache Qpid

KNIME Server 4.11 introduces an integrated message broker for communication between KNIME Server and KNIME Executor, based on Apache Qpid (https://qpid.apache.org/). This can be thought of as using an architecture that corresponds closely to distributed KNIME Executors, but runs on a single host, which is used by both KNIME Server and KNIME Executor.

Unlike distributed KNIME Executors, it is not necessary to install a separate message broker such as RabbitMQ. Instead, Apache Qpid is bundled as part of all KNIME Server installations. New installations of KNIME Server use Qpid by default. To switch your current installation to Qpid, changes are required in two locations:

Application Server

Activate queue in knime-server.config. The file can be found in <server-repository>/config/knime-server.config. Set the parameter

com.knime.enterprise.executor.embedded-broker= to true

KNIME Executor

In the KNIME Executor installation directory, add the following line to the knime.ini file, anywhere after the -vmargs line:

-Dcom.knime.enterprise.executor.msgq=amqp://knime:20knime16@localhost/

Unlike the RMI-based execution, using the Qpid message broker requires to startup KNIME Server and KNIME Executor as two separate services.

For increased security, we recommend to run both services by different users.

For setting up the KNIME Server service, please follow the steps outlined in the KNIME Server Installation Guide. Installing the executor as a service follows the same procedure as described for distributed KNIME Executors.

Changelog (KNIME Server 4.11)

KNIME Server 4.11.0 (released July 13, 2020)

Enhancement

  • [SRV-2033] - Dialogs of DB nodes not available in remote workflow editor

  • [SRV-2154] - Permissions for scheduled jobs in REST interface

  • [SRV-2155] - Permissions for scheduled jobs in KNIME Explorer

  • [SRV-2534] - Allow to reserve executors for exclusive use

  • [SRV-2562] - Improve message on forbidden symbols in mountID in server installer

  • [SRV-2580] - Allow re-connecting to existing jobs after server restart

  • [SRV-2600] - Add max core setting to preferences.epf.template

  • [SRV-2639] - Provide versions of server components in REST interface

  • [SRV-2696] - Enable dynamic port configuration in remote workflow editor

  • [SRV-2712] - Executor should retry connections to message queue

  • [SRV-2733] - Port for embedded message queue should be configurable

  • [SRV-2798] - [OAuth] Webportal landing page when using OIDC

  • [SRV-2807] - Improve job logging

  • [SRV-2820] - Dialog option to keep or change history when moving/copying workflows

  • [SRV-2821] - Add possibility to map JWT claims to groups

  • [SRV-2823] - Reuse OAuth Configuration From AuthenticationValve in KNIME Server

  • [SRV-2824] - Automatic OIDC endpoint discovery

  • [SRV-2830] - Define multiple queues in server config

  • [SRV-2871] - Prevent loss of OIDC configuration when war is updated

  • [SRV-2904] - Remove Glassfish support from KNIME Server Client

  • [SRV-2920] - Add option to force-enable "Create Snapshot"

  • [SRV-2921] - Add option to force-enable "Reset Workflow before upload"

  • [SRV-2922] - Change default executor starting port

  • [SRV-2926] - Installer should set path to embedded queue in executor knime.ini

  • [SRV-2927] - Add new template for server managed customizations specific to executors

  • [SRV-2928] - Remove preferences.epf.template from installer

  • [SRV-2930] - Fix behavior of Use REST option

  • [SRV-2932] - Show warning when using EJB mountpoints

  • [SRV-2938] - Extract user and group information for executor reservation

  • [SRV-2940] - Executor using embedded message queue should get as many core tokens as available

  • [SRV-2948] - Make table view chunk size editable and use smaller default value

  • [SRV-2958] - Do not send emails during Remote Job Edits

  • [SRV-2962] - 'Workflow summary' endpoint for jobs

  • [SRV-2963] - Default.epf template should use REST mountpoint

  • [SRV-2971] - New REST endpoint with executor information

  • [SRV-2977] - Add job IDs to all job related messages

  • [SRV-2982] - Executor should have a start script

  • [SRV-2985] - Server client should check for EJB support

  • [SRV-2988] - Improve UI of EJB warning dialog

  • [SRV-3002] - Support String claim as group-mapping-claim

  • [SRV-3013] - Allow force-reset of jobs before saving as workflow

  • [SRV-3015] - Update KNIME logo on OIDC login page

  • [SRV-3026] - Improve error message if no queue can be found due to group restrictions

  • [WEBP-454] - New WebPortal with improved layout and design

Bugfixes

  • [SRV-2352] - Improve error message if unexpected logout occurs

  • [SRV-2492] - IPv6 addresses are not correctly handled in canonical host name autodetection

  • [SRV-2718] - Closing KNIME OAuth Pop-Up (ESC Key) Will Block Port Used For Redirect

  • [SRV-2849] - Unexpected Basic Auth Pop-Up When "Opening API Definition" From the AP in the Browser Without an Authenticated Session

  • [SRV-2862] - Subnode-execution mode once set in server get’s overwritten with every job update

  • [SRV-2874] - Nullpointer Exception in AP when connecting to server stored as OAuth with OAuth disabled

  • [SRV-2883] - OAuth redirect page doesn’t show error

  • [SRV-2884] - Edit Mount Point Dialog does not remember Authentication Type

  • [SRV-2890] - Mount Point Dialog resizes when large error is displayed

  • [SRV-2892] - Wrong error message from executor when workflow cannot be loaded

  • [SRV-2899] - JavaNullPointer exception when trying to open configuration window of google updater on a remote job view

  • [SRV-2908] - Remote Job View does not refresh after an Executor restart

  • [SRV-2934] - Wrong error message when loading workflow failed in executor

  • [SRV-2941] - Email notification dialog in Call Workflow action advanced options not displayed correctly

  • [SRV-2943] - server_logs.zip sometimes contains folders with incomplete executor ID

  • [SRV-2967] - Overwriting workflow group via REST removes existing schedules

  • [SRV-2990] - Lazily loaded table rows are not updated in port table view (job view)

  • [SRV-3025] - Account settings of KNIME server are not applied on runtime

  • [SRV-3032] - Workflow pinning/reservation rules dont accept the @ character

  • [SRV-3034] - Failure to read updated server config prevents further updates

  • [SRV-3044] - Force reset workflow is ignored on temporary copies of workflows

  • [SRV-3051] - Unable to enter username in "Edit Mount Point" dialog

  • [SRV-3052] - Server sends multiple Status Emails for loops in workflows

  • [SRV-3060] - OAuth: NullPointerException when trying to refresh the access token