KNIME Server Release Notes and Update Guide


Release Notes

KNIME Server 4.11 is a feature release of the 4.x release line. All clients that have worked with KNIME Server 4.10 will continue to work with KNIME Server 4.11 without restrictions.

To find out which version of KNIME Server you are currently running, you can check the Administration pages on the WebPortal.

New Features

For a list that includes the new Analytics Platform 4.2 features see here.

Highlighted new functionality is:

A detailed changelog for KNIME Server 4.11 is also available.

Switch to Apache Tomcat

In previous releases, the application server component of KNIME Server was based on Apache TomEE - the Java Enterprise Edition of Apache Tomcat. Now, in the new KNIME Server 4.11, TomEE is replaced by the standard Apache Tomcat.

In line with the switch to the standard Apache Tomcat, the use of the old EJB mountpoints to connect to KNIME Server is now discouraged in favor of the newer REST implementation. Migrating to a REST mountpoint has been made very simple. As soon as the user logs in with an existing EJB mountpoint, there is a prompt from which they can switch to REST with a single click. REST provides many benefits in comparison to EJB - particularly advantages in terms of performance and stability.

The Tomcat version used is 9.0.36. Note that for existing customers who are hesitant to switch to Tomcat, we still provide a KNIME Server installer based on Apache TomEE. This has been updated to the latest version, TomEE 8.0.3.

Execution via Qpid

Previous releases of KNIME Server used RMI to establish a connection between an application server and KNIME Executor. RMI has now been replaced by an embedded message queue based on Apache Qpid. Rather than having direct communication between application Server and Executor, events such as requests for job execution are passed on via the message queue.

Qpid technology comes bundled with the KNIME Server installer, so no additional setup is necessary. All new KNIME Server installations are configured to use Qpid as default. In addition, the KNIME Server installer makes various adjustments to the Executor’s knime.ini file to facilitate the connection to Qpid.

Note that Qpid only supports a single KNIME Executor running on the same host as the application server. In case you want to run multiple distributed KNIME Executors, it is still necessary to set up RabbitMQ. Unlike with previous releases the Executor is not automatically started together with the Server, but instead has to be started individually. The necessary steps are described in the KNIME Server Installation Guide

Switch to REST mountpoints

Starting with the release of KNIME Server 4.11, the use of the old EJB mountpoints to connect to KNIME Server is now discouraged in favor of the newer REST implementation. In case of a Tomcat-based KNIME Server, this will be the only valid connection method. I.e., EJB is not supported anymore for KNIME Servers running on Apache Tomcat.

We have made migrating to a REST mountpoint very simple. Just log in with an existing EJB mountpoint and there will be a prompt from which you can switch to REST with a single click. REST provides many benefits in comparison to EJB - particularly advantages in terms of performance and stability.

After you switch to REST, you will notice some slight differences in behavior. Most notably, we do not show a popup dialog anymore when you start a workflow that uses either Workflow Variables or Workflow Credentials. These options have now been made accessible directly from the workflow execution dialog. To get there, right-click a workflow in the server repository → Execute. Then open the Configuration options tab. This allows you to enter Workflow Variables and Workflow Credentials.

In addition, the Configuration options tab also allows you to set values for all Configuration nodes inside a particular workflow that are located on the top-level of the workflow (i.e., that are not inside a component or metanode). This is available for all nodes with no incoming connection (including flow variables) included in the Workflow Abstraction → Configuration category of the KNIME Analytics Platform node repository

Local file system access by KNIME workflows

A growing number of KNIME nodes (see below) are being revised to use a new shared framework for file access. More nodes will follow over the next releases, so that eventually the file access of all KNIME nodes uses the shared framework.

When executing on KNIME Server, a preference controls whether those nodes may access the local file system of the KNIME Server Executor or not. With this release, local file system access is disallowed by default (previously it was allowed).

To allow local file system access (not recommended), you can add the following line to the customization profile used by your KNIME Server Executor(s):

/instance/org.knime.filehandling.core/allow_local_fs_access_on_server=true

This preference currently affects the following KNIME nodes: Excel Reader (XLS), Excel Writer (XLS), Excel Sheet Appender (XLS), Line Reader. In addition, this affects all nodes within the new File Handling (Labs) category. Any other KNIME nodes are not affected by this setting.

Update

This document guides you through the steps that are needed to update an existing KNIME Server installation from version 4.10 to 4.11 or apply a bugfix for version 4.11. You will find a complete guide to installing KNIME Server in the KNIME Server Installation Guide, and a complete description of all configuration options in the KNIME Server Administration Guide. If you have any questions or need assistance with the update process, please contact your dedicated KNIME support specialist.

For the sake of readability, this documentation uses <tomcat-folder>/ to refer to the installation directory of the server application. Apart from the directory name, the location will be the same on TomEE-based folders.

New installations of KNIME Server are based on Apache Tomcat, rather than Apache TomEE which was used in previous releases. It is not possible to change your installation from TomEE to Tomcat. Instead, if you plan to use Tomcat, please perform a fresh installation of KNIME Server. In addition, we continue to offer a TomEE based installer for the time being. The TomEE version for this has been updated to version 8.0.3 (from TomEE 7.0.5). If you want to update to the newest TomEE version, it is also necessary to perform a fresh installation of KNIME Server as upgrades are not supported. I.e., the remainder of this document assumes that you update your KNIME Server to version 4.11, while maintaining TomEE 7.0.5 as application server. KNIME Executors are not affected by these changes, and can be upgraded following the standard procedure.
Due to some updated libraries, the KNIME Server REST API documentation available at https://<your-server>/knime/rest/doc/index.html does not work anymore in installations based on TomEE 7.0.5. If you need this functionality, please install a fresh KNIME Server, which is based on TomEE 8.0.3 (or move directly to Tomcat). This does not affect the per-workflow Swagger pages (right-click in KNIME Explorer → Show API Definition), which continue to work on all TomEE versions.
In order to upgrade a TomEE-based KNIME Server, TomEE needs to be at least version 7.0.5. Older versions are not supported, and require a fresh installation.

Prerequisites

Server

  • This guide assumes you already have either KNIME Server 4.10.x or KNIME Server 4.11.x running. If you wish to update from a previous version you must study the intermediate update guides to identify configuration changes that may apply, and you may wish to consider a fresh installation in order to update the underlying Tomcat application server.

Client & Executor

  • KNIME Analytics Platform clients need at least KNIME ServerSpace 4.0 otherwise it will not be able to talk to the server.

  • KNIME Server 4.11 requires at least KNIME Analytics Platform 4.2 as executor with the corresponding KNIME Executor Connector version 4.11 installed on the server side.

  • New KNIME WebPortal features are available with any supported web browser.

Backup

Even though all the data on the server should be preserved during the update it is highly recommended to create a backup of all important data (see KNIME Server Administration Guide for backup information).

Application server

Backup existing configuration

Before performing an update, make a backup of <tomcat-folder>/conf/Catalina/localhost/knime.xml. The name of the file might be different if the .war file had a different name (e.g. com.knime.enterprise.server.xml or ROOT.xml).

Failure to backup this file will result in scheduled jobs failing to run since it contains the secret key that is used to encrypt the schedule.

Also make sure to create a backup of <tomcat-folder>/conf/server.xml.

  1. Download the updated war file war-4.11.x.y.war from the download-page. Make sure you use the correct file, depending on whether your KNIME Server is based on Tomcat or TomEE.

  2. Stop KNIME Server.

  3. Delete the existing folder <tomcat-folder>/webapps/knime (or similar if the .war file had a different name).

  4. Copy the new war-4.11.x.y.war over the existing war file (usually <tomcat-folder>/webapps/knime.war). Please note that it needs to be the same file name as this is what is referred to as context in the application.

  5. Check if the knime.xml file that you backed up before still exists. If not copy the backup to the previous location.

  6. Copy the downloaded file knime-tomcat.jar to <tomcat-folder>/lib

  7. Follow the steps below on updating the KNIME Server Executor.

  8. Start Tomcat.

Note that it is not possible to run multiple KNIME Server instances in one Tomcat installation.

KNIME Executor

Installing a new KNIME Executor

KNIME Server 4.11 requires KNIME Analytics Platform 4.2 or later. Since an update from KNIME Analytics Platform 4.1 to 4.2 isn’t possible you have to create a new installation of KNIME Analytics Platform 4.2 and install the KNIME Executor connector version 4.11 into it. The reason for this are an updated Eclipse version as well as an upgrade of the bundled JRE.

To install KNIME Analytics Platform 4.2 from scratch you will need to run it as the user that owns the Installation.

Please refer to the KNIME Server Administration Guide for more details about installing a KNIME Executor.

Due to a hotfix release, it is currently necessary to update the KNIME Executor right after performing the installation, as some important fixes are not in the release build. See below for how to perform an update.

You must update the KNIME Server Executor to at least version 4.2 and the KNIME Executor connector to version 4.11.

If possible, start the executor installation in graphical mode (using the correct installation user!). If you have internet access, go to File → Update KNIME… and it will suggest updating to the new version. Follow the steps in the wizard. If you don’t have direct internet access, you must download the zipped update sites from the commercial downloads page (contact the KNIME support team if you don’t have access to this page). Then register the ZIP files in File → Preferences → Install/Update → Available Software Sites and go to File → Update KNIME…

Under Linux, if you cannot start the graphical user interface, you can use the update-executor.sh script that is in the root of the executor installation.

Due to a bug, it is currently necessary to remove the lines referring to client preferences from the knime.ini file before running the update script. After the update, they can be added again. This will be fixed in the next bugfix release of KNIME Server.

This affects the following lines:

-profileLocation
http://<server-address>/knime/rest/v4/profiles/contents
-profileList
<list of profiles>

Call the script on the command line and provide a list of update sites that contain the new versions of the installed extensions and all installed extension will be updated (given that an update is available):

./update-executor.sh https://update.knime.com/analytics-platform/4.2

If you get error messages when executing update-executor.sh about missing "installable units", make sure that you have provided all necessary update sites.

knime.ini file

In case you want to use custom settings from you old KNIME Executor’s knime.ini file, please copy only the lines following -vmargs to the new knime.ini. The only exception to this are the four lines related to Server Managed Customizations (-profileLocation…​), in case those are set. The remaining lines above -vmargs contain paths to resources that have been changed in the new release, so they are not compatible.

server.xml file

For fresh installations, please note that a server.xml file (<tomcat-folder>/conf/server.xml) from an older installation cannot be copied over due to some larger changes. Any custom changes applied to your existing server.xml need to be manually copied over to the new server.xml. For upgrades the existing server.xml continues to work, and does not need any adjustments.

KNIME Server license

The KNIME Server license continues working with the new Server. If you haven’t received a license file or if it is not working correctly, please contact KNIME support or your dedicated KNIME account manager.

Enabling execution via Apache Qpid

KNIME Server 4.11 introduces an integrated message broker for communication between KNIME Server and KNIME Executor, based on Apache Qpid (https://qpid.apache.org/). This can be thought of as using an architecture that corresponds closely to distributed KNIME Executors, but runs on a single host, which is used by both KNIME Server and KNIME Executor.

Unlike distributed KNIME Executors, it is not necessary to install a separate message broker such as RabbitMQ. Instead, Apache Qpid is bundled as part of all KNIME Server installations. New installations of KNIME Server use Qpid by default. To switch your current installation to Qpid, changes are required in two locations:

Application Server

Activate queue in knime-server.config. The file can be found in <server-repository>/config/knime-server.config. Set the parameter

com.knime.enterprise.executor.embedded-broker= to true

In addition, comment out the line com.knime.server.executor.knime_exe= by placing a # character at the start of the line.

KNIME Executor

In the KNIME Executor installation directory, add the following line to the knime.ini file, anywhere after the -vmargs line:

-Dcom.knime.enterprise.executor.msgq=amqp://knime:20knime16@localhost/

Unlike the RMI-based execution, using the Qpid message broker requires to startup KNIME Server and KNIME Executor separately. For starting up KNIME Server, you can still use the startup.sh/.bat file located in <tomcat-folder>/bin. For starting the Executor, you can use <executor directory>/start-executor.sh (Linux) or <executor directory>/start-executor.bat (Windows)

In case the Executor startup script is not present, you can start the Executor from command line by running

./knime -nosplash -consolelog -application com.knime.enterprise.slave.KNIME_REMOTE_APPLICATION

For setting up the KNIME Server service, please follow the steps outlined in the KNIME Server Installation Guide. Installing the Executor as a service follows the same procedure as described for distributed KNIME Executors.

For increased security, we recommend to run both services by different users.

Preferences file

Note that when switching from RMI to Qpid, the old preferences.epf file in ./workflow_repository/config is not used anymore. In order to set preferences on the Executor (e.g. database drivers or Python configuration), please use a preference profile as described in the KNIME Server Admin Guide.

Server temp directory

Switching to Qpid also uses a different temp directory than RMI. On Linux, this defaults to /tmp, which might have size restrictions. In order to manually set a different temp directory, please add the following to the knime.ini file of the KNIME Executor, anywhere below the -vmargs line: -Djava.io.tmpdir=/path/to/tmpdir

Qpid port

In some environments, it is necessary to explicitly open a port for connections to Qpid on localhost (even though Server and Executor are running on the same host). By default, the port is 5672. This can be changed in the knime-server.config file.

Server host name in /hosts

By default, the host name of the machine is available in /etc/hosts/. If this is not the case, KNIME Server will not work using Qpid. This can be resolved by adding the server’s host name to /etc/hosts/, e.g. 127.0.0.1 <hostname>.

Changelog (KNIME Server 4.11)

KNIME Server 4.11.2 (released September 28, 2020)

Enhancement

  • [SRV-3153] - Check during server startup if a tomee.war is used in a Tomcat server (and vice versa)

  • [SRV-3154] - Update COMMUNITY mount point to use OAuth

Bugfixes

  • [SRV-3091] - Social Workflow Repository not reachable anymore

  • [SRV-3135] - Mail session is not forwarded to server properly

  • [SRV-3136] - Cannot update the executor from command line due to -profileLocation

  • [SRV-3148] - Authenticator may throw NullpointerException if token verification fails

  • [WEBP-530] - Nuxt routing depends on final '/'

  • [WEBP-536] - File upload: old webportal displays file location with object store path

  • [WEBP-541] - Workflows with execute-permission only can’t be executed

KNIME Server 4.11.1 (released August 26, 2020)

Enhancement

  • [SRV-2952] - Propagate JWT Parsing Errors to the log

  • [SRV-2979] - Suppress warnings about missing metainformation for simple files during workflow group upload

  • [SRV-3001] - Save workflow summary together with the job

  • [SRV-3064] - Improve Explorer perfomance for many jobs

  • [SRV-3100] - Allow knime.system.default to be used in Configuration Options

  • [SRV-3110] - Increase maximum number of in-memory jobs in local queue-based executor

Bugfixes

  • [SRV-2869] - Server Executor dialog does not scale

  • [SRV-3035] - "Reset workflow" gets checked when I select a workflow group in "deploy to server"

  • [SRV-3039] - OIDC: nullpointer in authenticator

  • [SRV-3054] - No different font for disabled schedules visible via REST connection to the server

  • [SRV-3069] - Potential deadlock when modifying scheduled job on old workflows

  • [SRV-3078] - OAuth: admin rights missing in Analytics Platform

  • [SRV-3088] - OpenAPI index page not loaded properly

  • [SRV-3095] - Installer does not contain the files to start KNIME Server as service on Windows

  • [SRV-3097] - Repository items named 'services' are not accessible any more

  • [SRV-3113] - Authentication Tokens of jobs lose their validity after server restart

  • [SRV-3117] - Reading files from the server repository randomly fails with 403

  • [SRV-3126] - Credentials Configuration isn’t filled with Server Login

  • [SRV-3130] - Embedded user database from pre-4.11 installations is not readable any more

KNIME Server 4.11.0 (released July 13, 2020)

Enhancement

  • [SRV-2033] - Dialogs of DB nodes not available in remote workflow editor

  • [SRV-2154] - Permissions for scheduled jobs in REST interface

  • [SRV-2155] - Permissions for scheduled jobs in KNIME Explorer

  • [SRV-2534] - Allow to reserve executors for exclusive use

  • [SRV-2562] - Improve message on forbidden symbols in mountID in server installer

  • [SRV-2580] - Allow re-connecting to existing jobs after server restart

  • [SRV-2600] - Add max core setting to preferences.epf.template

  • [SRV-2639] - Provide versions of server components in REST interface

  • [SRV-2696] - Enable dynamic port configuration in remote workflow editor

  • [SRV-2712] - Executor should retry connections to message queue

  • [SRV-2733] - Port for embedded message queue should be configurable

  • [SRV-2798] - [OAuth] Webportal landing page when using OIDC

  • [SRV-2807] - Improve job logging

  • [SRV-2820] - Dialog option to keep or change history when moving/copying workflows

  • [SRV-2821] - Add possibility to map JWT claims to groups

  • [SRV-2823] - Reuse OAuth Configuration From AuthenticationValve in KNIME Server

  • [SRV-2824] - Automatic OIDC endpoint discovery

  • [SRV-2830] - Define multiple queues in server config

  • [SRV-2871] - Prevent loss of OIDC configuration when war is updated

  • [SRV-2904] - Remove Glassfish support from KNIME Server Client

  • [SRV-2920] - Add option to force-enable "Create Snapshot"

  • [SRV-2921] - Add option to force-enable "Reset Workflow before upload"

  • [SRV-2922] - Change default executor starting port

  • [SRV-2926] - Installer should set path to embedded queue in executor knime.ini

  • [SRV-2927] - Add new template for server managed customizations specific to executors

  • [SRV-2928] - Remove preferences.epf.template from installer

  • [SRV-2930] - Fix behavior of Use REST option

  • [SRV-2932] - Show warning when using EJB mountpoints

  • [SRV-2938] - Extract user and group information for executor reservation

  • [SRV-2940] - Executor using embedded message queue should get as many core tokens as available

  • [SRV-2948] - Make table view chunk size editable and use smaller default value

  • [SRV-2958] - Do not send emails during Remote Job Edits

  • [SRV-2962] - 'Workflow summary' endpoint for jobs

  • [SRV-2963] - Default.epf template should use REST mountpoint

  • [SRV-2971] - New REST endpoint with executor information

  • [SRV-2977] - Add job IDs to all job related messages

  • [SRV-2982] - Executor should have a start script

  • [SRV-2985] - Server client should check for EJB support

  • [SRV-2988] - Improve UI of EJB warning dialog

  • [SRV-3002] - Support String claim as group-mapping-claim

  • [SRV-3013] - Allow force-reset of jobs before saving as workflow

  • [SRV-3015] - Update KNIME logo on OIDC login page

  • [SRV-3026] - Improve error message if no queue can be found due to group restrictions

  • [WEBP-454] - New WebPortal with improved layout and design

Bugfixes

  • [SRV-2352] - Improve error message if unexpected logout occurs

  • [SRV-2492] - IPv6 addresses are not correctly handled in canonical host name autodetection

  • [SRV-2718] - Closing KNIME OAuth Pop-Up (ESC Key) Will Block Port Used For Redirect

  • [SRV-2849] - Unexpected Basic Auth Pop-Up When "Opening API Definition" From the AP in the Browser Without an Authenticated Session

  • [SRV-2862] - Subnode-execution mode once set in server get’s overwritten with every job update

  • [SRV-2874] - Nullpointer Exception in AP when connecting to server stored as OAuth with OAuth disabled

  • [SRV-2883] - OAuth redirect page doesn’t show error

  • [SRV-2884] - Edit Mount Point Dialog does not remember Authentication Type

  • [SRV-2890] - Mount Point Dialog resizes when large error is displayed

  • [SRV-2892] - Wrong error message from executor when workflow cannot be loaded

  • [SRV-2899] - JavaNullPointer exception when trying to open configuration window of google updater on a remote job view

  • [SRV-2908] - Remote Job View does not refresh after an Executor restart

  • [SRV-2934] - Wrong error message when loading workflow failed in executor

  • [SRV-2941] - Email notification dialog in Call Workflow action advanced options not displayed correctly

  • [SRV-2943] - server_logs.zip sometimes contains folders with incomplete executor ID

  • [SRV-2967] - Overwriting workflow group via REST removes existing schedules

  • [SRV-2990] - Lazily loaded table rows are not updated in port table view (job view)

  • [SRV-3025] - Account settings of KNIME server are not applied on runtime

  • [SRV-3032] - Workflow pinning/reservation rules dont accept the @ character

  • [SRV-3034] - Failure to read updated server config prevents further updates

  • [SRV-3044] - Force reset workflow is ignored on temporary copies of workflows

  • [SRV-3051] - Unable to enter username in "Edit Mount Point" dialog

  • [SRV-3052] - Server sends multiple Status Emails for loops in workflows

  • [SRV-3060] - OAuth: NullPointerException when trying to refresh the access token