KNIME Server Release Notes and Update Guide


Release Notes

KNIME Server 4.12 is a feature release of the 4.x release line. All clients that have worked with KNIME Server 4.11 will continue to work with KNIME Server 4.12 without restrictions.

To find out which version of KNIME Server you are currently running, you can check the Administration pages on the WebPortal.

New Features

For a list that includes the new Analytics Platform 4.3 features see here.

Highlighted new functionality is:

A detailed changelog for KNIME Server 4.12 is also available.

Java 11 support

KNIME Server 4.12 adds support for Java 11. While it is possible to continue using Java 8 for existing installations, we recommend the use of Java 11 for fresh installations. Note: Java 11 support is only confirmed for Tomcat-based installations of KNIME Server. TomEE-based installations should continue to use Java 8

Configuration for email notifications

Configuration for email notifications sent by KNIME Server has moved from the knime.xml file to knime-server.config. In order for email notifications to still work, please migrate your settings to the new location. This can be done at runtime, it is not necessary to restart KNIME Server.

Revised job states

Job state handling has been revised to be more consistent. Among other things, it is now easier to identify jobs that have failed. See here for a list of all job states.

New configuration options

Various new configuration options were added to KNIME Server to improve usability:

com.knime.server.job.max_schedule_failures=<number>: It is now possible to configure the number of attempts to create a job before a schedule is disabled. Set to -1 to deactivate auto-disabling of schedules. This was previously hardcoded to 3.

com.knime.server.gateway.timeout=<duration with unit, e.g. 30s, 1m>: Allows increasing the timeout for loading jobs in the Remote Editor. This was previously hardcoded to 20s

com.knime.server.action.job.force_discard_on_failure=<true|false> com.knime.server.action.job.force_discard_on_success=<true|false> com.knime.server.action.job.enable_discard_checkboxes=<true|false>: Automatic discarding of jobs after execution is now more flexible. Previously, it was only possible to discard jobs after successful execution. We have now added an option to also discard on failure. Default values for discard on success/failure can be configured using the first two options listed above. The third option allows the admin to enforce these settings by disabling the checkbox that would change the behavior.

Local file system access by KNIME workflows

A growing number of KNIME nodes are being revised to use a new shared framework for file access. More nodes will follow over the next releases, so that eventually the file access of all KNIME nodes uses the shared framework.

When executing on KNIME Server, a preference controls whether those nodes may access the local file system of the KNIME Server Executor or not. With this release, local file system access is disallowed by default (previously it was allowed).

To allow local file system access (not recommended), you can add the following line to the customization profile used by your KNIME Server Executor(s):

/instance/org.knime.filehandling.core/allow_local_fs_access_on_server=true

This preference affects all nodes that are part of the revised file handling framework. Old nodes that have not yet been ported to the new framework are not affected by this setting.

Update

This document guides you through the steps that are needed to update an existing KNIME Server installation from version 4.11 to 4.12 or apply a bugfix for version 4.12. You will find a complete guide to installing KNIME Server in the KNIME Server Installation Guide, and a complete description of all configuration options in the KNIME Server Administration Guide. If you have any questions or need assistance with the update process, please contact your dedicated KNIME support specialist.

For the sake of readability, this documentation uses <apache-tomcat>/ to refer to the installation directory of the server application. Apart from the directory name, the location will be the same on TomEE-based folders.

New installations of KNIME Server are based on Apache Tomcat, rather than Apache TomEE which was used in previous releases. It is not possible to change your installation from TomEE to Tomcat. Instead, if you plan to use Tomcat, please perform a fresh installation of KNIME Server. In addition, we continue to offer a TomEE based installer for the time being. The TomEE version for this has been updated to version 8.0.3 (from TomEE 7.0.5). If you want to update to the newest TomEE version, it is also necessary to perform a fresh installation of KNIME Server as upgrades are not supported. I.e., the remainder of this document assumes that you update your KNIME Server to version 4.11, while maintaining TomEE 7.0.5 as application server. KNIME Executors are not affected by these changes, and can be upgraded following the standard procedure.
Due to some updated libraries, the KNIME Server REST API documentation available at https://<your-server>/knime/rest/doc/index.html does not work anymore in installations based on TomEE 7.0.5. If you need this functionality, please install a fresh KNIME Server, which is based on TomEE 8.0.3 (or move directly to Tomcat). This does not affect the per-workflow Swagger pages (right-click in KNIME Explorer → Show API Definition), which continue to work on all TomEE versions.
In addition, the newly-added configuration page on the KNIME WebPortal is read-only for TomEE 7.0.5 Servers. While configuration values are shown, you will not be able to modify them.
In order to upgrade a TomEE-based KNIME Server, TomEE needs to be at least version 7.0.5. Older versions are not supported, and require a fresh installation.

Prerequisites

Server

  • This guide assumes you already have either KNIME Server 4.11.x or KNIME Server 4.12.x running. If you wish to update from a previous version you must study the intermediate update guides to identify configuration changes that may apply, and you may wish to consider a fresh installation in order to update the underlying Tomcat application server.

Client & Executor

  • KNIME Analytics Platform clients need at least KNIME ServerSpace 4.0 otherwise it will not be able to talk to the server.

  • KNIME Server 4.12 requires at least KNIME Analytics Platform 4.3 as executor with the corresponding KNIME Executor Connector version 4.12 installed on the server side.

  • New KNIME WebPortal features are available with any supported web browser.

Backup

Even though all the data on the server should be preserved during the update it is highly recommended to create a backup of all important data (see KNIME Server Administration Guide for backup information).

Application server

Backup existing configuration

Before performing an update, make a backup of <apache-tomcat>/conf/Catalina/localhost/knime.xml. The name of the file might be different if the .war file had a different name (e.g. com.knime.enterprise.server.xml or ROOT.xml).

Failure to backup this file will result in losing access to your scheduled jobs since it contains the secret key that is used to encrypt the schedule.

Also make sure to create a backup of <apache-tomcat>/conf/server.xml.

  1. Download the updated war file war-4.12.x.y.war from the download-page. Make sure you use the correct file, depending on whether your KNIME Server is based on Tomcat or TomEE.

  2. Stop KNIME Server.

  3. Delete the existing folder <apache-tomcat>/webapps/knime (or similar if the .war file had a different name).

  4. Copy the new war-4.12.x.y.war over the existing war file (usually <apache-tomcat>/webapps/knime.war). Please note that it needs to be the exact same file name as this is what is referred to as context in the application.

  5. Check if the knime.xml file that you backed up before still exists. If not copy the backup to the previous location.

  6. Copy the downloaded file knime-tomcat.jar to <apache-tomcat>/lib. Delete the existing knime-tomcat.jar file that was already in that directory.

  7. Follow the steps below on updating the KNIME Server Executor.

  8. Start Tomcat.

Note that it is not possible to run multiple KNIME Server instances in one Tomcat installation.

With KNIME Server 4.12 the email configuration moved from the knime.xml to the <server repository>/config/knime-server.config, so any existing custom configuration must be moved. More information concerning the email configuration can be found in the KNIME Server Administration Guide

KNIME Executor

Installing a new KNIME Executor

KNIME Server 4.12 requires KNIME Analytics Platform 4.3 or later. Due to some larger changes, it is currently not possible to upgrade to KNIME Analytics Platform 4.3.0 from a previous version. Therefore, you need to install KNIME Analytics Platform 4.3.0 from scratch and and install the KNIME Executor connector version 4.12 into it.

To install KNIME Analytics Platform 4.3 from scratch you will need to run it as the user that owns the Installation.

Please refer to the KNIME Server Administration Guide for more details about installing a KNIME Executor.

Upgrading using the standard procedures will be possible again from KNIME Analytics Platform 4.3.1 onwards.

knime.ini file

In case you want to use custom settings from you old KNIME Executor’s knime.ini file, please copy only the lines following -vmargs to the new knime.ini. The only exception to this are the four lines related to Server Managed Customizations (-profileLocation…​), in case those are set. The remaining lines above -vmargs contain paths to resources that have been changed in the new release, so they are not compatible.

server.xml file

For fresh installations, please note that a server.xml file (<apache-tomcat>/conf/server.xml) from an older installation cannot be copied over due to some larger changes. Any custom changes applied to your existing server.xml need to be manually copied over to the new server.xml. For upgrades the existing server.xml continues to work, and does not need any adjustments.

KNIME Server license

The KNIME Server license continues working with the new Server. If you haven’t received a license file or if it is not working correctly, please contact KNIME support or your dedicated KNIME account manager.

Enabling execution via Apache Qpid

KNIME Server 4.11 introduced an integrated message broker for communication between KNIME Server and KNIME Executor, based on Apache Qpid (https://qpid.apache.org/). This can be thought of as using an architecture that corresponds closely to distributed KNIME Executors, but runs on a single host, which is used by both KNIME Server and KNIME Executor.

Unlike distributed KNIME Executors, it is not necessary to install a separate message broker such as RabbitMQ. Instead, Apache Qpid is bundled as part of all KNIME Server installations. New installations of KNIME Server use Qpid by default. To switch your current installation to Qpid, changes are required in two locations:

Application Server

Activate queue in knime-server.config. The file can be found in <knime-server-repository>/config/knime-server.config. Set the parameter

com.knime.enterprise.executor.embedded-broker= to true

In addition, comment out the line com.knime.server.executor.knime_exe= by placing a # character at the start of the line.

KNIME Executor

In the KNIME Executor installation directory, add the following line to the knime.ini file, anywhere after the -vmargs line:

-Dcom.knime.enterprise.executor.msgq=amqp://knime:20knime16@localhost/

Unlike the RMI-based execution, using the Qpid message broker requires to startup KNIME Server and KNIME Executor separately. For starting up KNIME Server, you can still use the startup.sh/.bat file located in <apache-tomcat>/bin. For starting the Executor, you can use <executor directory>/start-executor.sh (Linux) or <executor directory>/start-executor.bat (Windows)

In case the Executor startup script is not present, you can start the Executor from command line by running

./knime -nosplash -consolelog -application com.knime.enterprise.slave.KNIME_REMOTE_APPLICATION

For setting up the KNIME Server service, please follow the steps outlined in the KNIME Server Installation Guide. Installing the Executor as a service follows the same procedure as described for distributed KNIME Executors.

For increased security, we recommend to run both services by different users.

Preferences file

Note that when switching from RMI to Qpid, the old preferences.epf file in ./workflow_repository/config is not used anymore. In order to set preferences on the Executor (e.g. database drivers or Python configuration), please use a preference profile as described in the KNIME Server Admin Guide.

Server temp directory

Switching to Qpid also uses a different temp directory than RMI. On Linux, this defaults to /tmp, which might have size restrictions. In order to manually set a different temp directory, please add the following to the knime.ini file of the KNIME Executor, anywhere below the -vmargs line: -Djava.io.tmpdir=/path/to/tmpdir

Qpid port

In some environments, it is necessary to explicitly open a port for connections to Qpid on localhost (even though Server and Executor are running on the same host). By default, the port is 5672. This can be changed in the knime-server.config file.

Server host name in /hosts

By default, the host name of the machine is available in /etc/hosts/. If this is not the case, KNIME Server will not work using Qpid. This can be resolved by adding the server’s host name to /etc/hosts/, e.g. 127.0.0.1 <hostname>.

Troubleshooting

All schedules are lost after updating

One important step during update process is the backup of the knime.xml file so that it can be copied back into the KNIME Server installation folder. The knime.xml file contains a secret key for encrypting the communication between the server and the installation folder. If this is not restored before starting the server, the server will not be able to load jobs and schedules previously created.

Access to KNIME WebPortal version > 4.11 is not possible

If you cannot access the URL of the KNIME WebPortal version > 4.11
https://<server-address>/knime/webportal/
and it fails with 500 errors, usually indicates that the update did not work as expected.

Please contact KNIME support team via support@knime.com to get this fixed.

Workflows deployed on the server cannot be executed

  • If the following error message appears:

Unable to load workflow, it was created with a future version of KNIME

The KNIME Analytics Platform client with which you are connecting to the KNIME Server and with which the server was built needs to have a lower or equal version to the KNIME Executor. If this is not the case, please ask your KNIME Server Administrator to update the KNIME Server and/or KNIME Executor to the release version of the KNIME Analytics Platform client you are using locally.

  • If a missing node is reported, instead, check if the node is already installed and if the version of the respective extension is the same in both the KNIME Analytics Platform client and the KNIME Executor.

  • Finally, if your scripting (R or Python) snippets fail, it is possible that the respective packages are missing on KNIME Server. Please refer to the respective guides to install Python and R packages. Please also make sure the user under which you install those packages is the same user running the KNIME Executor.

Connection to KNIME Executor hosts after switching to Qpid is not possible

  • Make sure the hostname of the KNIME Server is mentioned in the /etc/hosts file

  • Check that the following ports are accessible on localhost. They do not need (and in fact should not) to be accessible from other systems (usually this is the case).

    • With RMI: com.knime.server.executor.start_port (default 60100)

    • With QPid: com.knime.enterprise.executor.embedded-broker.port (default 5672)

    • With RabbitMQ: For KNIME Server and for the KNIME Executor that is setup on a separate machine (default 5672)

  • Starting with QPid, KNIME Executor needs to be started manually. Please follow the instructions in the KNIME Server Installation Guide to do so.

  • It might be necessary to change the user running the service. Please follow the instructions in the KNIME Server Installation Guide to do so.

Reading from my local file system after update to KNIME Server 4.11+ is not possible

Starting with KNIME Analytics Platform version 4.2, which is the KNIME Executor version for the KNIME Server 4.11 we are, by default, not allowing access to the local file system on the KNIME Server via the new File Util nodes. You can enable this again as described in KNIME Server Admin Guide.

Please note that this only affects nodes that are part of the revised file handling framework.

The authentication field is empty

Please update the knime-tomcat.jar as described in the Application server update section. This must be done before restarting the KNIME Server.

Changelog (KNIME Server 4.12)

KNIME Server 4.12.0 (released December 6, 2020)

Enhancement

  • [SRV-1000] - Enforce discarding (successful) jobs

  • [SRV-1346] - Update KNIME Server to be compliant with Java 11 LTS

  • [SRV-2201] - Number of failures until schedule is deactivated should be configurable

  • [SRV-2272] - Move job state logic from server to executor

  • [SRV-2408] - REST API: Allow loading of repository for a specified number of levels

  • [SRV-2695] - Add TLSv1.2 support for email notifications

  • [SRV-3071] - Allow changing server configuration via REST interface

  • [SRV-3073] - Move E-Mail configuration from knime.xml to server configuration

  • [SRV-3075] - Add meta information for server configuration to REST interface

  • [SRV-3076] - Adding a feature to set an executor to a draining state and stop it

  • [SRV-3081] - Store source of job request

  • [SRV-3082] - Definite state mapping for jobs

  • [SRV-3083] - Add log file for tracing individual jobs

  • [SRV-3086] - Add more executor information

  • [SRV-3093] - Logs should be retrievable with date range.

  • [SRV-3094] - Add reset before execution option to Call Workflow Action

  • [SRV-3120] - Let admins assign a maximum number of core tokens to executor groups

  • [SRV-3123] - RabbitMQ HA support

  • [SRV-3151] - Remove SysV init scripts

  • [SRV-3157] - Executors using RabbitMQ should get as many core tokens as available

  • [SRV-3164] - Add multiple 'My-KNIME-Hub' mountpoints via the 'Add Mountpoint' dialog

  • [SRV-3179] - Add last execution date and job ID to scheduled jobs

  • [SRV-3211] - Throttle deep repository listing requests

  • [SRV-3212] - Make repository read timeout configurable

  • [SRV-3225] - Allow concurrent triggering of scheduled jobs

  • [SRV-3228] - Update to Tomcat 9.0.39

  • [SRV-3232] - Add new server configuration option for noVNC launch script

  • [SRV-3233] - Add new REST call for launching a noVNC container

  • [SRV-3234] - Create launch script for noVNC docker container

  • [SRV-3235] - Create Dockerfile for noNVC image

  • [SRV-3247] - Option to hide folders without view permissions

  • [SRV-3287] - Increase default job load timeout to 3 minutes

  • [SRV-3289] - Make creation of workflow summary configurable

  • [WEBP-58] - Toggle “full screen” mode

  • [WEBP-464] - Add Fullscreen controls to the WebPortal UI

  • [WEBP-486] - Implement new Admin/Monitoring page routing

  • [WEBP-488] - Forward to new admin portal

  • [WEBP-497] - Documentation page

  • [WEBP-502] - WebPortal/PB File Upload API for widget rewrite

  • [WEBP-506] - Navigation bar for monitoring and admin portal

  • [WEBP-507] - Side bar component

  • [WEBP-508] - Data retrieval for schedules page in monitoring portal

  • [WEBP-510] - Data retrieval for executors overview page in monitoring portal

  • [WEBP-512] - Jobs page in monitoring portal

  • [WEBP-514] - License page for admin portal

  • [WEBP-515] - User/group pages for admin portal

  • [WEBP-516] - Config page for admin portal

  • [WEBP-517] - New REST endpoint to change server configuration

  • [WEBP-518] - Editable server config in admin portal

  • [WEBP-519] - Implement table view for admin portal

  • [WEBP-522] - Change password page

  • [WEBP-540] - Navigation bar responsive design

  • [WEBP-551] - Change default title of WebPortal to 'WebPortal'

  • [WEBP-567] - Toggle component

  • [WEBP-568] - Include table on schedules page

  • [WEBP-569] - Donut chart

  • [WEBP-570] - Include table on executors page

  • [WEBP-577] - Logs page in monitoring portal

  • [WEBP-581] - Do not show empty workflow groups in WebPortal

  • [WEBP-586] - Button to open VNC instance

  • [WEBP-587] - Deactivate legacy WebPortal by default

  • [WEBP-597] - Monitoring Permissions

  • [WEBP-611] - Set admin client type header for admin requests

  • [WEBP-612] - License page needs to display expired license

Bugfixes

  • [SRV-3138] - Timeouts in remote workflow editor are too low

  • [SRV-3169] - Script install-executor-as-service.bat fails if path contains blanks

  • [SRV-3215] - Race condition when updating job states in interactive execution

  • [SRV-3221] - Incorrect credentials warning when opening a job in Remote Workflow Editor

  • [SRV-3227] - Timeouts while swapping a job don’t mark the job as non-swappable

  • [SRV-3229] - Executor group is not set in case server is restarted

  • [SRV-3230] - Race condition between job swapping and status update

  • [SRV-3231] - Username not extracted from JWT for Hub mount points

  • [SRV-3249] - KNIME Server Installer adds mail resources even though it hasn’t been configured

  • [SRV-3252] - Deadlock in executor during job state notifications

  • [SRV-3259] - Typo in log error/exception messaging

  • [SRV-3260] - Cannot set configuration nodes via WebPortal when using OAuth

  • [SRV-3264] - Allow user to replace a connection in remote workflow editor

  • [SRV-3269] - Maximum JWT lifetime not set if parameter cannot be parsed

  • [SRV-3273] - Wrong state returned for cancelled job

  • [SRV-3278] - Invalid token on discard prohibits onClose operations of nodes

  • [SRV-3286] - Multiple error dialogs when connecting to community server

  • [SRV-3290] - Possible null pointer when OAuthAuthenticator tries to bring KAP to front

  • [WEBP-325] - Fix keyboard navigation in JobList for Firefox

  • [WEBP-441] - Missing required input on last page of executed job succeeded but have to throw an error

  • [WEBP-552] - SubMenu component throws error when re-render occurs after menu item has been selected

  • [WEBP-557] - Focus of navigation bar tabs not working

  • [WEBP-564] - Monitoring in the Menu misses hover state when using keyboard navigation

  • [WEBP-576] - Remove duplicate notifications from Schedules/Executor polling

  • [WEBP-651] - URL parameters should not be comma separated list

  • [WEBP-652] - Configuration node’s value can only be passed via WebPortal URL if the parameter name contains node ID