KNIME File Handling Guide

When reading from Local File System the path syntax to use will be dependent on the system on which the workflow is executing, i.e. if Windows or UNIX operative system.

The working directory will be implicit and will correspond to the system root directory.

Please be aware that when executing a workflow on KNIME Server version 4.11 or higher the local file system access is disabled for security reasons. The KNIME Server administrators can activate it, but this is not recommended. For more information about this please refer to the KNIME Server Administration Guide.

With the Mountpoint option you will have access to KNIME mount points, such as LOCAL, your KNIME Server mount points, if any, and the KNIME Hub. You have to be logged in to the specific mountpoint to have access to it.

The path syntax will be UNIX-like, i.e. /folder1/folder2/file.txt and relative to the implicit working directory, which corresponds to the root of the mountpoint.

Please note that workflows inside the mountpoints are treated as files, so it is not possible to read or write files inside a workflow.

With the Relative to option you will have access to three different file systems:

All the paths used with the option Relative to are of the type folder/file and they must be relative paths.

In the example above you will read a .csv file from a folder data which is in:

Please note that workflows are treated as files, so it is not possible to read or write files inside a workflow.

When the workflow is executing on KNIME Server the options Relative to → Current mountpoint or Current workflow will access the workflow repository on the server. The option Relative to → Current workflow data area, instead, will access the data area of the job copy of the workflow. Please be aware that files written to the data area will be lost if the job is deleted.

This option works with a pseudo-file system that allows to access single files via URL. It supports the following URLs:

For this option, you can also set up manually a timeout parameter (in milliseconds) for reading and writing.

The URL syntax should be as follows:

Using this option you can read and write single files, but you would not be able to move and copy files or folders. However, listing files in a folder, i.e. browsing, is not supported.

With KNIME Analytics Platform version 4.5 new connectors nodes for standard file systems have been introduced. These nodes allow you to prototype workflows using a specific file system, by providing access to the standard file systems. The resulting output port allows downstream nodes to access files, e.g. to read or write, or to perform other file system operations, e.g. browse/list files, copy, move.

Local File System Connector node

This node provides access to the file system of the local machine.

In the Local File System node configuration dialog you can choose to use a custom working directory. If this option is not set, the default working directory is the home directory of the current operating system.

Mountpoint Connector node

This node provides a file system connection with access to a mountpoint, for example "LOCAL", or "My-KNIME-Hub". It can also provide a file system connection with access to the mountpoint which contains the current workflow, similar to the option Relative to > Current mountpoint.

In the Mountpoint Connector node configuration dialog you can specify the mountpoint to access.

Finally, you can specify a Working directory using a Path syntax.

Workflow Data Area Connector

This node provides a file system connection with access to the data area of the current workflow.

If you are not using this connector node, this option is equivalent to choosing Read from > Relative to > Current workflow data area.

In the Workflow Data Area Connector node configuration dialog you can specify a Working directory using a Path syntax.

The path syntax varies according to the connected file system, but in most cases it will be UNIX-like. Information on this are indicated in the respective Connector node descriptions.

Typically in the configuration dialog of the Connector node you will be able to:

Most connectors will require a network connection to the respective remote service. The connection is then opened when the Connector node executes and closed when the Connector node is reset or the workflow is closed.

It is important to note that the connections are not automatically re-established when loading an already executed workflow. To connect to the remote service you will then need to execute again the Connector node.

All the Connector nodes that do not need an external Authentication node will connect upon execution to a specific file system. This allows the downstream nodes to access the files of the remote server or file system.

The new table based reader nodes e.g. Excel Reader or CSV Reader allow you to transform the data while reading it via the Transformation tab. To do so the node analyzes the structure of the file(s) to read in the node dialog and stores the file structure and the transformation that should be applied. When the node is executed this information is used to read the data and apply the transformations before creating the KNIME data table. This allows the nodes to return a table specification during configuration of the node and not only once the node is executed, assuming that the file structure does not change. The benefits of this is that you can configure downstream nodes without executing the reader node first and improves execution speed of the node.

To change the transformation in the updated reader nodes configuration dialog, go to the Transformation tab after selecting the desired file. This tab displays every column as a row in a table that allows modifying the structure of the output table. It supports reordering, filtering and renaming columns. It is also possible to change the type of the columns. Reordering is done via drag and drop. Just drag a column to the position it should have in the output table. Note that the positions of columns are reset in the dialog if a new file or folder is selected.

If you are reading multiple files from a folder, i.e. with the option Files in folder in the Settings tab, you can also choose to take the resulting columns from Union or Intersection of the files you are reading in.

The Transformation tab provides two options which are important when dealing with changing file structures if you control the reader via flow variables.

The Enforce types option controls how columns whose type changes are dealt with. If selected, the node attempts to map to the KNIME type you configured in the transformation tab and fails if that is not possible. If unselected, the KNIME type corresponding to the new type is used whether it is compatible to the configured type or not. This might cause downstream nodes to fail if the type of a configured column has changed.

The <any unknown new column> option is a place holder for all previously unknown columns and specifies whether and where these columns should be added during execution.

As described in the Transformation tab section, the reader assumes that the structure of the file will not change after closing the node dialog. However when a reader is controlled via a flow variable the resulting table specification can change. This happens if settings that affect which data should be read are changed via a flow variable. One example is the reading of several CSV files with different columns in a loop where the path to the file is controlled by a flow variable.

During configuration the node checks if the settings have changed. If that is the case the node will not return any table specification during configuration and will analyze the new file structure with each execution. In doing so it will apply the transformations to the new discovered file structure using the <any unknown new column> and Enforce types option.

As described in the Transformation tab section, the reader assumes that the structure of the file will not change after closing the node dialog. However the structure of the file can change (e.g. by adding additional columns) if it gets overwritten. In this case you need to enable the Support changing file schemas option on the Advanced Settings tab. Enabling this option forces the reader to compute the table specification during each execution thus adapting to the changes of the file structure.

In the Advanced Settings tab you can also check the option Append path column under Path column pane. If this option is checked, the node will append a path column with the provided name to the output table. This column will contain for each row which file it was read from. The node will fail if adding the column with the provided name causes a name collision with any of the columns in the read table. This will allow you to then distinguish the file from which a specific row is read from in case you are reading multiple files and are concatenating them into a single data table.

The Image Writer (Table) node is also available with a table based input.

You can give to this node an input data table where a column contains, in each data cell, images. Image Writer (Table) is able to process the images contained in a column of the input data table and write them as files in a specified output location.

The Path to String node converts a path data cell to a string data cell. By default the node will create a String representation of the path that can be used in a subsequent node that still requires the old string or URI type, e.g. JSON Reader.

If you only want to extract the plain path you can disable the _Create KNIME URL for 'Relative to' and 'Mountpoint' file system option in the Path to String node configuration dialog.

Similar to the Path to String node, the Path to String (Variable) node converts the selected path flow variables to a string variable.

If you want to use a node that requires a URI cell you can use the String to URI node after the Path to String node.

In order to convert a string path to the path type you can use the String to Path node. The node has a dynamic File System port that you need to connect to the corresponding file system if you want to create a path for a connected file system such as Amazon S3.

Similar to the String to Path node the String to Path (Variable) node converts a string flow variable into a path flow variable.

You can identify the new file handling nodes by the three dots for the dynamic port on the node icon that allow you to connect to a connected file system.

Another way to identify a new file handling node is by the Input location section in the node configuration dialog. Here, for nodes that have been migrated to the new file handling framework you will typically find a drop-down menu that allows you to specify the file system that should be used. The following image shows the Input location section of the configuration dialog of a typical reader node in the new and old file handling framework.

Using flow variables you can specify the location of a file to read or write automatically. Because not all nodes provided by KNIME or the Community have been migrated yet you might face the problem that the newer file nodes support the new Path type whereas older nodes still support a string flow variable which can be either a file path or a URI.

The List Files/Folders node returns a table with a list of files/folders in the new Path type. To use the Path type in a node that has not been migrated yet (e.g. in the Table Reader), you need to first convert the new Path type using the Path to String node into a string.

Some older nodes require a URI as input. To convert the Path type to an URI you first need to convert it to a string using the Path to String node and then convert the string to an URI using the String to URI node.

If you already have a Path type flow variable you can use the Path to String (Variable) node to convert the Path type flow variable into a string flow variable.

If you want to convert an existing file path or string to the new Path type you can either us the String to Path node to convert data columns or the String to Path (Variable) node to convert string variable.

Using the old file handling framework you had to use loops in order to read multiple files into a single KNIME data table. So your workflows looked like the following image.

With the new file handling framework you no longer need to use loops since the reading of multiple files into a single KNIME data table is now built into the reader nodes them self. All you have to do is to change the reading mode to Files in Folder. You can than open the Filter options dialog by clicking on the Filter options button. Here you can filter the files that should be considered e.g. based on their file extension. In addition you can specify if files in subfolders should be included or not.

With the old file handling framework, when working with remote file systems such as Amazon S3 or HDFS you had two options when reading files:

In order to write a file to a remote file system you had to first write the file to your local hard drive and then upload it to the remote file system.

With the new file handling framework you no longer need to use any additional nodes to work with files in remote file systems but simply connect the reader and writer nodes to the corresponding connector node via the dynamic ports.

When working with the nodes of the new file handling framework you no longer need to create KNIME URLs but can use the built-in convenience file systems Relative to and Mountpoint. The following table lists the relation between the KNIME URL and the two new file systems.

If you need the KNIME URL for a Relative to or Mountpoint path you can use the Path to String and Path to String (Variable) nodes with the Create KNIME URL for 'Relative to' and 'Mountpoint' file systems option enabled, as it is by default.

The functionality of the Excel Appender node has been integrated into the new Excel Writer node. The node allows you to create new Excel files with one or many spreadsheets but also to append any number of spreadsheets to an existing file. The number of sheets to write can be changed via the dynamic ports option of the node.

With KNIME Analytics Platform release 4.4.0 we:

The File Reader node now supports the new File Handling framework and can use path flow variables and connect to file systems. The File Reader node is able to read the most common text files.

The newly introduced File Reader (Complex Format) node also supports the new File Handling framework, and it is able to read complex format files. We recommend to use the File Reader (Complex Format) node only in case the File Reader node is not able to read your file.

With the File Reader node you can also enable the Support changing file schemas option in the Advanced Settings tab of its configuration dialog. This allows the node to support eventual input file structure changes between different invocations. This is instead not possible with the File Reader (Complex Format) node that does not support files changing schemas. Thus, if the File Reader (Complex Format) node is used in a loop, you should make sure that all files have the same format (e. g. separators, column headers, column types). The node saves the configuration only during the first execution. Alternatively, the File Reader node can be used.