KNIME Documentation

File handling ​

A file system manages how data is organized, stored, and accessed — think of it as a tree, with folders as branches and files as leaves.

File handling covers every way a workflow interacts with files and folders in a file system: reading and writing data, but also browsing, listing, copying, moving, and deleting them.

A file handling node is any node that performs one of these operations — reader and writer nodes, and utility nodes such as List Files/Folders, Transfer Files, or Delete Files/Folders.

Where files can live ​

A file can live in a space, within the workflow itself (its embedded data), or in an external file system. How you reference or operate on it — reading, writing, listing, moving, or deleting — depends on where it lives and how the node connects to it.

Every file handling node connects to a file system in one of two ways:

  • Standard (default). Browse and select files directly in the space the workflow lives in, or in the workflow's embedded data.
  • Connected. Add a file system connection to the node, and browse and select files in the connected file system instead.

You can also reference a single file directly by URL, without browsing. See Reference files with URL.

Some locations, such as the current space or the embedded data, are reachable both with standard and connected options. See Standard vs connected.

Files in a space ​

You can reference files in one of the spaces you have access to. By default, file handling nodes can only access the space the workflow lives in. In that case, the file can be referenced:

  • Via a space-relative path — relative to the space root, independent of which workflow references it.
  • Via a workflow-relative path — relative to the workflow's own location within the space.

After selecting a file from the file chooser panel, the node configuration shows the path to the file. The path is:

  • knime://knime.space/path-to file for space-relative paths.
  • knime://knime.workflow/path-to-file for workflow-relative paths.

Using the Space Connector node before the file handling node, you can select files from any of the spaces you have access to. In this case, the path to the file is always relative to the connected space.

See how to Connect to a space.

Files within the workflow (embedded data) ​

Every workflow has an embedded data folder where data can be read and written. The embedded data within a workflow is copied, moved and deleted with the workflow itself.

Each run of the workflow works on its own copy of the embedded data. Changes made while a run executes affect only that run's copy, not the embedded data stored with the workflow itself, and are lost once the run is deleted.

Using the Workflow Data Area Connector node you can create a file system connection for a file handling node. Since the embedded data is already accessible with the standard connection of the file handling node, this node is useful only when the embedded data has subfolders and you want downstream nodes to browse from a specific one, instead of the root. See Standard vs. connected file systems.

See how to Connect to the workflow's embedded data.

Files in a connected file system ​

You can connect to a different file system using one of the connector nodes before a file handling node. In the connector nodes configuration dialog it is possible to configure the most convenient working directory.

See Connector nodes for a complete list of connectors and authenticator nodes.

Standard vs. connected file systems ​

When a file handling node has an incoming file system connection, the Connected option replaces the Standard ones — you can only browse the connected file system from the file chooser panel.

The current space and the workflow's embedded data are accessible with or without a connector. While the final result is the same, the resulting path to the file is different. See Path syntax for path data cells and flow variable.

Use a connector node to connect to the current space or the embedded data if you plan to change the file system later. For example, test a workflow against the embedded data, then connect it to a remote file system before deployment: you only need to reconfigure the connector node, leaving the downstream file handling node as they are.

A connector node also lets you set a working directory. This doesn't change the resulting path, it only makes browsing more convenient, for example when several file handling nodes downstream browse from the same starting point instead of the file system root.

Most connectors require a network connection to the respective remote service. The connection is opened when the Connector node executes and closed when the Connector node is reset or the workflow is closed. The connections are not automatically re-established when loading an already executed workflow. To connect to the remote service, you need to execute again the Connector node. Read the guide on how to connect to a file system.

Reference files with URL ​

With the standard file system you can also access single files by entering their URL. The URL syntax should be as follows:

  • scheme:[//authority]path[?query][#fragment]​
  • The URL must be encoded, e.g. spaces and some special characters that are reserved, as ?. To encode the URL you can use any available online URL encoder tool.

It supports the following protocols:

  • knime:// to directly type workflow- and space-relative paths. See Files in a space
  • http(s):// if authentication is not needed
  • ssh:// if authentication is not needed
  • ftp:// if authentication is not needed

You can also set up manually a timeout parameter (in milliseconds) for reading and writing.

Using the URL you can read and write single files, but you would not be able to move and copy files or folders. Listing files in a folder, i.e. browsing, is not supported.

Path syntax for path data cells and flow variables ​

Files and folders can be uniquely identified via their path syntax within a file system. Within a workflow, such a path is represented via a path type. A path type consists of three parts:

  • Type: Specifies the file system type: RELATIVE, CUSTOM_URL, or CONNECTED.

  • Specifier: Optional string that contains additional file system specific information e.g. relative to space or workflow.

  • Path: Specifies the location within the file system with the file system specific notation.

Path examples are:

  • Relative
    • (RELATIVE, knime.workflow, file1.csv)
    • (RELATIVE, knime.space, file1.csv)
  • Custom URL
    • (CUSTOM_URL, , \https://server:443/my%20example?query=value#frag)
    • (CUSTOM_URL, 10000, https://my-website.com/folder/file%201.csv)
  • Connected
    • (CONNECTED, amazon-s3:eu-west-1, /mybucket/file1.csv)
    • (CONNECTED, microsoft-sharepoint, /myfolder/file1.csv)
    • (CONNECTED, ftp:server:port, /home/user/file1.csv)
    • (CONNECTED, ssh:server:port, /home/user/mybucket/file1.csv)
    • (CONNECTED, http:server:port, /file.asp?key=value)
    • (CONNECTED, knime-relative-workflow-data, /table.csv)
    • (CONNECTED, knime-relative-space, /files/my-table.csv)
    • (LOCAL, , \\fileserver\file1.csv)
    • (LOCAL, , /home/user)

A path can be packaged into either a path data cell (i.e., a cell in a data table column that holds a path) or a path flow variable. By default, the path data cell within a data table only displays the path part. This behavior can be changed in the data renderer option of the output table.

Key concepts ​

  • File system: a tree of folders and files that a workflow can read from and write to, such as a space, the embedded data, or an external storage service.
  • File handling node: any node that reads, writes, lists, moves, copies, or deletes files, including reader, writer, and utility nodes.
  • Embedded data: the folder stored inside a workflow. It's copied, moved, and deleted together with the workflow, and each run works on its own copy.
  • Standard file system: the default option of a file handling node. It browses the space the workflow lives in, or the workflow's embedded data.
  • Connected file system: a file system that a connector node provides to a file handling node. It replaces the standard options in the file chooser.
  • Connector node: a node that opens a connection to a file system, such as Space Connector or Amazon S3 Connector. Some connectors need an authenticator node first.
  • Space-relative and workflow-relative paths: paths that start from the root of the space (knime://knime.space/) or from the workflow's own location (knime://knime.workflow/).
  • Working directory: the folder a connector node opens in when you browse. It makes browsing easier but doesn't change the resulting path.
  • Path type: how a workflow represents a file location, made of a type, a specifier, and a path. It's stored in a path data cell or a path flow variable.

Next steps ​