KNIME Documentation

Connect to a file system ​

Use this guide when a file handling node needs to reach a file system beyond the standard space or embedded data option: a different space, or an external service such as Amazon S3, Google Drive, or SharePoint Online. It also covers connecting explicitly to the current space or the embedded data, instead of using the standard option.

A file handling node is any node that can browse a file system — reader and writer nodes, and utility nodes such as List Files/Folders, Transfer Files, or Delete Files/Folders. See File handling.

Before you start ​

  • Know which space or external file system you're connecting to
  • For file systems needing an authenticator: have credentials ready
  • Optionally, save credentials as a Secret (see Manage and use secrets)

Generic steps to connect to a file system ​

  1. If needed, configure the authenticator node to authenticate against the file system.
  2. Add the connector node.
  3. Optionally configure the working directory and a timeout.
  4. Drag the connector's output port onto the downstream file handling node.
  5. From the file handling node configuration, browse files and directories of the connected file system.

Connect to a space ​

You can connect to the current space i.e. the space of the workflow you are editing, from any file handling node, without the need of a connector.

  1. Open the configuration of the file handling node.
  2. Open the file chooser panel.
  3. Select the Current Space tab.
  4. Browse the current space.

Use the steps below to connect explicitly to the current space, or to a different one. An explicit connection to the current space is optional. See Standard vs. connected file systems for when it's worth it. Connecting to a different space requires a connector node.

  1. Optionally add a KNIME Authenticator node, to connect as a different user or to a space on another Hub.
  2. Add a Space Connector node.
  3. Select the space you want to connect to from the list.
  4. Optionally configure the working directory.
  5. Drag the connector's output port onto the downstream file handling node.
  6. In the file handling node, open the file chooser panel and browse the connected space.

Connect to the workflow's embedded data ​

You can connect to the embedded data of the workflow you are editing from any file handling node, without the need of a connector.

  1. Open the configuration of the file handling node.
  2. Open the file chooser panel.
  3. Select the Embedded data tab.
  4. Browse the workflow's embedded data.

Follow the steps below to connect explicitly to the workflow's embedded data. See Standard vs. connected file systems for when it's worth it.

  1. Add a Workflow Data Area Connector node.
  2. Optionally configure the working directory.
  3. Drag the connector's output port onto the downstream file handling node.
  4. In the file handling node, open the file chooser panel and browse the embedded data.

Connect to an external file system ​

Amazon S3 ​

  1. Optionally add a Secrets Retriever node to fetch the credentials from the secret store. See Create a secret.
  2. Add the Amazon Authenticator node.
  3. Select the bucket's region (e.g. eu-west-1).
  4. Add the Amazon S3 Connector node.
  5. Optionally select the working directory.
  6. Drag the connector's output port onto the downstream file handling node.

Amazon S3 file system normalizes paths. Amazon S3 allows paths such as /mybucket/.././file, where ".." and "." must not be removed during path normalization because they are part of the name of the Amazon S3 object. When such a case is present you will need to uncheck Normalize paths option from the Amazon S3 Connector node.

Google Cloud Storage ​

To connect to Google Cloud Storage you need to create a project at console.developers.google.com. The specific Google API you want to use has to be enabled under APIs. After you create your Service Account, you receive a p12 key file that you can use to authenticate.

  1. Use the Google Authenticator node to authenticate against the service.
  2. Alternatively, use a Secrets Retriever node to fetch the Google credentials from the secret store. See Create a secret and Google secret type.
  3. Add the Google Cloud Storage connector node.
  4. Enter the project ID to which the Cloud Storage data belong.
  5. Optionally select the working directory.
  6. Drag the connector's output port onto the downstream file handling node.

Google Cloud Storage normalizes paths. Google Cloud Storage allows paths such as /mybucket/.././file, where ".." and "." must not be removed during path normalization because they are part of the name of the object. When such a case is present you will need to uncheck Normalize paths option from the Google Cloud Storage Connector node.

Google Drive ​

  1. Use the Google Authenticator node to authenticate against the service.
  2. Alternatively, use a Secrets Retriever node to fetch the Google credentials from the secret store. See Create a secret and Google secret type.
  3. Add the Google Drive Connector node.
  4. Optionally select the working directory.
  5. Drag the connector's output port onto the downstream file handling node.

The root directory of the Google Drive file system contains your Shared drives (if available) and the My Drive folder. The path to a Shared drive will be /shared_drive1/, while the path to your My Drive folder will be /My Drive/.

Microsoft file systems ​

The Microsoft Authenticator​ node offers OAuth authentication for Azure and Office365 clouds.

It supports the following authentication modes:

  • Interactive authentication: Performs an interactive, web browser based login by clicking on Login in the node dialog. In the browser window that pops up, you may be asked to consent to the requested level of access. The login results in a token being stored in a configurable location. The token will be valid for a certain amount of time that is defined by your Microsoft Entra ID settings.
  • Username/password authentication: Performs a non-interactive login to obtain a fresh token every time the node executes. Since this login is non-interactive and you get a fresh token every time, this mode is well-suited for deployed workflows. However, it also has some limitations. First, you cannot consent to the requested level of access, hence consent must begiven beforehand, e.g. during a previous interactive login, or by an Microsoft Entra ID directory admin. Second, accounts that require multi-factor authentication (MFA) will not work.
  • Shared key authentication (Azure Storage only): Specific to Azure Blob Storage and Azure Data Lake Storage Gen2. Performs authentication using an Azure storage account and its secret key.
  • Shared access signature (SAS) authentication (Azure Storage only): Specific to Azure Blob Storage and Azure Data Lake Storage Gen2. Performs authentication using shared access signature (SAS). For more details on shared access signatures see the Azure storage documentation.

Alternatively, you can store the credentials in the secret store and fetch them using the Secrets Retriever node. See Create a secret and Microsoft secret type.

SharePoint Online ​

  1. Use the Microsoft Authenticator node to authenticate against the service.
  2. Alternatively, fetch the credentials with the Secrets Retriever node.
  3. Add a SharePoint Connector node.
  4. In the node configuration dialog you can choose to connect to the following sites:
    • Root Site: Root site of the your organization​
    • Web URL: https URL of the SharePoint site (same as in the browser)​
    • Group site: Group site of a Office365 user group​
    • Subsite: Connects to subsite or sub-sub-site of the above​
  5. Drag the connector's output port onto the downstream file handling node.

In SharePoint, document libraries are represented as top-level folders.

Azure Blob Storage ​

  1. Use the Microsoft Authenticator node to authenticate against the service.
  2. Alternatively, fetch the credentials with the Secrets Retriever node.
  3. Add a Azure Blob Storage Connector node.
  4. Optionally select the working directory.
  5. Drag the connector's output port onto the downstream file handling node.

Azure Blob Storage file system normalizes paths. Azure Blob Storage allows paths such as /mybucket/.././file, where ".." and "." must not be removed during path normalization because they are part of the name of the object. When such a case is present you will need to uncheck Normalize paths option from the Azure Blob Storage Connector node.

Azure Data Lake Storage Gen2 ​

  1. Use the Microsoft Authenticator node to authenticate against the service.
  2. Alternatively, fetch the credentials with the Secrets Retriever node.
  3. Add a Azure Data Lake Storage Gen2 Connector node.
  4. Optionally select the working directory.
  5. Drag the connector's output port onto the downstream file handling node.

The path syntax will be UNIX-like, i.e. /mycontainer/myfolder/myfile and relative to the root of the storage.