Read and write files
This guide covers configuring a reader or writer node once a file system is available. See Connect to a file system to set up the connection to the file system first.
Configure a reader node
- Authenticate and connect to the file system, if needed. See Connect to a file system.
- Add the reader node for the file type you want to read. See Reader and writer nodes.
- From the reader node configuration, open the file chooser panel and browse to the file to read.
- Execute the node. The content of the file appears in the node output.
Upload and read a file from the local file system
- Open a workflow in the workflow editor.
- Browse your local file system and locate the file you want to read.
- Drag and drop the file inside the workflow editor.
The file is uploaded to the space next to the workflow, and a reader node configured to read it is added to the workflow.
Read multiple files into one table
You can read multiple files from the same folder into one table with one reader node.
- Authenticate and connect to the file system, if needed. See Connect to a file system.
- Add the reader node for the file type you want to read. See Reader and writer nodes.
- From the reader node configuration, select Files in folder as the mode.
- Open the file chooser panel and browse to the folder containing the files to read.
- Optionally apply filters to exclude some of the files in the folder, for example by extension or file name.
- Optionally include files from the subfolders of the selected folder.
- Choose how to handle columns that don't match among all files:
- Let the node fail (default)
- Use the union of the columns
- Use the intersection of the columns
- Execute the node. The content of all files gets concatenated in the output table of the node.
Transform data while reading
Some reader nodes, such as the Excel Reader and the CSV Reader, allow you to transform the data while reading it. This lets you configure downstream nodes without executing the reader node first, and it speeds up execution. This is possible assuming that the file structure does not change. See Read a file with changing structure.
- Configure the reader node to read the desired file.
- In its configuration dialog, show the advanced settings.
- Optionally change or remove the limit of the scanned rows.
- In the Table Transformation section, modify the structure of the table:
- Deselect columns to filter them out.
- Use the arrows to reorder the columns.
- Change the column name or column type of each column.
- Define how to handle unknown columns with the special row
<any unknown new column>.
Read a file with changing structure
As described in Transform data while reading, the reader assumes the file structure doesn't change after closing the node dialog. Two situations override this assumption, and the node recomputes the table specification and reapplies the transformations using <any unknown new column> and the Enforce types option.
The file gets overwritten
If the file changes after configuration (for example, new columns are added), select Use new schema from the If schema changes setting. This forces the reader to compute the table specification during each execution.
INFO
The Use new schema option disables the Table Transformation options, and the node doesn't return a table specification during configuration.
The path changes via a flow variable
When a flow variable controls which file to read — for example, looping over several CSV files with different columns — the node detects the change automatically during configuration. No manual Use new schema selection is needed.
Configure a writer node
- Authenticate and connect to the file system, if needed. See Connect to a file system.
- Add the writer node for the file type you want to write. See Reader and writer nodes.
- From the writer node configuration, open the file chooser panel and navigate to the output folder.
- Either select an existing file or insert the output file name.
- Execute the node.