Process data in batches

When working with datasets larger than the available memory, you can use the batch processing capabilities of the Python Integration to handle data in smaller chunks.

Implementation pattern

To process an input table in batches, initialize a BatchOutputTable and iterate through the input. This ensures that only a small portion of the data is loaded into memory at any given time.

import knime.scripting.io as knio

# Initialize the output container
processed_table = knio.BatchOutputTable.create()

# Iterate through batches of the input table
for batch in knio.input_tables[0].batches():

    # Convert batch to a Pandas DataFrame
    df = batch.to_pandas()

    # Apply your processing logic here
    # Example: output_df = df.copy()

    # Append the processed data to the output container
    processed_table.append(df)

# Assign the final container to the output port
knio.output_tables[0] = processed_table

Performance

Batch processing is most effective when the workflow is configured to use the Columnar Table Backend.

Memory management

When using batches, the memory consumption remains constant regardless of the total table size, as only one chunk is active in the Python process at a time.