KNIME Documentation

Tables ​

class knime.extension.Table ​

This class serves as public API to create KNIME tables either from pandas or pyarrow. These tables can than be sent back to KNIME. This class has to be instantiated by calling either from_pyarrow() or from_pandas()

method __getitem__(slicing: Union[Union[slice, List[int], List[str]], Tuple[Union[slice, List[int], List[str]], slice]]) → _TabularView ​

Creates a view of this Table by slicing rows and columns. The slicing syntax is similar to that of numpy arrays, but columns can also be addressed as index lists or via a list of column names.

Notes

The syntax is [column_slice, row_slice]. Note that this is the exact opposite order than in the deprecated scripting API's ReadTable.

Parameters ​
  • slicing : slice, int, str, list, or tuple — Either a column selector — a column index, a column name, a slice object, a list of column indices, or a list of column names — or a (column_slice, row_slice) tuple, where row_slice is a slice object describing which rows to use.
Returns ​

TabularView — A _TabularView representing a slice of the original Table.

Examples ​
python
>>> row_sliced_table = table[:, :100] # Get the first 100 rows
... column_sliced_table = table[["name", "age"]] # Get all rows of the columns "name" and "age"
... row_and_column_sliced_table = table[1:5, :100] # Get the first 100 rows of columns 1,2,3,4
method __init__() ​

Do not use this constructor directly, use the from_ methods instead.

Raises ​
  • RuntimeError — Always raises a RuntimeError.
method append(other: Union[_Columnar, Sequence[_Columnar]]) → _ColumnarView ​

Append another _Columnar object (e.g. Table, Schema) or a sequence of _Columnar objects to the current _Columnar object.

Parameters ​
  • other : Union[_Columnar, Sequence[_Columnar]] — The _Columnar object or a sequence of _Columnar objects to be appended.
Returns ​

_ColumnarView — A _ColumnarView object representing the current _Columnar object after the append operation.

method batches() → Iterator[Table] ​

Returns a generator over the batches in this table. A batch is part of the table with all columns, but only a subset of the rows. A batch should always fit into memory (max size currently 64mb). The table being passed to execute() is already present in batches, so accessing the data this way is very efficient.

Returns ​

generator — A generator object that yields batches of the table.

Examples ​
python
>>> output_table = BatchOutputTable.create()
... for batch in my_table.batches():
...     input_batch = batch.to_pandas()
...     # process the batch
...     output_table.append(Table.from_pandas(input_batch))
property column_names → list ​

Get the names of the columns in a dataset.

staticmethod from_pandas(data: pandas.DataFrame, sentinel: Optional[Union[str, int]] = None, row_ids: str = 'auto') ​

Factory method to create a Table given a pandas.DataFrame. The index of the data frame will be used as RowKey by KNIME.

Examples ​
python
>>> Table.from_pandas(my_pandas_df, sentinel="min")
Parameters ​
  • data : pandas.DataFrame — A pandas DataFrame.
  • sentinel : str — Interpret the following values in integral columns as missing value: - "min": min int32 or min int64 depending on the type of the column - "max": max int32 or max int64 depending on the type of the column - a special integer value that should be interpreted as missing value
  • row_ids : (keep, generate, auto) — Defines what RowID should be used. Must be one of the following values: - "keep": Keep the DataFrame.index as the RowID. Convert the index to strings if necessary. - "generate": Generate new RowIDs of the format f"Row{i}" where i is the position of the row (from 0 to length-1). - "auto": If the DataFrame.index is of type int or unsigned int, use f"Row{n}" where n is the index of the row. Else, use "keep".
Returns ​

Table — The created Table object.

staticmethod from_pyarrow(data: pyarrow.Table, sentinel: Optional[Union[str, int]] = None, row_ids: str = 'auto') ​

Factory method to create a Table given a pyarrow.Table.

All batches of the table must have the same number of rows. Only the last batch can have less rows than the other batches.

Examples ​
python
>>> Table.from_pyarrow(my_pyarrow_table, sentinel="min")
Parameters ​
  • data : pyarrow.Table — A pyarrow.Table
  • sentinel : str — Interpret the following values in integral columns as missing value: _ "min" min int32 or min int64 depending on the type of the column _ "max" max int32 or max int64 depending on the type of the column * a special integer value that should be interpreted as missing value
  • row_ids : str — Defines what RowID should be used. Must be one of the following values: _ "keep": Use the first column of the table as RowID. The first column must be of type string. _ "generate": Generate new RowIDs of the format f"Row{i}" where i is the position of the row (from 0 to length-1). _ "auto": Use the first column of the table if it has the name "<RowID>" and is of type string or integer. _ If the "<RowID>" column is of type string, use it directly _ If the "<RowID>" column is of an integer type use f"Row{n} where n is the value of the integer column. _ Generate new RowIDs ("generate") if the first column has another type or name.
Returns ​

pyarrow.Table — The created Table instance.

method insert(other: _Columnar, at: int) → _Columnar ​

Insert a column or another _Columnar object (e.g. Table, Schema) into the current _Columnar object at a specific position.

Parameters ​
  • other : _Columnar` or `Column — The column or _Columnar object to be inserted.
  • at : int — The index at which the insertion should occur.
Returns ​

_Columnar — The _Columnar object after the insertion.

Raises ​
  • TypeError — If other is not of type _Columnar or Column.

Notes

The insertion is done in-place, meaning the current _Columnar object is modified.

property num_columns → int ​

Get the number of columns in the dataset.

property num_rows ​
method remove(slicing: Union[str, int, List[str]]) ​

Implements remove method for Columnar data structures. The input can be a column index, a column name or a list of column names.

If the input is a column index, the column with that index will be removed. If it is a column name, then the first column with matching name is removed. Passing a list of column names will filter out all (including duplicate) columns with matching names.

Parameters ​
  • slicing : int | list | str — Can be of type integer representing the index in column_names to remove. Or a list of strings removing every column matching from that list. Or a string of which first occurrence is removed from the column_names.
Returns ​

A View missing the columns to be removed.

Raises ​
  • ValueError: If no matching column is found given a list or str.
  • IndexError: If column is accessed by integer and is out of bounds.
  • TypeError: If the key is neither an integer nor a string or list of strings.
property schema → ks.Schema ​

The schema of this table, containing column names, types, and potentially metadata

property shape → Tuple[int, int] ​
method to_batches() → Iterator[Table] ​

Alias for Table.batches()

method to_pandas(sentinel: Optional[Union[str, int]] = None) → pandas.DataFrame ​

Access this table as a pandas.DataFrame.

Parameters ​
  • sentinel : str or int — Replace missing values in integral columns by the given value. It can be one of the following: _ "min" min int32 or min int64 depending on the type of the column _ "max" max int32 or max int64 depending on the type of the column * An integer value that should be inserted for each missing value
method to_pyarrow(sentinel: Optional[Union[str, int]] = None) → pyarrow.Table ​

Access this table as a pyarrow.Table.

Parameters ​
  • sentinel : str or int — Replace missing values in integral columns by the given value, which can be one of the following: - "min": minimum value of int32 or int64 depending on the type of the column - "max": maximum value of int32 or int64 depending on the type of the column - An integer value that should be inserted for each missing value

class knime.extension.BatchOutputTable ​

An output table generated by combining smaller tables (also called batches).

Notes

  • All batches must have the same number, names and types of columns.
  • All batches except the last batch must have the same number of rows.
  • The last batch can have fewer rows than the other batches.
  • This object does not provide means to continue to work with the data but is meant to be used as a return value of a Node's execute() method.
method __init__() ​
method append(batch: Union[Table, pandas.DataFrame, pyarrow.Table, pyarrow.RecordBatch]) → None ​

Append a batch to this output table. The first batch defines the structure of the table, and all subsequent batches must have the same number of columns, column names and column types.

Notes

Keep in mind that the RowID will be handled according to the "row_ids" mode chosen in BatchOutputTable.create.

staticmethod create(row_ids: str = 'keep') ​

Create an empty BatchOutputTable

Parameters ​
  • row_ids : str — Defines what RowID should be used. Must be one of the following values: - "keep": - For appending DataFrames: Keep the DataFrame.index as the RowID. Convert the index to strings if necessary. - For appending Arrow tables or record batches: Use the first column of the table as RowID. The first column must be of type string. - "generate": Generate new RowIDs of the format "Row{i}"
staticmethod from_batches(generator, row_ids: str = 'generate') ​

Create output table where each batch is provided by a generator

Parameters ​
  • generator : iterable — An iterable (e.g. a generator) yielding the batches to append to the output table. Each batch must be a Table.
  • row_ids : object — See BatchOutputTable.create.
property num_batches → int ​

The number of batches written to this output table.

class knime.extension.Schema ​

A schema defines the data types and names of the columns inside a table. Additionally, it can hold metadata for the individual columns.

method __getitem__(slicing: Union[slice, List[int], List[str]]) → _ColumnarView ​

Creates a view of this Table or Schema by slicing columns. The slicing syntax is similar to that of numpy arrays, but columns can also be addressed as index lists or via a list of column names.

Parameters ​
  • slicing : (int, str, slice, list) — A column index, a column name, a slice object, a list of column indices, or a list of column names. For single indices, the view will create a "Column" object. For slices or lists of indices, a new Schema will be returned.
Returns ​

_ColumnarView — A representation of a slice of the original Schema or Table.

Examples ​
python
>>> # Get columns 1,2,3,4
... sliced_schema = schema[1:5]
python
>>> # Get the columns "name" and "age"
... sliced_schema = schema[["name", "age"]]
method __init__(ktypes: List[Union[KnimeType, Type]], names: List[str], metadata: List = None) ​

Create a schema from a list of column data types, names and metadata.

method append(other: Union[_Columnar, Sequence[_Columnar]]) → _ColumnarView ​

Append another _Columnar object (e.g. Table, Schema) or a sequence of _Columnar objects to the current _Columnar object.

Parameters ​
  • other : Union[_Columnar, Sequence[_Columnar]] — The _Columnar object or a sequence of _Columnar objects to be appended.
Returns ​

_ColumnarView — A _ColumnarView object representing the current _Columnar object after the append operation.

property column_names → List[str] ​
classmethod deserialize(table_schema: dict) → Schema ​

Construct a Schema from a dict that was retrieved from KNIME in JSON encoded form as the input to a node's configure() method. KNIME provides table information with a RowKey column at the beginning, which we drop before returning the created schema.

classmethod from_columns(columns: Union[Sequence[Column], Column]) ​

Create a schema from a single column or a list of columns.

Parameters ​
  • columns : Union[Sequence[Column], Column] — A single column or a list of columns.
Returns ​

Schema — The constructed schema.

classmethod from_types(ktypes: List[Union[KnimeType, Type]], names: List[str], metadata: List = None) ​

Create a schema from a list of column data types, names and metadata.

Parameters ​
  • ktypes : List[Union[KnimeType, Type]] — A list of KNIME types or types known to KNIME.
  • names : List[str] — A list of column names.
  • metadata : List — An optional list of per-column metadata, one entry per column.
Returns ​

Schema — The constructed schema.

method insert(other: _Columnar, at: int) → _Columnar ​

Insert a column or another _Columnar object (e.g. Table, Schema) into the current _Columnar object at a specific position.

Parameters ​
  • other : _Columnar` or `Column — The column or _Columnar object to be inserted.
  • at : int — The index at which the insertion should occur.
Returns ​

_Columnar — The _Columnar object after the insertion.

Raises ​
  • TypeError — If other is not of type _Columnar or Column.

Notes

The insertion is done in-place, meaning the current _Columnar object is modified.

property num_columns ​
method remove(slicing: Union[str, int, List[str]]) ​

Implements remove method for Columnar data structures. The input can be a column index, a column name or a list of column names.

If the input is a column index, the column with that index will be removed. If it is a column name, then the first column with matching name is removed. Passing a list of column names will filter out all (including duplicate) columns with matching names.

Parameters ​
  • slicing : int | list | str — Can be of type integer representing the index in column_names to remove. Or a list of strings removing every column matching from that list. Or a string of which first occurrence is removed from the column_names.
Returns ​

A View missing the columns to be removed.

Raises ​
  • ValueError: If no matching column is found given a list or str.
  • IndexError: If column is accessed by integer and is out of bounds.
  • TypeError: If the key is neither an integer nor a string or list of strings.
method serialize() → Dict ​

Convert this Schema into dict which can then be JSON encoded and sent to KNIME as result of a node's configure() method. Because KNIME expects a row key column as first column of the schema, but we don't include this in the KNIME Python table schema, we insert a row key column here.

Raises ​
  • ** RuntimeError: if duplicate column names are detected**

class knime.extension.Column ​

A column inside a table schema consists of the KNIME datatype, a column name, and optional metadata.

method __init__(ktype: Union[KnimeType, Type], name: str, metadata: dict = None) ​

Construct a Column from type, name and optional metadata.

Parameters ​
  • ktype : Union[KnimeType, Type] — The KNIME type of the column or a type which can be converted via knime.api.schema.logical(ktype) to a KNIME type. Raises a TypeError if the type is not a KNIME type or cannot be converted to a KNIME type.
  • name : str — The name of the column. May not be empty. Raises a ValueError if the name is empty.
  • metadata : dict — Metadata of this column.
Returns ​

Column — The constructed column.

Raises ​
  • TypeError — If the type is not a KNIME type or cannot be converted to a KNIME type.
  • ValueError — If the name is empty.
attribute ktype : KnimeType ​
attribute metadata : Dict ​
attribute name : str ​

Data Types ​

Helper functions to create KNIME-compatible data types, for example when creating a new column.

function knime.extension.int32() ​

Create a KNIME integer type with 32 bits.

function knime.extension.int64() ​

Create a KNIME integer type with 64 bits

function knime.extension.double() ​

Create a KNIME floating point type with double precision (64 bits).

function knime.extension.bool_() ​

Create a KNIME boolean type.

function knime.extension.string(dict_encoding_key_type: DictEncodingKeyType = None) ​

Create a KNIME string type.

Parameters ​
  • dict_encoding_key_type : DictEncodingKeyType — The key type to use for dictionary encoding. If this is None (the default), no dictionary encoding will be used. Dictionary encoding helps to reduce storage space and read/write performance for columns with repeating values such as categorical data.

function knime.extension.blob(dict_encoding_key_type: DictEncodingKeyType = None) ​

Create a KNIME blob type for binary data of variable length.

Parameters ​
  • dict_encoding_key_type : DictEncodingKeyType — The key type to use for dictionary encoding. If this is None (the default), no dictionary encoding will be used. Dictionary encoding helps to reduce storage space and read/write performance for columns with repeating values such as categorical data.

function knime.extension.list_(inner_type: KnimeType) ​

Create a KNIME type that is a list of the given inner types

Parameters ​
  • inner_type : KnimeType — The type of the elements in the list. Must be a KnimeType.

function knime.extension.struct(*inner_types) ​

Create a KNIME structured data type where each given argument represents a field of the struct.

Parameters ​
  • inner_types : list — The argument list of this method defines the fields in this structured data type. Each inner type must be a KNIME type

function knime.extension.logical(value_type) → LogicalType ​

Create a KNIME logical data type of the given Python value type.

Parameters ​
  • value_type : type — The type of the values inside this column. A knime.api.types.PythonValueFactory must be registered for this type.
Raises ​
  • TypeError — If no PythonValueFactory has been registered for this value type with knime.api.types.register_python_value_factory.

function knime.extension.datetime(date: Optional[bool] = True, time: Optional[bool] = True, timezone: Optional[bool] = False) → LogicalType ​

Currently, KNIME supports the following date/time formats: - Local DateTime (date=True, time=True, timezone=False) - Local Date (date=True, time=False, timezone=False) - Local Time (date=False, time=True, timezone=False) - Zoned DateTime (date=True, time=True, timezone=True)

Parameters ​
  • date : Optional[bool] — Whether the column contains a date.
  • time : Optional[bool] — Whether the column contains a time.
  • timezone : Optional[bool] — Whether the column contains a timezone.
Returns ​

LogicalType — A LogicalType representing the given date/time format.

Raises ​
  • ValueError — If the combination of date, time and timezone is not supported or the datetime types are not registered in KNIME.