KNIME Documentation

Set up Python for an environment ​

Python nodes, such as the Python Script node, need a Python environment to run. Provide one in either of these ways:

If your workflow only needs common packages, use the bundled environment instead. It requires no setup. For the included packages, see Bundled packages.

Create the environment for each run ​

The Python Environment Provider node passes a Python environment specification to downstream Python nodes. This option works on both managed and sovereign environments.

To provide a Python environment with this node:

  1. Add the Python Environment Provider node to the workflow.
  2. In the node, define the packages your Python scripts need.
  3. Connect the node to the downstream Python nodes.

The executor creates the environment from scratch each time a downstream Python node runs. When the run ends, the executor discards the environment. For detailed instructions, see Set up Pixi for Python environment management.

Use a permanent environment ​

Use a permanent environment when you want workflows to run with a Python environment that you control. You install the environment once on your sovereign executor, and workflow builders select it with the Python Environment Reader (Labs) node.

INFO

Permanent environments work only on sovereign executors, as you need access to the executor installation folder. Environments that Knime manages don't support them.

Install the environment ​

To install the environment:

  1. Go to your executor installation folder.
  2. Open the bundling folder, next to the configuration, dropins, and features folders. This folder exists only if the KNIME Python Integration is installed.
  3. In the bundling folder, create a folder for your environment, for example my-python-env. Workflow builders select the environment by this folder name.
  4. In this folder, create the Pixi environment. It must contain knime-python-base and the packages your workflow builders need. knime-python-base provides pandas, pyarrow, py4j, and the other packages that Knime Python nodes require.

The result is a complete Pixi project. The node uses only the default Pixi environment:

text
knime/
└── bundling/
    └── my-python-env/
        ├── .pixi/
        │   └── envs/
        │       └── default/
        ├── pixi.lock
        └── pixi.toml

Make the environment read-only ​

Knime deletes writable environments in bundling/

Every time the executor restarts, Knime deletes the folders in bundling/ that don't belong to an installed extension. To keep your environment, make it read-only and owned by root. You can't turn this cleanup off.

To make the environment read-only:

  1. Import pandas once in the environment. Python writes cache files on the first import, which isn't possible after the next step.
  2. Change the permissions of the environment folder:
    • On Linux, make root the owner and remove write access for all users.
    • On Windows, give administrators and SYSTEM full control, and give users read and execute access only.
  3. Restart the executor.

After each restart, the log shows Failed to delete unused Conda environment directory once for your environment. This error is expected and means that the protection works.

Use the environment in a workflow ​

To use the environment in a workflow:

  1. Add the Python Environment Reader (Labs) node to the workflow.
  2. In the node, keep Input Mode on Bundled Environment and select my-python-env from the dropdown menu.
  3. On the Python node, for example the Python Script node, add the Python environment input port from the node's + menu.
  4. Connect the Python Environment Reader (Labs) node to this port.

Update the environment ​

To change the packages in a permanent environment, repeat the installation steps with the updated pixi.toml and pixi.lock files. Running workflows keep using the files that are already there.

Prevent workflow builders from creating their own environments ​

To prevent workflow builders from creating their own environments, blacklist the Python Environment Provider node:

  1. Under Global administration, open the Environments page.
  2. On the tile of the sovereign environment, click the three dots and select Edit.
  3. Under Advanced settings, add org.knime.pixi.nodes.PythonEnvironmentReaderNodeFactory to block the node.
  4. Save the changes.

Workflow builders can still point the Python Environment Reader (Labs) node to a pixi.toml file. This doesn't let them create an environment, because the node never installs or updates anything and needs a complete Pixi project next to the file.