Python Environments on Executors
When a workflow runs on a KNIME Executor inside KNIME Business Hub, Python nodes need access to a Python environment. There are three types of environments available on executors, each with different characteristics regarding performance, flexibility, governance, and persistence.
Environment types
Bundled environment
The bundled environment is pre-packaged with KNIME and requires no configuration. It is always available on the executor and does not depend on any external package repositories at runtime. For the list of included packages, see Bundled Packages.
Pre-installed custom environments
Python environments can be embedded into the executor image at build time, using the Execution Image Builder or by installing conda environments directly into a custom image. These environments are available immediately when the executor starts and survive executor restarts.
Pre-installed environments are configured for use in Python Script nodes via:
- executor preferences (globally for all workflows on that executor)
- the
python3_commandflow variable (per workflow) - the Conda Environment Propagation node in Check name only mode, which matches a pre-installed environment by name without overwriting it
Pre-installed environments are static: updating them requires rebuilding the executor image. They also increase the size of the executor image.
On-the-fly environments
On-the-fly environments are created during workflow execution using the Conda Environment Propagation node or the Python Environment Provider node. They are not present in the executor image; instead, they are installed at runtime when the workflow runs.
On-the-fly environments are matched to existing environments on the executor by name. Once created, they persist on the executor until it is restarted. In multi-executor setups, each executor instance creates its own copy of the environment independently.
Because environments are matched by name, naming collisions between workflows are possible. A specific naming format helps avoid this:
knimeenv_<workflowName>_<envName>On-the-fly environments require the executor to have network access to conda and pip package repositories at runtime. In restricted network environments, this requires configuring channel mirrors or proxy settings (see Channel Mirrors & Air-Gapped Setups).
Security
Installing environments at runtime pulls packages from external sources. These packages may be subject to CVEs and can execute code during installation. Consider using vetted mirrors or internal channels.
Comparison
| Aspect | Pre-installed environments | On-the-fly environments |
|---|---|---|
| Performance | No install overhead at runtime | Slower on first run; environment must be created |
| Update flexibility | Requires rebuilding the executor image | Changeable per workflow or per run |
| Governance | Centrally defined; easy to audit | Decentralized; all workflows can contribute environments |
| Resilience to restarts | Survive executor restarts | Lost on restart; recreated on next execution |
| Multi-executor setups | Available on all instances from image | Must be created on each executor instance separately |
| Runtime network access | Not required | Required (conda/pip repositories or configured mirrors) |