Build a plugin

Turn your CWL knowledge into a tool others can install and run. This guide builds process-inventory, a custom plugin that writes a JSON inventory of the processes in a CWL document. You will create a Python package, define validated options, implement the transformation, register it with the runtime, test it, and build a distributable wheel.

The example targets the contracts in transpiler-mate-api 1.0.1 and transpiler-mate-runtime 1.3.1, using Python 3.10 or newer. Dependency ranges below are starting points; test the versions you intend to support before publishing.

1. Understand the plugin lifecycle

A plugin is an independently packaged Python object with four members:

Member Responsibility
name The command name, such as process-inventory.
description A short explanation shown in command help.
options_model A Pydantic BaseModel subclass defining inputs, defaults, and validation.
execute(context, options) The transformation; returns None and writes its own artifacts.

The runtime handles the surrounding work:

Installed package entry point
    → load the plugin object
    → generate CLI options from its Pydantic model
    → validate options
    → load CWL and validate software metadata
    → execute(context, options)
    → report success or failure

Keep the transformation independent of Click and runtime internals. Depend on transpiler-mate-api for the contract; install transpiler-mate-runtime in the environment that will run the command. The same plugin can then be tested directly or invoked by another host that provides the API context.

The decorator used below creates a PluginRegistration implementing the TranspilerPlugin protocol. A class instance exposing the same members is also supported; no base-class inheritance is required. Entry points load objects, so a class-based implementation should export an instance, not the class or an uncalled factory.

2. Create your project

Option A: start from the project template

The Copier project template includes Hatch packaging, tests, documentation, and CI:

pipx install copier
copier copy --vcs-ref main \
  https://github.com/transpiler-mate/transpiler-mate-plugin-project-template.git \
  transpiler-mate-process-inventory

Use transpiler-mate-process-inventory for the project slug and process_inventory for the Python package. Prefer a published template tag over main when you need repeatable generation. Keep .copier-answers.yml if you want to use copier update later.

When adapting the generated project, keep its versioning, license, and quality configuration. Merge the dependencies and entry-point configuration below into the existing tables, replace its sample plugin, and update the generated tests. In particular, replace the generated entry point with process-inventory = "process_inventory.plugin:plugin" so it matches this guide's registration.

Option B: build the minimal example by hand

The rest of this tutorial is self-contained. Create this layout:

transpiler-mate-process-inventory/
├── pyproject.toml
├── src/
│   └── process_inventory/
│       ├── __init__.py
│       └── plugin.py
├── examples/
│   └── workflow.cwl
└── tests/
    └── test_plugin.py

mkdir transpiler-mate-process-inventory
cd transpiler-mate-process-inventory
mkdir -p src/process_inventory examples tests
python3 -m venv .venv
source .venv/bin/activate

These commands use a POSIX shell. On Windows, create the directories with your shell and activate the environment with .venv\Scripts\Activate.ps1 in PowerShell. All subsequent python and transpiler-mate commands should use this environment.

3. Define packaging and discovery

Create pyproject.toml:

[build-system]
requires = ["hatchling>=1.32.4"]
build-backend = "hatchling.build"

[project]
name = "transpiler-mate-process-inventory"
version = "0.1.0"
description = "A CWL process inventory plugin for Transpiler-Mate"
requires-python = ">=3.10"
dependencies = [
  "transpiler-mate-api>=1.0.1,<2",
  "pydantic>=2.13.4,<3",
]

[project.optional-dependencies]
test = ["pytest>=8,<10", "transpiler-mate-runtime>=1.3.1,<2"]

[project.entry-points."transpiler_mate.plugins"]
process-inventory = "process_inventory.plugin:plugin"

[tool.hatch.build.targets.wheel]
packages = ["src/process_inventory"]

[tool.pytest.ini_options]
testpaths = ["tests"]

There are three different names here:

Name Example Used by
Distribution transpiler-mate-process-inventory pip install and package indexes.
Import package process_inventory Python imports; use underscores.
Plugin / entry-point name process-inventory CLI subcommand and batch-plan keys.

The entry-point value uses module:object syntax. The runtime imports process_inventory.plugin and retrieves its plugin object. The entry-point name must match plugin.name. Registering the same name in two installed distributions causes a duplicate-plugin error.

The entry-point group belongs to the runtime's discovery mechanism, not the framework-independent API. An importable plugin is only discoverable after its package metadata has been installed. Reinstall after changing entry points, even when using an editable installation.

4. Implement the plugin

Create src/process_inventory/plugin.py:

"""Generate a JSON inventory from the runtime's resolved CWL context."""

from __future__ import annotations

import json
from pathlib import Path

from pydantic import BaseModel, ConfigDict, Field
from transpiler_mate.api import (
    PluginExecutionError,
    PluginFailureError,
    TranspilerContext,
    transpiler_plugin,
)


class InventoryOptions(BaseModel):
    """Control the inventory destination and JSON formatting."""

    model_config = ConfigDict(extra="forbid")

    output: Path = Field(description="Destination JSON file; must not exist")
    indent: int = Field(default=2, ge=0, le=8, description="JSON indentation")


@transpiler_plugin(
    name="process-inventory",
    description="Write a JSON inventory of every resolved CWL process",
    options_model=InventoryOptions,
)
def plugin(context: TranspilerContext, options: InventoryOptions) -> None:
    """Write the inventory, refusing to overwrite an existing artifact.

    Raises:
        PluginFailureError: The document is empty or the destination exists.
        PluginExecutionError: The destination cannot be created or written.
    """
    rows = [
        {"id": process_id, "class": type(process).__name__}
        for process_id, process in sorted(context.document.items())
    ]
    if not rows:
        raise PluginFailureError("the CWL document contains no processes")

    payload = {
        "software": context.metadata.name,
        "processes": rows,
    }
    content = json.dumps(payload, indent=options.indent) + "\n"

    try:
        options.output.parent.mkdir(parents=True, exist_ok=True)
        with options.output.open("x", encoding="utf-8") as stream:
            stream.write(content)
    except FileExistsError as error:
        raise PluginFailureError(
            f"destination {options.output} exists; choose a new output path"
        ) from error
    except OSError as error:
        raise PluginExecutionError(
            f"could not write inventory to {options.output}"
        ) from error

Create src/process_inventory/__init__.py:

"""Public registration for the process-inventory plugin."""

from process_inventory.plugin import plugin

__all__ = ["plugin"]

The decorator replaces the function binding with a registration object. Call plugin.execute(context, options) in Python, not plugin(context, options).

This plugin sorts the process IDs for stable output, creates missing parent directories, and opens the output in exclusive creation mode. An existing artifact is preserved. A write failure may leave a partial new file; use a temporary file and an appropriate commit strategy if your production output must be atomic.

How options become a command

The runtime creates --output PATH and --indent INTEGER from the model. output is required, indent defaults to 2, and Pydantic rejects indentation outside 0..8 before CWL resolution begins. Field descriptions become CLI help.

Model feature CLI behavior
output_path: Path --output-path PATH; underscores become hyphens.
Defaulted bool Paired flags such as --verbose/--no-verbose.
Literal["json", "text"] A case-sensitive choice.
Enum A case-insensitive choice of member names.
list[str] Repeatable option, such as --tags one --tags two.
Field(ge=0) or model validators Full Pydantic validation before execution.

Use EmptyOptions from the API when a plugin needs no options. Avoid names that collide with runtime options such as oci_hostname, oci_username, oci_password, authfile, and oauth2_bearer. The runtime already owns the required SOURCE argument and authentication setup.

5. Supply a complete CWL example

Create examples/workflow.cwl:

cwlVersion: v1.2
$namespaces:
  s: https://schema.org/

s:name: Inventory tutorial
s:description: A small workflow used to demonstrate a custom plugin.
s:dateCreated: "2026-01-01"
s:license: https://spdx.org/licenses/Apache-2.0.html
s:softwareVersion: "0.1.0"
s:softwareHelp:
  class: s:CreativeWork
  s:name: Tutorial documentation
  s:url: https://example.org/inventory/docs
s:publisher:
  class: s:Organization
  s:name: Example Research Group
s:author:
  class: s:Person
  s:givenName: Alex
  s:familyName: Example
  s:email: alex@example.org
  s:affiliation:
    class: s:Organization
    s:name: Example Research Group

$graph:
  - id: main
    class: Workflow
    inputs: []
    outputs: []
    steps: []

The names and URLs are illustrative. The empty workflow is sufficient for this inventory: Transpiler-Mate inspects the document rather than executing its steps.

Metadata is validated before your plugin runs

The runtime requires document-level metadata that validates as SoftwareApplication, even if a particular plugin only uses the CWL structure. Put it alongside $graph: the current loader/extractor relies on metadata preserved from that wrapper. A bare standalone process is not an equivalent fixture.

The required software fields are name, description, dateCreated, license, softwareVersion, softwareHelp, publisher, and author; nested objects have their own requirements. In particular, the person above includes both email and affiliation. Use the metadata generator or consult the metadata model reference for real projects.

6. Install, discover, and run

From the project root, with your environment active:

python -m pip install -e '.[test]'
transpiler-mate --version
transpiler-mate --help
transpiler-mate process-inventory --help
transpiler-mate process-inventory examples/workflow.cwl \
  --output build/inventory.json
python -m json.tool build/inventory.json

The first help command lists process-inventory; the second shows its generated options. The output is:

{
  "software": "Inventory tutorial",
  "processes": [
    {
      "id": "main",
      "class": "Workflow"
    }
  ]
}

Running the same command again fails intentionally because build/inventory.json exists. Choose another filename to produce another inventory.

For an explicit discovery check, run:

python -c "from transpiler_mate.runtime.plugin_loader import load_plugin_by_name; print(load_plugin_by_name('process-inventory').name)"

That checks installed metadata, import resolution, and the runtime's structural protocol check. It does not replace tests of execution or static type checking.

7. Work with the context

TranspilerContext provides the data your plugin should consume:

Member Meaning and use
source A Pydantic AnyUrl; local paths become absolute file: URLs.
document Mapping of process IDs to parsed cwl_utils process objects.
processes Iterable view of the mapping's values.
process_id Optional fragment requested in SOURCE, such as main.
resolved_process Looks up the requested ID; raises PluginExecutionError if no fragment was provided or the ID is absent.
metadata Validated SoftwareApplication, with Python attributes such as name and software_version.
resolver Resolves another source with the host's configured loading and authentication policy.

The context model is frozen, but nested process objects are not deeply immutable. Treat the context and its contents as read-only: batch executions share the same instance.

Whole-document versus selected-process plugins

This tutorial deliberately inventories every process in context.document. Adding #main does not filter that mapping automatically. If your plugin operates on one selected process, read context.resolved_process and document that users must supply a fragment:

transpiler-mate your-plugin 'examples/workflow.cwl#main' --output build/result.json

Here your-plugin is a placeholder for your own selected-process implementation. Do not assume that a single-process document automatically supplies a selection.

To filter by CWL class, use context.get_processes_by_type(...). Pass the appropriate version-specific classes from cwl_utils.parser (or a union of supported classes); fail_if_empty=False allows an empty result. A filter using only the CWL v1.2 Workflow class will not match the v1.0 or v1.1 equivalents.

Metadata and additional sources

Use context.metadata.model_dump(by_alias=True) when exporting metadata with Schema.org serialization aliases. Its model dump defaults to JSON-compatible values and excludes None. Reuse these generated models; do not recreate the software metadata schema in your plugin.

For comparison or multi-source plugins, use context.resolver.resolve(location) to obtain another context. Supply an explicit path or URL accepted by the runtime; do not assume a relative path is resolved against the original source. Preserve existing PluginError exceptions, and wrap other technical failures in PluginExecutionError with exception chaining when you can add useful context.

8. Test the contract and the installed command

Create tests/test_plugin.py:

"""Test artifacts, error behavior, discovery, and real CWL resolution."""

import json
import subprocess
import sys
from pathlib import Path

import pytest
from cwl_utils.parser.cwl_v1_2 import Workflow
from pydantic import AnyUrl, ValidationError
from transpiler_mate.api import (
    PluginExecutionError,
    PluginFailureError,
    SoftwareApplication,
    TranspilerContext,
)
from transpiler_mate.runtime.plugin_loader import load_plugin_by_name

from process_inventory import plugin
from process_inventory.plugin import InventoryOptions

PROJECT = Path(__file__).resolve().parents[1]


class UnusedResolver:
    """Reject unexpected source loading in unit tests."""

    def resolve(self, location: str) -> TranspilerContext:
        """Fail if inventory generation attempts to load another source."""
        raise AssertionError(f"unexpected resolution: {location}")


@pytest.fixture
def context(tmp_path: Path) -> TranspilerContext:
    """Provide only the metadata this unit under test reads."""
    return TranspilerContext(
        source=AnyUrl((tmp_path / "workflow.cwl").as_uri()),
        metadata=SoftwareApplication.model_construct(name="Inventory tutorial"),
        document={"main": Workflow(id="main", inputs=[], outputs=[], steps=[])},
        resolver=UnusedResolver(),
    )


def test_writes_inventory(context: TranspilerContext, tmp_path: Path) -> None:
    output = tmp_path / "nested" / "inventory.json"
    plugin.execute(context, InventoryOptions(output=output))
    assert json.loads(output.read_text(encoding="utf-8")) == {
        "software": "Inventory tutorial",
        "processes": [{"id": "main", "class": "Workflow"}],
    }


def test_preserves_existing_output(context: TranspilerContext, tmp_path: Path) -> None:
    output = tmp_path / "inventory.json"
    output.write_text("keep me", encoding="utf-8")
    with pytest.raises(PluginFailureError, match="exists"):
        plugin.execute(context, InventoryOptions(output=output))
    assert output.read_text(encoding="utf-8") == "keep me"


def test_rejects_empty_document(context: TranspilerContext, tmp_path: Path) -> None:
    empty = context.model_copy(update={"document": {}})
    output = tmp_path / "inventory.json"
    with pytest.raises(PluginFailureError, match="no processes"):
        plugin.execute(empty, InventoryOptions(output=output))
    assert not output.exists()


def test_reports_write_failure(context: TranspilerContext, tmp_path: Path) -> None:
    parent = tmp_path / "not-a-directory"
    parent.write_text("a file blocks the output directory", encoding="utf-8")
    with pytest.raises(PluginExecutionError) as raised:
        plugin.execute(context, InventoryOptions(output=parent / "child" / "out.json"))
    assert isinstance(raised.value.__cause__, OSError)


def test_validates_options(tmp_path: Path) -> None:
    with pytest.raises(ValidationError):
        InventoryOptions(output=tmp_path / "out.json", indent=9)
    with pytest.raises(ValidationError):
        InventoryOptions.model_validate({"output": "out.json", "indnet": 2})


def test_installed_entry_point() -> None:
    assert load_plugin_by_name("process-inventory") is plugin


def test_cli_loads_real_cwl(tmp_path: Path) -> None:
    output = tmp_path / "inventory.json"
    result = subprocess.run(
        [
            sys.executable,
            "-c",
            "from transpiler_mate.runtime.cli import main; main()",
            "process-inventory",
            str(PROJECT / "examples" / "workflow.cwl"),
            "--output", str(output),
        ],
        capture_output=True,
        text=True,
        check=False,
    )
    assert result.returncode == 0, result.stdout + result.stderr
    assert json.loads(output.read_text(encoding="utf-8")) == {
        "software": "Inventory tutorial",
        "processes": [{"id": "main", "class": "Workflow"}],
    }

Run the tests after installing the package:

python -m pytest

The unit fixture intentionally uses model_construct() to bypass unrelated metadata validation. It supplies the name that this plugin reads. The integration test then loads the actual CWL example, validates all its metadata, discovers the installed entry point, and checks the generated artifact. Do not use model_construct() to bypass validation in production.

For a production plugin, add cases for the CWL versions and constructs you support, multiple processes, selected-process behavior if applicable, and external-tool failures. Run your project's formatter, linter, type checker, security checks, and test matrix. The Copier template provides these development environments; inspect its generated pyproject.toml for the available commands.

9. Report failures deliberately

Situation Signal CLI result
Missing required option, wrong type, invalid range Pydantic / Click validation Exit 2; options rejected before execution.
Unsupported CWL construct, missing domain data, existing destination PluginFailureError Exit 1; expected failure without a traceback.
File write failure, failed subprocess, unavailable required service PluginExecutionError Exit 1; technical error with a traceback.
Completed transformation Return None Exit 0.

Catch narrow exception types and use raise ... from error to preserve the cause. Include the failed operation and an actionable remedy when known. Let the runtime control process exit and presentation; avoid sys.exit() or Click exceptions inside the plugin's transformation.

For subprocesses, use argument lists rather than shell interpolation, check exit status, and translate errors into the API's failure categories. Avoid logging credentials or sensitive source contents.

10. Run your plugin in a batch

No extra adapter is needed. Create tmom.yaml in the project root:

process-inventory:
  - output: build/inventory-pretty.json
    indent: 2
  - output: build/inventory-compact.json
    indent: 0

Then run:

transpiler-mate batch examples/workflow.cwl --file tmom.yaml

Batch validates each execution's options and invokes the same plugin with the same resolved context. Use model field names in YAML, not CLI flags. Paths are relative to the working directory, not to the plan file. indent: 0 removes indentation but still includes JSON line breaks.

Executions run sequentially and stop at the first failure. Earlier artifacts are not rolled back, so rerunning this example requires new output paths or deliberate removal of the previous outputs. See batch execution for larger plans.

11. Build and distribute

Build a source distribution and wheel:

python -m pip install build
python -m build

For this minimal project, the wheel is dist/transpiler_mate_process_inventory-0.1.0-py3-none-any.whl. Check that it works without your editable source tree by installing it in another environment:

python3 -m venv /tmp/inventory-wheel-check
/tmp/inventory-wheel-check/bin/python -m pip install \
  dist/transpiler_mate_process_inventory-0.1.0-py3-none-any.whl \
  'transpiler-mate-runtime>=1.3.1,<2'
/tmp/inventory-wheel-check/bin/transpiler-mate process-inventory \
  examples/workflow.cwl --output build/inventory-from-wheel.json

Adapt these environment paths on Windows. Run from the project root so the example input path resolves; use a new output path if repeating the check.

Before releasing, choose a unique distribution name and plugin command, add your license and project URLs, document supported CWL/API/runtime versions and output semantics, and record changes in a changelog. Include external executables and optional system dependencies in installation instructions. The template already supplies a starting structure for much of this.

Users can install your wheel directly or install a published distribution from your chosen package index alongside the runtime. Follow that index's publishing and authentication process. Runtime discovery comes from the wheel's entry-point metadata; no change to the runtime repository is needed.

Troubleshooting

Symptom What to check
No such command 'process-inventory' Install the plugin into the runtime's environment; check python -m pip show transpiler-mate-process-inventory and reinstall after entry-point changes.
Entry-point import fails Confirm process_inventory.plugin:plugin is importable and that the wheel includes src/process_inventory.
Plugin name mismatch Make the entry-point key and decorator's name identical.
Duplicate plugin error Two distributions registered the same command; rename yours or remove the unintended installation.
Impossible to load a CWL document Check CWL validity, source access, $graph, and all required document-level and nested metadata fields. This occurs before plugin execution.
No process selected Supply #main when reading resolved_process; inspect context.document for valid IDs.
Invalid plugin options Check required fields, types, bounds, aliases, and unknown keys; CLI option names use hyphens.
Destination already exists This example preserves outputs. Choose a new destination.
Direct tests pass but the CLI fails Run the integration test: direct contexts bypass installed discovery and real source/metadata loading.

API and runtime references

This guide combines the API contract with the runtime's discovery and CLI behavior. For deeper details, consult: