v1/systemone endpoint

@@ -27,6 +27,7 @@ The `transformers serve` command spawns a local server compatible with the [Open - `/v1/completions` for legacy text completions from a freeform prompt - `/v1/responses` supports the [Responses API](https://platform.openai.com/docs/api-reference/responses) - `/v1/audio/transcriptions` for audio transcriptions+- `/v1/systemone` answers typed questions (choice, score, yes/no) with a probability for every allowed answer - `/v1/models` lists available models for third-party integrations - `/load_model` streams model loading progress via SSE @@ -1649,6 +1650,144 @@ The follow-up question "How many people live there?" relies on the prior context As of 2021, Paris has a population of approximately 2.8 million people. ``` +## v1/systemone++The `v1/systemone` endpoint answers typed questions about a shared `state`, following [TypeSafe’s System One API](https://api.typesafe.ai/openapi.json). Use it for decisions such as routing a support ticket, rating urgency, or choosing an action.++The model scores the allowed answers in one batched forward pass without generating text. This endpoint supports autoregressive models with a chat template;++Send a model, the state to evaluate, and questions keyed by name:++<hfoptions id="systemone-http">+<hfoption id="curl">++```shell+curl http://localhost:8000/v1/systemone \+  -H "Content-Type: application/json" \+  -d '{+    "model": "Qwen/Qwen2.5-1.5B-Instruct",+    "state": "Our integration has been failing for 3 days and we are losing sales!",+    "questions": {+      "team": {+        "type": "choice",+        "instructions": "Which team handles this?",+        "criteria": {"billing": null, "technical": "Bugs or integration problems"}+      },+      "urgency": {+        "type": "score",+        "instructions": "How urgent is this ticket?",+        "criteria": ["low", "medium", "high"]+      },+      "upset": {"type": "noul", "instructions": "Is the customer upset?"}+    }+  }'+```++</hfoption>+<hfoption id="TypeSafe (TypeScript)">++Install the [TypeSafe SDK](https://github.com/typesafe-ai/typesafe-sdk-js) for Node.js 20 or newer:++```shell+npm install @typesafe-ai/sdk+```++Set `baseURL` to the server root; the SDK adds `/v1/systemone`. The SDK requires an API key, but a placeholder works with the local server.++```typescript+import { choice, noul, score, TypeSafeClient } from "@typesafe-ai/sdk";++const client = new TypeSafeClient({+  baseURL: "http://localhost:8000",+  apiKey: "<random_string>",+});++const response = await client.systemOne({+  model: "Qwen/Qwen2.5-1.5B-Instruct",+  state: "Our integration has been failing for 3 days and we are losing sales!",+  questions: {+    team: choice("Which team handles this?", {+      billing: null,+      technical: "Bugs or integration problems",+    }),+    urgency: score("How urgent is this ticket?", ["low", "medium", "high"]),+    upset: noul("Is the customer upset?"),+  },+});++console.log(response.answers.team.choice);+console.log(response.answers.urgency.score);+console.log(response.answers.upset.noul);+```++The client prints the selected team, urgency score, and probability that the customer is upset. For example:++```shell+technical+1.6739138662815094+0.851952850818634+```++</hfoption>+</hfoptions>++Answers use the same question names. For example:++```json+{+  "model": "Qwen/Qwen2.5-1.5B-Instruct",+  "answers": {+    "team": {"type": "choice", "choice": "technical", "probabilities": {"billing": 0.000006962256520637311, "technical": 0.999993085861206}, "confidence": 0.9999861717224121},+    "urgency": {"type": "score", "score": 1.6739138662815094, "legend": {"0": "low", "1": "medium", "2": "high"}, "probabilities": {"0": 0.0031353600788861513, "1": 0.31981542706489563, "2": 0.6770492196083069}, "confidence": 0.5108707791659981},+    "upset": {"type": "noul", "noul": 0.851952850818634}+  },+  "usage": {"input_tokens": 213, "output_tokens": 0}+}+```++| Question type | `criteria` | Answer |+| :--- | :--- | :--- |+| `choice` | Option names mapped to descriptions, which can be `null`. | The most likely option, option probabilities, and confidence. |+| `score` | Level descriptions ordered from lowest to highest. | The probability-weighted level index, probabilities, a legend, and confidence. |+| `noul` | Optional descriptions for `true` and `false`. | The probability of `true`. |++`state`, optional `instructions`, and descriptions can contain text or JSON objects and arrays. Each request accepts up to 64 questions, with 1–26 choices or score levels per question.++For vision-capable models, add an `images` array with up to four `data:image/...;base64,...` URLs. Every question receives the same images. The model's processor and chat template must support images; remote image URLs are not accepted.++### Decision configuration++Decision models do not yet share a standard for storing prompts, answer labels, and calibration settings. These vary across model implementations, so `--decision-config` lets you provide them explicitly. Models with custom decision heads still require a compatible implementation; this file only configures prompt formatting and next-token scoring.++By default, questions use a fallback prompt and the model's chat template with thinking disabled. Answer labels are `A`–`Z` for choices and scores, and `Yes`/`No` for booleans. Probabilities are the softmax over these labels at temperature 1.++Pass `--decision-config` to override these defaults for a model:++```shell+transformers serve openjev/openjev --decision-config decision.json+```++The JSON file supports three optional fields:++| Field | Meaning |+| :--- | :--- |+| `labels` | Answer labels by question type, in option order (`true`, then `false` for `noul`). Each must be a distinct, known single token. |+| `temperature` | Positive temperatures by question type, applied as `softmax(label_logits / temperature)`. |+| `chat_template` | An inline Jinja template replacing the full prompt, including chat formatting and the assistant prefix. |++For example, to change labels and calibration while keeping the default prompt:++```json+{+  "labels": {"noul": ["A", "B"]},+  "temperature": {"choice": 0.85, "score": 0.85, "noul": 1.5547129}+}+```++A custom template receives the usual `messages`, plus raw `state`, `images`, and `question` fields. The question exposes `name`, `type`, `instructions`, `options`, `labels`, `descriptions`, and `legend`; options, labels, and descriptions share the same order. Use these fields to replace the fallback wording, and `tojson` to render structured values. Templates for image inputs must include the model's image placeholders.++The file is loaded at startup and applies to `/v1/systemone` only. Omitted settings keep their defaults; no decision configuration is read automatically from the checkpoint.+ ## v1/audio/transcriptions  The `v1/audio/transcriptions` endpoint transcribes audio using speech-to-text models. It follows the [Audio transcription API](https://platform.openai.com/docs/api-reference/audio/createTranscription) format.@@ -2148,4 +2287,4 @@ Pass a model as a positional argument to avoid per-request model hints and produ transformers serve Qwen/Qwen2.5-0.5B-Instruct \   --continuous-batching \   --dtype "bfloat16"-```\ No newline at end of file+```

@@ -81,6 +81,10 @@ def __init__(                 )             ),         ] = None,+        decision_config: Annotated[+            str | None,+            typer.Option(help="Path to a JSON file with decision labels, temperatures, and a chat template."),+        ] = None,         device: Annotated[str, typer.Option(help="Device for inference (e.g. 'auto', 'cuda:0', 'cpu').")] = "auto",         dtype: Annotated[str | None, typer.Option(help="Override model dtype. 'auto' derives from weights.")] = "auto",         trust_remote_code: Annotated[bool, typer.Option(help="Trust remote code when loading.")] = False,@@ -123,6 +127,7 @@ def __init__(         from .serving.model_manager import ModelManager         from .serving.response import ResponseHandler         from .serving.server import build_server+        from .serving.systemone import SystemOneHandler         from .serving.transcription import TranscriptionHandler         from .serving.utils import GenerationState @@ -194,12 +199,20 @@ def __init__(          self._transcription_handler = TranscriptionHandler(self._model_manager, self._generation_state) +        self._systemone_handler = SystemOneHandler(+            model_manager=self._model_manager,+            generation_state=self._generation_state,+            chat_template_kwargs=chat_template_kwargs,+            decision_config=decision_config,+        )+         app = build_server(             self._model_manager,             self._chat_handler,             completion_handler=self._completion_handler,             response_handler=self._response_handler,             transcription_handler=self._transcription_handler,+            systemone_handler=self._systemone_handler,             generation_state=self._generation_state,             enable_cors=enable_cors,         )@@ -242,6 +255,7 @@ def kill_server(self): Endpoints:     POST /v1/chat/completions — Chat completions (streaming + non-streaming).     POST /v1/completions      — Legacy text completions from a prompt.+    POST /v1/systemone        — System One typed decisions (choice, score, yes/no) with probabilities.     GET  /v1/models           — Lists available models.     GET  /health              — Health check. 

@@ -32,6 +32,7 @@     from .chat_completion import ChatCompletionHandler     from .completion import CompletionHandler     from .response import ResponseHandler+    from .systemone import SystemOneHandler     from .transcription import TranscriptionHandler  from .model_manager import ModelManager@@ -47,6 +48,7 @@ def build_server(     completion_handler: "CompletionHandler",     response_handler: "ResponseHandler",     transcription_handler: "TranscriptionHandler",+    systemone_handler: "SystemOneHandler",     generation_state: GenerationState,     enable_cors: bool = False, ) -> "FastAPI":@@ -56,6 +58,7 @@ def build_server(         model_manager: Handles model loading, caching, and cleanup.         chat_handler: Handles `/v1/chat/completions` requests.         response_handler: Handles `/v1/responses` requests.+        systemone_handler: Handles `/v1/systemone` requests.         generation_state: Owns the per-model generation managers (regular and CB). Passed             in here so `/health` can check whether the CB worker has died and respond with             503 instead of a misleading 200.@@ -112,6 +115,10 @@ async def completions(request: Request, body: dict):     async def responses(request: Request, body: dict):         return await response_handler.handle_request(body, request.state.request_id) +    @app.post("/v1/systemone")+    async def systemone(request: Request, body: dict):+        return await systemone_handler.handle_request(body, request.state.request_id)+     @app.post("/v1/audio/transcriptions")     async def audio_transcriptions(request: Request):         return await transcription_handler.handle_request(request)

@@ -0,0 +1,423 @@+# Copyright 2026 The HuggingFace Team. All rights reserved.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+#     http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+"""+Handler for the /v1/systemone endpoint (System One typed decisions, in Jev's request and answer shapes).++Answers typed questions (`choice`, `score`, `noul`) about a `state` with a probability for every allowed answer. No+text is generated: each allowed answer gets a single-token label, and its probability is read from the model's logits+in one forward pass.+"""++import json+import string+from dataclasses import dataclass+from pathlib import Path+from typing import TYPE_CHECKING, Annotated, Any, Literal++from ...utils import logging+from ...utils.import_utils import is_serve_available+++if is_serve_available():+    from fastapi import HTTPException+    from fastapi.responses import JSONResponse+    from pydantic import BaseModel, ConfigDict, Field, ValidationError, field_validator++from .utils import BaseHandler, GenerateManager, Modality+++if TYPE_CHECKING:+    from transformers import PreTrainedModel, PreTrainedTokenizerFast, ProcessorMixin++    from .model_manager import ModelManager+    from .utils import GenerationState+++logger = logging.get_logger(__name__)+++class DecisionConfig(BaseModel):+    """Explicit server settings for decision prompts and next-token scoring."""++    model_config = ConfigDict(extra="forbid")++    labels: dict[Literal["choice", "score", "noul"], list[Annotated[str, Field(min_length=1)]]] = Field(+        default_factory=dict+    )+    temperature: dict[Literal["choice", "score", "noul"], Annotated[float, Field(gt=0, allow_inf_nan=False)]] = Field(+        default_factory=dict+    )+    chat_template: Annotated[str, Field(min_length=1)] | None = None++    @classmethod+    def from_file(cls, path: str | Path | None = None) -> "DecisionConfig":+        """Load explicit JSON settings, or return defaults when no file is supplied."""+        return cls() if path is None else cls.model_validate_json(Path(path).read_text(encoding="utf-8"))++    @field_validator("labels")+    @classmethod+    def validate_labels(cls, labels):+        for kind, values in labels.items():+            if len(values) < (2 if kind == "noul" else 1) or len(set(values)) != len(values):+                raise ValueError(f"labels.{kind} must contain distinct labels (at least two for noul).")+        return labels+++# Request types follow https://api.typesafe.ai/openapi.json.+class ChoiceQuestion(BaseModel):+    model_config = ConfigDict(extra="forbid")++    type: Literal["choice"]+    instructions: str | dict[str, Any] | list[Any] | None = None+    # One letter label per option, at most 26.+    criteria: dict[str, str | dict[str, Any] | list[Any] | None] = Field(min_length=1, max_length=26)+++class ScoreQuestion(BaseModel):+    model_config = ConfigDict(extra="forbid")++    type: Literal["score"]+    instructions: str | dict[str, Any] | list[Any] | None = None+    criteria: list[str | dict[str, Any] | list[Any]] = Field(min_length=1, max_length=26)+++class NoulCriteria(BaseModel):+    model_config = ConfigDict(extra="forbid")++    true: str | dict[str, Any] | list[Any] | None = None+    false: str | dict[str, Any] | list[Any] | None = None+++class NoulQuestion(BaseModel):+    model_config = ConfigDict(extra="forbid")++    type: Literal["noul"]+    instructions: str | dict[str, Any] | list[Any] | None = None+    criteria: NoulCriteria | None = None+++class SystemOneRequest(BaseModel):+    model: str+    state: str | dict[str, Any] | list[Any]+    images: list[Annotated[str, Field(pattern=r"^data:image/[^;]+;base64,")]] = Field(+        default_factory=list, max_length=4+    )+    questions: dict[str, Annotated[ChoiceQuestion | ScoreQuestion | NoulQuestion, Field(discriminator="type")]] = (+        Field(min_length=1, max_length=64)+    )+++class TransformersSystemOneRequestParams(SystemOneRequest):+    model_config = ConfigDict(extra="forbid")++    chat_template_kwargs: dict[str, Any] = Field(default_factory=dict)+++@dataclass+class Question:+    name: str+    type: Literal["choice", "score", "noul"]+    instructions: str | dict[str, Any] | list[Any] | None+    # Response keys in label order: choice keys, score indices, or true/false.+    options: list[str]+    labels: list[str]+    descriptions: list[str | dict[str, Any] | list[Any] | None]+    legend: list[str | dict[str, Any] | list[Any]] | None = None+++def render_content(content: str | dict[str, Any] | list[Any] | None) -> str:+    """Render validated System One content as text, preserving JSON values in objects and arrays."""+    if content is None:+        return ""+    return content if isinstance(content, str) else json.dumps(content, ensure_ascii=False)+++def parse_question(+    name: str, question: ChoiceQuestion | ScoreQuestion | NoulQuestion, labels: list[str] | None = None+) -> Question:+    """Extract question metadata for prompt rendering and answer scoring, preserving raw JSON values.++    Labels default to Yes/No for booleans and A/B/C/... otherwise.+    """+    legend = None+    n_options = 2 if isinstance(question, NoulQuestion) else len(question.criteria)+    if labels is None:+        labels = ["Yes", "No"] if isinstance(question, NoulQuestion) else list(string.ascii_uppercase[:n_options])+    else:+        labels = labels[:n_options]+    if len(labels) != n_options:+        raise HTTPException(+            status_code=400,+            detail=f"labels.{question.type} needs at least {n_options} labels for question {name!r}.",+        )++    if isinstance(question, NoulQuestion):+        options = ["true", "false"]+        criteria = question.criteria+        descriptions = [criteria.true, criteria.false] if criteria is not None else [None, None]+    elif isinstance(question, ChoiceQuestion):+        options = list(question.criteria)+        descriptions = list(question.criteria.values())+    else:+        options = [str(i) for i in range(n_options)]+        legend = question.criteria+        descriptions = list(question.criteria)++    return Question(+        name=name,+        type=question.type,+        instructions=question.instructions,+        options=options,+        labels=labels,+        descriptions=descriptions,+        legend=legend,+    )+++def render_decision_prompt(state: str | dict[str, Any] | list[Any], question: Question) -> str:+    """Render the fallback decision prompt without chat role markers.++    A custom chat template can use the raw state and question fields to replace this wording.+    """+    labels = question.labels+    answers = [render_content(description) for description in question.descriptions]+    if question.type == "noul":+        if labels != ["Yes", "No"]:+            answers = [answer or option for answer, option in zip(answers, question.options)]+        lines = [f"{label}: {answer}" for label, answer in zip(labels, answers) if answer]+        closing = f"Answer {labels[0]} or {labels[1]} only."+    else:+        if question.type == "choice":+            answers = [+                f"{option}: {answer}" if answer else option for option, answer in zip(question.options, answers)+            ]+        answer_kind = "option" if question.type == "choice" else "level"+        lines = [f"{label}. {answer}" for label, answer in zip(labels, answers)]+        label_kind = "letter" if labels == list(string.ascii_uppercase[: len(labels)]) else "label"+        closing = (+            f"Answer with the {label_kind} of the {answer_kind} that fits best ({labels[0]} to {labels[-1]}) only."+        )++    text = "\n".join(line for line in [render_content(question.instructions), *lines] if line)+    return f"{render_content(state)}\n\n{text}\n{closing}"+++def build_answers(questions: list[Question], probabilities: list[list[float]]) -> dict[str, dict]:+    """Build System One answers keyed by question name from each question's label probabilities."""+    answers = {}+    for question, question_probabilities in zip(questions, probabilities):+        option_probabilities = dict(zip(question.options, question_probabilities))+        best = max(option_probabilities, key=option_probabilities.get)+        if question.type != "noul":+            n = len(question_probabilities)+            if n == 1:+                confidence = 1.0+            elif question.type == "choice":+                confidence = max(0.0, (option_probabilities[best] - 1 / n) / (1 - 1 / n))+            else:+                distance = sum(p * abs(i - int(best)) for i, p in enumerate(question_probabilities))+                uniform_distance = sum(abs(i - (n - 1) / 2) for i in range(n)) / n+                confidence = max(0.0, 1 - distance / uniform_distance)+        if question.type == "choice":+            answer = {+                "type": "choice",+                "choice": best,+                "probabilities": option_probabilities,+                "confidence": confidence,+            }+        elif question.type == "score":+            answer = {+                "type": "score",+                "score": sum(int(level) * p for level, p in option_probabilities.items()),+                "legend": dict(zip(question.options, question.legend)),+                "probabilities": option_probabilities,+                "confidence": confidence,+            }+        else:+            answer = {"type": "noul", "noul": option_probabilities["true"]}+        answers[question.name] = answer+    return answers+++class SystemOneHandler(BaseHandler):+    """Handler for the `/v1/systemone` endpoint.++    Autoregressive models get one prompt per question and answer with their next token, all questions in one batched+    forward pass. Returns one JSON response; streaming is not supported.+    """++    def __init__(+        self,+        model_manager: "ModelManager",+        generation_state: "GenerationState",+        chat_template_kwargs: dict | None = None,+        decision_config: DecisionConfig | str | None = None,+    ):+        super().__init__(model_manager, generation_state, chat_template_kwargs)+        self.decision_config = (+            decision_config+            if isinstance(decision_config, DecisionConfig)+            else DecisionConfig.from_file(decision_config)+        )++    def _validate_request(self, body: dict) -> TransformersSystemOneRequestParams:+        """Validate the entire request before resolving or loading a model."""+        if "model" not in body and self.model_manager.force_model is not None:+            body = {**body, "model": self.model_manager.force_model}+        try:+            return TransformersSystemOneRequestParams.model_validate(body, strict=True)+        except ValidationError as error:+            detail = [{**item, "loc": ("body", *item["loc"])} for item in error.errors(include_url=False)]+            raise HTTPException(status_code=422, detail=detail) from error++    async def handle_request(self, body: dict, request_id: str) -> "JSONResponse":+        """Validate the request, load the model, and answer every question.++        Args:+            body (`dict`): The raw JSON request body (System One format).+            request_id (`str`): Unique request identifier (from header or auto-generated).++        Returns:+            `JSONResponse`: `model`, `answers` keyed by question name, and `usage`.+        """+        request = self._validate_request(body)+        model_id, model, processor = self._resolve_model(body)++        logger.warning(f"[Request received] Model: {model_id}, questions: {len(request.questions)}")++        gen_manager: GenerateManager = self.generation_state.get_manager(model_id, use_cb=False)  # type: ignore[assignment]++        inputs, questions, labels, input_tokens = self._prepare_inputs(model, processor, request)+        temperatures = [self.decision_config.temperature.get(question.type, 1.0) for question in questions]+        probabilities = await gen_manager.async_submit(+            self._compute_probabilities, model, inputs, labels, temperatures+        )+        answers = build_answers(questions, probabilities)++        return JSONResponse(+            {+                "model": body["model"],+                "answers": answers,+                "usage": {"input_tokens": input_tokens, "output_tokens": 0},+            }+        )++    def get_processor_inputs_from_request(+        self, request: SystemOneRequest, modality: Modality+    ) -> tuple[list[list[dict]], list[Question]]:+        """Build one processor-compatible conversation per question and retain its answer metadata."""+        if request.images and modality not in (Modality.VLM, Modality.MULTIMODAL):+            raise HTTPException(status_code=400, detail="The selected model does not support image input.")+        questions = []+        processor_inputs = []+        for name, request_question in request.questions.items():+            question = parse_question(name, request_question, self.decision_config.labels.get(request_question.type))+            questions.append(question)+            text = render_decision_prompt(request.state, question)+            if modality == Modality.LLM:+                content = text+            else:+                content = [{"type": "image", "url": image} for image in request.images]+                content.append({"type": "text", "text": text})+            processor_inputs.append([{"role": "user", "content": content}])+        return processor_inputs, questions++    def _prepare_inputs(+        self,+        model: "PreTrainedModel",+        processor: "ProcessorMixin | PreTrainedTokenizerFast",+        request: TransformersSystemOneRequestParams,+    ) -> tuple[dict, list[Question], list[list[int]], int]:+        """Apply the model's chat template and batch text and images for one forward pass."""+        tokenizer = getattr(processor, "tokenizer", processor)+        modality = self.model_manager.get_model_modality(model, processor=processor)+        processor_inputs, questions = self.get_processor_inputs_from_request(request, modality)++        chat_template_kwargs = {+            **self.chat_template_kwargs,+            **request.chat_template_kwargs,+            "enable_thinking": False,+        }+        prompts = []+        for messages, question in zip(processor_inputs, questions):+            if self.decision_config.chat_template is not None:+                chat_template_kwargs.update(+                    chat_template=self.decision_config.chat_template,+                    state=request.state,+                    question=question,+                    images=request.images,+                )+            prompt = processor.apply_chat_template(+                messages, add_generation_prompt=True, tokenize=False, **chat_template_kwargs+            )+            prompts.append(prompt)++        processor_kwargs = {"images": [request.images for _ in questions]} if request.images else {}+        inputs = processor(+            text=prompts,+            padding=True,+            padding_side="left",+            return_tensors="pt",+            add_special_tokens=False,+            **processor_kwargs,+        )+        if modality == Modality.LLM:+            inputs["position_ids"] = (inputs["attention_mask"].cumsum(-1) - 1).clamp(min=0)++        labels = [self._tokenize_labels(tokenizer, question) for question in questions]+        input_tokens = int(inputs["attention_mask"].sum().item())+        return inputs, questions, labels, input_tokens++    @staticmethod+    def _compute_probabilities(+        model: "PreTrainedModel", inputs: dict, labels: list[list[int]], temperatures: list[float] | None = None+    ) -> list[list[float]]:+        """Run the model and normalize each question's logits over its allowed answer tokens."""+        import torch++        inputs = {+            name: value.to(model.device) if isinstance(value, torch.Tensor) else value+            for name, value in inputs.items()+        }+        with torch.inference_mode():+            logits = model(**inputs, use_cache=False, logits_to_keep=1).logits[:, -1]+        probabilities = []+        for i, (question_logits, label_ids) in enumerate(zip(logits, labels)):+            answer_logits = question_logits[label_ids].float()+            if temperatures is not None:+                answer_logits = answer_logits / temperatures[i]+            answer_probabilities = answer_logits.softmax(dim=-1)+            probabilities.append(answer_probabilities.tolist())+        return probabilities++    @staticmethod+    def _tokenize_labels(tokenizer: "PreTrainedTokenizerFast", question: Question) -> list[int]:+        """Tokenize each label independently and require distinct single-token answers."""+        label_tokens = [tokenizer(label, add_special_tokens=False)["input_ids"] for label in question.labels]+        if any(len(ids) != 1 for ids in label_tokens):+            raise HTTPException(+                status_code=400,+                detail=f"The labels of question {question.name!r} are not single tokens for this model.",+            )+        label_ids = [ids[0] for ids in label_tokens]+        if tokenizer.unk_token_id in label_ids:+            raise HTTPException(+                status_code=400,+                detail=f"The labels of question {question.name!r} contain an unknown token for this model.",+            )+        if len(set(label_ids)) != len(label_ids):+            raise HTTPException(+                status_code=400, detail=f"Two labels of question {question.name!r} share a token for this model."+            )+        return label_ids

@@ -31,6 +31,7 @@ from transformers.cli.serving.model_manager import ModelManager, TimedModel from transformers.cli.serving.response import ResponseHandler, compute_usage from transformers.cli.serving.server import build_server+from transformers.cli.serving.systemone import SystemOneHandler from transformers.cli.serving.transcription import TranscriptionHandler from transformers.cli.serving.utils import (     _RESPONSE_TEMPLATE_FALLBACKS,@@ -579,12 +580,14 @@ def setUpClass(cls):         cls.completion_handler = MagicMock(spec=CompletionHandler)         cls.response_handler = MagicMock(spec=ResponseHandler)         cls.transcription_handler = MagicMock(spec=TranscriptionHandler)+        cls.systemone_handler = MagicMock(spec=SystemOneHandler)         cls.app = build_server(             cls.model_manager,             cls.chat_handler,             cls.completion_handler,             cls.response_handler,             cls.transcription_handler,+            systemone_handler=cls.systemone_handler,             generation_state=GenerationState(),         )         cls.transport = httpx.ASGITransport(app=cls.app)@@ -615,6 +618,62 @@ def test_request_id_passthrough(self):         self.assertEqual(resp.headers["x-request-id"], "my-id")  +@slow+@require_serve+class TestSystemOneEndpoint(ServeIntegrationTestCase):+    MODEL = "Qwen/Qwen2.5-0.5B-Instruct"++    def setUp(self):+        super().setUp()+        self.body = {+            "model": self.MODEL,+            "state": "Our integration has been failing for 3 days and we are losing sales!",+            "questions": {+                "team": {+                    "type": "choice",+                    "instructions": "Which team handles this?",+                    "criteria": {"billing": None, "technical": "Bugs or integration problems"},+                },+                "urgency": {+                    "type": "score",+                    "instructions": "How urgent is this ticket?",+                    "criteria": ["low", "medium", "high"],+                },+                "upset": {"type": "noul", "instructions": "Is the customer upset?"},+            },+        }++    def test_response_format(self):+        response = httpx.post(f"{self.base_url}/v1/systemone", json=self.body, timeout=120)+        self.assertEqual(response.status_code, 200)+        data = response.json()+        self.assertEqual(data["model"], self.MODEL)+        answers = data["answers"]+        self.assertEqual(set(answers), {"team", "urgency", "upset"})+        self.assertEqual(answers["team"]["type"], "choice")+        self.assertEqual(answers["urgency"]["type"], "score")+        self.assertEqual(answers["upset"]["type"], "noul")+        self.assertEqual(answers["urgency"]["legend"], {"0": "low", "1": "medium", "2": "high"})+        self.assertAlmostEqual(sum(answers["team"]["probabilities"].values()), 1.0, places=5)+        self.assertAlmostEqual(sum(answers["urgency"]["probabilities"].values()), 1.0, places=5)+        self.assertGreater(data["usage"]["input_tokens"], 0)+        self.assertEqual(data["usage"]["output_tokens"], 0)++    def test_decision_with_text(self):+        response = httpx.post(f"{self.base_url}/v1/systemone", json=self.body, timeout=120)+        self.assertEqual(response.status_code, 200)+        answers = response.json()["answers"]+        self.assertEqual(answers["team"]["choice"], "technical")+        self.assertGreater(answers["urgency"]["score"], 1)+        self.assertGreater(answers["upset"]["noul"], 0.5)++    def test_invalid_question(self):+        self.body["questions"]["team"]["criteria"] = {}+        response = httpx.post(f"{self.base_url}/v1/systemone", json=self.body, timeout=30)+        self.assertEqual(response.status_code, 422)+        self.assertIn("criteria", response.json()["detail"][0]["loc"])++ @slow @require_serve class TestChatCompletion(ServeIntegrationTestCase):@@ -1895,6 +1954,7 @@ def _build_app(self, generation_state, chat_handler=None):             completion_handler=MagicMock(),             response_handler=MagicMock(),             transcription_handler=MagicMock(),+            systemone_handler=MagicMock(spec=SystemOneHandler),             generation_state=generation_state,         )