Streaming supported for all models, plus reasoning_format parameter for reasoning models

Streaming is now supported for all models, including with structured output requests (response_format, tools).

Reasoning Format Controls

Added reasoning_format parameter for reasoning models, giving you control over how reasoning text appears in responses. Options include:

• parsed - Returns reasoning in a separate reasoning field with logprobs separated into reasoning_logprobs
• text_parsed - Returns reasoning in a separate reasoning field (logprobs stay together)
• raw - Includes reasoning inline with content using <think> tags
• hidden - Drops reasoning text from the response (tokens still counted)

See Reasoning for details.

Logprobs with Structured Outputs

logprobs are now supported for structured output requests (response_format, tools), matching OpenAI's API behavior.

When constrained decoding limits the valid token set below the requested top_logprobs, the response is padded with additional tokens from the model vocabulary, each assigned a logprob of -100. If only a single valid token remains after constraint masking, that token will have a logprob of 0 (probability 1). The output will not contain duplicate tokens.

Multi-turn Tool Calling for Llama 3.3 70B

Multi-turn tool calling is now supported for llama-3.3-70b. A bug preventing this functionality has been resolved.