Thinking Machines Releases Trillion-Parameter Inkling Model

Thinking Machines has released Inkling on Hugging Face, presenting it as a large open multimodal language model designed to reason across text, images, and audio. The model has approximately one trillion parameters, accepts a context of up to one million tokens, and was trained on 45 trillion tokens spanning text, images, audio, and video. Its intended role is not limited to general-purpose prompting: the company emphasizes domain adaptation through fine-tuning and sees Inkling as a foundation for a new generation of applications that must combine information from several media types. Its central proposition is native multimodal reasoning at an unusually large scale, with agentic capabilities and a one-million-token context window.

Inkling is a decoder-only, causal autoregressive model, meaning it generates outputs step by step from the information already available in its context. It uses a sparse Mixture-of-Experts architecture with 975 billion total parameters but activates only about 41 billion parameters for a given token. This design is intended to provide much of the capacity of a very large network without using every parameter during each inference step. The model contains 256 experts. Its router evaluates both routed and shared experts, selects six routed experts, and keeps two shared experts active, creating what the authors describe as a shared-expert sink.

The attention system differs from the rotary positional embeddings commonly used in modern language models. Inkling instead applies relative attention, allowing each attention layer to learn positional relationships directly in its attention scores. In addition to the standard key, query, and value projections, it creates a fourth, per-token and per-head relative feature. This feature is modified using the distance between the relevant key and query positions before entering the attention calculation. The goal is to encode how tokens relate by distance without relying on RoPE.

Inkling also alternates between two attention patterns to balance reach and computational cost. Global-attention layers can examine the entire available context, while sliding-window layers operate over a limited moving region. The architecture uses five sliding-window layers for every global-attention layer, and the final layer returns to global attention to form richer representations from the complete context. A short one-dimensional convolution, called SConv, supplements this arrangement by processing the current hidden state together with the preceding states inside its window. The stated intuition is that SConv can capture nearby patterns, leaving the attention and expert modules more capacity for other relationships.

For images, Inkling employs a comparatively simple hierarchical MLP patchifier rather than a large independent vision encoder. Successive linear layers progressively merge pixels until the system produces one embedding for each image patch. The image tower also supports an additional temporal dimension intended for video inputs. Thinking Machines expects that feature to become useful after downstream fine-tuning, but the published description explicitly says that the model's out-of-the-box video performance has not yet been evaluated. Neighboring patches are folded together by stacking a small local grid into the channel dimension before it is processed by the hierarchical MLP.

Audio follows a similarly direct route into the common model. The waveform is divided into chunks of 100 milliseconds and converted to the mel scale, a representation organized around perceived sound frequencies. Each chunk is classified into a discrete mel-spectrogram bin. Those bin values are embedded by the audio tower, and the resulting embeddings are summed to construct the model's audio input. Images and audio therefore reach the decoder through relatively compact embedding modules instead of fully separate, heavyweight encoders. This unified approach is meant to support reasoning that crosses modality boundaries rather than treating each medium as an isolated task.

The release includes full BF16 weights and a calibrated NVFP4 version aimed at substantially reducing memory requirements. It also includes speculative multi-token-prediction layers intended to accelerate inference by helping the system propose more than one future token at a time. Even after quantization, Inkling remains infrastructure-intensive: the BF16 checkpoint requires about 2 TB of VRAM, while the NVFP4 checkpoint requires roughly 600 GB. The BF16 model is intended for Nvidia Hopper-generation or newer hardware, whereas the NVFP4 checkpoint targets Nvidia Blackwell GPUs.

Thinking Machines and Hugging Face are positioning software availability as an important part of the launch. Inkling arrives with immediate support in Transformers and compatibility with major inference engines, including SGLang and vLLM; the release also points to llama.cpp and GGML quantizations as a route toward local deployment. Users without the necessary hardware can access the model through serverless inference routers such as Hugging Face Inference Providers. For direct use through Transformers, the recommended interface is the any-to-any pipeline, with users instructed to update to Transformers 5.14.0.

The combination of open weights, multimodal inputs, sparse expert routing, long context, and established inference integrations makes Inkling primarily a platform for experimentation and specialized adaptation rather than an ordinary desktop model. Its practical accessibility will depend heavily on hosted services, aggressive quantization, or large GPU systems, despite the efficiency gained by activating only 41 billion parameters at a time. The launch description reports demonstrations and architectural exploration by the Hugging Face team, but it does not provide benchmark results or evidence for native video quality. Claims about its suitability for multimodal applications should therefore be separated from capabilities that have not yet been formally evaluated.