Google Brings Sign-to-Text AI to Pixel Users

Google has introduced sign-language-to-text, or SL2T, a multilingual AI model designed to translate signing into written language. The company says the system brings sign language AI into consumer products rather than keeping it confined to research demonstrations. It initially supports American Sign Language-to-English translation in Gboard and Live Transcribe on Pixel 11, with support for more devices and sign languages planned. The broader aim is to extend language technology to some of the estimated 70 million Deaf and hard of hearing people who use more than 200 sign languages worldwide.

The Gboard integration gives Deaf users a sign-based alternative to typing. A person can sign to their phone when searching the web, writing a message or document, or interacting with Gemini to answer a question or perform a task. In Live Transcribe, users can sign their side of a conversation instead of repeatedly typing responses. Testers told Google that communicating in ASL this way felt faster, more natural, and more enjoyable than entering the same ideas in written English. The central product change is that signing can now function as a general-purpose input method wherever users would ordinarily type.

Google emphasizes that sign languages are independent natural languages and an essential part of Deaf culture, not visual encodings of surrounding spoken languages. Deaf and hard of hearing people also differ in their fluency across signing, speech, reading, and writing, so accessibility requires support for multiple ways of communicating. Sign language processing could offer Deaf communities conveniences comparable to speech recognition and voice dictation while also helping bridge communication between Deaf and hearing people.

Building such a system is substantially harder than converting speech into text. Speech transcription usually maps a sequence of sounds to words in the same language. Sign-to-text systems must instead perform genuine translation because languages such as ASL have their own vocabulary and grammar. They cannot reliably translate a sentence by assigning an English word to each individual sign. This is why earlier concepts such as sensor-equipped signing gloves were inherently limited: they focused mainly on hand motion even though sign languages are not simply “English on the hands.”

The second challenge is visual understanding. Meaning can be expressed simultaneously through the hands, arms, torso, head, facial expression, and the position of movements in space. An AI system must track these fine physical details at a high frame rate and interpret how they work together. That demands both sophisticated computer vision and a language model capable of translating structures that may not follow the order or conventions of written English. SL2T is intended to combine whole-body visual perception with full machine translation rather than treating signing as a sequence of isolated gestures.

Google trained the model on more than 100,000 hours of material covering over 50 sign languages, with ASL accounting for approximately one quarter of the training data. The dataset includes variation in languages, dialects, and users’ signing proficiency. According to the company’s experiments, joint multilingual training helps the model discover structures shared across sign languages and produces better results than training separate systems for only one language. This approach also underpins Google’s claim that the architecture is general enough to expand beyond its initial ASL-to-English release.

The privacy design avoids sending raw camera video to the translation server. MediaPipe Holistic, an on-device model, identifies pose landmarks representing points on the signer’s body. The original video can then be discarded immediately, while only the resulting sequence of geometric coordinates is transmitted for translation. SL2T processes those changing coordinates and generates a streaming text response. This reduces the amount of visually identifiable information handled by the server, although translation itself still relies on server-side processing.

SL2T also avoids an intermediate representation called a gloss. Glosses are written labels commonly used to annotate signs in research datasets, but Google argues that they cannot fully represent features such as facial and other non-manual signals or spatial constructions. A fixed gloss vocabulary can additionally restrict what a model learns. By translating pose landmarks directly into text, SL2T removes that artificial vocabulary bottleneck and lets performance improve as more varied training data becomes available.

Google describes SL2T as its most capable sign language translation model so far and cites the FLEURS-ASL sd-test benchmark, which measures ASL-to-English translation quality. The company reports a zero-shot result of 70 BLEURT, meaning the model is evaluated on the benchmark without being specifically trained on its test examples. The significance lies not only in the reported benchmark performance but in deploying the underlying model as an everyday accessibility feature. The initial release remains limited to ASL-to-English and selected Pixel software, so its practical impact will depend on real-world reliability, device expansion, and how successfully Google adds the many sign languages used by distinct communities.