gemma-tokenizer

The text tokenizer used in Google's Gemma models.

Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("much1na/gemma-tokenizer")

ids = tokenizer("hello world").input_ids
print(ids)  # [2, 23391, 1902, 1]
print(tokenizer.decode(ids, skip_special_tokens=True))  # hello world
Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support