What are the technical hurdles to removing tokenization?
What are the leading alternative approaches? Character-by-character prediction?
Thinking about human brains, it seems we operate on an ever-evolving set of symbols, at various levels of abstraction. I believe LLMs capture higher level symbols in their weights, but would it make sense to try to include a dynamic, self-updating set of symbols into the “what is being predicted” part of the training?
> What are the technical hurdles to removing tokenization?
In a sense, none, character (with whatever encoding scheme you want) models have been done. OTOH, you get effectively a smaller context window (since tokens represent one or more characters), and since tokens are semantic unions you push more work into the main network to acheive similar quality.
> Thinking about human brains, it seems we operate on an ever-evolving set of symbols, at various levels of abstraction.
Analogies between human brains and LLMs are often more obscuring than illuminating, but I think if you want to go down that route here, that’s equivalent to doing continuous fine tuning on both the tokenizer (that takes sensory input and maps to semantically meaningful symbols) and the main model.
> I believe LLMs capture higher level symbols in their weights, but would it make sense to try to include a dynamic, self-updating set of symbols into the “what is being predicted” part of the training?
“Dynamic, self-updating” simply isn’t part of the way modern LLMs work, independent of whether you are looking at the tokenizer or main model. If someone could make it practical, sure, there are lots of things it would potentially improve.
(I'm not an expert on LLMs, this is just my guess)
No tokens is like no compression. It's much less efficient. My previous sentence is 12 tokens (24 bytes), or 58 characters (58 bytes). Your self attention block will be able to handle a much smaller effective context.
Also, especially in english, tokens by itself have quite a lot of semantic meaning by themselves, while characters are basically meaningless. So it probably needs extra work to piece those characters together vs operating on a token that already has meaning.
> tokens by itself have quite a lot of semantic meaning by themselves, while characters are basically meaningless
maybe one approach would be to tokenize not on tokens automatically found using BPE (Byte Pair Encoding) but tokens that are semantically meaningful - word prefixes, suffixes and syllables. Of course we will have to decide whether to divide the word helicopter into heli-copter (pronunciation) or helico-pter (meaning), but both alternatives encode much more transferable meaning than GPT's he-lic-opter.
Of course the difficulty is that that's ... difficult. Your tokenizer would first need to understand what language a word is in, then how to split it in the context of that language. Maybe a task for a whole ML model, where current tokenizers can be simple largest-substring matchers.
Hmmm, is there any way for a model to predict / edit its own tokens as it learns?
Kind of a recursive, self-evolving tokenization process?
I’m pretty new to deep learning, so this analogy might be off. But I’m reminded of convolutional layers inside image-based models. Could we have “tokenization layers” in a GPT model?
---
Edit: I asked GPT-4 about my idea.
It said it the comparison between tokenization and convoluational feature detection is a “someawhat accurate” analogy. And it agreed that making the encoder a fixed separate process that doesn’t evolve during training does limit the GPT model in certain ways and introduce quirks.
But it said that it might increase the computational requirements significantly, and that the transformer-architecture doesn’t lend itself to having “tokenization layers”, and it isn’t clear how one could do that.
It did say that there may be ways to work around the main challenges, and that there is some research in this direction.
That's not "meaning", that's etymology. If you divide it by "meaning" you get heli-copter, which is evidenced by the way the speakers use those morphemes: helipad, heliskiing, heliport,... and parcelcopter, gyrocopter, quadcopter,...
There is more than enough info out there on how LLMs work. Suggesting to remove tokenization is like suggesting to not use neural nets anymore. Why do we need all these weights and all this training? What would be the technical hurdles to just create a library that does all of the above?
Imho, you cannot expect to have a conversation if you don't put the minimum amount of required effort to understand what it's being talked about.
Throwing in wild ideas maybe works when you know what you're talking about and want to get people to think ouside the box, but does not work when you don't have a clue and expect people to explain things to you and to find a solution to your wild ideas.
Re: removing tokenization. Tell me how you would remove tokenization and keep NN. Go ahead.
Look at the interesting discussion that happened as a result of my comment. My comment was completely fine.
Your incredulity and exasperation is misplaced. If you can’t realize that then I think we are done here.
Btw, since we are already in the thick of things… Even if you were 100% correct, which you aren’t, there is really no need to be rude. It just makes things worse.
> Re: removing tokenization. Tell me how you would remove tokenization and keep NN. Go ahead.
You can make a language model using character-by-character prediction, but I believe they don’t perform as well. This is now my 3rd time mentioning this.
It seems like using multiple tokenizations simultaneously would help a lot. It also seems likely that we'll need at least an order of magnitude more processing power for it to be feasible. I think these are good things to be asking.
Yes, character by character is the future. At least for low-density languages like English (around one bit per character), it can be hard for an LLM to start getting the idea of what's going on statistically. Current tokenization methods are quite naive, but progress is being made on improving semantic meaning of tokens and also in improving performance of character-based models.
That might work fine for English, but I would be very surprised if it performed better for non-ASCII logograms e.g. hanzi/kanji. Or would the model just learn those characters on its own?
Interestingly enough, for Japanese, the average is more than 1 token per Japanese character. It's not at the byte level, but it's clearly at a sub-utf-8 level. And yet, considering this, it works amazingly well.
It's probably not necessary though. Just cleaning the input data well enough and then ensuring the vocabulary is matched with the actual training data should be sufficient?
Edit: the token list for GPT-4 looks pretty clean. Overwhelmingly dominated by code but just eyeballing fragments of the set I didn't see any tokens in there that were obviously going to be rare.
The most effective architecture for LLMs without tokenization is Meta's MEGABYTE architecture: https://arxiv.org/abs/2305.07185. The sacrifice here is that they have two different types of layers, instead of the simple uniform transformer layers.
