Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Why would it make latency go up? It's incredibly parallelizable just copy the weights on more machines.


>It's incredibly parallelizable

No, it isn't. It generates 1 token at a time in a loop until the response is finished. It's a highly serialized task. Parallelization increases the throughput of how many queries can be processed simultaneously, but you wouldn't be able to speed up a single query.


Ah the ol’ have a baby in a month by impregnating 9 women strategy.


The additional cpu to gpu to cpu round trip time I'm guessing?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: