Comment by amelius
15 hours ago
I personally think our academia should be training and curating these kinds of models and the data they are based on, but this is an acceptable second best.
15 hours ago
I personally think our academia should be training and curating these kinds of models and the data they are based on, but this is an acceptable second best.
IMO there are much better ways to spend 300 million in research beyond firing up a cluster for 60 days to train on internet content.
Spending $100 million one time on a GPT4-level model that is open-source would help with a lot of that research. Especially after all the 3rd party groups fine-tuned it or layered their tools on it.
I think the Copilot-equivalent tools alone would make it quickly pay itself off in productivity gains. Research summaries, PDF extraction, and OCR would add more to that.
there's already https://www.goody2.ai/chat
basically no one in the entire world was willing to spend the kind of money on massive compute and data centers that Meta did spend, is spending and will spend. The actual numbers are (I think) rare to find and so large that it is hard to comprehend it.