I think it’s important to come up with other forms of generating synthetic data that doesn’t come from distilling other models. Translating documents, OCRing old documents and using Digital Twins to train visual models come to mind. I’ve never successfully trained any model text-related, but I think the quality of the original text should be critical in how it will perform.
Not only the big players extract data from the common citizen, but it also enforces information upon them. AI will make people interact through exchange of knowledge less, and concentrate all the “talk” and information on the hands of few. I think this is a big problem, especially as we near the quantum computation era. How can individuals and smaller organizations possibly compete in AI quality on that scenario? But maybe hardware power won’t be the greatest force in Artificial Intelligence.
You mentioned smaller models achieving better results than ChatGPT, but those models have trouble extending their knowledge to a wide variety of topics, which is shown by their subpar performance in GPQA (general knowledge) tests.