You can't get GI through spicy autocorrect ? π±
this post was submitted on 14 Apr 2024
274 points (92.0% liked)
Futurology
1762 readers
115 users here now
founded 1 year ago
MODERATORS
Lotta you meatbags are awful confident in your own complexity.
Apparently not, given the content of this article
Even if the model stops here - did you imagine it'd get this far?
Humans do all their civilization brouhaha on three pounds of wet meat powered by corn flakes. Most of which evolved for marginal improvements on "grab branch and pull" or "do not pet tiger." It's a cosmic accident that's given us language and music and dubstep. And this stupid trick with a pile of video cards can fake a lot of that, to the point we're worried the average human will be able to spot the fakes.
Point being: the miraculous birth of a computer intellect may well arise from "the fact blender." Or "fancy Wikipedia." Or "twenty questions, hard mode." Or any other stupid gimmick that some grad students can cobble together after a 4 AM what-if. Calling this hot mess "spicy autocorrect" is accurate, and in some sense damning, but we had no fucking idea where it'd stop. Emergent properties are chaos. Approximate knowledge of conditions cannot predict approximate outcomes.
LLMs are still liable to figure out math. That's a process which gigabytes of linear algebra can obviously do, which would massively improve its ability to guess the next letter in a word problem. It won't be the kind of AI you can explain calculus to, and then expect it to remember, next time - but getting any portion of the way there is deeply spooky.
Humans do all their civilization brouhaha on three pounds of wet meat powered by corn flakes
Dude you're a poet
load more comments
(1 replies)
OP, you do realize that this paper is about image generation and classification based on related data sets and only relates to the image processing features of multimodal models, right?
How do you see this research as connecting to the future scope of LLMs?
And why do you think that the same leap we've now seen with synthetic data transmitting abstract capabilities in text data won't occur with images (and eventually video)?
Edit: Which LLMs do you see in the models they tested:
Models. We test CLIP [91] models with both ResNet [53] and Vision Transformer [36] architecture, with ViT-B-16 [81] and RN50 [48, 82] trained on CC-3M and CC-12M, ViT-B-16, RN50, and RN101 [61] trained on YFCC-15M, and ViT-B-16, ViT-B-32, and ViT-L-14 trained on LAION400M [102]. We follow open_clip [61], slip [81] and cyclip [48] for all implementation details.
I don't see how that paper has anything to do with OPs theory.
I mean, if we're playing devil's advocate to the "WTF is OP talking about" position, I can kind of see the argument around how exponential needs for additional training data combined with the ways in which edge cases are underrepresented from synthetic data sources leading to model collapse could be extrapolated to believing that we've hit a plateau resulting from a training data bottleneck.
In theory there's an inflection point at which models become sophisticated enough that they can self-sustain with generating training data to recursively improve and whether we will hit that point or not is an open question with arguments on both sides.
I agree that this paper in relation to the title isn't exactly the best form of the argument, but I can see how someone only kind of understanding what's being covered could have felt it was confirming their existing beliefs around where models currently are at and will be in the future.
The only thing I'll add is that I was just getting a nice laugh out of looking at if Gary Marcus (a common AI skeptic) has ever been right about anything to date, and saw he had a long post about how deep learning was hitting a wall and we were a far way off from LLMs understanding human text...four days before GPT-4 released.
In my experience, while contrarian positions to continuing research trends can be correct in a "even a broken clock is right twice a day" sense, personally I wouldn't put my bets on a reversal of a trend that in its pacing and replication seems to be accelerating, not decelerating.
In particular regarding OP's claim, the work over the past 18 months with synthetic data sets from GPT-4 giving tiny models significant boosts in critical reasoning skills during fine tuning should give anyone serious pause on "we're hitting diminishing returns and model collapse."
In theory thereβs an inflection point at which models become sophisticated enough that they can self-sustain with generating training data to recursively improve
That sounds surprising. Do you have a source?
Added to this finding, there's a perhaps greater reason to think LLMs will never deliver AGI. They lack independent reasoning. Some supporters of LLMs said reasoning might arrive via "emergent behavior". It hasn't.
People are looking to get to AGI in other ways. A startup called Symbolica says a whole new approach to AI called Category Theory might be what leads to AGI. Another is βobjective-driven AIβ, which is built to fulfill specific goals set by humans in 3D space. By the time they are 4 years old, a child has processed 50 times more training data than the largest LLM by existing and learning in the 3D world.
They can quite possibly be a useful component. They're the language center of the brain.
People who ever thought they would actually resemble intelligence were woefully uninformed of how complex intelligence is.
How complex is intelligence, though? People who were sure they don't were drawing from information we don't actually have.
Yeah, so many people are confidently stating "LLMs can't think like humans do!" When we're actually still pretty unclear on how humans think.
Sure, an LLM on its own may not be an AGI. But they're remarkably closer than we would have predicted they could get just a few years ago, and it may well be that we just need to add a bit more "special sauce" (memory, prompting strategies, perhaps a couple of parallel LLMs that specialize in different types of reasoning) to get them over the hump. At this point a lot of the research isn't going into simply "make it bigger!", it's going into "use LLMs smarter."
load more comments
(1 replies)
Obscenely.
The brain is stacks on stacks of insanely complicated systems. The fact that we know a ridiculous amount about the brain and are barely scratching the surface is exactly the point.
load more comments
(9 replies)
I wonder where the line is drawn between an emergent behavior and a hallucination.
If someone expects factual information and gets a hallucination, they will think the llm is dumb or not helpful.
But if someone is encouraging hallucinations and wants fiction, they might think it's an emergent behavior.
In humans, what is the difference between an original thought, and a hallucination?
Hallucinations are unlike Human creative output. For one, ai hallucinations are unintentional. There's plenty of reasons if you actually think about the question why they are not the same. They are at best dreamlike, but dreams are an intentional process.
load more comments
(4 replies)
If you're thinking about clicking the link to find out what AGI is, don't bother π
Artificial General Intelligence. Basically what most people think of when they hear AI compared to how its often used by computer scientists.
If youβre unsure, it stands for artificial general intelligence, an actual full AI like weβre used to from Sci-fi
Ah, tyvm, tvkoy
It stands for adjusted gross income. Ignore the AI wave. Do your taxes!
What about LLM? Does it say what it means?
I know that's Large Language Model because the phrase has been bandied about for a while now
load more comments
(1 replies)
I'm glad, you know. Now we're talking about preparing for AGI, but if it's not imminent we also have some time to actually do it.
view more: next βΊ