Technology

59672 readers

2852 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 1 year ago

MODERATORS

[email protected]

595

AI worse than humans in every way at summarising information, government trial finds (www.crikey.com.au)

submitted 2 months ago by [email protected] to c/[email protected]

123 comments fedilink hide all child comments

Artificial intelligence is worse than humans in every way at summarising documents and might actually create additional work for people, a government trial of the technology has found.

Amazon conducted the test earlier this year for Australia’s corporate regulator the Securities and Investments Commission (ASIC) using submissions made to an inquiry. The outcome of the trial was revealed in an answer to a questions on notice at the Senate select committee on adopting artificial intelligence.

The test involved testing generative AI models before selecting one to ingest five submissions from a parliamentary inquiry into audit and consultancy firms. The most promising model, Meta’s open source model Llama2-70B, was prompted to summarise the submissions with a focus on ASIC mentions, recommendations, references to more regulation, and to include the page references and context.

Ten ASIC staff, of varying levels of seniority, were also given the same task with similar prompts. Then, a group of reviewers blindly assessed the summaries produced by both humans and AI for coherency, length, ASIC references, regulation references and for identifying recommendations. They were unaware that this exercise involved AI at all.

These reviewers overwhelmingly found that the human summaries beat out their AI competitors on every criteria and on every submission, scoring an 81% on an internal rubric compared with the machine’s 47%.

you are viewing a single comment's thread
view the rest of the comments

[–] [email protected] 29 points 2 months ago (2 children)

On July 18, 2023, in partnership with Microsoft, Meta announced Llama 2 On April 18, 2024, Meta released Llama-3

L2 it’s one year old. A study like that takes time. What is your point? I bet if they would do it with L3 and the result came back similar, you would say L3 is „insanely outdaded“ as well?

Can you confirm that you think with L3, the result would look completely opposite and the summaries of the AI would always beat the human summaries? Because it sounds like you are implying that.

[–] [email protected] 18 points 2 months ago* (last edited 2 months ago)

We know the performance of L2-70b to be on par with L3-8b, just to put the difference in perspective. Surely they models continue to improve and we can only hope the same improvements will be found in L4, but I think the point is that models have improved dramatically since this study was run and they have put in way more attention in the fine-tuning and alignment phase of training, specifically for these kinds of tasks. Not saying this means the models would beat the human summaries everytime (very likely not), but at the very least the disparity between them wouldn't be nearly as large. Ultimately, human summaries will always be "ground truth", so it's hard to see how models will beat humans, but they can get close.

[–] [email protected] 3 points 2 months ago (1 children)

Can you confirm that you think with L3, the result would look completely opposite and the summaries of the AI would always beat the human summaries? Because it sounds like you are implying that.

Lemmy users try not to make a strawman argument (impossible challenge)

No, that's not what I said, and not even close to what I was implying. If Llama 2 scored a 47% then 3.1 would score significantly better, easily over 60% at least. No doubt humans can be better at summarizing but A) It needs someone that's very familiar with the work and has great English skills and B) It needs a lot of time and effort.

The claim was never that AI can summarize better than people, it was that it can do it in 10 seconds and the result would be "good enough". People are already doing AI summaries of longer articles without much complaints.

[–] [email protected] -1 points 2 months ago (1 children)

Lemmy users try not to make a strawman argument (impossible challenge)

This was not a strawman. Please don’t assume lemmy users make logical fallacies when it’s only you who thinks that.

[–] [email protected] 2 points 2 months ago (1 children)

I guess you missed the part where he said "Oh you said X but you're actually implying Y? Did you mean Y? Please confirm you actually meant Y."

[–] [email protected] 0 points 2 months ago* (last edited 2 months ago)

That’s my point, from my perspective, there was no switch. Using a one year old model is fine.

My comment was about how people looking at the same thing, one might think it’s a bait and switch while the other one always knew the second item was being implied.

The headline never said all AI or latest AI.