- cross-posted to:
- technology@beehaw.org
- cross-posted to:
- technology@beehaw.org
You must log in or # to comment.
Microsoft says its Agent Mode in Excel has an accuracy rate of 57.2 percent in SpreadsheetBench, a benchmark for evaluating an AI model’s ability to edit real world spreadsheets.
It generates 42.8% bullshit.
They probably view that as a statistic worth bragging about. It’s not. If Excel got calculations right 57.2% of the time it would be completely worthless.
I asked copilot to look through my every spreadsheet and find how many instances of a category occurred. I was curious to see if it was any good. Gave me 2 different numbers. Neither were correct.
Copilot: Putting the “Artificial” in Artificial Intelligence.
The tech behind LLMs could have just been Clippy and everyone would be happy.
Did you read the next sentence? Humans only get like 72% right. It’s not far off at all.
I wonder where that “human accuracy” statistic is coming from. Plenty of people don’t know how to read and interpret data, much less use excel in the first place. There’s a difference between 1/4 of people in the workforce not being able to complete a task, and a specialized AI not being able to complete a task. Additionally, this is how you get into the KPI as a goal rather than a proxy issue. AI will never understand context isn’t directly provided in the workbook. If you introduced a new drink at your restaurant in 2020 AI will tell you that the introduction of the drink caused a 100% decrease in foot traffic since there’s no line item for “global pandemic”. I’m not saying AI will never be there, but people using this version of AI instead of actual analysis don’t care about the facts and just want an answer and for that answer to be cheap.
As I’ve said many times, though not in this topic - AI is a tool to be used, and using it is a skill that needs to be learned.
For your pandemic example, that’s something that you would need to provide the AI with the context of. The joke of a “prompt engineer” being a job soon actually has merit, in that you want people who know how to use their tools the best. It’s constantly learning through iteration to give the AI a specific instruction set to get the results you want/need.
Depending on where you go to school, 70% is passing while 50% is not. While “not far off,” one is a C, the other a F.
That’s not at all what this means. In this instance, 70% is basically “human level”. For AI to already get 57% it means that it’s approaching the same level as people do in Excel.
So it achieved the actual proficiency of a middle manager…
Decades ago. The company that replaced it’s CEO with a LLM thrives.
Nice. Basically a coin flip
Slightly better than Vegas. Unfortunately, plenty of people are okay with Vegas odds.
Not enough accuracy to be useful. Not enough bullshit for politics.
The best cancers of both worlds.
Oh it’s going to do it for Word too?
Prompt: Termination letter telling my boss and bosses to kindly go fuck themselves and make it professional
The best you can do in any job is to care as little about them as they care about you.
They will barely read it, and they won’t care nearly as much as you do.
I resign my position as a [position], effective [DATE].
They’re out smarting the sheet that’s for sure.
So let me fast forward a bit, ->underpaid stressed out techworkers in the global south pretending to be AI for incompetent upper management in wealthy countries?
Not related but does global south refer to south of the equator or just everything south of north America?
I don’t know if it is a perfect term, but it doesn’t literally refer to any specific “South”, rather I think it is a reference to the coincidence that many of the heavily industrialized empires of the 18th, 19th and 20th centuries have been in the northern hemisphere, and the general colonial power dynamic therein set up has lead to the term “Global South” meaning pretty much anywhere that has gotten the short end of the colonialism stick, vs the long end.
https://en.m.wikipedia.org/wiki/Global_North_and_Global_South
Excel is one place where AI makes sense. All the data is there, in a nice structured and typed format with headings etc. Easily verifiable and to provide the reasoning for its work.
LLMs can’t count. Can’t add. Can’t deal with actually large datasets
How is excel a good fit for vibe-coding?
This isn’t just an LLM. It uses excel functions and features to do the counting and adding and dealing with large data sets.
It’s not “vibe coding” as much as “vibe performing steps in excel”.
Also LLMs absolutely can deal with large data sets anyway. Not sure where you got that from.
Until it starts pulling data from nonexistent worksheets
You tell it not to.
I swear none of you guys have even attempted to use AI to do data analysis. I have, I built a MCP and integrated a copilot agent into Teams which has access to specific database data, and refined the rules for it to the point where the CFO rigorously tested it (and still does) and trusts the results it returns.
It could be good to layer in standard machine learning (ML), and it already does have some features (like line of best fit).
However, in today’s context AI means LLMs, and that is not a good fit due to its unpredictability.
