The issue is I have 30ndifferent harnesses that I proof and stress test all my ideas against. So by the time I have gone back and forth, pangram always labels me 100% AI....it's really frustrating
This is a useful model with a logical basis: Ryle's "knowing that" vs "how", which ties nicely to propositional vs procedural knowing. It feels true that a model can hand you the "thats" (code, text, even arguments) to varying levels of validity, but not the "how" as easily. In part I think it's that transferring the how requires both the model and user know where the user is at in terms of their current capabilities to help get them improve. So does that imply it's got to evaluate you, while you as the user have to be honest about your self-evaluation?
The piece also made me wonder if there is an unseen challenge: we might assume the model's "how" is a process that a user can learn from. But the way the models do things might not actually work for a human if they tried it the way the model does. I know that's not quite the point of this piece, but especially in education and even in basic LLM-human collaboration, if we expect some transfer of (or even transparency in the) "how", the two processes (the machine's "how" vs the human's "how") may not match. For example it's often hard to understand WHY the LLM did certain things.
To put it another way, I've been increasingly thinking that LLMs may have a hidden limit in education because how they "think" or "reason" about things isn't the same way a human would. If so, can they really show us the "how"? I don't mean they can't generate/repeat/parrot a good "customized" lecture, I mean can they truly tutor or mentor you through a process if theirs is so different from ours?
One last, nice bit of hope comes out of this piece for anyone that feels like they're "falling behind." If you can reflect on where you are on this framework, it may help offset that fear a bit. Especially if you realize the model can't do the climb but you can. And that by doing it you make the combined output better. Maybe this map can at least point you in a useful direction.
I really like your point in the second paragraph. It wasn't my focus but I think you are spot on that the "how" for the AI system and for the human might be quite different. That does seem like a hidden limit to me as well
The part that i liked is that the person describing the application isn't always the one fixing it six months later.
and this is usually when you find out whether the understanding was built alongside the application or skipped along the way.
Oh that’s an excellent point! It needs to be made clearer in future iterations of this idea.
Tests as 100% AI. Geez Louise.
The issue is I have 30ndifferent harnesses that I proof and stress test all my ideas against. So by the time I have gone back and forth, pangram always labels me 100% AI....it's really frustrating
Everything I write is gonna test this way
My “Geez Louise“ was really more about the existence of the “detector.“ As you know, I think they are ridiculous.
Oh I know but I am so frustrated by this!!!
This is a useful model with a logical basis: Ryle's "knowing that" vs "how", which ties nicely to propositional vs procedural knowing. It feels true that a model can hand you the "thats" (code, text, even arguments) to varying levels of validity, but not the "how" as easily. In part I think it's that transferring the how requires both the model and user know where the user is at in terms of their current capabilities to help get them improve. So does that imply it's got to evaluate you, while you as the user have to be honest about your self-evaluation?
The piece also made me wonder if there is an unseen challenge: we might assume the model's "how" is a process that a user can learn from. But the way the models do things might not actually work for a human if they tried it the way the model does. I know that's not quite the point of this piece, but especially in education and even in basic LLM-human collaboration, if we expect some transfer of (or even transparency in the) "how", the two processes (the machine's "how" vs the human's "how") may not match. For example it's often hard to understand WHY the LLM did certain things.
To put it another way, I've been increasingly thinking that LLMs may have a hidden limit in education because how they "think" or "reason" about things isn't the same way a human would. If so, can they really show us the "how"? I don't mean they can't generate/repeat/parrot a good "customized" lecture, I mean can they truly tutor or mentor you through a process if theirs is so different from ours?
One last, nice bit of hope comes out of this piece for anyone that feels like they're "falling behind." If you can reflect on where you are on this framework, it may help offset that fear a bit. Especially if you realize the model can't do the climb but you can. And that by doing it you make the combined output better. Maybe this map can at least point you in a useful direction.
I really like your point in the second paragraph. It wasn't my focus but I think you are spot on that the "how" for the AI system and for the human might be quite different. That does seem like a hidden limit to me as well