Just how good are locally hostable code models on Cambridge first year OCaml assignments? @anil.recoil.org , @jon.recoil.org and I wanted to find out, so ran some tests. TL;DR Qwen3 means we might need new assignments. toao.com/blog/ocaml-l... - ThreadSky

sadiq.toao.com • 23 days ago

Just how good are locally hostable code models on Cambridge first year OCaml assignments? @anil.recoil.org , @jon.recoil.org and I wanted to find out, so ran some tests. TL;DR Qwen3 means we might need new assignments. https://toao.com/blog/ocaml-local-code-models

Comments

edwintorok.bsky.social•23 days ago

Some thoughts on what could confuse AI models:
*newer language feature, e.g. `{|` for strings containing " chars.

* a recursive function, and a non-recursive wrapper with the same name that supplies the initial value

* String vs Bytes

* Use Stdlib modules introduced in newer versions

edwintorok.bsky.social•23 days ago

It'd be interesting if AI models came with a plagiarism detector, watermark their output. Wouldn't help against self-hosted models.
Perhaps run the exam questions through all the models and use plagiarism detection tools, although that requires unique assignments that admit multiple solutions, hard.

asbradbury.org•23 days ago

I think the degree to which assignments should change is an interesting one. For beginning programming questions, Google and StackOverflow etc have been there for a long time.

asbradbury.org•23 days ago

At least for unsupervised exercises, maybe the risk is that students no longer feel inclined to limit the use of such tools if they believe they're not gaining much in the way of useful skills when doing so.

asbradbury.org•23 days ago

Asking an LLM to spit out a red-black tree implementation in language X seems like cheating (and cheating yourself from better understanding how it works). But writing detailed pseudocode and having an LLM translate it to the syntax of the target language seems much less clear cut.

sadiq.toao.com•23 days ago

This is a thorny question and mostly comes down to what we're trying to teach. I wonder if a progressive approach where at early stages of teaching there is no automatic tooling but as critical skills are learnt more can be automated. It's a bit of a moving target at the moment though.

sadiq.toao.com•23 days ago

One thing I probably should highlight more in the post is that the proprietary models (like Claude and Gemini) that most students currently have access to can already ace the assignments.

Comments

Posting Rules

Reply