I hardly see it changed to be honest. I work in the field too and I can imagine LLMs being good at producing decent boilerplate straight out of documentation, but nothing more complex than that.
I think one of the top lists on advent of code this year is a cheater that fully automated the solutions using LLMs. Not sure which LLM though, I use LLMs quite a bit and ChatGPT 4o frequently tells me nonsense like “perhaps subtracting by zero is affecting your results” (issues I thought were already gone in GPT 4, but I guess not, Sonnet 3.5 does a bit better in this regard).
Maybe some postmortem analysis will be interesting.
The AoC is also a context in which the domain is self-contained and there is probably a ton of training material on similar problems and tasks.
I can imagine LLM might do decently there.
Also there is no big consequence if they don’t and it’s probably possible to bruteforce (which is how many programming tasks have been solved).
I think one of the top lists on advent of code this year is a cheater that fully automated the solutions using LLMs. Not sure which LLM though, I use LLMs quite a bit and ChatGPT 4o frequently tells me nonsense like “perhaps subtracting by zero is affecting your results” (issues I thought were already gone in GPT 4, but I guess not, Sonnet 3.5 does a bit better in this regard).
Maybe some postmortem analysis will be interesting. The AoC is also a context in which the domain is self-contained and there is probably a ton of training material on similar problems and tasks. I can imagine LLM might do decently there.
Also there is no big consequence if they don’t and it’s probably possible to bruteforce (which is how many programming tasks have been solved).