🤖KillSwitch Programming Language Confuses LLMs With Adverb-Based Code
LLMs Struggle to Decode Adverb-Based Code
TL;DR
KillSwitch, a new programming language, uses adverbs and anagrams to confuse LLMs. Most models score below 50% on coding tasks, highlighting the language's effectiveness in testing AI capabilities.
KillSwitch, a programming language designed to confuse LLMs, uses adverbs as identifiers and anagrams for variable names. Most LLM models score below 50% on coding tasks, with the best performing at 66.8%. This highlights the language's effectiveness in testing AI's ability to discern fact from fiction. The language's author, Dominik Picheta, aims to challenge LLMs' understanding and security guardrails, making it a unique tool for AI research.
Key Points
KillSwitch uses adverbs as identifiers, requiring anagrams for variable names, making code difficult for LLMs to parse.
The best-performing LLM model achieved a 66.8% composite benchmark score on KillSwitch coding tasks.
Most LLM models scored below 50% on the same tasks, highlighting the language's effectiveness in testing AI capabilities.
KillSwitch is stack-based and interpreted, with a limited set of constructs including functions, variables, and if statements.
The language's specification document includes lies and a 'hack statement' to test LLMs' ability to discern fact from fiction.
Why It Matters
If you're researching AI capabilities or security guardrails, KillSwitch offers a unique way to test how LLMs handle unconventional programming. The language's adverb-based syntax and anagram identifiers challenge AI's ability to parse and understand code, making it a valuable tool for developers and researchers.
Comments
Be the first to comment
Enjoyed this article?
Get it daily. 7am. Free. Reads in 5 minutes.
Join 3,566 builders reading daily.