Ben Lipkin - Home

About Me

I am a PhD student in the Brain & Cognitive Sciences (BCS) department at MIT, where I work with Roger Levy, Ev Fedorenko, and many other wonderful mentors and collaborators. I am grateful for funding support from the NSF GRFP and an MIT Presidential Fellowship. My research sits at the interface of cognitive science and AI, where I work on building robust and pragmatic AI systems that can solve problems spanning natural language, programming, and mathematics. Towards this goal, I jointly leverage LLMs alongside tools from Bayesian machine learning and symbolic AI. See ProbSem and LINC for early outputs of this research program, the latter of which won an outstanding paper award at EMNLP '23. Prior to starting my PhD, I studied Computational Neuroscience and Complex Systems at the University of Michigan, and worked for several years on machine learning applications to neuroscience, including publications in Nature, PNAS, and NeurIPS.

Open Source
I care about open source and allocate a portion of my time towards community contributions. These have previously included the development and evaluation of code models with the Star Coder project by Hugging Face and Service Now, the first implementation of CFG-guided text generation for the Outlines library by .txt, and contributions to AI for mathematics with Project Numina. I am also developing the Decoding library, a framework that makes it easy to design and implement custom LLM search and inference algorithms, built from a set of pure composable building blocks.

News

Dec 2024: We presented the EWoK project at Google DeepMind.
Dec 2024: "Models can and should embrace the communicative nature of human-generated math" presented at NeurIPS Workshop on Mathematical Reasoning and AI in Vancouver, BC.
Nov 2024: Initial release of the decoding library.
Oct 2024: Invited research talks on Neurosymbolic AI at Microsoft Research and Pitt NLP Seminar.
Sept 2024: Co-organizer for SFI working group on "Assessing Representation in Minds and Artificial Systems" in Santa Fe, NM.
Aug 2024: Program chair for ACL workshop on "Natural Language Reasoning and Structured Explanations" in Bangkok, Thailand.
June 2024: Joined Project Numina as a contributor, facilitating the release of the models, datasets, and code used to win the first AIMO progress prize.
May 2024: "Elements of World Knowledge (EWOK): A cognition-inspired framework for evaluating basic world knowledge in language models" released as preprint alongside the companion open-source software library and dataset.
May 2024: Student organizer for NSF workshop on "New Horizons in Language Science: Large Language Models, Language Structure, and the Cognitive and Neural Basis of Language" in Arlington, VA.
Apr 2024: "Modeling uncertainty in semantic parsing" presented at the New England NLP Meeting in Providence, RI.
Apr 2024: Awarded NSF GRFP Fellowship.
Apr 2024: "Log probability scores provide a closer match to human plausibility judgments than prompt-based evaluations" presented at the SouthNLP symposium in Atlanta, GA.
Dec 2023: Designed and implemented the first version of CFG-guided generation for the open-source Outlines library.
Dec 2023: "LINC: a neurosymbolic approach for logical reasoning by combining language models with first-order logic provers" wins outstanding paper award at EMNLP conference in Singapore.
Dec 2023: "StarCoder: may the source be with you!" published at TMLR.
Aug 2023: Started at SFI summer program on intelligence and representation at the Isaac Newton Institute in Cambridge, UK.
July 2023: "Evaluating statistical language models as pragmatic reasoners" presented at CogSci in Sydney, Australia, and the ACL NLRSE workshop in Toronto, ON.
May 2023: BigCode evaluation harness released, enabling secure containerized evaluation of code generation language models.
Dec 2022: "This is your brain. This is your brain on code." promotes our work in MIT News and Communications of the ACM.
Nov 2022: "Convergent representations of computer programs in human and artificial neural networks" presented at NeurIPS conference in New Orleans, LA.
Sept 2022: Awarded MIT Presidential Fellowship.
Sept 2022: Started PhD at MIT BCS.