My research interests broadly lie in the fields of Machine Learning, Natural Language Processing and Speech Processing. In the past, I have worked on time-series and sequence-to-sequence tasks. Recently, I have been working on reasearch promblems in prompt-based learning and in-context learning with pretrained large language models in zero-shot and few-shot settings as well as evaluation model-generated step-by-step reasoning. I am excited about problems involving (task) instruction-based learning, evaluation of consistency and reasoning abilities of large language models. I am also interested in building machine learning models that robust to distributional shifts and are not dependent on dataset biases or spurious correlations.

Publications and Patents [Google Scholar] [Semantic Scholar]

• Soft Self-Consistency Improves Language Model Agents
  Han Wang*, Archiki Prasad*, Elias Stengel-Eskin*, and Mohit Bansal
  Arxiv Preprint
  [abstract] [code]

  
  

  Generations from large language models (LLMs) can be improved by sampling and scoring multiple solutions to select a final answer. Current "sample and select" methods such as self-consistency (SC) rely on majority voting to score answers. However, when tasks have many distinct and valid answers, selection by voting requires a large number of samples. This makes SC prohibitively expensive for interactive tasks that involve generating multiple actions (answers) sequentially. After establishing that majority voting fails to provide consistent gains on such tasks, we demonstrate how to increase success rates by softening the scoring criterion. We introduce Soft Self-Consistency (Soft-SC), which replaces SC's discontinuous scoring with a continuous score computed from model likelihoods, allowing for selection even when actions are sparsely distributed. Soft-SC improves both performance and efficiency on long-horizon interactive tasks, requiring half as many samples as SC for comparable or better performance. For a fixed number of samples, Soft-SC leads to a 1.3% increase over SC in absolute success rate on writing bash programs, a 6.6% increase on online shopping (WebShop), and a 4.7% increase for an interactive household game (ALFWorld). Finally, we show that Soft-SC can be applied to both open-source and black-box models.

• ReGAL: Refactoring Programs to Discover Generalizable Abstractions
  Elias Stengel-Eskin*, Archiki Prasad*, and Mohit Bansal
  Arxiv Preprint
  [abstract] [code]

  
  

  While large language models (LLMs) are increasingly being used for program synthesis, they lack the global view needed to develop useful abstractions; they generally predict programs one at a time, often repeating the same functionality. Generating redundant code from scratch is both inefficient and error-prone. To address this, we propose Refactoring for Generalizable Abstraction Learning (ReGAL), a gradient-free method for learning a library of reusable functions via code refactorization, i.e. restructuring code without changing its execution output. ReGAL learns from a small set of existing programs, iteratively verifying and refining its abstractions via execution. We find that the shared function libraries discovered by ReGAL make programs easier to predict across diverse domains. On three datasets (LOGO graphics generation, Date reasoning, and TextCraft, a Minecraft-based text game), both open-source and proprietary LLMs improve in accuracy when predicting programs with ReGAL functions. For CodeLlama-13B, ReGAL results in absolute accuracy increases of 11.5% on graphics, 26.1% on date understanding, and 8.1% on TextCraft, outperforming GPT-3.5 in two of three domains. Our analysis reveals ReGAL's abstractions encapsulate frequently-used subroutines as well as environment dynamics.

• ADaPT: As-Needed Decomposition and Planning with Language Models
  Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot
  NAACL 2024 (Findings)
  [abstract] [code] [project page]

  
  

  Large Language Models (LLMs) are increasingly being used for interactive decision-making tasks requiring planning and adapting to the environment. Recent works employ LLMs-as-agents in broadly two ways: iteratively determining the next action (iterative executors) or generating plans and executing sub-tasks using LLMs (plan-and-execute). However, these methods struggle with task complexity, as the inability to execute any sub-task may lead to task failure. To address these shortcomings, we introduce As-Needed Decomposition and Planning for complex Tasks (ADaPT), an approach that explicitly plans and decomposes complex sub-tasks as-needed, i.e., when the LLM is unable to execute them. ADaPT recursively decomposes sub-tasks to adapt to both task complexity and LLM capability. Our results demonstrate that ADaPT substantially outperforms established strong baselines, achieving success rates up to 28.3% higher in ALFWorld, 27% in WebShop, and 33% in TextCraft -- a novel compositional dataset that we introduce. Through extensive analysis, we illustrate the importance of multilevel decomposition and establish that ADaPT dynamically adjusts to the capabilities of the executor LLM as well as to task complexity.

• Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models
  Archiki Prasad, Elias Stengel-Eskin and Mohit Bansal
  ICLR 2024
  [abstract] [code]

  
  

  An increasing number of vision-language tasks can be handled with little to no training, i.e., in a zero and few-shot manner, by marrying large language models (LLMs) to vision encoders, resulting in large vision-language models (LVLMs). While this has huge upsides, such as not requiring training data or custom architectures, how an input is presented to a LVLM can have a major impact on zero-shot model performance. In particular, inputs phrased in an underspecified way can result in incorrect answers due to factors like missing visual information, complex implicit reasoning, or linguistic ambiguity. Therefore, adding visually grounded information to the input as a preemptive clarification should improve model performance by reducing underspecification, e.g., by localizing objects and disambiguating references. Similarly, in the VQA setting, changing the way questions are framed can make them easier for models to answer. To this end, we present Rephrase, Augment and Reason (RepARe), a gradient-free framework that extracts salient details about the image using the underlying LVLM as a captioner and reasoner, in order to propose modifications to the original question. We then use the LVLM's confidence over a generated answer as an unsupervised scoring function to select the rephrased question most likely to improve zero-shot performance. Focusing on three visual question answering tasks, we show that RepARe can result in a 3.85% (absolute) increase in zero-shot accuracy on VQAv2, 6.41%, and 7.94% points increase on A-OKVQA, and VizWiz respectively. Additionally, we find that using gold answers for oracle question candidate selection achieves a substantial gain in VQA accuracy by up to 14.41%. Through extensive analysis, we demonstrate that outputs from RepARe increase syntactic complexity, and effectively utilize vision-language interaction and the frozen language model in LVLMs.

• ReCEval: Evaluating Reasoning Chains via Correctness and Informativeness
  Archiki Prasad, Swarnadeep Saha, Xiang Zhou and Mohit Bansal
  EMNLP 2023
  [abstract] [code]

  
  

  Multi-step reasoning ability is fundamental to many natural language tasks, yet it is unclear what constitutes a good reasoning chain and how to evaluate them. Most existing methods focus solely on whether the reasoning chain leads to the correct conclusion, but this answer-oriented view may confound the quality of reasoning with other spurious shortcuts to predict the answer. To bridge this gap, we evaluate reasoning chains by viewing them as informal proofs that derive the final answer. Specifically, we propose ReCEval (Reasoning Chain Evaluation), a framework that evaluates reasoning chains through two key properties: (1) correctness, i.e., each step makes a valid inference based on the information contained within the step, preceding steps, and input context, and (2) informativeness, i.e., each step provides new information that is helpful towards deriving the generated answer. We implement ReCEval using natural language inference models and information-theoretic measures. On multiple datasets, ReCEval is highly effective in identifying different types of errors, resulting in notable improvements compared to prior methods. We demonstrate that our informativeness metric captures the expected flow of information in high-quality reasoning chains and we also analyze the impact of previous steps on evaluating correctness and informativeness. Finally, we show that scoring reasoning chains based on ReCEval can improve downstream performance of reasoning tasks.

• MeetingQA: Extractive Question-Answering on Meeting Transcripts
  Archiki Prasad, Trung Bui, Seunghyun Yoon, Hanieh Deilamsalehy, Franck Dernoncourt and Mohit Bansal
  ACL 2023
  [abstract] [code + data] [project page]

  
  

  With the ubiquitous use of online meeting platforms and robust automatic speech recognition systems, meeting transcripts have emerged as a new and interesting domain for natural language tasks. Most recent works on meeting transcripts are restricted to summarization and extraction of action items. However, meeting discussions also have a useful question-answering (QA) component, crucial to understanding the discourse or meeting content, and can be used to build interactive interfaces on top of long transcripts. Hence, in this work, we leverage this inherent QA component of meeting discussions and introduce MeetingQA, an extractive QA dataset comprising questions asked by meeting participants and corresponding responses. As a result, questions can be open-ended and seek active discussions, while the answers can be multi-span and spread across multiple speakers. Our comprehensive empirical study of several robust baselines including long-context language models and recent instruction-tuned models reveals that models perform poorly on this task (F1 = 57.3) and severely lag behind human performance (F1 = 84.6), thus presenting a useful, challenging new task for the community to improve upon.

• GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Models
  Archiki Prasad, Peter Hase, Xiang Zhou and Mohit Bansal
  EACL 2023
  [abstract] [code]

  
  

  Providing natural language instructions in prompts is a useful new paradigm for improving task performance of large language models in a zero-shot setting. Recent work has aimed to improve such prompts via manual rewriting or gradient-based tuning. However, manual rewriting is time-consuming and requires subjective interpretation, while gradient-based tuning can be extremely computationally demanding for large models and requires full access to model weights, which may not be available for API-based models. In this work, we introduce Gradient-free Instructional Prompt Search (GrIPS), a gradient-free, edit-based search approach for improving task instructions for large language models. GrIPS takes in instructions designed for humans and automatically returns an improved, edited prompt, while allowing for API-based tuning. The instructions in our search are iteratively edited using four operations (delete, add, swap, paraphrase) on text at the phrase-level. With InstructGPT models, GrIPS improves the average task performance by up to 4.30 percentage points on eight classification tasks from the Natural-Instructions dataset (with similar improvements for OPT, BLOOM, and FLANT5). We see improvements for both instruction-only prompts and for k-shot example+instruction prompts. Notably, GrIPS outperforms manual rewriting following the guidelines in Mishra et al. (2022) and also outperforms purely example-based prompts while controlling for the available compute and data budget. Further, performance of GRIPS is comparable to select gradient-based tuning approaches. Qualitatively, we show our edits can simplify instructions and at times make them incoherent but nonetheless improve accuracy.

• The Effectiveness of Intermediate-Task Training for Code-Switched Natural Language Understanding
  Archiki Prasad*, Mohammad Ali Rehan*, Shreya Pathak* and Preethi Jyothi
  Workshop on Multilingual Representation Learning (MRL) 2021, EMNLP 2021
  🏆 Best Paper Honorable Mention
  [abstract] [code] [slides]

  
  

  While recent benchmarks have spurred a lot of new work on improving the generalization of pretrained multilingual language models on multilingual tasks, techniques to improve code-switched natural language understanding tasks have been far less explored. In this work, we propose the use of bilingual intermediate pretraining as a reliable technique to derive large and consistent performance gains on three different NLP tasks using code-switched text. We achieve substantial absolute improvements of 7.87%, 20.15%, and 10.99%, on the mean accuracies and F1 scores over previous state-of-the-art systems for Hindi-English Natural Language Inference (NLI), Question Answering (QA) tasks, and Spanish-English Sentiment Analysis (SA) respectively. We show consistent performance gains on four different code-switched language-pairs (Hindi-English, Spanish-English, Tamil-English and Malayalam-English) for SA. We also present a code-switched masked language modelling (MLM) pretraining technique that consistently benefits SA compared to standard MLM pretraining using real code-switched text.

• An Investigation of End-to-End Models for Robust Speech Recognition
  Archiki Prasad, Preethi Jyothi and Rajbabu Velmurugan
  IEEE-ICASSP 2021
  [abstract] [code] [poster]

  
  

  End-to-end models for robust automatic speech recognition (ASR) have not been sufficiently well-explored in prior work. With end-to-end models, one could choose to preprocess the input speech using speech enhancement techniques and train the model using enhanced speech. Another alternative is to pass the noisy speech as input and modify the model architecture to adapt to noisy speech. A systematic comparison of these two approaches for end-to-end robust ASR has not been attempted before. We address this gap and present a detailed comparison of speech enhancement-based techniques and three different model-based adaptation techniques covering data augmentation, multi-task learning, and adversarial learning for robust ASR. While adversarial learning is the best-performing technique on certain noise types, it comes at a cost of degrading clean speech WER. On other relatively stationary noise types, a new speech enhancement technique outperformed all the model-based daptation techniques. This suggests that knowledge of the underlying noise type can meaningfully inform the choice of adaptation technique.

• Decentralized Age-of-Information Bandits
  Archiki Prasad, Vishal Jain and Sharayu Moharir
  IEEE-WCNC 2021
  [abstract] [long-form with proofs]

  
  

  Age-of-Information (AoI) is a performance metric for scheduling systems that measures the freshness of the data available at the intended destination. AoI is formally defined as the time elapsed since the destination received the recent most update from the source. We consider the problem of scheduling to minimize the cumulative AoI in a multi-source multi-channel setting. Our focus is on the setting where channel statistics are unknown and we model the problem as a distributed multi-armed bandit problem. For an appropriately defined AoI regret metric, we provide analytical performance guarantees of an existing UCB-based policy for the distributed multi-armed bandit problem. In addition, we propose a novel policy based on Thomson Sampling and a hybrid policy that tries to balance the trade-off between the aforementioned policies. Further, we develop AoI-aware variants of these policies in which each source takes its current AoI into account while making decisions. We compare the performance of various policies via simulations.

• How Accents Confound: Probing for Accent Information in End-to-End Speech Recognition Systems
  Archiki Prasad and Preethi Jyothi
  ACL 2020
  [abstract] [code] [talk]

  
  

  In this work, we present a detailed analysis of how accent information is reflected in the internal representation of speech in an end-to-end automatic speech recognition (ASR) system. We use a state-of-the-art end-to-end ASR system, comprising convolutional and recurrent layers, that is trained on a large amount of US-accented English speech and evaluate the model on speech samples from seven different English accents. We examine the effects of accent on the internal representation using three main probing techniques: a) Gradient-based explanation methods, b) Information-theoretic measures, and c) Outputs of accent and phone classifiers. We find different accents exhibiting similar trends irrespective of the probing technique used. We also find that most accent information is encoded within the first recurrent layer, which is suggestive of how one could adapt such an end-to-end model to learn representations that are invariant to accents.

• Time Series Forecasting for Cold-Start Items by Learning from Related Items using Memory Networks
  Ayush Chauhan, Archiki Prasad, Parth Gupta, Amiredddy Prashanth Reddy and Shiv Kumar Saini
  The Web Conference (WWW) 2020
  [abstract]

  
  

  Time series forecasting for new items is very important in a wide variety of applications. Existing solutions for time series forecasting, however, do not address this cold start problem. The underlying machine learning models in these solutions rely heavily on the availability of the past data points of the time series. Here, we propose to use a modified Dynamic Key-Value Memory Network (DKVMN) that enables knowledge sharing across items. The network is conventionally used for binary tasks in knowledge tracing. We modify it for our regression-based forecasting use-case. Specifically, we change the output layer, include feedback for error correction, add a mechanism to handle scale across items. We test our solution on the SKU level data of a large e-commerce company and compare the results to the widely used LSTM model, outperforming it by over 25% across multiple metrics.

• Key-Value Memory Networks for Predicting Time Series Metrics of Target Entities
  Shiv Kumar Saini, Ayush Chauhan, Parth Gupta, Archiki Prasad, Amiredddy Prashanth Reddy and Ritwick Chaudhary
  Patent filed at the US Patent and Trademarks Office 2020 | Adobe Inc.
  [summary] [application no. US16/868942]

  
  

  This disclosure involves using key-value memory networks to predict time-series data. For instance, a computing system retrieves, for a target entity, static feature data and target time-series feature data. The computing system can normalize the target time-series feature data based on a normalization scale. The computing system also generates input data by, for example, concatenating the static feature data, the normalized time-series feature data, and time-specific feature data. The computing system generates predicted time-series data for the target metric of the target entity by applying a key-value memory network to the input data. The key-value memory network can include a key matrix learned from training static feature data and training time-series feature data, a value matrix representing time-series trends, and an output layer with a continuous activation function for generating predicted time-series data.

Research Projects

Please have a look at my Curriculum Vitae for a comprehensive list of my projects.