Experience

Exploring LLM-based audio synthesis using neural audio codecs and its codebook compression with knowledge distillation.

Developing CLIP-style audio-representation learning framework with Whisper encoder and synthetic dense audio captions.

Developed audio-understanding LLMs that surpass prior open-weight models by 10+ points on multi-domain benchmarks (speech, music, sound) like MMAU and AirBench, leveraging a simpler architecture, single-stage training, and under 30K hours of public data.

Developed VisCon, a novel LLM fine-tuning dataset with leaky visual conversations and contextual information for enhanced visual QA.

Integrated image understanding in text-only LLMs via distributed fine-tuning with CLIP’s vision encoder.

Enhanced distributed pretraining of Falcon3 LLM, integrating Triton and optimizing backward-pass for architectural changes.

Accelerated LLM Inference through self-distillation, intermediate layers prediction, and recurrent multi-token prediction.

Collaborated on the development Jais, an English-Arabic LLM.

Managed the LLM Arena framework, benchmarking models (in-house, open-source, and GPT-4) with human annotators using Elo ratings.

Leveraged LLM Arena data to align LLMs with harmlessness and usefulness through RLHF and DPO techniques using the TRL package.

Extensively evaluated the design choices for Text-To-Speech systems and open-sourced state-of-the-art models for 13 Indian languages. [ICASSP Paper]

Demonstrated the target classifier and attack-agnostic nature of the Double Variational Autoencoder Network (DoENet), leading to improved unsupervised adversarial image classification.

Enhanced retail sales forecasting with deep neural networks (RNNs, LSTNet) and a novel N-gram method for dynamic pricing. [US Patent]

Implemented RNN-based spatio-temporal travel time prediction, benchmarking against temporal difference-based methods. [IJCNN Paper]

Built an employee profile retrieval system using information parsing and natural language query processing. [Special Initiative Award]

Developed a winning yield prediction model for hybrid corn crops, recommending effective species crossing (internal hackathon). [Innovation Pride Award]

Designed winning computer vision pipelines for curved text extraction and perspective correction (internal hackathon with 100+ folks). [Innovation Pride Award]

Developed a scene text translation system for Indian languages using synthetic datasets, incorporating an Efficient and Accurate Scene Text Detector (EAST) for detection, and Convolutional Recurrent Neural Networks (CRNN) for Classification and Recognition. [Dataset] [Detection] [Classification] [Recognition]

Set up programming competitions for the One Fourth Labs Deep Learning course. [Course]

Investigated attention models in Deep Learning, analyzing foreground region detection through proxy data with distinct statistical properties. [Report]