Publications & Preprints
SmolTulu: Higher Learning Rate to Batch Size Ratios can lead to Better Reasoning in SLMs
Demonstrated that task-dependent learning rate to batch size ratios significantly impact small language model performance during SFT, achieving state-of-the-art results in sub-2B parameter SLMs.
Read Paper →Fineweb-Edu-Ar: Machine Translated Educational Corpus for Small Arabic Language Models
Created largest open-source machine translated Arabic educational dataset to support development of Arabic small language models.
Read Paper →Tutoring tutors: Online Generation of Educational Preference Data
Novel pipeline for generating educational preference alignment data and improving model tutoring performance through online feedback. (Under Review at ARR)
ALLaM: Large Language Models for Arabic and English
Paper describing ALLaM's training process and benchmarks leading to state-of-the-art Arabic-English performance.
Read Paper →When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards
Analysis of LLM evaluation sensitivity to structural perturbations in benchmarks.
Read Paper →Experience
Research Specialist
First research specialist hired by Prof. Francesco Orabona in the OPTIMAL Lab, conducting multilingual natural language processing research alongside PhD students and postdocs.
Research Engineer - Joint SDAIA & KAUST Fellowship Program
- Proposed and implemented parallelized translation pipeline using NLLB, creating largest open-source machine translated Arabic dataset
- Part of team to release the first version of Taleem presented at 3rd GAIN summit
- Engineered AI components of Kaleem, a real-time AI tutoring system
- Architected multilingual vector database containing Saudi Arabian curriculum
Artificial Intelligence Engineer
- Founding member of the ALLaM team, developing state-of-the-art multilingual Arabic-English language model
- Lead engineer in multimodal alignment team
- Extended Megatron-LM to support diverse dataset structures
- Developed scalable reinforcement learning with human feedback framework
Peer-Assisted Study Session (PASS) Leader
Led initiative to mentor students in computer science algorithms and mathematical foundations of machine learning. Organized technical workshops on programming concepts.
Projects
HuggingFace Text Data Analyzer
A Python pip package for analyzing text datasets from HuggingFace. Combines basic text statistics with NLP capabilities like language detection and sentiment analysis. Optimized for large-scale datasets with batch processing and two-level caching.
View Project →Sage LLMs
A collection of finetuned Gemma-2-it models with LoRAs using educational content filtered from Fineweb and writing advice generated by Claude-3.5.
View Project →Environment Encoder
An approach to making reinforcement learning agents more environment agnostic by training them on vision-language model embeddings instead of raw environment states.
View Project →EasyRogue
A roguelike game written in Python that runs in the command line and supports RGB GUI. Built as a testbed for comparing perfect vs imperfect information in reinforcement learning agents.
View Project →Next-Token Agent
An experiment in training tiny language models from scratch to play ASCII games by predicting successive frames.
View Project →minLLMTrain
A barebones LLM pretraining codebase implementing DeepSpeed and dataset packing while maintaining readable code and minimal abstraction.
View Project →Cheatsheet
A computer vision project exploring novel image augmentation strategy and training objective where the model learns ordering but downstreams to classification.
View Project →