CS 7052: Advanced Topics in Natural Language Processing
University of Cincinnati
Spring 2025

Instructor: Tianyu Jiang
Time: TuTh 11 am - 12:20 pm
Location: Baldwin 757
Office Hour:
     Tianyu Jiang, Mon 10-11 am, Rhodes Hall 889

Course Description
The course explores current developments in natural language processing (NLP), emphasizing emerging research topics and state-of-the-art techniques. The course is aimed to prepare you to conduct cutting-edge research in NLP. The course focuses on conceptual understanding and research rather than engineering or software usage. You will engage in substantial reading, ask critical questions and brainstorm new ideas. Discussions and active participation is required for the class.

Grading
  • Attendance: 20%
  • Presentation: 40%
  • Review: 20%
  • Scribe's Notes: 20%

    • Late Policy: 24 hour grace period with 10% penalty. No points after 24 hours.
    Electronic Submission: We use a shared Google Drive folder for all materials.

    Prerequisites
    This course assumes you have taken CS5134/6034 Natural Language Processing or have a equivalent background (good knowledge of NLP basics and deep learning). The class is mainly for graduate students in computer science. Senior undergraduates or students from other departments will need Prof Jiang's permission.

    Schedule (Tentative)
    Week Date Topic Reading Optional Reading
    101/14Introduction-
    01/16Attention & TransformersAttention Is All You Need
    201/21PretrainingLanguage Models are Few-Shot LearnersLanguage Models are Unsupervised Multitask Learners, Generating Long Sequences with Sparse Transformers, An Empirical Model of Large-Batch Training
    01/23Scaling LawsTraining Compute-Optimal Large Language ModelsScaling laws for neural language models, Quasi-Newton Matrices with Limited Storage
    301/28Instruction TuningScaling Instruction-Finetuned Language ModelsChain-of-Thought Prompting Elicits Reasoning in Large Language Models
    01/30The Flan Collection: Designing Data and Methods for Effective Instruction TuningScaling Instruction-Finetuned Language Models, Finetuned Language Models Are Zero-Shot Learners
    402/04PromptingChain-of-Thought Prompting Elicits Reasoning in Large Language ModelsTree of Thoughts: Deliberate Problem Solving with Large Language Models
    02/06Self-Consistency Improves Chain-of-Thought Reasoning in Language ModelsUniversal Self-Consistency for Large Language Model Generation, Early-Stopping Self-Consistency for Multi-step Reasoning, Ask One More Time: Self-Agreement Improves Reasoning of Language Models
    502/11ART: Automatic Multi-Step Reasoning and Tool-Use for Large Language ModelsTALM: Tool Augmented Language Models, LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models
    02/13Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations, Active Learning Principles for In-Context Learning with Large Language Models, Larger language models do in-context learning differently, Lost in the Middle: How Language Models Use Long Contexts
    602/18LLM AbilitiesEmergent Abilities of Large Language ModelsA Latent Space Theory for Emergent Abilities in Large Language Models, Are Emergent Abilities in Large Language Models just In-Context Learning?
    02/20Are Emergent Abilities of Large Language Models a Mirage?Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models, Predicting Emergent Capabilities by Finetuning, Training on the Test Task Confounds Evaluation and Emergence, State of What Art? A Call for Multi-Prompt LLM Evaluation
    702/25Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP ResearchersResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models
    02/27Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis RefinementLarge Language Models can Learn Rules, Context-Aware Testing: A New Paradigm for Model Testing with Large Language Models
    803/04Alignment and AgentTraining Language Models to Follow Instructions with Human FeedbackAligning Language Models with Self-Generated Instruction, Direct Preference Optimization: Your Language Model is Secretly a Reward Model, LIMA: Less Is More for Alignment
    03/06Toolformer: Language Models Can Teach Themselves to Use Tools
    903/11MoESwitch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
    03/13GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
    1003/18Spring Break-
    03/20-
    1103/25RAGImproving Language Models by Retrieving from Trillions of Tokens
    03/27RLDeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
    1204/01MultimodalLearning Transferable Visual Models From Natural Language Supervision (CLIP)
    04/03Improved Baselines with Visual Instruction Tuning (LLaVA)
    1304/08Distillation and QuantizationTinyBERT: Distilling BERT for Natural Language Understanding
    04/10LoRA: Low-Rank Adaptation of Large Language Models
    1404/15No Class-
    04/17Long ContextA Controlled Study on Long Context Extension and Generalization in LLMs
    1504/22Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention
    04/24Fact CheckingSelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models