Advanced NLP
강의 주제: Natural Language Processing Instructor : Graham Neubig(Associate Professor, Carnegie Mellon University),Robert Frederking(Associate Dean for Ph.D. Programs, CMU Language Technologies Institute) [[schedule, codes](https://phontron.com/class/anlp2022/schedule.html)] | [[youtube](https://youtube.com/playlist?list=PL8PYTP1V4I8D0UkqW2fEhgLrnlDW9QK7z&si=3vCrtwi-s7LRntEl)]
:bulb: 목표
- 자연어 처리의 개념을 이해하고, 최신 기법을 파악한다.
syntactic, semantic, discourse analysis 등 자연어 처리의 기초적인 개념을 이해하고, 관련된 최신 기법을 파악한다.
🚩 정리한 문서 목록
📖 Basics of Natural Language Processing
Generative Text Classification: Count-based Unigram Models, Bag-of-Words Generative Classifier(BoW) / Discriminative Text Classification: BOW Discriminative Classifier
Evaluation: accuracy, precision, recall, F1 score, statistical testing
📐 Modeling
Ancestral Sampling, Greedy Search, Beam Search
Evaluation(Human Evaluation, BLEU Score, Embedding-based Metrics, Perplexity), Meta-Evaluation, Difficulties(bad model + big beam), Alternative Methods(worse search for better outputs, minimize Bayes risk, Train Better Models)
Attention, Attention Score Functions(MLP, Bilinear, Dot Product, Scaled Dot Product), Self Attention, Multi-Head Attention
Transformer: Transformer Architecture, Attention Tricks, Training Tricks, Masked Multi-Head Attention
Extensions to Attention: Incorporating Markov Properties, Hard Attention, Monotonic Attention, Coverage, Bidirectional Training, Alignment Attention
📔 Representation
Multi-task Learning: Standard, Pre-train and Fine-tune, Prompting
Pre-trained LMs: BERT, RoBERTa, ELECTRA, XLNet, DeBERTa
Auto-regressive LMs for Generation/Prompting: GPT-2, GPT-3, PaLM, OPT, BLOOM
Pre-training Pros and Cons, Design Choices(Data, Transform, Representation, Output), Scaling Law
domain, domain shift(covariate shift, concept shift), domain adaptation
parameter sharing(domain tag, adapter, regularization-based), task weighting(uniform, proportional, temperature-based, uncertainty-based)
Types of Prompts(filled, answered, prefix, cloze), Prompt Workflows, Pre-trained LM(MASS, BART, mBART, UNiLM, T5)
Prompt Engineering: Cloze, Prefix, Hand-crafted, Automated(Prompt Mining, Prompt Parapharasing, Gradient-based Search, Prefix/Prompt Tuning)