NLP-Project: Building a Word Predictor
01 Jan 2015The intended audience.
This page will serve as a sort of navigation index to all the various parts involved in getting a Word Prediction app up and running. The intended audience is someone looking to get into NLP (natural language processing), has an understanding of R or at least enough programming experience to read through snippets or R code, look around for online resources, and get a general understanding of what's going on.
That being said I'll keep any technical jargon limited and ideally anyone with an interest in word prediction will be able to take something away from the pages ahead.
The Aim
The aim is to start from raw, English: News, Blog and Twitter data and produce an NLP data product (interactive online word prediction app) hosted at ShinyApps .
Throughout the project I found that R on its own isn't a very popular language to produce an word predictor with. You'll find quite a few Python packages and C++ based solutions. Although I'm familiar with Python and have dabbled in C/C++, neither of these languages would interface in an obvious way with my aim to build a Shiny App. That being said, when it comes to making a word predictor the steps will be the same regardless of what language and packages you decide to use.
Index of Topics
Below are the individual pages that I plan on making posts about. When they are written and posted I'll make the following list into links for easy navigation.
- How I got involved with NLP and my Coursera Capstone Experience.
- The Data: Getting, cleaning, and dividing.
- The Data: Analysing the data.
- Tokenizing.
- Building N-grams.
- Interpolation or Backoff?
- Stupid Backoff and Knesser-Ney.
- Testing Accuracy.
- The ShinyApp and a five-slide pitch.
Just note, these pages and topic titles might change if I find a better way of presenting the info.
