Some good resources for NNMT
Tutorial:
- NMT tutorial written by Thang Luong - my impression is that it is a shorter tutorial with step-by-step procedure. The part which is slightly disappointing is that it doesn't quite record exactly how the benchmarking experiments were run and evaluated. Of course, it's kind of trivial to fix it, but it did take me a bit of time.
- The original Tensorflow seq2seq tutorial - more a big gun of SMT, the first experiment I played with. Now we are talking about the WMT15 set.
- tf-seq2seq (blog post: here)
- Graham Neubig's tutorial
- Nematus
- OpenNMT
- NeuralMonkey (Tensorflow-based)
- Prof. Philip Koehn's new chapter on NMT
a bit special: Tensor2Tensor uses a novel architecture instead of pure RNN/CNN decoder/encoder. It gives a surprisingly large amount of gain. So it's likely that it will become a trend in NNMT in the future.
Important papers:
- Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation by Cho Et al. (link) - Very innovative and smart paper by Kyunghyun Cho. It also introduces GRU.
- Sequence to Sequence Learning with Neural Networks by Ilya Sutskever (link) - By Google's researchers, and perhaps it shows for the first time an NMT system is comparable to the traditional pipeline.
- Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (link)
- Neural Machine Translation by Joint Learning to Align and Translate by Dzmitry Bahdanau (link) - The paper which introduce attention
- Neural Machine Translation by Min-Thuong Luong (link)
- Effective Approaches to Attention-based Neural Machine Translation by Min-Thuong Luong (link) - On how to improve attention approach based on local attention.
- Massive Exploration of Neural Machine Translation Architectures by Britz et al (link)
- Recurrent Convolutional Neural Networks for Discourse Compositionality by Kalchbrenner and Blunsom (link)
Important Blog Posts/Web page:
- Attention and Augmented Recurrent Neural Networks: Only partially relevant to attention-based RNN, but Olah's writing is always worthwhile to read.
- Stanford NMT research page: Related to Luong, See and Manning's work on NMT. Very entertaining to look at recent techniques. Tutorial/code/models are available.
Others: (Unsorted, and seems less important)
- JayPark's Github https://github.com/JayParks/tf-seq2seq
- https://github.com/ematvey/tensorflow-seq2seq-tutorials
- https://indico.io/blog/the-good-bad-ugly-of-tensorflow/
- https://www.reddit.com/r/MachineLearning/comments/43fw8s/simple_seq2seq_example_in_tensorflow/
- https://r2rt.com/recurrent-neural-networks-in-tensorflow-iii-variable-length-sequences.html
- http://forums.fast.ai/t/the-wonders-of-the-new-version-of-tensorflow-1-2rc0/3491
- http://monik.in/a-noobs-guide-to-implementing-rnn-lstm-using-tensorflow/
- http://suriyadeepan.github.io/2016-12-31-practical-seq2seq/
Usage in Chatbot and Summarization (again unsorted, and again perhaps less important.....)