Fixing Weight Decay Regularization in Adam

Here is a read on the paper “Fixing Weight Decay Regularization in Adam”, a major correction of how weight decay with ADAM should be used together.

• The key to understand this paper is that weight decay is often implemented as L2 regularization in the optimization function. And we often think that the two concepts, weight decay and L2 regularization, are the same.
• But then the authors, Loshchilov and Hutter observed a very simple fact: which is if you implement weight decay through L2 regularization often reduce the effect of weight decay. So that explain why ADAM has poorer generalization power.
• We will just refer you guys to Algorithm 1 and 2 on p.3. And if you follow the text you will quickly realize the past implementation was just wrong. The authors also proposed how you can fix the update to get weight decay correctly. (The green highlights.)
• Reading the paper also required you to understand the idea of warm restarts. You can read it from the paper SGDR: Stochastic Gradient Descent with Warm Restarts.
• The authors went ahead to show that the idea works well on Top-1 Error of CIFAR-10 and Top-5 Error of Imagenet32x32, a downsampled Imagenet. Looks good.
• This sounds like a great idea. In fact, the author of ADAM, DP Kingma, thought so too. So is Jeremy Howard, the technique is already implemented in fast.ai’s code.

The original Link can be found at here.

arxiv.org

Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm

• As you know the story, AlphaZero is not only just playing Go, and is now playing Chess and Shogi. By itself this is a significant event, because most stoa board game engine are specific to games. General game playing engines are seen as novelties but not a norm.
• Another note, most Chess and Shogi engines are based on alpha-beta search. But then AlphaZero is now using Monte-Carlo Tree Search which simulate board positions. Positions are order by scores from a board NN. State is entered in the order of visit counts and value of the board according to NN. So you can see this is not just AlphaZero is beating up more games, it will be more a paradigm shift of both computer Chess and Shogi community.
• As you know, AlphaZero beats the strongest program in 2016, Stockfish. But one analysis which caught my eyes: In chess, DeepMind researchers also fix the first few moves of AlphaZero so that it follows the top 12 most-play openings for black and white. If you are into chess, Queen’s Gambit, several Sicilian Defences, The French, KID. They show that AlphaZero can beat Stockfish in multiple type of situations, and openings doesn’t matter too much.
• But then, would AlphaZero beat all computer players such as Shredder or Komodo? No one knows the answers yet.
• One more thing: AlphaZero doesn’t assume zero knowledge neither. As Denny Britz points out in his tweet, AlphaZero was provided with perfect knowledge in terms of rules. So intriguing rules such as castling, threefold repetition or 50-move drawing rules are all provided to the machine. Perhaps Britz points out, may be we want to focus on how to let the machine to figure out the rules themselves in the future.

arxiv.org

Fixing Weight Decay Regularization in Adam

Here is a read on the paper “Fixing Weight Decay Regularization in Adam”, a major correction of how weight decay with ADAM should be used together.

• The key to understand this paper is that weight decay is often implemented as L2 regularization in the optimization function. And we often think that the two concepts, weight decay and L2 regularization, are the same.
• But then the authors, Loshchilov and Hutter observed a very simple fact: which is if you implement weight decay through L2 regularization often reduce the effect of weight decay. So that explain why ADAM has poorer generalization power.
• We will just refer you guys to Algorithm 1 and 2 on p.3. And if you follow the text you will quickly realize the past implementation was just wrong. The authors also proposed how you can fix the update to get weight decay correctly. (The green highlights.)
• Reading the paper also required you to understand the idea of warm restarts. You can read it from the paper SGDR: Stochastic Gradient Descent with Warm Restarts.
• The authors went ahead to show that the idea works well on Top-1 Error of CIFAR-10 and Top-5 Error of Imagenet32x32, a downsampled Imagenet. Looks good.
• This sounds like a great idea. In fact, the author of ADAM, DP Kingma, thought so too. So is Jeremy Howard, the technique is already implemented in fast.ai’s code.

The original Link can be found at here.

arxiv.org

Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm

• As you know the story, AlphaZero is not only just playing Go, and is now playing Chess and Shogi. By itself this is a significant event, because most stoa board game engine are specific to games. General game playing engines are seen as novelties but not a norm.
• Another note, most Chess and Shogi engines are based on alpha-beta search. But then AlphaZero is now using Monte-Carlo Tree Search which simulate board positions. Positions are order by scores from a board NN. State is entered in the order of visit counts and value of the board according to NN. So you can see this is not just AlphaZero is beating up more games, it will be more a paradigm shift of both computer Chess and Shogi community.
• As you know, AlphaZero beats the strongest program in 2016, Stockfish. But one analysis which caught my eyes: In chess, DeepMind researchers also fix the first few moves of AlphaZero so that it follows the top 12 most-play openings for black and white. If you are into chess, Queen’s Gambit, several Sicilian Defences, The French, KID. They show that AlphaZero can beat Stockfish in multiple type of situations, and openings doesn’t matter too much.
• But then, would AlphaZero beat all computer players such as Shredder or Komodo? No one knows the answers yet.
• One more thing: AlphaZero doesn’t assume zero knowledge neither. As Denny Britz points out in his tweet, AlphaZero was provided with perfect knowledge in terms of rules. So intriguing rules such as castling, threefold repetition or 50-move drawing rules are all provided to the machine. Perhaps Britz points out, may be we want to focus on how to let the machine to figure out the rules themselves in the future.

arxiv.org

Nature ML Journal Launching in 2017

It’s online, but will it be free? We will see. Regardless, it says a lot of the importance of ML in scientific research. Though some in the community does raise concern if Nature is the best host of the new journal. Some use JMLR as an example that an academic journal could just be organized and edited by academicians themselves.

nature.com

CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning

This is a note on CheXNet, the paper. As you know it is the widely circulated paper from Stanford, purportedly outperform human’s performance on Chest X-ray diagnostic.

• BUT, after we read it in detail, my impression is slightly different from just reading the popular news including the description on github.
• Since the ML part is not very interesting. We will just briefly go through it – it’s a 121-layer Densenet, basically it means there are feed-forward connection from every previous layers. Given the data size, it’s likely a full training.
• There was not much justification on the why of the architecture. Our guess: the team first transfer learning, but decide to move on to full-training to get better performance. A manageable setup would be Densenet.
• Then there was fairly standard experimental comparison using AUC. In a nut shell, CheXNet did perform better than humans in every one of the 14 classes of ChestX-ray-14, which is known to be the largest of the similar databases.
• Now here is the caveat popular news hadn’t mentioned:
  1, First of all, humans weren’t allow to access previous medical records of a patient.
  2, Only frontal images were shown to human doctors. But prior work did show when the lateral view was also shown.
• That’s why on p.3 of the article, the authors note:
  “We thus expect that this setup provides a conservative estimate of human radiologist performance.”

Reading so far should make you realize that may be it will still take a bit for deep learning to “replace radiologists”.

See the original discussion at AIDL-LD.

arxiv.org

Weighted Transformer

This is one of the two papers SalesForce Einstein lab published last week. Both of them requires understanding of MT, NNMT and purely attention-based NNMT. Since this first one is not too difficult to understand, I would just give you some background on NNMT first.

When NNMT was first perceived. The original form starts with an Encoder of the text, convert it to what usually known as a “thought vector”. The thought vector will then decode by the Decoder. In the original setting, both Encoder and Decoder are usually LSTMs

Then there is the idea of attention. Well, you can think of it as more like an extra layer just on the thought vector on the decoder side. The goal is decide how much attention you want to pay on the thought vector.

Now of course, people have then played with various architecture for these Enc-Dec structure. The first to notice is that such structure usually has a giant LSTM or CNN. But notice that no one really like them! LSTM is hard to parallelize and CNN can consume a lot of memories.

That makes Google work mid of this year, “Attention is all you need” a stunning and useful result. What the authors were saying is proposing is to just use the idea of attention to create a system, they call it transformer. There are multiple tricks to get it work but perhaps the most important one is “multi-head attentions”, in a way this is like the concepts of channels in Convnet, but now instead of doing one single attention, we are now attend in multiple places. Each head will learn to attend differently.

Naturally the method is fast because you can also parallelize it, but then Google’s researchers also find it to be better in the BLEU score. That’s why top house are switching to purely attention-based method these days.

Now finally I can talk about what the Salesforce paper is about. In the original Google’s paper, representation learned by multi-attention heads are only concatenate with each other to form one “supervector” But then the authors of the paper decide to use another set of weighting. This again, further improve the performance on WMT14 by 0.4 BLEU score, which is quite significant.

einstein.ai

Non-Autoregressive Neural Machine Translation

This is the second of the two papers from Salesforce, “Non-Autoregressive Neural Machine Translation” . Unlike the “Weighted Transformer, I don’t believe it improves SOTA results. But then it introduces a cute idea into a purely attention-based NNMT, I would suggest you my previous post before you read on.

Okay. The key idea introduced in the paper is fertility. So this is to address one of the issues introduced by a purely attention-based model introduced from “Attention is all you need”. If you are doing translation, the translated word can 1) be expanded to multiple words, 2) transform to a totally different word location.

In the older world of statistical machine translation, or what we called IBM models. The latter model is called “Model 2” which decide the “absolute alignment” of source/target language pair. The former is called fertility model or “Model 3”. Of course, in the world of NNMT, these two models were thought to be obsolete. Why not just use RNN in the Encoder/Decoder structure to solve the problem?
(Btw, there are totally 5 types IBM Models. If you are into SMT, you should probably learn it up.)

But then in the world of purely attention-based NNMT, idea such as absolute alignment and fertility become important again. Because you don’t have memory within your model. So in the original “Attention is all you need” paper, there is already the thought of “positional encoding” which is to model absolute alignment.

So the new Salesforce paper actually introduces another layer which brought back fertility. Instead of just feeding the output of encoder directly into the decoder. It will feed to a fertility layer to decide if a certain word should have higher fertility first. e.g. a fertility of 2 means that it should be copied twice. 0 means the word shouldn’t be copy.

I think the cute thing about the paper is two-fold. One is that it is an obvious expansion of the whole idea of attention-based NNMT . Then there is the Socher’s group is reintroducing classical SMT idea back to NNMT.

The result though is not working as well as the standard NNMT. As you can see in Table 1. There is still some degradation using the attention-based approach. That’s perhaps why when the Google Research Blog mention the Salesforce results : it said “towards non-autoregressive translation”. It implies that the results is not yet satisfying.

einstein.ai

Medical Image Segmentation Based on Multi-Modal Convolutional Neural Network: Study on Image Fusion Schemes

This is a well-written summary by one of our members. We decide to feature it here without further comments. So check it out! Only in our Literature Discussion group.

facebook.com

What We Know About Capsules So Far

We wrote the previous piece on Monday on our Literature Discussion group. But then it triggered very interesting discussion which we learnt couple of things:

1. Hinton and students might have published another paper on ICLR 2018. It’s very likely to involve EM as the routing mechanism.
2. There are already two available implementations of CapsNet. (See the Implementation Section.)

facebook.com

Capsules

This is the Hinton’s new invention of capsules algorithm. Here is a write up: It’s TL;DR but we doubt we completely grok the idea anyway.

• The first mention of “capsule” is perhaps in the paper “Transforming Auto-encoders” which Hinton and students coauthored.
• It’s important to understand what capsules try to solve before you delve into the details. If you look at Hinton’s papers and talks, capsule is really an idea which improve upon Convnet. Hinton has two major complaints.
• First, the general settings of Convnet assumes that one filter is being used across different locations. This is also known as “location invariance”. In this setting, the exact location of a feature doesn’t matter. That has a lot to do with robust feature parameter estimation. It also drastically simplify backprop with weight sharing.
• But then location invariance also removes one important information of an image: the apparent location.
• Second assumption is max pooling. As you know, pooling usually removes a high percentage of information from the previous layer. In early architectures, usually pooling is the key to shrink the size of a representation down. Of course, later architectures had changed. But pooling is still an important component.
• So the design of capsule has a lot of do to tackle problems of max pooling: Instead of losing information, can we “route” pixel values from previous layer correctly so that they are in optimal use?
• Generally “capsule” represents a certain entity of an image, “such as pose (position, size, orientation), deformation, velocity, albedo, hue, texture etc”. Notice that they are not hard-wired and automatically discovered.
• Then there is how the low level information can “route” to higher level. The mechanism is intriguing in this current implementation:
• First, your goal is to calculate a softmax in the form of
  exp(b{ij} / Sum_k exp(b{ik} where b_{ij} is the output of lower level capsule i to a higher level capsule j. This is something you can train.
• Then what you do is iteratively estimate b_{ij}. This appears in Procedure 1. The 4 steps are:
  a, calculate the softmax weight b.
  b, compute the prediction vector from a capsule i, then form a weighted sum,
  c, squash the weighted sum
  d, update softmax weight b based on the squash value and weighted sum.
• So why the squash function, our guess is it is to normalize the value computed in b. According to Hinton, a good function is
  v_j = |s_j|^2 / (1 + |s_j|^2) * s_j / |s_j|
• The rest of the architecture actually looks very much like a Convnet. The first layer was a Convnet with ReLU activation.
• Would this work? The authors say yes. Not only it reaches the state of art benchmark of MNIST. It can also tackle more difficult tasks such as CIFAR-10, SVNH. In fact, the authors found that in both task they already achieve better results when first Convnet was first used to tackle these tasks.
• It also works well for two tasks called affMNISt and multiMNIST. First is MNIST go through affine transform, second is MNIST regenerated with many overlappings. This is quite impressive, because you will need to use much data augmentation and effort of object detection to get these cases working.
• The part, we have some doubts – is this model more complex than convnet? It’s possible that we are just fitting a more complex model to get better results.
• Nice thing about the implementation: it’s in Tensorflow, so we can play with it in the near future.
  Have fun!

arxiv.org

Unsupervised Machine Translation Using Monolingual Corpora

This is an impressive paper by FAIR authors which claims that one only need to use monolingual corpora to train a usable translation model. So how does it work? Here are some notes.

• For starter, indeed you don’t need to use a parallel corpora, but you still need a bidirectional dictionary to generate translation. You also need to have monolingual corpora in both languages. That’s why the title is about monolingual corpora (plural) but not monolingual corpus (singular).
• Then, there is the issue of how you actually create translation. It’s actually much simpler than you thought, first imagine there is a latent language which both your source and target languages mapped to.
• How do you train? So let’s just use the source language as an example first. What you can do is create an encoder-decoder architecture which translate your source to the latent space, then translate it back. Using BLEU score, you can then setup an optimization criteria.
• Now this doesn’t quite do the translation. Now you apply the same procedure on both source and target language. Don’t you now have a common latent space? One you train up such common latent subspace, in actual translation, what you need to do is to first map the target language in the common latent space, then map it back to the source language.
• Many of you might recognize that such encoder-decoder scheme which map the language to itself as very similar to autoencoder. Indeed, the authors in the paper actually use a version of autoencoder: denoising autoencoder(dA) to train the model.
• The final interesting idea I spot is to idea of iterative training. In this case, you can imagine that you can first train an initial translator, but then use its output as the truth and retrain another one. The authors found tremendous gain in BLEU score in the process.
• The results are stunning if you consider no parallel corpus is involved. BLEU score is around 10 points lower, but do remember: deep learning has pretty much improved BLEU scores by absolute 7-8 points anyway from the classical phrased based translation models.

Member Ben Davis also wrote a fairly good summary for the paper. Check it in our thread?

arxiv.org

Swish: a Self-Gated Activation Function

Perhaps the most interesting paper last week is the Swish function. Here are some notes:

• Swish is extraordinarily simple. It’s just
  swish(x) = x * sigmoid(x).
• Derivative? swish'(x) = swish(x) + sigmoid(x) (1 – swish (x)) Simple calculus.
• Can you tune it? Yes, there is a tunable version which the parameter is trainable. It’s call Swish-Beta which is x * sigmoid( Beta * x)
• So here’s an interesting part of why it is a “self-gating function”. So…. if you understand LSTM, essentially it introduced a multiplication sign. e.g. input gate and forget gate, give you are weight of “how much you want to consider the input” and “how much much you want to forget”. (http://colah.github.io/posts/2015-08-Understanding-LSTMs/)
• So swish is not too different – there is the activation function but it is weighted by the input itself. Thus the term self-gating. In a nutshell, in plain English, “because we multiply”.
• It’s all good, but does it work? The experimental results look promising. It works on Cifar-10, Cifar-100. On Imagenet, it beats Inception-v2 and v3 when swish replace ReLU.
• It’s worthwhile to point out the latest Inception is in v4. So the imagenet number is not beating stoa even within Google, not to say the best number in Imagenet 2016. But that shouldn’t matter, if something consistently improve on some models of Imagenet, it is a very good sign it is working.
• Of course, looking at the activation function. It introduces a multiplication. So it does increase computation when compare with a simple ReLU. And that seems to be the complaint I heard.
  That’s what I have. Enjoy!

arxiv.org

Tutorial on Variational Autoencoders

We were trying to read DP Kingma’s thesis on VAE. As you might know, he wrote the paper on reparametization trick on VAE. But this is not my topic this time. We just want to talk about a simple tutorial paper by Carl Doersch, which we found it to be better for beginners. Here are some notes:

• VAE is really quite different sparse AE or denoising AE except you can think of both of them like having an encoder structure.
• The Tutorial would guide you through the setup of VAE. e.g. You probably know that VAE is based on latent variable. But then the setup is special which the latent variable is randomly sampled from a Gaussian.
• Then there is a detail readable section on how the common optimization evidence lower bound (ELBO) is formulated. What it bugs me a bit is that it doesn’t quite use the term ELBO.
• Lastly it’s the reparametization trick. we don’t fully grok it but that’s why we still need to read Kingma

So far, this seems to be a great tutorial on VAE and it’s a great first read on the topic.

arxiv.org

Tutorial on Variational Autoencoders

We were trying to read DP Kingma’s thesis on VAE. As you might know, he wrote the paper on reparametization trick on VAE. But this is not my topic this time. We just want to talk about a simple tutorial paper by Carl Doersch, which we found it to be better for beginners. Here are some notes:

• VAE is really quite different sparse AE or denoising AE except you can think of both of them like having an encoder structure.
• The Tutorial would guide you through the setup of VAE. e.g. You probably know that VAE is based on latent variable. But then the setup is special which the latent variable is randomly sampled from a Gaussian.
• Then there is a detail readable section on how the common optimization evidence lower bound (ELBO) is formulated. What it bugs me a bit is that it doesn’t quite use the term ELBO.
• Lastly it’s the reparametization trick. we don’t fully grok it but that’s why we still need to read Kingma

So far, this seems to be a great tutorial on VAE and it’s a great first read on the topic.

arxiv.org

Unsupervised Hypernym Detection by Distributional Inclusion Vector Embedding

Modified from my note at here

Some notes and thoughts:

• This work is to model relationship such as “poodle is-a dog” or generally the word entailment problem in computational linguistic.
• But what it follows is the more modern word-embedding-based approach. The hypernym problems are more often solved by Bag of Words (BOW) model. So this paper is perhaps one of the more earlier papers which used word embedding type of method on hypernym detection.
• It’s good to think of why this method works. So in the case of synonyms, we all know about the distributional hypothesis – or from Firth: “a word is characterized by the company it keeps”.
• What about hypernyms? the corresponding hypotheses are perhaps two “Distribution Informativeness Hypothesis” and “Distributional Inclusion Hypothesis”. First stated that semantically general word occur more often. Second states that context sets of the words tends to be subset of the hypernym. So combining the two, “chihuahua” would occur less often than “dog”, and adjective which can be applied to “chihuahua” can all be applied to “dog”, but not necessarily vice versa.
• Then there is DIVE. So in essence, it is a modification of Mikolov’s skipgram but with two major modificiations: Non-negativity of skip-gram as well as weighting the negative sample with the inverse of how often a word appears. That’s pretty much is what DIVE is.
• Then there is another part about PMI (point-wise mutual information). In a nutshell, Levy and Goldberg proved that the skip-gram formulation is equivalent to matrix factorization with elements equals to PMI or (p(w,c)/p(w)p(c). The authors also give a similar formulation to DIVE. The nifty part is they also apply a filter to the PMI matrix if the occurring words are too small.
• Then there is a fairly extensive evaluation on the technique which include 11 datasets. And the authors have tested their technique with breakdown such as with/without DIVE, with/without PMI filtering. There is a very good improvement from SBOW approach.

Interesting enough. This doesn’t seem to be picked by arxiv-sanity. Perhaps hypernym is more a side topic in linguistic. But it’s certainly a good read. Let’s see how other members think about this?

arxiv.org