Thoughts From Your Humble Curators
A number of very interesting developments this past week:
- Google’s TPU,
- The Vector Institute,
- Newly released cs224n 2017 videos,
- CNTK 2.0,
- BEGAN
Last Saturday was April Fools Day, so we round up the best jokes and pranks about AI. Did you fall for any of them? Some of them, like OpenAI’s spam detection are fairly sophisticated.
As always if you like our newsletter, share it with your friends and colleagues. If you haven’t done it yet, don’t forget to subscribe!
Corrections on Issue #7
In the email edition of Issue 7, we erroneously reported that an autonomous vehicle was involved in a fatal accident. It turns out that there were no serious injuries resulted. We promptly corrected the web version and post correction notice at AIDL. We apologize for causing any misunderstanding.
April Fools’ Jokes on AI and Deep Learning – 2017
It’s almost a week after April Fools, have you fallen for any of the following pranks? Here are some of the best April Fools’ jokes we gathered this year:
- Google Cheese Master – not that subtle……
- Google Wind – despite urban legends, weather control is still an unfeasible task in large scale…..
- Google Translate doesn’t quite add an alien language In case you don’t know, we haven’t been contacted by alien yet; also zero-shot learning from Translate is really not that zero-shot…..
- Amazon Petlexa Amazon’s Alexa haven’t quite solved how to connect with canines and felines. So hasn’t “rival” Google.
- Nvidia G-Assist Automatic gaming bot is feasible but probably not on a thumb drive at least. May be a K40 board.
- Google doesn’t have a data center at Mars. To see through this one, you just need to know how much payload today’s spacecraft can transport.
- And of course, DeepMind hadn’t solved AGI yet. We are faaaaaaaaaaaaaaaaar far away from having an AGI.
The one which we felt confused about: OpenAI’s result on spam detection, because the claim on using simulation to improve real-life training is possible. But the “future plan” on “phishing” and “adversarial spam” give it out. 🙂
(Photo Credit: Open AI)
Sponsor
Cyclops.io turns your plain old whiteboard into a remote collaboration tool
No Download. No login. Get up and running in seconds. Ever wanted to do a quick whiteboarding session by pointing the camera at an actual whiteboard but remote guys can’t read the content or collaborate together? We got you covered. We use computer vision to enhance the writing/drawing and allow anyone to annotate on it, enabling your team to work together as if you’re are all in the same room. You can email or post a snapshot of the whiteboard with your annotations into any Slack channels at any time. #GetStuffDone
News
Quantifying Google’s TPU
Perhaps the biggest news last week is Google’s technical paper on Tensor processing unit (TPU) which you might know is known to be the one secret source which speed up deep-learning research within Google. It was involved in projects such as Translate and AlphaGo. Here is the full paper version. It is a very impressive work. It is both faster (15x-30x) and more energy efficient (30x-80x) than current GPUs and CPUs.
A.I. versus M.D.
This is an interesting, but tl;dr type of article. We were curious on how Prof. Hinton thinks on the future of AI on M.D. I think the following paragraph summarize his view the best:
“I think that if you work as a radiologist you are like Wile E. Coyote in the cartoon,” Hinton told me. “You’re already over the edge of the cliff, but you haven’t yet looked down. There’s no ground underneath.” Deep-learning systems for breast and heart imaging have already been developed commercially. “It’s just completely obvious that in five years deep learning is going to do better than radiologists,” he went on. “It might be ten years. I said this at a hospital. It did not go down too well.”
We would also include Pathologists in this camp. In fact, we think it’s more at risk because pathology scans are generally a lot more hi resolution than radiology scans. The delta between human and machine performance on feature detection would be much larger.
However, many AIDL members cite interesting opinions on why Prof. Hinton could be wrong for the case of using AI on medicine. Check out this thread to join the discussion!
Canada AI Moment
This piece, penned by Prof. Geoff Hinton, gives us more detail of the plan of Vector Institute. Prof. Hinton said that the new institute will be applied on fields such as healthcare, financial services and advanced manufacturing. That’s exciting because they are exactly fields which are still under research and can see many real-life applications in the future.
3 More AI-related April Fools’ Jokes
VB is going above-and-beyond in their post. They found couple of April Fools’ jokes which we didn’t quite see before. Enjoy!
Blog Posts
Facebook AI Academy
This is a glimpse on how Facebook is building a learning community within the company. The system is very close to apprentice system which there is first a series of hand-on deep learning class, then the engineers would be able to follow researchers from FAIR to work on deep learning problems.
This is fairly smart system. For the most part, programming is one type of tasks a machine learning department need to source. In real-life though, programming in machine/deep learning requires a specific mindset. For example, you need to be willing to make sure your machine is numerically accurate. And only painstaking matching would accomplish such goals. Not every programmers train in todays university curriculum would prepare for this type of work. So Facebook system allows researchers from FAIR to easily “in-source” high quality programmers within the company.
Btw, Google has a very similar program in-house. Now AI/deep learning is getting more prominent, we can expect more companies will follow the suit to beef up their own internal AI/ML education.
The $64K question now is – if you are not FB and Google, what’s your strategy?
Why Momentum Really Works
This is a very interesting note, written UC Davis’ Gabriel Goh, on why momentum works so nicely in practice. Goh starts from breaking the errors of a gradient descent based on the eigenvalues of the Hessian A (or Fisher Information Matrix depends on your formulation), he proceeds to define condition number which is ratio between the largest and smallest eigenvalues of A, which define how poorly gradient descent would perform. Goh then explains given such thinking, how would you choose the right step size (or learning rate)?
Goh carries out the same analysis on momentum-based method. Now we don’t want to spoil the rest of the paper. All we will say is he was able to explain several heuristics widely used in fairly simple arguments.
We find this paper to be must-read, not only because it explains momentum well, but also gives you insight on gradient descent, which beginner DL classes won’t teach.
BEGAN
GAN, GAN, GAN. Generative Adversarial Network is one of the hotter optic in unsupervised learning. As Prof. Yann Lecun once said it is the most important recent development in deep learning. In practice, GAN training is tough. So there are many improvements upon the basic idea which was first proposed by Ian Goodfellow.
Our summary:
- The authors take away the usual discriminator, instead they uses an autoencoder as the discriminator. The authors argue that we could assume the loss to be normal, and suggest autoencoder can be used to model a normal loss. This is not new, EBGAN is first to propose similar change in the discriminator network. BEGAN assume a pixel-wise loss function, that makes discrimination just a problem comparing distribution of real and generated images.
- After that, all you have to do is to come up with a distance function, they are using an approximate Wasserstein distance as in Martin Arjovsky’s Wasserstein GAN (WGAN). It is proven to lead to better convergence property than Jensen-Shannon distance.
- There is also a new term which balances the generator loss and discriminator loss. In practice, these issues matter because one of the issues of training GAN is to not make either one of the networks too “strong”.
The Google team then reported that the resulting method, boundary equilibrium GAN (BEGAN), outperforms many existing benchmarks. The extra plus here is that BEGAN can train both the generator and discriminator networks in parallel. Looking at the comparison between EBGAN and BEGAN. Indeed, the image generated is less ghoulish. Objectively, it also beats WGAN in the inception scores.
It’s also interesting to compare with the paper “Improved Training of Wasserstein GANs” by NYU authors. It seems like both papers were trying to issues with WGAN, which usually require weight-clipping and can occasionally lead to instability of training. The NYU authors suggest using gradient penalties to resolve the problem. In contrast, BEGAN seems to only requires tuning the equilibrium term so it sounds more handy.
Again, we don’t want to judge if BEGAN is the ultimate form of GAN. From DCGAN, to EBGAN to WGAN. The GAN space is rapidly changing. Though, if you like to play with one, BEGAN sounds like a good implementation to start with. Here is an unofficial implementation by Taehoon Kim.
Open Source
CNTK 2.0
CNTK is getting an upgrade. This time, it has Convnet and Resnet examples, interfacing with Tensorboard and Fast R-CNN examples. CNTK is not Tensorflow, so third-party development is lagging. But MSFT Research has always been a power house of deep learning research. So you should certainly check out if CNTK is fit for your company.
Video
cs224n 2017 Videos
After waiting for close to 3 months, Stanford is kind enough to share the complete videos of cs224n 2017. There are 18 lectures and it a merged course from the old cs224n which focused on more traditional NLP, and Socher’s cs224d which is more deep-learning based. Both sets of lecture used to take 16 lectures, so using 18 lectures some of materials are gone. So far though, we found most comments on the new course are positive. And just looking at the indices, there are juicy topics such as dependency grammar as well coreference resolution. So we would strongly recommend you to watch this series as well even if you have already taken the previous cs224d and cs224n before.
Bob Ross Posessed by DeepDream
After Grocery Trip, Fear and Loathing, we have the third video, by user artBoffin which is entirely processed by DeepDream, and it’s about Bob Ross, our beloved painting teacher. I can only assure your the video is both soothing and horrifying at the same time……
Member’s Question
Should You Learn Lisp?
Q: [As subject]
A: “Learning programming languages, like human languages, or generally different skills, is a way to enlighten you. LISP is a cool language because it does things differently. So sure, in that sense, Lisp may be worth your time.
On the other hand, if you do want to learn modern-day A.I. though, perhaps probability and statistics are the first “language” you want to learn well. As one member, Ernest Szeto said, nowadays A.I. usually use at least some probability-based logic. And if you think probability and statistics as a language, they are fairly difficult to learn on their own.
And yes, at AIDL, we recommend python as the first language, because it allows you to use several stacks in deep learning. You can also use R and java, but notice that there will be a gap between your work and what many people are doing.”
First publish as blog message: