This week, we cover Google I/O: TPU v2 is a monster, equivalent to 32 P100s. Then there is new free TPU research clusters, TensorFlow lite which is a package for easy TF development on embedded device, automatic search of network architecture. All the amazing features!
Google could have taken quite a bit of attention away from Nvidia GTC by announcing TPU v2 last week, as The Next Platform's Nicole Hemsoth nicely put it:
[......] we have to harken back to Google’s motto from so many years ago… “Don’t be evil.” Because let’s be honest, going public with this beast during the Volta unveil would have been…yes, evil.
In other news, AMD also announced a competing product, the Radeon RX Vega. However, AMD has been having a hard time against Nvidia due to software issues (more details in this issue) even when its specs are slightly better and the card is cheaper. This is the power of the software moat. Hardware commoditizes fast but software makes things sticky.
Other than Google I/O and AMD new GPU card, we also include several nice resources and links this week including:
Arthur's Review on Five Basic Deep Learning Classes
Adit Despande's github on using Tensorflow.
As always, if you like our letter, feel free to subscribe and forward it to your colleagues!!
We also just reach 20000 members in our AIDL forum, so come join us!
Artificial Intelligence and Deep Learning Weekly
A Correction on Issue #13 Editorial
About the Editorial of Issue 13: Peter Morgan is kind enough to correct us - both Nervanna and TPU are based on ASIC, rather than FPGA. We have corrected the web version since.
Imagine this, Nvidia P100 is probably the best GPU you can buy last year, you bought one, but realize there is yet another device which is as fast as 32 P100 combined! So here is TPU v2, a monster device quietly developed last 2 years and it's only released after v1's specification is opened. We chose the article from The Next Platform, because it has the best writeup.
If you were Google, perhaps an important question you would ask is how to sell the powerful TPU. Google's natural strategy is cloud. The way they market and test the water is through opening 1000 TPUs to public. So here's the free sign-up links for everyone: not only researchers, but anyone who has computational need.
It says a lot of Google's cloud strategy - TPU is an important differentiator between Google and its competitors such as Amazon and lately Nvidia. Google tends to think that inference will be where the cost/compute/power bottlenecks will be.
While Tensorflow is very versatile in building AI-based tools, it is also notoriously difficult to port its code to embedded platform. Tensorflowlite is going to change that. We don't have too much detail yet, but we expect convenience features, or utilities, which can quickly transform models onto Android.
In the last issue (issue 13) , we argued that Nvidia has incredible moat on deep learning hardware. Here's AMD's counterstrike! And so far it looks respectable. The most noticeable feature is the 16G on-board memory. That's a new in the industry.
Would AMD makes researchers switch? It's still hard to say, as we argued last issue, Nvidia has both hardware and software advantage. It's hard to move away from CUDA from existing development. Since many CUDA core libraries are close source, it's hard for AMD to come up with an equivalent architecture using say OpenCL.
One of us (Arthur) wrote an article of on five basic classes of deep learning. That includes some discussion on why these classes are important, what you can learn from them, and what you can do after taking the classes. The list includes,
Andrew Ng's Coursera Machine Learning,
Fei-Fei Li and Andrew Karpathy's Convolutional Neural Networks for Visual Recognition or Stanford cs231n 2015/2016,
Richard Socher's Deep Learning and Natural Language Processing or Stanford cs224d,
Architecture search becomes more a respectable research. Perhaps an interesting results was Zoph and Le published last year - it comes up with state of the art results on CIFAR-10, which 0.09% more accurate and 5% faster. This is stunning because before most automatic tuning methods are slightly worse than human tuning.
You can go into the details of how reinforcement learning is applied to such problem, but I want to point out one important trend here: architecture search is also applied on CIFAR-100, which usually requires one more order of computation. So if we continue to develop, the next target would be Imagenet. That would mean such search become a staple in all top research. You might find them in widespread use in the future.
Adit Despande wrote several popular introductory blog posts in the space of deep learning in the past. The one which is the most memorable perhaps is "The 9 Deep Learning Papers You Need To Know About" (Part 1, Part 2, Part 3). Recently, he also published a github about Tensorflow. The examples include GAN, RNN and CNN, which are more advanced variations of neural networks.
Google recent release the "Coarse" dataset - it's fairly interesting because it is the biggest dataset of online forum discussion, other the turns, there are also annotations on discourse types, which include announcement, question, answer, agreement, disagreement, appreciation, negative reaction, elaboration, and humor. That seems to be a very useful set if you want to do either conversation bot, or text classification type of applications.
Bay Area-based startup Dishcraft looking for a machine learning engineer. Well-funded by tier-1 brand-name investors (led by First Round Capital) and are doing extremely well. For the right candidate, willing to relocate the person.
Looking for basic traditional ML (SVM and boosting). Kaggle experience is a plus, Deep Learning for 2D images and 3D volumetric data (CNN focused), Tensorflow + Keras. Desirable computer vision skills: point cloud processing, signal and image processing, computational photography (familiarity with multi-view geometry and stereo vision, and color processing)
Ajay Juneja share this on AIDL:
"Thoughts from the admin of the Self-Driving Car group this week (I attended the Nvidia Conference):
Bi-LSTMs (Bi directional LSTMs) are everywhere, and working quite well. If you aren't using them yet, you really should. For those of us from the mechanical engineering world, think of them a bit like making closed-loop feedback control systems.
The convergence of AI, VR, AR, Simulation, and Autonomous Driving. It's happening. Need to generate good data for your neural nets, quickly? Build a realistic simulator using Unreal Engine or Unity, and work with gaming developers to do so. Want to make your VR and AR worlds more engaging? Add characters with personality and a witty voice assistant with emotion to them, while using cameras and audio to determine the emotional state of the players. Want to prototype a new car or building or surgery room? Build it in VR, create a simulator out of it. We need to cross pollinate these communities and have everyone working together 🙂
Toyota signed with Nvidia. That's 8 of 14 car companies... and they have signed the 2 largest ones (VW and Toyota). I hear rumblings from the AI community saying "If you want to build a self driving car TODAY, your choices are Nvidia and... nothing. What can I actually buy from Intel and Mobileye? Where are the engineers to support it? Qualcomm may have something for the 845 but they are drunk on mobile profits and again, no one knows their tools."
500K Nvidia Developers vs. next to nothing for Intel and Qualcomm's solutions.
I believe Nvidia has its moat now."
Artificial Intelligence and Deep Learning Weekly
Speech Recognition, Machine Learning, and Random Musing of Arthur Chan