The definitive weekly newsletter on A.I. and Deep Learning, published by Waikit Lau and Arthur Chan. Our background spans MIT, CMU, Bessemer Venture Partners, Nuance, BBN, etc. Every week, we curate and analyze the most relevant and impactful developments in A.I.
We also run Facebook’s most active A.I. group with 191,000+ members and host a weekly “office hour” on YouTube.
Editorial
Thoughts From Your Curators
This week we link you to the latest AI Index report, discuss the impact of reorganization of Udacity and analyze a blog post from OpenAI.
Join our community for real-time discussions here: Expertify
This newsletter is published by Waikit Lau and Arthur Chan. We also run Facebook’s most active A.I. group with 186,000+ members and host an occasional “office hour” on YouTube. To help defray our publishing costs, you may donate via link. Or you can donate by sending Eth to this address: 0xEB44F762c58Da2200957b5cc2C04473F609eAA65.
News
Udacity Reorg
When you think about it, democratization of AI also coincided with democratization of on-line learning. Remember Coursera Machine Learning by Andrew Ng? It is when many of us start to learn about machine learning, but it’s also the first time many of us learn an important skill through an on-line curriculum.
Yet these days both Coursera and Udacity seem to have issues to come up with a sustainable business models. For Coursera, they hired a new CEO back in July last year, and his focus is on searching for a viable business model. Udacity’s route is as winding and tortuous. They are thinking of filing an IPO earlier this year, yet they went through a round of layoff. And it seems like they will focus more on corporate training than the current models.
What does it mean to us AI/DL learners? For starter, it might mean that the beloved subsidies for underprivileged students would be gone from on-line learning sites. Or it can also mean development of new materials would slow down.
We hope our prediction is wrong. Hopefully, both Udacity and Coursera can come up with business models which can be both profitable yet cater to learners.
Blog Posts
Reading the AI Index 2018 Report
The new AI Index report for 2018 is just published last week. We are excited because you may think of it as one report which capture the state of AI development every year. Looking through the report, you may find the section on the growth of AI as a field and its public perception. You may also learn the trend of the state of the art of several fields such as computer vision and machine translation.
The 2017 report was criticized for focusing too much on North America development, but then this year you can see the report cover much of activities around the world. e.g. We learn that Tsinghua University has the highest increase of course enrollment in AI.
Notably missing in the report is automatic speech recognition (ASR). That perhaps has to do with the difficulty of searching for one golden benchmark for ASR. As you may know, ASR performance tremendously across different noise condition.
Anyway, we recommend you to read the report in details.
How to scale AI Training without voodoo?
Last few years, we several works in deep learning training which use large batch size. The advantage of the approach is that you may easily spread the computation of a large batch across several machines.
But here is the problem, how can you decide what batch size to use, or… if a task is suitable to have a large batch size. OpenAI’s work seems to give at least one answer to the problem. They found that a simple quantity, gradient noise scale correlate to the optimal batch size in a training task.
Reading the paper, there are many to unpack here. e.g. The authors favor the use of simplified version of the metric which just require calculation of a determinant of a matrix, and the trace of the covariance matrix. They found that such measure correlates with the batch size.
All-in-all, this is indeed an interesting finding because now you may have a guidance for tuning one important parameter, batch size, in your training. Perhaps the question in practice: can you just use a sample of your dataset to training set to measure gradient noise scale, and use it in a larger set? And more importantly, do we have other guidance on tuning other parameters? Those are interesting question to ask, and if we can solve them, may be we can truly call neural network training a science more than an art.
“Contributing and bringing machine learning to JavaScript ecosystem with TensorFlow.js” by Manraj Singh
There are many posts on AIDL which ask about how to use a deep learning framework. But there are many of them are about experience in contributing deep learning engine. Singh’s post is a notable exception.
Open Source
Papers With Code
Written by Zaur Fataliyev and suggested by one of our AIDL moderators, Zubair Ahmed, this github contains papers with its codebase, and it spans from 2013 to 2018. It’s a great resource for everyone who not just want to read deep learning papers, but also study the underlying implementations.
About Us
About Us
This newsletter is published by Waikit Lau and Arthur Chan. We also run Facebook’s most active A.I. group with 186,000+ members and host an occasional “office hour” on YouTube. To help defray our publishing costs, you may donate via link. Or you can donate by sending Eth to this address: 0xEB44F762c58Da2200957b5cc2C04473F609eAA65.
Join our community for real-time discussions here: Expertify