AI Winter isn’t Coming, but Variable Cloudiness with a Shower is Expected
After seeing a story about next AI winter in google now, then multiple times in twitter, then similar questions have appeared on quora, I got a little confused. I think it gets much more attention than it deserves. Why? Let’s sort through.
In general this story explains why next AI winter is near. And it’s arguments are… weak at least.
Decreasing tweets-per-day rate by Andrew Ng is not an argument. At all. Unlike us, bloggers-prophets, he has real and important work to do.
Lower than human performance of a DL model on MURA challenge. First of all, it’s a BASELINE model. It’s not supposed to win the competition. Second, training dataset consists of ~40.000 training images. Run the same thing on a million examples and you’ll see a much better picture. Third, model’s competitors were doctors with highest-quality education. Not every hospital and every country can afford those. And, to make things clear, I have personal reasons to regard this work as extremely important. In the past it might have relieved me from years of limping.
“There are much fewer tweets praising deep learning as the ultimate algorithm, the papers are becoming less ‘revolutionary’ and much more ‘evolutionary’”
And that’s how it is supposed to be. DL is not a panacea.
“Deepmind hasn’t shown anything breathtaking since their Alpha Go zero”.
AlphaGo Zero was announced just about 6 months ago. You can’t blame them for not making major breakthroughs several times a year.
“In fact articles began showing up that even Google in fact does not know what to do with Deepmind, as their results are apparently not as practical as originally expected”.
First, after restructuring the whole thing into Alphabet they still have to figure out how to manage all of it’s divisions. But this is a structural issue and has nothing to do with AI. Second, if reducing cooling bills by 40% isn’t practical for you, I don’t know what is. Maybe WaveNet, which is hugely responsible for revolution in speech generation, or their new features for Android P announced just a few weeks ago will help you reconsider that. Third, DeepMind is not just a bunch of ML engineers, but also heavily focused on theoretical research right on the border between Neuroscience and AI, and showing amazing results(like this one).
“As for the prominent researchers, they’ve been generally touring around meeting with government officials in Canada or France to secure their future grants, Yann Lecun even stepped down (rather symbolically) from the Head of Research to Chief AI scientist at Facebook. This gradual shift from rich, big corporations to government sponsored institutes suggests to me that the interest in this kind of research within these corporations (I think of Google and Facebook) is actually slowly winding down. Again these are all early signs, nothing spoken out loud, just the body language”.
Yann Lecun will now serve as Chief AI Scientist at Facebook. And I think the main reason is much simpler, I guess he wants to do what he does best — science, not management. And I don’t see any decline in interest in AI from big corporations, only growing interest from governments(especially Chinese).
One of the key slogans repeated about deep learning is that it scales almost effortlessly. We had the AlexNet in 2012 which had ~60M parameters, we probably now have models with at least 1000x that number right? Well probably we do, the question however is — are these things 1000x as capable? Or even 100x as capable?”.
Model is not the only thing. We don’t have x1000 data and computational resources(except Google, Facebook etc).
Then there is lot of misinterpretation of this graph:
And a question in the end: “OK, so we can now train AlexNet in minutes rather than days, but can we train a 1000x bigger AlexNet in days and get qualitatively better results? Apparently not…”. In my view, we can take 1000x bigger computational resources and get significant improvements. Take AutoML for example and compare it’s performance to AlexNet (~83% vs ~57% in top-1 accuracy). Looks like a qualitatively better result to me.
“So in fact, this graph which was meant to show how well deep learning scales, indicates the exact opposite. We can’t just scale up AlexNet and get respectively better results — we have to fiddle with specific architectures, and effectively additional compute does not buy much without order of magnitude more data samples, which are in practice only available in simulated game environments.”.
I’m sorry, but this looks to me like a plain bullshit.
“Self driving crashes”.
Wow. Indeed it is. It is, however, astronomically better than it was 10 or 5 years ago. And crash statistics on semi-autonomous autopilot shows that it can already bring value. We haven’t got to completely autonomous cars yet, but buses, trains and other vehicles with predictable routes are on their way.
And the conclusion:
“Predicting the A.I. winter is like predicting a stock market crash — impossible to tell precisely when it happens, but almost certain that it will at some point. Much like before a stock market crash there are signs of the impending collapse, but the narrative is so strong that it is very easy to ignore them, even if they are in plain sight. In my opinion there are such signs visible already of a huge decline in deep learning (and probably in AI in general as this term has been abused ad nauseam by corporate propaganda), visible in plain sight, yet hidden from the majority by the increasingly intense narrative. How “deep” will that winter be? I have no idea. What will come next? I have no idea. But I’m pretty positive it is coming, perhaps sooner rather than later.”
As I wrote on Quora:
Personally, I’m glad that this hype is going down. I’m already full of news with images from Terminator and titles like “AI is going to replace doctors by 2020”. Or this one:
I’m absolutely for realistic view on AI/ML/DL, not media-driven.
And, by the way, Gary Marcus is a great guy.
