If you’re using machine learning or deep learning then you’ve likely obsessed over making sure all your code can run on GPUs or, for the brave souls, even TPUs.
I hate to be the bearer of bad news, but your models are already likely pretty optimal for GPUs! (especially if you’re using frameworks like PyTorch Lightning that automatically let you switch between GPUs and CPUs with no code changes).
The real culprit is data thoroughput. Data can be a bottleneck in subtle ways you may not be aware of.
When dealing with data for deep learning, you are very very…
PyTorch is an amazing framework for building neural networks. It’s easy to get started and get value very quickly. But for realistic research or production use-cases, your laptop or local server won’t do.
In this tutorial, I’ll show you how to run ANY PyTorch code on the cloud without making any code changes.
For this tutorial I’m going to pick the dcgan from the PyTorch examples repository.
Whether you’re a data scientist, research engineer, AI researcher, or machine learning engineer, baselines are non-negotiable. Don’t build a fancy GAN or try a complex idea before setting up a good foundation.
In this tutorial, we’ll use Flash to build two PyTorch baselines in minutes. After that, we’ll iterate on that baseline using Lightning to get to a custom implementation tailored to your particular circumstances and squeeze more performance from your models.
Deep learning models have been shown to improve with more data and more parameters. Even with the latest GPT-3 model from Open AI which uses 175B parameters, we have yet to see models plateau as the number of parameters grow.
For some domains like NLP, the workhorse model has been the Transformer which requires massive amounts of GPU memory. For realistic models, they just don’t fit in memory. The latest technique called Sharded was introduced by Microsoft’s Zero paper in which they develop a technique to bring us closer to 1 trillion parameters.
In this article, I will give the…
It’s likely that you’ve searched for VAE tutorials but have come away empty-handed. Either the tutorial uses MNIST instead of color images or the concepts are conflated and not explained clearly.
You’re in luck!
This tutorial covers all aspects of VAEs including the matching math and implementation on a realistic dataset of color images.
The outline is as follows:
Follow along with this colab.
Code is also available on Github here (don’t forget to star!).
For a production/research-ready implementation simply install…
As an open-source developer, the question I hear the most is “why would you want to give that away for free.?”
In the field of AI, there are many reasons why opensource is key. First, the code for building models does not give away any competitive advantage because the value comes from models+your own data. Second, it lets the whole world help you find and correct mistakes. Imagine building a house where every architect in the world can contribute one tiny idea. But more importantly, AI is a really hard problem to solve.
The problems in the field cannot be…
Neural networks get a bad reputation for being black boxes. And while it certainly takes creativity to understand their decision making, they are really not as opaque as people would have you believe.
In this tutorial, I’ll show you how to use backpropagation to change the input as to classify it as whatever you would like.
Follow along using this colab.
(This work was co-written with Alfredo Canziani ahead of an upcoming video)
Let’s consider the case of humans. If I show you the following input:
This is the partner blog matching our new paper: A Framework For Contrastive Self-Supervised Learning And Designing A New Approach (by William Falcon and Kyunghyun Cho).
In the last year, a stream of “novel” self-supervised learning algorithms have set new state-of-the-art results in AI research: AMDIM, CPC, SimCLR, BYOL, Swav, etc…
In our recent paper, we formulate a conceptual framework for characterizing contrastive self-supervised learning approaches. …
Today we released 0.8.1 which is a major milestone for PyTorch Lightning. With incredible user adoption and growth, we’re continuing to build tools to easily do AI research.
This major release puts us on track for final API changes for our v1.0.0 coming soon!
PyTorch Lightning is a very light-weight structure for PyTorch — it’s more of a style guide than a framework. But once you structure your code, we give you free GPU, TPU, 16-bit precision support and much more!
Throughout the last 10 months, while working on PyTorch Lightning, the team and I have been exposed to many styles of structuring PyTorch code and we have identified a few key places where we see people inadvertently introducing bottlenecks.
We’ve taken great care to make sure that PyTorch Lightning does not make any of these mistakes for the code we automate for you, and we even try to correct it for users when we detect them. …
⚡️PyTorch Lightning Creator • PhD Student, AI (NYU, Facebook AI research).