When I have to deal with Huge image datasets, this is what I do. Working with image datasets in Kaggle competitions can be quite problematic, your computer could just freeze and don't care about you anymore. To stop this things from happening, I'm going to be sharing with you here the 5 Major Steps to work with Image datasets.
This is like the most usefull resource on practical AI that you can possibly find today! Have you ever wanted to play around with the AMAZING Kaggle datasets? If this interests you, buckle in!
Having some conversations about the Monty Hall experiment and got quite surprised on how many people even after the explanation, still think that they would have more chances of winning by sticking with the first choice. I started to think, there has to be a way that I can explain this better. Elaborating on this that let me to this analysis that I present to you today.
Convolutional Neural Networks are heavily used in computer vision and recomendation systems. The magic behind Neural Networks in general lies on hidden layers as well as training, but how these layers work on Convolutional Neural Nets?
If you haven't heard about Convolutional Neural Networks yet ( Do you live in a cave? Just kidding ), or Neural Networks in general, I bet you've heard all about it's applications. They're used in recognition systems such as Face recognition, handwritten digits recognition and also used in my loved example, self-driving cars ( I really enjoy self-driving cars if you haven't noticed ). Ok, now you know what this is all about, but how Convolutional Neural Networks work?
Let’s talk about how you can bring your AI application into the world.
Machine Learning and AI projects are getting a lot of attention these days. A great deal of libraries are in the game. You can easily play around and fast enough create your on model, maybe even test it on some dataset on kaggle. But have you ever thought about taking AI to the real world, like self-driving cars?
Neural networks are all over the place and have been extremely successfull. Have you ever stopped to think about how a computer learns to see the world as well as a person? How they can recognize your face in smart devices authentication process and furthermore your retina? To do this smart devices use deep learning and computer vision, got interested? Buckle up and let's go for it!
It's time to get your hand's on training your Neural Network, because you are going are going to be capable of working through parameters with real data coming from the MNIST Kaggle competion.
If you haven't been following the previous steps, you can check out how Kaggle works and how to start with the MNIST competion. After getting confortable with the data you could look how to create basic Neural Networks and how to train them. If you are already up to date and ready for a challenge, buckle up and let's go for it.
When working with Neural Networks in PyTorch, some functions expect specific types of tensors, you can look up the types here. But what caught me on this specific Neural net was the difference between torch.Tensor and torch.tensor. The images we pass in the forward pass is expected to be torch.FloatTensor, but the labels are expected to be torch.LongTensor, you can create the first type with torch.Tensor, and the second one with torch.tensor.
After creating your model you need to train it. This is where things got interesting, we have in total 42000 images, we're going to create simple batches of 64 images and loop it through. The epochs are how many times we're going to repeat this process.
We can see that our model can predict that the handwritten digit is a zero
There are innumerous learning opportunities here, I've started with a Softmax function in the end layer and discover that the LogSoftmax worked way better. Also the Loss function and the optmizer could be chosen differently. PyTorch also has some cool functions to normalize and clean the data. An important thing also is validation and testing datasets, I'll be making more posts on this.
If you want more content like this, I have a YouTube Channel