I have been learning and implementing Deep Learning models using Keras Framework. Recently, a friend of mine shared his work on GitHub from the course that he is been attending and mentioned they use PyTorch to build and train neural networks. That is when I started exploring PyTorch and I found that PyTorch makes it easier and cleaner in building Deep Learning models -much easier than TensorFlow!

A short Intro

PyTorch is a python based deep learning framework and a scientific computing package. PyTorch was initially released in October 2016, primarily developed by Facebook’s AI Research lab and was based on Torch library (open source machine learning library, a scientific computing framework and a scripting language based on Lua programming language). As of 2018, Torch is no longer in active development and PyTorch is been actively developed and maintained.

PyTorch Tensors — Super Important!

In math, there is a special name for the generalization of vectors and matrices to a higher dimensional space — Tensor. Tensors are the core data structures for PyTorch, used for building and training our neural networks. Tensors and their associated operations are similar to NumPy ndarrays (another python based scientific computing package), however unlike NumPy , Tensors use the power of GPUs (Graphical Processing Units) to accelerate computing, this helps PyTorch provide maximum flexibility and speed.

Tensor is an entity with a defined number of dimensions called an order (rank). 
Scalar can be considered as a rank-0-tensor.
Vector can be introduced as a rank-1-tensor.
Matrices can be considered as a rank-2-tensor.

Let’s dive into few tensor operations and we can see how similar they are to NumPy

Before that, firstly, lets see the primary packages required to build and train neural networks using PyTorch

• torch : The top-level PyTorch package and tensor library.
• torch.nn : A subpackage that contains modules and extensible classes for building neural networks.
• torch.autograd : A sub package that supports all the differentiable Tensor operations in PyTorch.
• torch.nn.functional : A functional interface that contains typical operations used for building neural networks like loss functions, activation functions, and convolution operations.
• torch.optim : A sub package that contains standard optimization operations like SGD and Adam.
• torch.utils : A sub package that contains utility classes like data sets and data loaders that make data preprocessing easier.
• torchvision : This provides access to popular datasets, model architectures, and image transformations for computer vision.

Note — At present, torchvision package is separate from top-level torch package.

Tensors can be created in 4 ways:
– torch.Tensor(data)
– torch.tensor(data)
– torch.from_numpy(data)
– torch.as_tensor(data)

Let’s first import the torch library and create a tensor using above functions , like so

import torch

t = torch.tensor([
[1,1,1,1],
[2,2,2,2],
[3,3,3,3]
])

print(t)
print("n Type of tensor", t.dtype)

t = torch.Tensor([
[1,1,1,1],
[2,2,2,2],
[3,3,3,3]
])

print(t)
print("n Type of tensor", t.dtype)

The first option is what we call a factory function that constructs torch.tensor objects and returns them to the caller and the second option with the uppercase T is the constructor of the torch.Tensor class.
torch.Tensor() constructor uses the default dtype when building the tensor. The variants of tensor creation commands choose dtypebased on the incoming data.

# Create a numpy array
data = np.array([
[1,1,1,1],
[2,2,2,2],
[3,3,3,3]
])

# Create a tensor using as_tensor() function
t1 = torch.as_tensor(data)

# Create a tensor using as_tensor() function
t2 = torch.from_numpy(data)

torch.Tensor() and torch.tensor() copy their input data while torch.as_tensor() and torch.from_numpy() share their input data in memory with the original input object. Any changes done to the elements within original input or the new tensor — t1 or t2, is reflected within all the three tensors — data, t1 and t2.

Given all the above details the best option to create tensors is using
torch.tensor()
torch.as_tensor()

The torch.tensor() call is the sort of go-to call, while torch.as_tensor() should be employed when tuning our code for performance.

Shape of Tensor

We can view the shape of the tensor in two ways:
– t.shape
– t.size()

To determine the shape of the tensor, we first look at rows (i.e 3 in this case) and then column (i.e 4 in this case) , so the shape would be 3 X 4. The rank of the tensor would be 2 (because of two dimensions and also we can derive the rank using the number of ‘[[‘ (square brackets). We can also, get the rank of the tensor using len of shape

Number of Tensor Elements

We can get the number of elements within a tensor using two ways
* run a product operation on the shape of the tensor

* there is also a direct function available with torch package — numel()

Reshape

Reshaping operations are perhaps the most important type of tensor operations,we can change the shape of a tensor without the memory copying overhead. Tensor reshape is a key concept while building deep learning models, we can reshape the data using various method however we need to ensure that we maintain the same number of elements as in the original tensor. 
i.e In the above scenario, we have 3 X 4 = 12 elements. We can reshape this to any dimension — (2,6) (6,2) (4,3) (12,1) (1,12) (2, 2, 3) etc, so that the total number of elements unchanged.

There are two methods for reshaping : view() and reshape() essentially perform same operation by returning a reshaped tensor without changing the original tensor in place.

• view tries to return a tensor, and it shares the same memory with the original tensor. In case, if it cannot reuse the same memory due to some reason, it just fails.
• reshape always returns the tensor with the desired shape and tries to reuse the memory. If it cannot, it creates a copy

Views and reshape can infer the correct size, by passing in -1 PyTorch will infer the correct value

Changing Shape By Squeezing And Unsqueezing

Another we can change the shape of tensors is by squeezing and unsqueezing them.
– Squeezing a tensor removes the dimensions or axes that have a length of one.
– Unsqueezing a tensor adds a dimension with a length of one.
These functions allow us to expand or shrink the rank (number of dimensions) of our tensor.

Notice how the shape changes as we squeeze and unsqueeze the tensor.

Flatten a tensor

A flatten operation on a tensor reshapes the tensor to have a shape that is equal to the number of elements contained in the tensor. This is the same thing as a 1d-array of elements, Flattening a tensor means to remove all of the dimensions except for one.

Conclusion

We covered various tensor operations and their usage, however we have just scratched the surface there are many more functions and operations available in pytorch, which can be referenced from the official pytorch documentation — https://pytorch.org/docs/stable/tensors.html

References

https://pytorch.org/docs/stable/tensors.html
https://deeplizard.com/learn/playlist/PLZbbT5o_s2xrfNyHZsM6ufI0iZENK9xgG
https://jovian.ml/forum/c/pytorch-zero-to-gans/18
https://jovian.ml/divya-r-kamat/01-tensor-operations-experiment