# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---
# #  Introduction to Numpy and Pandas
# > This blog gives introduction to basic commands
#
# - toc: true
# - branch: master
# - badges: true
# - comments: true
# - author: <NAME>
# - categories: ['fastpages', 'jupyter']
# - image: images/numpy1.png
# # Introduction to Python Libraries: Numpy and Pandas
# **This blog explores Numpy Library For Basic Array/Matrix operations and Pandas Library for reading and manipulating different types of files.<br>
# <br>
# Prerequisites:** Basic Data structures in python like list, tuple, dictionaary and control structures in python<br>
# <br>
# ## Numpy
#
# NumPy, which stands for Numerical Python, is a library consisting of multidimensional array objects and a collection of routines for processing those arrays. Using NumPy, mathematical and logical operations on arrays can be performed. NumPy provides the foundation data structures and operations for SciPy.
# Install Numpy package using command **pip install numpy** or **conda install numpy** command
# ### Array
# The most important object defined in NumPy is an N-dimensional array type called ndarray. It describes the collection of items of the same type. Items in the collection can be accessed using a zero-based index.
# Every item in an ndarray takes the same size of block in the memory. Each element in ndarray is an object of data-type object (called dtype).
# Any item extracted from ndarray object (by slicing) is represented by a Python object of one of array scalar types. The following diagram shows a relationship between ndarray, data type object (dtype) and array scalar type.
# The basic ndarray is created using an array function in NumPy as follows −
# ### Defining Arrays and Basic Array Operations
# Import numpy library before using it
import numpy as np
x=np.array([1,2,3,4])# Define 1D Array
print(x) # Print the array
x1=np.array([[1,2,3],[4,5,6]]) #Define 2D Array
print(x1)
np.shape(x) # Return the shape of an array. i.e. Number of rows and columns in array
np.shape(x1)
x1.shape # This also gives shape of array
x1.ndim # ndim gives no. of dimensions of array. x1 is 2 dimensional array
x.ndim # Check dimension of x
print(len(x1))
len(x)
type(x1) # It gives type of x1
x1.dtype # Gives data type of elements in x1
# creating array from nested python lists
list1=[1,2,3,4]
list2=[2,3,4,5]
list3=[3,4,5,6]
array1=np.array([list1,list2,list3])
print(array1)
print(array1.shape)
# Generate array of random numbers in the interval[0,1]
# If the given shape is, e.g., (m, n, k), then m * n * k samples are drawn. 
# Default is None, in which case a single value is returned.
x3=np.random.random((4,3))
print(x3)
# Return a new array of given shape and type, filled with zeros.
z=np.zeros((3,3))
print(z)
# Return a new array of given shape and type, filled with zeros.
z1=np.ones((3,3))
print(z1)
len(z1)
md=np.random.rand(2,3)# Generate matrix of shape 2*3 with matrix elements between 0 and 1
print(md)
a=np.arange(1,5,1)# generate sequence of numbers excluding stop value, start, stop, interval
print(a)
a=np.arange(10,50,4)
print(a)
print(a.dtype)
type(a)
np.arange(3) # if no interval is mentioned, Default interval is one
# Return evenly spaced numbers over a specified interval.
#Returns num evenly spaced samples, calculated over the interval [start, stop].
#The endpoint of the interval can optionally be excluded.
b=np.linspace(10,50,4)# generate sequence of numbers, start, stop,how many numbers
print(b)
# numpy.full(shape, fill_value, dtype=None, order='C')[source]
#Return a new array of given shape and type, filled with fill_value
m=np.full((5,5),11)
print(m)
a=np.arange(10,50,4)
print(a)
print(a.dtype)
type(a)
# ### Array slicing:
# Contents of ndarray object can be accessed and modified by indexing or slicing, just like Python's in-built container objects.
# As mentioned earlier, items in ndarray object follows zero-based index. Three types of indexing methods are available − field access, basic slicing and advanced indexing.
#
# Basic slicing is an extension of Python's basic concept of slicing to n dimensions. A Python slice object is constructed by giving start, stop, and step parameters to the built-in slice function. This slice object is passed to the array to extract a part of array.
# Extracting required elements of array
#If only one parameter is put, a single item corresponding to the index will be returned. 
#If a : is inserted in front of it, all items from that index onwards will be extracted. 
#If two parameters (with : between them) is used, items between the two indexes (not including the stop index)
#with default step one are sliced.
print(a)
print(a[2]) # element with index 2
print(a[2:5]) # elements from index 2 to 5-1 are printed
print(a[2:]) # 2nd element onwards all elements are printed
print(a[:3]) # element from index 0 to 3-1 are printed
print(a[2::2])# from index 2, with interval 2
print(a[2::3]) # from index 2, with interval 3
print(a[::-1]) # elements from last to first, i.e. reverse order as step is -1
print(a[::-2]) # Elements in reverse order with step -2
print(a[2::-1]) # Elements in reverse order from index 2 to 0 with step -1
print(a[-4::2]) # print elements from 4th last element onwards with step 2
# +
a = np.array([[1,2,3],[3,4,5],[4,5,6]]) 
print(a)
# slice items starting from index
print ('Now we will slice the array from the index a[1:]')
print(a[1:])# Print first row onwards
# -
# Slicing can also include ellipsis (…) to make a selection tuple of the same length as the dimension of an array. 
# If ellipsis is used at the row position, it will return an ndarray comprising of items in rows.
# +
# this returns array of items in the second column 
print('The items in the second column are:')  
print(a[...,1])
print('\n')
# Now we will slice all items from the second row 
print('The items in the second row are:')
print(a[1,...]) 
print('\n')
# What is output of this? Execute and check
print(a[1:,...])
# Now we will slice all items from column 1 onwards 
print('The items column 1 onwards are:') 
print(a[...,1:])
# -
# ### Array manipulation
# Several routines are available in NumPy package for manipulation of elements in ndarray object.
# Reshape,Joining and broadcasting are covered here 
#
# Concatenation refers to joining. This function is used to join two or more arrays of the same shape along a specified axis.
#
# NumPy has in-built support for broadcasting. This function mimics the broadcasting mechanism. It returns an object that encapsulates the result of broadcasting one array against the other.
np.reshape(x1,(3,2)) # Reshapes array to specified dimension
x1.reshape(3,-1)#for large arrays, we may not know the dimension of any one of the axis, in such case provide that unknown dimension as -1
np.transpose(a)# interchange rows and columns of array
# +
import numpy as np 
a = np.array([[1,2],[3,4]]) 
print('First array:')
print(a) 
print('\n')
b = np.array([[5,6],[7,8]]) 
print('Second array:') 
print(b) 
print('\n')
# both the arrays are of same dimensions 
print('Joining the two arrays along axis 0:') 
print(np.concatenate((a,b)))
print('\n')  
print('Joining the two arrays along axis 1:') 
print (np.concatenate((a,b),axis = 1))
# +
# Broadcasting
a = np.array([[0.0,0.0,0.0],[10.0,10.0,10.0],[20.0,20.0,20.0],[30.0,30.0,30.0]]) 
b = np.array([1.0,2.0,3.0])
print(a)
print(b)
print(a+b)
# -
# ### Arithmatic, Statistical and Linear algebraic functions in Numpy
# #### Arithmatic Operations
# Add two arrays
print(a+b)
print(np.add(a,b))
a=np.array([[1,2],[3,4]])
b=np.array([[2,2],[3,3]])
print(a)
print(b)
# Subtract two arrays
print(a-b)
print(np.subtract(a,b))
# Multiply 2 matrices elementwise
print(a*b)
print(np.multiply(a,b))
# Divide one array by another, elementwise
print(a/b)
print(np.divide(a,b))
##the floor division // rounds the result down to the nearest whole number
print(a//b)
print(a)# a is 2x2 matrix
# Sum of the array elements (a scalar value if axis is none) or array with sum values along the specified axis.
np.sum(a)# sum of elements in matrix
# Sum of the array elements (a scalar value if axis is none) or array with sum values along the specified axis.
np.sum(a,axis=0)
np.sum(a,axis=1)
# #### Statistical Functions in Numpy
# NumPy has quite a few useful statistical functions for finding minimum, maximum, mean,median from the given elements in the array.
# Using np.min, max, mean,median,argmax,argmin 
print(np.min(a))
print(np.max(a))
print(np.mean(a))
print(np.median(a))
print(np.argmin(a))
print(np.argmax(a))
# #### Algebraic Functions in Numpy
# Matrix product of two arrays. If both arguments are 2-D they are multiplied like conventional matrices.
np.matmul(a,b)
print(a*b)# element wise multiplication
# +
# This function returns the dot product of two arrays. 
#For 2-D vectors, it is the equivalent to matrix multiplication. For 1-D arrays, it is the inner product of the vectors. 
#For N-dimensional arrays, it is a sum product over the last axis of a and the second-last axis of b
print(a.dot(b))# matrix multiplication
# +
# try np.mod
# np.reminder
# +
# Generating array with random values
ar1=np.random.rand(3,4) # Generate 3x4 matrix of nubers between 0 and 1
print(ar1)
# -
ar2=np.random.randn(3,4) #Return a sample (or samples) from the “standard normal” distribution.
print(ar2)
# numpy.random.randint(low, high=None, size=None, dtype=int)
#Return random integers from low (inclusive) to high (exclusive).
#Return random integers from the “discrete uniform” distribution of the specified dtype in the “half-open” interval [low, high).
#If high is None (the default), then results are from [0, low).
ar3=np.random.randint(1,10,8).reshape(4,2)
print(ar3)
# +
# Use statistical functions on array a3
# find min element of array
print(np.min(ar3))
# Find maximum element of array
print(np.max(ar3))
# Find mean of all array elements
print(np.mean(ar3))
# Find Median
print(np.median(ar3,axis=0))
# index of smallest number in array
print(np.argmin(ar3))
# index of smallest number at given axis
print(np.argmin(ar3,axis=0))
print(np.argmin(ar3,axis=1))
# index of largest number in array
print(np.argmax(ar3))
# -
#Return random floats in the half-open interval [0.0, 1.0).
#Results are from the “continuous uniform” distribution over the stated interval. 
ar4=np.random.random_sample(3)
print(ar4)
print(np.argmax(ar3))
print(np.argmax(ar3,axis=0))
print(np.argmax(ar3,axis=1))
# ## PANDAS
# Pandas is an open-source, BSD-licensed Python library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language. Python with Pandas is used in a wide range of fields including academic and commercial domains including finance, economics, Statistics, analytics, etc.
# ### Series
# Series is a one-dimensional labeled array capable of holding any data type (integers, strings, floating point numbers, Python objects, etc.). The axis labels are collectively referred to as the index. The basic method to create a Series is to call:
# **s = pd.Series(data, index=index)**
# Here, data can be many different things:
# a Python dict, an ndarray, a scalar value (like 5)
# The passed index is a list of axis labels. Thus, this separates into a few cases depending on what data is.
import os
import numpy as np
import pandas as pd
# Generate a series as random array whose elements are drawn from normal distribution
s = pd.Series(np.random.randn(5), index=['a', 'b', 'c', 'd', 'e'])
s
s.index
#Series can be instantiated from dicts:
#When the data is a dict, and an index is not passed, the Series index will be ordered by the dict’s insertion order 
d = {'b': 1, 'a': 0, 'c': 2}
pd.Series(d)
#When working with raw NumPy arrays, looping through value-by-value is usually not necessary. 
#The same is true when working with Series in pandas. Series can also be passed into most NumPy methods expecting an ndarray.
s+s
# ### Dataframe
# DataFrame is a 2-dimensional labeled data structure with columns of potentially different types. You can think of it like a spreadsheet or SQL table, or a dict of Series objects. It is generally the most commonly used pandas object. Like Series, DataFrame accepts many different kinds of input:
#
# Dict of 1D ndarrays, lists, dicts, or Series, 2-D numpy.ndarray,structured or record ndarray,A Series, another DataFrame.
#
# Along with the data, you can optionally pass index (row labels) and columns (column labels) arguments. If you pass an index and / or columns, you are guaranteeing the index and / or columns of the resulting DataFrame. Thus, a dict of Series plus a specific index will discard all data not matching up to the passed index.
#
# If axis labels are not passed, they will be constructed from the input data based on common sense rules.
# Creating Dataframe from series
d = {'one': pd.Series([1., 2., 3.], index=['a', 'b', 'c']),'two': pd.Series([1., 2., 3., 4.], index=['a', 'b', 'c', 'd'])}
pd.DataFrame(d)
# Read .csv file
data=pd.read_csv('C:/Users/Swift 3/AppData/Local/Programs/Python/Python38/Iris_data_sample.csv')
# To view a small sample of a Series or DataFrame object, use the head() and tail() methods. 
#The default number of elements to display is five, but you may pass a custom number.
data.head()
# By default, idex column is labeled from 0 onwards.
#index_col=0 will remove default index column
# and Makes passed column as index instead of 0, 1, 2, 3…r
data=pd.read_csv('C:/Users/Swift 3/AppData/Local/Programs/Python/Python38/Iris_data_sample.csv',index_col=0)
data.head()
data.index # to get row labels
data.columns # to get column labels
data.shape# rows n columns in dataframe
data.size # no of elements in dataframe
data.ndim # no of dimensions of dataframe
data.memory_usage()#Return the memory usage of each column in bytes.
# ### Indexing and Slicing of DataFrame
data.head(5)
data.tail(4)
data.loc[:,'SepalLengthCm']# to access rows or columns by label
data.loc[:,'Species']
data.loc[:,['SepalLengthCm','Species']]# access multiple columns by label
data.loc[4,:] # access 4th row of dataframe
data.loc[[4,7],:] # access 4th and 7th row of dataframe
data.iloc[0:4,0:2]# access first 4 rows and first 2 columns using index of rows and columns
data.iloc[34:37,:]# dataframename.iloc[start:end,start:end]    end value is excluded
# create a copy of dataframe
iris1=data.copy(deep='False')# new variable shares only reference of original object
iris2=data.copy(deep='True')#new variable creates copy of original object
# ### Data Types
# There are two main types of data
# Numeric and character types.
# Strings are known as objects in Pandas
data.dtypes # to know the data type of each column
data.select_dtypes(exclude=[object])# exclude columns with 'object' data type
# Print a concise summary of a DataFrame.
#This method prints information about a DataFrame including the index dtype and columns, non-null values and memory usage.
# Syntax:  DataFrame.info(verbose=None, buf=None, max_cols=None, memory_usage=None, null_counts=None)
data.info()
#print(np.unique(data['Species'])) i.e. unique values in Species column
data['Species'].unique()
# value_counts() Return a Series containing counts of unique rows in the DataFrame.
# DataFrame.value_counts(subset=None, normalize=False, sort=True, ascending=False)
data['Species'].value_counts()# print count 
# The missing values exist in the form of ‘nan','??','???'
#
# Python, by default replace blank values with ‘nan'
# convert '??' and '???' values to NaN
data.isnull().sum()
data['SepalLengthCm'][0:5]# check the value before using na_values
data=pd.read_csv('C:/Users/Swift 3/AppData/Local/Programs/Python/Python38/Iris_data_sample.csv',index_col=0,na_values=['??','###'])
data['SepalLengthCm'][0:5]# check the value before using na_values
data.info()# observe data type before and after using na_values attribute
data.isnull().sum()# count no of missing values in each column
# reading excel file using Pandas
# before reading excel file, you need to import xlrd package to read excel files, Install it using 'pip install xlrd' command
iris_excel=pd.read_excel('C:/Users/Swift 3/AppData/Local/Programs/Python/Python38/Iris_data_sample.xlsx')