Pandas Selecting document summary

6/5/2018

This walk-through jupyter notebook outlined the important techniques shown in pandas document for selecting and modifying. The website can be referred as: Pandas advanced indexing with labels

Dataset uses the housing price competition from Kaggle.com. The website is Housing price competition

Selecting and modifying with Pandas is very useful tools for data scientists to grasp. Let’s start.

import pandas as pd
test = pd.read_csv('C:/Users/t_louji/Downloads/test.csv',index_col='Id')
test.head()
#test.info()

Selecting

1. .loc (select by label)

.loc is primarily label based, but may also be used with a boolean array.

Allowed inputs are:

• A single label, e.g. 5 or ‘a’ (Note that 5 is interpreted as a label of the index. This use is not an integer position along the index.).

• A list or array of labels [‘a’, ‘b’, ‘c’].

• A slice object with labels ‘a’:’f’ (Note that contrary to usual python slices, both the start and the stop are included, when present in the index! See Slicing with labels.).

• A boolean array

• A callable function with one argument (the calling Series, DataFrame or Panel) and that returns valid output for indexing (one of the above).

# test.loc[0] #key error
test.loc[1461:1500]
test.loc[test.MSZoning=='RH']
test.loc[test.MSZoning!='RH','Alley'].head()

Id
1462    NaN
1463    NaN
1464    NaN
1465    NaN
1466    NaN
Name: Alley, dtype: object

df1.loc['a', 'A']   #row label a and column A

**Note: ** When slicing, both the start bound AND the stop bound are included, if present in the index. Integers are valid labels, but they refer to the label and not the position.

2. .iloc (select by position)

.iloc is primarily integer position based (from 0 to length-1 of the axis), but may also be used with a boolean array.

Allowed inputs are:

• An integer e.g. 5.

• A list or array of integers [4, 3, 0].

• A slice object with ints 1:7.

• A boolean array.

• A callable function with one argument (the calling Series, DataFrame or Panel) and that returns valid output for indexing (one of the above).

test.iloc[0:3]
test.iloc[0:3,0:3]
test.iloc[3,3]  # a scalar value
test.iloc[1]

#Out of range slice indexes are handled gracefully just as in Python/Numpy.
test.iloc[3000:30001]
#A single indexer that is out of bounds will raise an IndexError. 
#A list of indexers where any element is out of bounds will raise an IndexError.

** Note ** :
Object Type Indexers

        Series	      s.loc[indexer]

        DataFrame	df.loc[row_indexer,column_indexer]

3. [ ]

Series series[label] scalar value

DataFrame frame[colname] Series corresponding to colname

test['MSZoning'][test.index[0]]  #get scala element

'RH'

4. Swap two columns

# The correct way to swap column values is by using raw values:

#df.loc[:,['B', 'A']] = df[['A', 'B']].values

5. Attribute Access

.COL_Name

test1=test.copy()
test1.Street.head()

Id
1461    Pave
1462    Pave
1463    Pave
1464    Pave
1465    Pave
Name: Street, dtype: object

x = pd.DataFrame({'x': [1, 2, 3], 'y': [3, 4, 5]})
x.iloc[1] = dict(x=9, y=99) # assign a dict to a row of a dataframe

x

**Note: ** You can use attribute access to modify an existing element of a Series or column of a DataFrame, but be careful; if you try to use attribute access to create a new column, it creates a new attribute rather than a new column.

6. Slicing ranges

df[:n]

test[::2].head() ##every odd column
s2[:5] = 0

ix indexer is deprecated

7. Scalar value getting and setting

Similarly to loc, at provides label based scalar lookups, while, iat provides integer based lookups analogously to iloc

test.at[1461,'MSZoning']

'RH'

test.iat[0,1]

'RH'

8. Boolean indexing

test[test.MSSubClass>100].head()
##multiple conditions
test[(test.MSSubClass>100)&(test.MSSubClass<160)].head()

Comprehensive methods

criterion = test['Street'].map(lambda x: x.startswith('P'))
test[criterion].head()

# equivalent but slower
# test[[x.startswith('t') for x in df2['a']]]