Pandas学习笔记(二)—— Pandas索引

导入需要使用的库和文件:

>>> import numpy as np
>>> import pandas as pd
>>> df = pd.read_csv('data/table.csv',index_col='ID')
>>> df.head()
     School Class Gender   Address  Height  Weight  Math Physics
ID
1101    S_1   C_1      M  street_1     173      63  34.0      A+
1102    S_1   C_1      F  street_2     192      73  32.5      B+
1103    S_1   C_1      M  street_2     186      82  87.2      B+
1104    S_1   C_1      F  street_2     167      81  80.4      B-
1105    S_1   C_1      F  street_4     159      64  84.8      B+

一、单级索引

最常用的索引方法可能就是这三类,其中iloc表示位置索引,loc表示标签索引,[]也具有很大的便利性,各有特点。

1. loc方法

(a)单行索引:

>>> df.loc[1103]
School          S_1
Class           C_1
Gender            M
Address    street_2
Height          186
Weight           82
Math           87.2
Physics          B+
Name: 1103, dtype: object

(b)多行索引:

>>> df.loc[[1102, 2304]]
     School Class Gender   Address  Height  Weight  Math Physics
ID
1102    S_1   C_1      F  street_2     192      73  32.5      B+
2304    S_2   C_3      F  street_6     164      81  95.5      A-

注:所有在loc中使用的切片全部包含右端点!这是因为如果作为Pandas的使用者,那么肯定不太关心最后一个标签再往后一位是什么,但是如果是左闭右开,那么就很麻烦,先要知道再后面一列的名字是什么,非常不方便,因此Pandas中将loc设计为左右全闭

>>> df.loc[1304:2103].head
     School Class Gender   Address  Height  Weight  Math Physics
ID
1304    S_1   C_3      M  street_2     195      70  85.2       A
1305    S_1   C_3      F  street_5     187      69  61.7      B-
2101    S_2   C_1      M  street_7     174      84  83.3       C
2102    S_2   C_1      F  street_6     161      61  50.6      B+
2103    S_2   C_1      M  street_4     157      61  52.5      B->
>>> df.loc[2402::-1].head()
     School Class Gender   Address  Height  Weight  Math Physics
ID
2402    S_2   C_4      M  street_7     166      82  48.7       B
2401    S_2   C_4      F  street_2     192      62  45.3       A
2305    S_2   C_3      M  street_4     187      73  48.9       B
2304    S_2   C_3      F  street_6     164      81  95.5      A-
2303    S_2   C_3      F  street_7     190      99  65.9       C

(c)单列索引

>>> df.loc[:,'Height'].head()
ID
1101    173
1102    192
1103    186
1104    167
1105    159
Name: Height, dtype: int64

(d)多列索引

>>> df.loc[:,['Height', 'Math']].head()
      Height  Math
ID
1101     173  34.0
1102     192  32.5
1103     186  87.2
1104     167  80.4
1105     159  84.8
>>> df.loc[:,'Height':'Math'].head()
      Height  Weight  Math
ID
1101     173      63  34.0
1102     192      73  32.5
1103     186      82  87.2
1104     167      81  80.4
1105     159      64  84.8

(e)联合索引

>>> df.loc[1102:2401:3,'Height':'Math'].head()
      Height  Weight  Math
ID
1102     192      73  32.5
1105     159      64  84.8
1203     160      53  58.8
1301     161      68  31.5
1304     195      70  85.2

(f)函数式索引

>>> df.loc[lambda x:x['Gender']=='M'].head()
     School Class Gender   Address  Height  Weight  Math Physics
ID
1101    S_1   C_1      M  street_1     173      63  34.0      A+
1103    S_1   C_1      M  street_2     186      82  87.2      B+
1201    S_1   C_2      M  street_5     188      68  97.0      A-
1203    S_1   C_2      M  street_6     160      53  58.8      A+
1301    S_1   C_3      M  street_4     161      68  31.5      B+

>>> def f(x):
...     return [1101, 1105]
>>> df.loc[f]
     School Class Gender   Address  Height  Weight  Math Physics
ID
1101    S_1   C_1      M  street_1     173      63  34.0      A+
1105    S_1   C_1      F  street_4     159      64  84.8      B+

(g)布尔索引

>>> df.loc[df['Address'].isin(['street_7', 'street_4'])].head()
     School Class Gender   Address  Height  Weight  Math Physics
ID
1105    S_1   C_1      F  street_4     159      64  84.8      B+
1202    S_1   C_2      F  street_4     176      94  63.5      B-
1301    S_1   C_3      M  street_4     161      68  31.5      B+
1303    S_1   C_3      M  street_7     188      82  49.7       B
2101    S_2   C_1      M  street_7     174      84  83.3       C
>>> df.loc[[True if i[-1]=='4' or i[-1]=='7' else False for i in df['Address'].values]].head()
     School Class Gender   Address  Height  Weight  Math Physics
ID
1105    S_1   C_1      F  street_4     159      64  84.8      B+
1202    S_1   C_2      F  street_4     176      94  63.5      B-
1301    S_1   C_3      M  street_4     161      68  31.5      B+
1303    S_1   C_3      M  street_7     188      82  49.7       B
2101    S_2   C_1      M  street_7     174      84  83.3       C

小结:本质上说,loc中能传入的只有布尔列表和索引子集构成的列表,只要把握这个原则就很容易理解上面那些操作

2. iloc方法(注意与loc不同,切片右端点不包含)

(a)单行索引

>>> df.iloc[3]
School          S_1
Class           C_1
Gender            F
Address    street_2
Height          167
Weight           81
Math           80.4
Physics          B-
Name: 1104, dtype: object

(b)多行索引

>>> df.iloc[3:5]
     School Class Gender   Address  Height  Weight  Math Physics
ID
1104    S_1   C_1      F  street_2     167      81  80.4      B-
1105    S_1   C_1      F  street_4     159      64  84.8      B+

(c)多列索引

>>> df.iloc[:,3].head()
ID
1101    street_1
1102    street_2
1103    street_2
1104    street_2
1105    street_4
Name: Address, dtype: object

(d)多列索引

>>> df.iloc[:,7::-2].head()
     Physics  Weight   Address Class
ID
1101      A+      63  street_1   C_1
1102      B+      73  street_2   C_1
1103      B+      82  street_2   C_1
1104      B-      81  street_2   C_1
1105      B+      64  street_4   C_1

(e)混合索引

>>> df.iloc[3::4,7::-2].head()
     Physics  Weight   Address Class
ID
1104      B-      81  street_2   C_1
1203      A+      53  street_6   C_2
1302      A-      57  street_1   C_3
2101       C      84  street_7   C_1
2105       A      81  street_4   C_1

(f)函数式索引

>>> df.iloc[lambda x:[3]].head()
     School Class Gender   Address  Height  Weight  Math Physics
ID
1104    S_1   C_1      F  street_2     167      81  80.4      B-

小结:iloc中接收的参数只能为整数或整数列表或布尔列表,不能使用布尔Series,如果要用就必须如下把values拿出来


>>> df.iloc[(df['School']=='S_1').values].head()
     School Class Gender   Address  Height  Weight  Math Physics
ID
1101    S_1   C_1      M  street_1     173      63  34.0      A+
1102    S_1   C_1      F  street_2     192      73  32.5      B+
1103    S_1   C_1      M  street_2     186      82  87.2      B+
1104    S_1   C_1      F  street_2     167      81  80.4      B-
1105    S_1   C_1      F  street_4     159      64  84.8      B+

3. []操作符

3.1 Series的[]操作

(a)单元素索引:

>>> s = pd.Series(df['Math'], index=df.index)
>>> s[1101]
34.0

(b)多行索引

>>> s[0:4]
ID
1101    34.0
1102    32.5
1103    87.2
1104    80.4
Name: Math, dtype: float64

(c)函数式索引


>>> s[lambda x: x.index[16::-6]]
ID
2102    50.6
1301    31.5
1105    84.8
Name: Math, dtype: float64

(d)布尔索引

>>> s[s > 80]
ID
1103    87.2
1104    80.4
1105    84.8
1201    97.0
1302    87.7
1304    85.2
2101    83.3
2205    85.4
2304    95.5
Name: Math, dtype: float64

注:如果不想陷入困境,请不要在行索引为浮点值时使用[]操作符,因为在Series中[]的浮点切片并不是进行位置比较,而是值比较,非常特殊

>>> s_int = pd.Series([1, 2, 3, 4], index=[1, 3, 5, 6])
>>> s_float = pd.Series([1,2,3,4],index=[1.,3.,5.,6.])
>>> s_int
1    1
3    2
5    3
6    4
dtype: int64
>>> s_int[2:]
5    3
6    4
dtype: int64
>>> s_float
1.0    1
3.0    2
5.0    3
6.0    4
dtype: int64

>>> s_float[2:]
3.0    2
5.0    3
6.0    4
dtype: int64

3.2 DataFrame的[]操作

(a)单行索引:


>>> df[1:2]
     School Class Gender   Address  Height  Weight  Math Physics
ID
1102    S_1   C_1      F  street_2     192      73  32.5      B+
>>> row = df.index.get_loc(1102)
>>> df[row:row+1]
     School Class Gender   Address  Height  Weight  Math Physics
ID
1102    S_1   C_1      F  street_2     192      73  32.5      B+

(b)多行索引


>>> df[3:5]
     School Class Gender   Address  Height  Weight  Math Physics
ID
1104    S_1   C_1      F  street_2     167      81  80.4      B-
1105    S_1   C_1      F  street_4     159      64  84.8      B+

(c)单列索引

>>> df['School'].head()
ID
1101    S_1
1102    S_1
1103    S_1
1104    S_1
1105    S_1
Name: School, dtype: object

(d)多列索引

>>> df[['School', 'Math']].head()
     School  Math
ID
1101    S_1  34.0
1102    S_1  32.5
1103    S_1  87.2
1104    S_1  80.4
1105    S_1  84.8

(e)函数式索引

>>> df[lambda x: ['Math', 'Physics']].head()
      Math Physics
ID
1101  34.0      A+
1102  32.5      B+
1103  87.2      B+
1104  80.4      B-
1105  84.8      B+

(f)布尔索引

>>> df[df['Gender']=='F'].head()
     School Class Gender   Address  Height  Weight  Math Physics
ID
1102    S_1   C_1      F  street_2     192      73  32.5      B+
1104    S_1   C_1      F  street_2     167      81  80.4      B-
1105    S_1   C_1      F  street_4     159      64  84.8      B+
1202    S_1   C_2      F  street_4     176      94  63.5      B-
1204    S_1   C_2      F  street_5     162      63  33.8       B

小结:一般来说,[]操作符常用于列选择或布尔选择,尽量避免行的选择

(a)布尔符号:’&’,’|’,’~’:分别代表和and,或or,取反not

>>> df[(df['Gender']=='F')&(df['Address']=='street_2')].head()
     School Class Gender   Address  Height  Weight  Math Physics
ID
1102    S_1   C_1      F  street_2     192      73  32.5      B+
1104    S_1   C_1      F  street_2     167      81  80.4      B-
2401    S_2   C_4      F  street_2     192      62  45.3       A
2404    S_2   C_4      F  street_2     160      84  67.7       B
>>> df[df['Math']>85 | (df['Address']=='street_2')].head()
     School Class Gender   Address  Height  Weight  Math Physics
ID
1101    S_1   C_1      M  street_1     173      63  34.0      A+
1102    S_1   C_1      F  street_2     192      73  32.5      B+
1103    S_1   C_1      M  street_2     186      82  87.2      B+
1104    S_1   C_1      F  street_2     167      81  80.4      B-
1105    S_1   C_1      F  street_4     159      64  84.8      B+
>>> df[~((df['Math']>75)|(df['Address']=='street_1'))].head()
     School Class Gender   Address  Height  Weight  Math Physics
ID
1102    S_1   C_1      F  street_2     192      73  32.5      B+
1202    S_1   C_2      F  street_4     176      94  63.5      B-
1203    S_1   C_2      M  street_6     160      53  58.8      A+
1204    S_1   C_2      F  street_5     162      63  33.8       B
1205    S_1   C_2      F  street_6     167      63  68.4      B-

loc和[]中相应位置都能使用布尔列表选择:


>>> df.loc[df['Math']>60, df.columns=='Physics'].head()
     Physics
ID
1103      B+
1104      B-
1105      B+
1201      A-
1202      B-

(b)isin方法

>>> df[df['Address'].isin(['street_1', 'street_4']) & df['Physics'].isin(['A', 'A+'])]
     School Class Gender   Address  Height  Weight  Math Physics
ID
1101    S_1   C_1      M  street_1     173      63  34.0      A+
2105    S_2   C_1      M  street_4     170      81  34.2       A
2203    S_2   C_2      M  street_4     155      91  73.8      A+

>>> df[df[['Address','Physics']].isin({'Address':['street_1','street_4'],'Physics':['A','A+']}).all(1)]
     School Class Gender   Address  Height  Weight  Math Physics
ID
1101    S_1   C_1      M  street_1     173      63  34.0      A+
2105    S_2   C_1      M  street_4     170      81  34.2       A
2203    S_2   C_2      M  street_4     155      91  73.8      A+

当只需要取一个元素时,at和iat方法能够提供更快的实现:

>>> print(df.at)
df.at       df.at_time( df.attrs
S_1
df.loc     df.lookup(
>>> print(df.loc[1101, 'School'])
S_1
>>> print(df.iat[0, 0])
S_1
>>> print(df.iloc[0, 0])
S_1

此处介绍并不是说只能在单级索引中使用区间索引,只是作为一种特殊类型的索引方式,在此处先行介绍

(a)利用interval_range方法


>>> pd.interval_range(start=0, end=5)
IntervalIndex([(0, 1], (1, 2], (2, 3], (3, 4], (4, 5]], dtype='interval[int64, right]')

(b)利用cut将数值列转为区间为元素的分类变量,例如统计数学成绩的区间情况:


>>> math_interval = pd.cut(df['Math'], bins=[0, 40, 60, 80, 100])
>>> math_interval.head()
ID
1101      (0, 40]
1102      (0, 40]
1103    (80, 100]
1104    (80, 100]
1105    (80, 100]
Name: Math, dtype: category
Categories (4, interval[int64, right]): [(0, 40] < (40, 60] < (60, 80] < (80, 100]]

(c)区间索引的选取

>>> df_i = df.join(math_interval, rsuffix='_interval')[['Math', 'Math_interval']].reset_index().set_index('Math_interval')
>>> df_i.head()
                 ID  Math
Math_interval
(0, 40]        1101  34.0
(0, 40]        1102  32.5
(80, 100]      1103  87.2
(80, 100]      1104  80.4
(80, 100]      1105  84.8
>>> df_i.loc[65].head()
                 ID  Math
Math_interval
(60, 80]       1202  63.5
(60, 80]       1205  68.4
(60, 80]       1305  61.7
(60, 80]       2104  72.2
(60, 80]       2202  68.5
>>> df_i.loc[[65, 90]].head()
                 ID  Math
Math_interval
(60, 80]       1202  63.5
(60, 80]       1205  68.4
(60, 80]       1305  61.7
(60, 80]       2104  72.2
(60, 80]       2202  68.5

如果想要选取某个区间,先要把分类变量转化为区间变量,再使用 overlap方法


>>> df_i[df_i.index.astype('interval').overlaps(pd.Interval(70, 85))].head()
                 ID  Math
Math_interval
(80, 100]      1103  87.2
(80, 100]      1104  80.4
(80, 100]      1105  84.8
(80, 100]      1201  97.0
(60, 80]       1202  63.5

二、多级索引

1. 通过from_tuple或from_arrays

(a)直接创建元组

>>> tuples = [('A','a'),('A','b'),('B','a'),('B','b')]
>>> mul_index = pd.MultiIndex.from_tuples(tuples, names=('Upper', 'Lower'))
>>> mul_index
MultiIndex([('A', 'a'),
            ('A', 'b'),
            ('B', 'a'),
            ('B', 'b')],
           names=['Upper', 'Lower'])
>>> pd.DataFrame({'Score':['perfect','good','fair','bad']},index=mul_index)
               Score
Upper Lower
A     a      perfect
      b         good
B     a         fair
      b          bad

(b)利用zip创建元组

>>> L1 = list('AABB')
>>> L2 = list('abab')
>>> tuples = list(zip(L1,L2))
>>> mul_index = pd.MultiIndex.from_tuples(tuples, names=('Upper', 'Lower'))
>>> pd.DataFrame({'Score':['perfect','good','fair','bad']},index=mul_index)
               Score
Upper Lower
A     a      perfect
      b         good
B     a         fair
      b          bad

(c)通过Array创建

>>> arrays = [['A','a'],['A','b'],['B','a'],['B','b']]
>>> mul_index = pd.MultiIndex.from_tuples(arrays, names=('Upper', 'Lower'))
>>> pd.DataFrame({'Score':['perfect','good','fair','bad']},index=mul_index)
               Score
Upper Lower
A     a      perfect
      b         good
B     a         fair
      b          bad

>>> mul_index
MultiIndex([('A', 'a'),
            ('A', 'b'),
            ('B', 'a'),
            ('B', 'b')],
           names=['Upper', 'Lower'])

2.通过from_product

>>> L1 = ['A','B']
>>> L2 = ['a','b']
>>> pd.MultiIndex.from_product([L1,L2],names=('Upper', 'Lower'))
MultiIndex([('A', 'a'),
            ('A', 'b'),
            ('B', 'a'),
            ('B', 'b')],
           names=['Upper', 'Lower'])

3. 指定df中的列创建(set_index方法)

>>> df_using_mul = df.set_index(['Class','Address'])
>>> df_using_mul.head()
               School Gender  Height  Weight  Math Physics
Class Address
C_1   street_1    S_1      M     173      63  34.0      A+
      street_2    S_1      F     192      73  32.5      B+
      street_2    S_1      M     186      82  87.2      B+
      street_2    S_1      F     167      81  80.4      B-
      street_4    S_1      F     159      64  84.8      B+
>>> df_using_mul.head()
               School Gender  Height  Weight  Math Physics
Class Address
C_1   street_1    S_1      M     173      63  34.0      A+
      street_2    S_1      F     192      73  32.5      B+
      street_2    S_1      M     186      82  87.2      B+
      street_2    S_1      F     167      81  80.4      B-
      street_4    S_1      F     159      64  84.8      B+

1. 一般切片


>>> df_using_mul.sort_index().loc['C_2','street_5']
               School Gender  Height  Weight  Math Physics
Class Address
C_2   street_5    S_1      M     188      68  97.0      A-
      street_5    S_1      F     162      63  33.8       B
      street_5    S_2      M     193     100  39.1       B

>>> df_using_mul.sort_index().loc[('C_2','street_6'):('C_3','street_4')]
               School Gender  Height  Weight  Math Physics
Class Address
C_2   street_6    S_1      M     160      53  58.8      A+
      street_6    S_1      F     167      63  68.4      B-
      street_7    S_2      F     194      77  68.5      B+
      street_7    S_2      F     183      76  85.4       B
C_3   street_1    S_1      F     175      57  87.7      A-
      street_2    S_1      M     195      70  85.2       A
      street_4    S_1      M     161      68  31.5      B+
      street_4    S_2      F     157      78  72.3      B+
      street_4    S_2      M     187      73  48.9       B

>>> df_using_mul.sort_index().loc[('C_2','street_7'):'C_3'].head()
               School Gender  Height  Weight  Math Physics
Class Address
C_2   street_7    S_2      F     194      77  68.5      B+
      street_7    S_2      F     183      76  85.4       B
C_3   street_1    S_1      F     175      57  87.7      A-
      street_2    S_1      M     195      70  85.2       A
      street_4    S_1      M     161      68  31.5      B+

2. 第一类特殊情况:由元组构成列表


>>> df_using_mul.sort_index().loc[[('C_2','street_7'),('C_3','street_2')]]
               School Gender  Height  Weight  Math Physics
Class Address
C_2   street_7    S_2      F     194      77  68.5      B+
      street_7    S_2      F     183      76  85.4       B
C_3   street_2    S_1      M     195      70  85.2       A

3. 第二类特殊情况:由列表构成元组


>>> df_using_mul.sort_index().loc[(['C_2','C_3'],['street_4','street_7']),:]
               School Gender  Height  Weight  Math Physics
Class Address
C_2   street_4    S_1      F     176      94  63.5      B-
      street_4    S_2      M     155      91  73.8      A+
      street_7    S_2      F     194      77  68.5      B+
      street_7    S_2      F     183      76  85.4       B
C_3   street_4    S_1      M     161      68  31.5      B+
      street_4    S_2      F     157      78  72.3      B+
      street_4    S_2      M     187      73  48.9       B
      street_7    S_1      M     188      82  49.7       B
      street_7    S_2      F     190      99  65.9       C
>>> L1,L2 = ['A','B','C'],['a','b','c']
>>> mul_index1 = pd.MultiIndex.from_product([L1,L2],names=('Upper', 'Lower'))
>>> L3,L4 = ['D','E','F'],['d','e','f']
>>> mul_index2 = pd.MultiIndex.from_product([L3,L4],names=('Big', 'Small'))
>>> df_s = pd.DataFrame(np.random.rand(9,9),index=mul_index1,columns=mul_index2)
>>> df_s
Big                 D                             E                             F
Small               d         e         f         d         e         f         d         e         f
Upper Lower
A     a      0.865934  0.752678  0.992263  0.471948  0.101374  0.750520  0.029240  0.841838  0.202736
      b      0.358436  0.315506  0.141048  0.179118  0.579804  0.387298  0.731970  0.504881  0.664886
      c      0.688227  0.076362  0.447927  0.897414  0.990657  0.577089  0.885058  0.242146  0.551289
B     a      0.622251  0.583529  0.970421  0.798430  0.075585  0.453897  0.196744  0.243493  0.407374
      b      0.660225  0.383329  0.884619  0.646215  0.251076  0.753128  0.857983  0.240076  0.391556
      c      0.336650  0.051452  0.472089  0.750627  0.920971  0.131141  0.160800  0.567003  0.608006
C     a      0.875975  0.545787  0.449710  0.062922  0.931482  0.595037  0.124742  0.016393  0.221201
      b      0.608213  0.789482  0.744773  0.768816  0.364518  0.787751  0.536297  0.282383  0.828840
      c      0.972491  0.477164  0.000541  0.236157  0.951343  0.572702  0.270309  0.225364  0.027862

索引Slice的使用非常灵活:

>>> idx=pd.IndexSlice

>>> df_s.loc[idx['B':,df_s['D']['d']>0.3],idx[df_s.sum()>4]]
Big                 D                   E
Small               d         f         d         e         f
Upper Lower
B     a      0.622251  0.970421  0.798430  0.075585  0.453897
      b      0.660225  0.884619  0.646215  0.251076  0.753128
      c      0.336650  0.472089  0.750627  0.920971  0.131141
C     a      0.875975  0.449710  0.062922  0.931482  0.595037
      b      0.608213  0.744773  0.768816  0.364518  0.787751
      c      0.972491  0.000541  0.236157  0.951343  0.572702

1. swaplevel方法(两层交换)

>>> df_using_mul.head()
               School Gender  Height  Weight  Math Physics
Class Address
C_1   street_1    S_1      M     173      63  34.0      A+
      street_2    S_1      F     192      73  32.5      B+
      street_2    S_1      M     186      82  87.2      B+
      street_2    S_1      F     167      81  80.4      B-
      street_4    S_1      F     159      64  84.8      B+
>>> df_using_mul.swaplevel(i=1,j=0,axis=0).sort_index().head()
               School Gender  Height  Weight  Math Physics
Address  Class
street_1 C_1      S_1      M     173      63  34.0      A+
         C_2      S_2      M     175      74  47.2      B-
         C_3      S_1      F     175      57  87.7      A-
street_2 C_1      S_1      F     192      73  32.5      B+
         C_1      S_1      M     186      82  87.2      B+

2. reorder_levels方法(多层交换)

>>> df_muls = df.set_index(['School','Class','Address'])
>>> df_muls.head()
                      Gender  Height  Weight  Math Physics
School Class Address
S_1    C_1   street_1      M     173      63  34.0      A+
             street_2      F     192      73  32.5      B+
             street_2      M     186      82  87.2      B+
             street_2      F     167      81  80.4      B-
             street_4      F     159      64  84.8      B+
>>> df_muls.reorder_levels([2,0,1],axis=0).sort_index().head()
                      Gender  Height  Weight  Math Physics
Address  School Class
street_1 S_1    C_1        M     173      63  34.0      A+
                C_3        F     175      57  87.7      A-
         S_2    C_2        M     175      74  47.2      B-
street_2 S_1    C_1        F     192      73  32.5      B+
                C_1        M     186      82  87.2      B+

>>> df_muls.reorder_levels(['Address','School','Class'],axis=0).sort_index().head()
                      Gender  Height  Weight  Math Physics
Address  School Class
street_1 S_1    C_1        M     173      63  34.0      A+
                C_3        F     175      57  87.7      A-
         S_2    C_2        M     175      74  47.2      B-
street_2 S_1    C_1        F     192      73  32.5      B+
                C_1        M     186      82  87.2      B+

三、索引设定

index_col是read_csv中的一个参数,而不是某一个方法:

>>> pd.read_csv('data/table.csv',index_col=['Address','School']).head()
                Class    ID Gender  Height  Weight  Math Physics
Address  School
street_1 S_1      C_1  1101      M     173      63  34.0      A+
street_2 S_1      C_1  1102      F     192      73  32.5      B+
         S_1      C_1  1103      M     186      82  87.2      B+
         S_1      C_1  1104      F     167      81  80.4      B-
street_4 S_1      C_1  1105      F     159      64  84.8      B+

reindex是指重新索引,它的重要特性在于索引对齐,很多时候用于重新排序:

>>> df.head()
     School Class Gender   Address  Height  Weight  Math Physics
ID
1101    S_1   C_1      M  street_1     173      63  34.0      A+
1102    S_1   C_1      F  street_2     192      73  32.5      B+
1103    S_1   C_1      M  street_2     186      82  87.2      B+
1104    S_1   C_1      F  street_2     167      81  80.4      B-
1105    S_1   C_1      F  street_4     159      64  84.8      B+
>>> df.reindex(index=[1101,1203,1206,2402])
     School Class Gender   Address  Height  Weight  Math Physics
ID
1101    S_1   C_1      M  street_1   173.0    63.0  34.0      A+
1203    S_1   C_2      M  street_6   160.0    53.0  58.8      A+
1206    NaN   NaN    NaN       NaN     NaN     NaN   NaN     NaN
2402    S_2   C_4      M  street_7   166.0    82.0  48.7       B
>>> df.reindex(columns=['Height','Gender','Average']).head()
      Height Gender  Average
ID
1101     173      M      NaN
1102     192      F      NaN
1103     186      M      NaN
1104     167      F      NaN
1105     159      F      NaN

可以选择缺失值的填充方法:fill_value和method(bfill/ffill/nearest),其中method参数必须索引单调:


>>> df.reindex(index=[1101,1203,1206,2402],method='bfill')
     School Class Gender   Address  Height  Weight  Math Physics
ID
1101    S_1   C_1      M  street_1     173      63  34.0      A+
1203    S_1   C_2      M  street_6     160      53  58.8      A+
1206    S_1   C_3      M  street_4     161      68  31.5      B+
2402    S_2   C_4      M  street_7     166      82  48.7       B

>>> df.reindex(index=[1101,1203,1206,2402],method='nearest')
     School Class Gender   Address  Height  Weight  Math Physics
ID
1101    S_1   C_1      M  street_1     173      63  34.0      A+
1203    S_1   C_2      M  street_6     160      53  58.8      A+
1206    S_1   C_2      F  street_6     167      63  68.4      B-
2402    S_2   C_4      M  street_7     166      82  48.7       B

reindex_like的作用为生成一个横纵索引完全与参数列表一致的DataFrame,数据使用被调用的表

>>> df_temp = pd.DataFrame({'Weight':np.zeros(5),
...                         'Height':np.zeros(5),
...                         'ID':[1101,1104,1103,1106,1102]}).set_index('ID')
>>> df_temp.reindex_like(df[0:5][['Weight','Height']])
      Weight  Height
ID
1101     0.0     0.0
1102     0.0     0.0
1103     0.0     0.0
1104     0.0     0.0
1105     NaN     NaN

如果df_temp单调还可以使用method参数:

>>> df_temp = pd.DataFrame({'Weight':range(5),
...                         'Height':range(5),
...                         'ID':[1101,1104,1103,1106,1102]}).set_index('ID').sort_index()

>>> df_temp.reindex_like(df[0:5][['Weight','Height']],method='bfill')
      Weight  Height
ID
1101       0       0
1102       4       4
1103       2       2
1104       1       1
1105       3       3

先介绍set_index:从字面意思看,就是将某些列作为索引

使用表内列作为索引:

>>> df.head()
     School Class Gender   Address  Height  Weight  Math Physics
ID
1101    S_1   C_1      M  street_1     173      63  34.0      A+
1102    S_1   C_1      F  street_2     192      73  32.5      B+
1103    S_1   C_1      M  street_2     186      82  87.2      B+
1104    S_1   C_1      F  street_2     167      81  80.4      B-
1105    S_1   C_1      F  street_4     159      64  84.8      B+
>>> df.set_index('Class').head()
      School Gender   Address  Height  Weight  Math Physics
Class
C_1      S_1      M  street_1     173      63  34.0      A+
C_1      S_1      F  street_2     192      73  32.5      B+
C_1      S_1      M  street_2     186      82  87.2      B+
C_1      S_1      F  street_2     167      81  80.4      B-
C_1      S_1      F  street_4     159      64  84.8      B+

利用append参数可以将当前索引维持不变:

>>> df.set_index('Class', append=True).head()
           School Gender   Address  Height  Weight  Math Physics
ID   Class
1101 C_1      S_1      M  street_1     173      63  34.0      A+
1102 C_1      S_1      F  street_2     192      73  32.5      B+
1103 C_1      S_1      M  street_2     186      82  87.2      B+
1104 C_1      S_1      F  street_2     167      81  80.4      B-
1105 C_1      S_1      F  street_4     159      64  84.8      B+

当使用与表长相同的列作为索引时间(需要先转化为Series,否则报错):

>>> df.set_index(pd.Series(range(df.shape[0]))).head()
  School Class Gender   Address  Height  Weight  Math Physics
0    S_1   C_1      M  street_1     173      63  34.0      A+
1    S_1   C_1      F  street_2     192      73  32.5      B+
2    S_1   C_1      M  street_2     186      82  87.2      B+
3    S_1   C_1      F  street_2     167      81  80.4      B-
4    S_1   C_1      F  street_4     159      64  84.8      B+

可以直接添加多级索引:

>>> df.set_index([pd.Series(range(df.shape[0])),pd.Series(np.ones(df.shape[0]))]).head()
      School Class Gender   Address  Height  Weight  Math Physics
0 1.0    S_1   C_1      M  street_1     173      63  34.0      A+
1 1.0    S_1   C_1      F  street_2     192      73  32.5      B+
2 1.0    S_1   C_1      M  street_2     186      82  87.2      B+
3 1.0    S_1   C_1      F  street_2     167      81  80.4      B-
4 1.0    S_1   C_1      F  street_4     159      64  84.8      B+

下面介绍reset_index方法,它的主要功能是将索引重置

默认状态直接恢复到自然数索引:

>>> df.reset_index().head()
     ID School Class Gender   Address  Height  Weight  Math Physics
0  1101    S_1   C_1      M  street_1     173      63  34.0      A+
1  1102    S_1   C_1      F  street_2     192      73  32.5      B+
2  1103    S_1   C_1      M  street_2     186      82  87.2      B+
3  1104    S_1   C_1      F  street_2     167      81  80.4      B-
4  1105    S_1   C_1      F  street_4     159      64  84.8      B+

用level参数指定哪一层被reset,用col_level参数指定set到哪一层:

>>> L1,L2 = ['A','B','C'],['a','b','c']
>>> mul_index1 = pd.MultiIndex.from_product([L1,L2],names=('Upper', 'Lower'))
>>> L3,L4 = ['D','E','F'],['d','e','f']
>>> mul_index2 = pd.MultiIndex.from_product([L3,L4],names=('Big', 'Small'))
>>> df_temp = pd.DataFrame(np.random.rand(9,9),index=mul_index1,columns=mul_index2)
>>> df_temp.head()
Big                 D                             E                             F
Small               d         e         f         d         e         f         d         e         f
Upper Lower
A     a      0.996924  0.779796  0.198003  0.876215  0.801679  0.740366  0.072776  0.172737  0.103133
      b      0.856929  0.384369  0.988760  0.300426  0.109809  0.445339  0.735657  0.109474  0.632733
      c      0.631834  0.748637  0.378666  0.696078  0.404629  0.747714  0.237205  0.988239  0.260963
B     a      0.740106  0.995469  0.005640  0.204483  0.958359  0.737188  0.696751  0.900894  0.275091
      b      0.026315  0.251426  0.594558  0.313601  0.145479  0.433199  0.704520  0.366411  0.473218
>>> df_temp1 = df_temp.reset_index(level=1,col_level=1)
>>> df_temp1.head()
Big                 D                             E                             F
Small Lower         d         e         f         d         e         f         d         e         f
Upper
A         a  0.996924  0.779796  0.198003  0.876215  0.801679  0.740366  0.072776  0.172737  0.103133
A         b  0.856929  0.384369  0.988760  0.300426  0.109809  0.445339  0.735657  0.109474  0.632733
A         c  0.631834  0.748637  0.378666  0.696078  0.404629  0.747714  0.237205  0.988239  0.260963
B         a  0.740106  0.995469  0.005640  0.204483  0.958359  0.737188  0.696751  0.900894  0.275091
B         b  0.026315  0.251426  0.594558  0.313601  0.145479  0.433199  0.704520  0.366411  0.473218

>>> df_temp1.columns
MultiIndex([( '', 'Lower'),
            ('D',     'd'),
            ('D',     'e'),
            ('D',     'f'),
            ('E',     'd'),
            ('E',     'e'),
            ('E',     'f'),
            ('F',     'd'),
            ('F',     'e'),
            ('F',     'f')],
           names=['Big', 'Small'])

>>> df_temp1.index
Index(['A', 'A', 'A', 'B', 'B', 'B', 'C', 'C', 'C'], dtype='object', name='Upper')

rename_axis是针对多级索引的方法,作用是修改某一层的索引名,而不是索引标签

>>> df_temp.rename_axis(index={'Lower':'LowerLower'},columns={'Big':'BigBig'})[['D', 'E']]
BigBig                   D                             E
Small                    d         e         f         d         e         f
Upper LowerLower
A     a           0.996924  0.779796  0.198003  0.876215  0.801679  0.740366
      b           0.856929  0.384369  0.988760  0.300426  0.109809  0.445339
      c           0.631834  0.748637  0.378666  0.696078  0.404629  0.747714
B     a           0.740106  0.995469  0.005640  0.204483  0.958359  0.737188
      b           0.026315  0.251426  0.594558  0.313601  0.145479  0.433199
      c           0.642152  0.803119  0.869278  0.643841  0.933842  0.373142
C     a           0.419632  0.187484  0.420311  0.136625  0.512117  0.167024
      b           0.123571  0.571580  0.201483  0.788676  0.067141  0.955275
      c           0.075575  0.832965  0.934871  0.549695  0.511443  0.286503

rename方法用于修改列或者行索引标签,而不是索引名:

>>> df_temp.rename(index={'A':'T'},columns={'e':'changed_e'}).head()
Big                 D                             E                             F
Small               d changed_e         f         d changed_e         f         d changed_e         f
Upper Lower
T     a      0.996924  0.779796  0.198003  0.876215  0.801679  0.740366  0.072776  0.172737  0.103133
      b      0.856929  0.384369  0.988760  0.300426  0.109809  0.445339  0.735657  0.109474  0.632733
      c      0.631834  0.748637  0.378666  0.696078  0.404629  0.747714  0.237205  0.988239  0.260963
B     a      0.740106  0.995469  0.005640  0.204483  0.958359  0.737188  0.696751  0.900894  0.275091
      b      0.026315  0.251426  0.594558  0.313601  0.145479  0.433199  0.704520  0.366411  0.473218

四、常用索引型函数

当对条件为False的单元进行填充时:

>>> df.head()
     School Class Gender   Address  Height  Weight  Math Physics
ID
1101    S_1   C_1      M  street_1     173      63  34.0      A+
1102    S_1   C_1      F  street_2     192      73  32.5      B+
1103    S_1   C_1      M  street_2     186      82  87.2      B+
1104    S_1   C_1      F  street_2     167      81  80.4      B-
1105    S_1   C_1      F  street_4     159      64  84.8      B+

>>> df.where(df['Gender']=='M').head()
     School Class Gender   Address  Height  Weight  Math Physics
ID
1101    S_1   C_1      M  street_1   173.0    63.0  34.0      A+
1102    NaN   NaN    NaN       NaN     NaN     NaN   NaN     NaN
1103    S_1   C_1      M  street_2   186.0    82.0  87.2      B+
1104    NaN   NaN    NaN       NaN     NaN     NaN   NaN     NaN
1105    NaN   NaN    NaN       NaN     NaN     NaN   NaN     NaN

通过这种方法筛选结果和[]操作符的结果完全一致:

>>> df.where(df['Gender']=='M').dropna().head()
     School Class Gender   Address  Height  Weight  Math Physics
ID
1101    S_1   C_1      M  street_1   173.0    63.0  34.0      A+
1103    S_1   C_1      M  street_2   186.0    82.0  87.2      B+
1201    S_1   C_2      M  street_5   188.0    68.0  97.0      A-
1203    S_1   C_2      M  street_6   160.0    53.0  58.8      A+
1301    S_1   C_3      M  street_4   161.0    68.0  31.5      B+

第一个参数为布尔条件,第二个参数为填充值:

>>> df.where(df['Gender']=='M',np.random.rand(df.shape[0],df.shape[1])).head()
        School     Class    Gender   Address      Height     Weight       Math   Physics
ID
1101       S_1       C_1         M  street_1  173.000000  63.000000  34.000000        A+
1102  0.147943  0.670993   0.93367   0.16424    0.314864   0.121429   0.433781  0.074907
1103       S_1       C_1         M  street_2  186.000000  82.000000  87.200000        B+
1104  0.749106    0.9844  0.184485  0.674807    0.738321   0.525289   0.019779   0.19905
1105  0.534726  0.657987  0.370359   0.89066    0.613029   0.456765   0.389943  0.756956

mask函数与where功能上相反,其余完全一致,即对条件为True的单元进行填充:

>>> df.mask(df['Gender']=='M').dropna().head()
     School Class Gender   Address  Height  Weight  Math Physics
ID
1102    S_1   C_1      F  street_2   192.0    73.0  32.5      B+
1104    S_1   C_1      F  street_2   167.0    81.0  80.4      B-
1105    S_1   C_1      F  street_4   159.0    64.0  84.8      B+
1202    S_1   C_2      F  street_4   176.0    94.0  63.5      B-
1204    S_1   C_2      F  street_5   162.0    63.0  33.8       B
>>> df.mask(df['Gender']=='M',np.random.rand(df.shape[0],df.shape[1])).head()
        School     Class    Gender   Address      Height     Weight       Math   Physics
ID
1101  0.532138  0.841576  0.885163  0.169569    0.983056   0.714640   0.820599  0.012835
1102       S_1       C_1         F  street_2  192.000000  73.000000  32.500000        B+
1103  0.538961  0.155097  0.401648  0.283565    0.617196   0.260921   0.395324  0.478259
1104       S_1       C_1         F  street_2  167.000000  81.000000  80.400000        B-
1105       S_1       C_1         F  street_4  159.000000  64.000000  84.800000        B+
>>> df.head()
     School Class Gender   Address  Height  Weight  Math Physics
ID
1101    S_1   C_1      M  street_1     173      63  34.0      A+
1102    S_1   C_1      F  street_2     192      73  32.5      B+
1103    S_1   C_1      M  street_2     186      82  87.2      B+
1104    S_1   C_1      F  street_2     167      81  80.4      B-
1105    S_1   C_1      F  street_4     159      64  84.8      B+

query函数中的布尔表达式中,下面的符号都是合法的: 行列索引名、字符串、and/not/or/&/|/~/not in/in/==/!=、四则运算符

>>> df.query('(Address in ["street_6","street_7"])&(Weight>(70+10))&(ID in [1303,2304,2402])')
     School Class Gender   Address  Height  Weight  Math Physics
ID
1303    S_1   C_3      M  street_7     188      82  49.7       B
2304    S_2   C_3      F  street_6     164      81  95.5      A-
2402    S_2   C_4      M  street_7     166      82  48.7       B

五、重复元素处理

该方法返回了是否重复的布尔列表:

>>> df.duplicated('Class').head()
ID
1101    False
1102     True
1103     True
1104     True
1105     True
dtype: bool

可选参数keep默认为first,即首次出现设为不重复,若为last,则最后一次设为不重复,若为False,则所有重复项为True

>>> df.duplicated('Class',keep='last').tail()
ID
2401     True
2402     True
2403     True
2404     True
2405    False
dtype: bool
>>> df.duplicated('Class',keep=False).head()
ID
1101    True
1102    True
1103    True
1104    True
1105    True
dtype: bool

从名字上看出为剔除重复项,这在后面章节中的分组操作中可能是有用的,例如需要保留每组的第一个值:

>>> df.drop_duplicates('Class')
     School Class Gender   Address  Height  Weight  Math Physics
ID
1101    S_1   C_1      M  street_1     173      63  34.0      A+
1201    S_1   C_2      M  street_5     188      68  97.0      A-
1301    S_1   C_3      M  street_4     161      68  31.5      B+
2401    S_2   C_4      F  street_2     192      62  45.3       A

参数与duplicate函数类似:

>>> df.drop_duplicates('Class',keep='last')
     School Class Gender   Address  Height  Weight  Math Physics
ID
2105    S_2   C_1      M  street_4     170      81  34.2       A
2205    S_2   C_2      F  street_7     183      76  85.4       B
2305    S_2   C_3      M  street_4     187      73  48.9       B
2405    S_2   C_4      F  street_6     193      54  47.6       B

在传入多列时等价于将多列共同视作一个多级索引,比较重复项:

>>> df.drop_duplicates(['School','Class'])
     School Class Gender   Address  Height  Weight  Math Physics
ID
1101    S_1   C_1      M  street_1     173      63  34.0      A+
1201    S_1   C_2      M  street_5     188      68  97.0      A-
1301    S_1   C_3      M  street_4     161      68  31.5      B+
2101    S_2   C_1      M  street_7     174      84  83.3       C
2201    S_2   C_2      M  street_5     193     100  39.1       B
2301    S_2   C_3      F  street_4     157      78  72.3      B+
2401    S_2   C_4      F  street_2     192      62  45.3       A

六、抽样函数

这里的抽样函数指的就是sample函数

>>> df.sample(n=5)
     School Class Gender   Address  Height  Weight  Math Physics
ID
2205    S_2   C_2      F  street_7     183      76  85.4       B
2202    S_2   C_2      F  street_7     194      77  68.5      B+
2101    S_2   C_1      M  street_7     174      84  83.3       C
1103    S_1   C_1      M  street_2     186      82  87.2      B+
1301    S_1   C_3      M  street_4     161      68  31.5      B+
>>> df.sample(frac=0.05)
     School Class Gender   Address  Height  Weight  Math Physics
ID
1104    S_1   C_1      F  street_2     167      81  80.4      B-
2105    S_2   C_1      M  street_4     170      81  34.2       A
>>> df.sample(n=df.shape[0],replace=True).head()
     School Class Gender   Address  Height  Weight  Math Physics
ID
2302    S_2   C_3      M  street_5     171      88  32.7       A
1101    S_1   C_1      M  street_1     173      63  34.0      A+
2305    S_2   C_3      M  street_4     187      73  48.9       B
2101    S_2   C_1      M  street_7     174      84  83.3       C
1304    S_1   C_3      M  street_2     195      70  85.2       A
>>> df.sample(n=35,replace=True).index.is_unique
False
>>> df.sample(n=3,axis=1).head()
       Address Class  Weight
ID
1101  street_1   C_1      63
1102  street_2   C_1      73
1103  street_2   C_1      82
1104  street_2   C_1      81
1105  street_4   C_1      64
>>> df.sample(n=3,weights=np.random.rand(df.shape[0])).head()
     School Class Gender   Address  Height  Weight  Math Physics
ID
1101    S_1   C_1      M  street_1     173      63  34.0      A+
2302    S_2   C_3      M  street_5     171      88  32.7       A
1105    S_1   C_1      F  street_4     159      64  84.8      B+

>>> df.sample(n=3,weights=df['Math']).head()
     School Class Gender   Address  Height  Weight  Math Physics
ID
2304    S_2   C_3      F  street_6     164      81  95.5      A-
1201    S_1   C_2      M  street_5     188      68  97.0      A-
2205    S_2   C_2      F  street_7     183      76  85.4       B

Original: https://blog.csdn.net/qq_43300880/article/details/124985663
Author: HiSpring流云
Title: Pandas学习笔记(二)—— Pandas索引

原创文章受到原创版权保护。转载请注明出处:https://www.johngo689.com/676069/

转载文章受原作者版权保护。转载请注明原作者出处!

(0)

大家都在看

亲爱的 Coder【最近整理,可免费获取】👉 最新必读书单  | 👏 面试题下载  | 🌎 免费的AI知识星球