目录
- 1.缺失值处理
* - 1.1删除缺失值dropna
- 1.2填充/替换缺失数据 – fillna、replace
- 1.3缺失值插补(mean,median,mode,ffill,lagrange)
- 2.异常值处理
* - 2.1 3σ原则
- 2.2箱型图分析
- 3.数据归一化和标准化
* - 3.1 0-1标准化
- 3.2 Z-score标准化
- 4.数据连续属性离散化(cut,qcut)
* - 4.1等宽法(cut)
- 4.2等频法(qcut)
- 5.查看数据(info,describle,enumerate,iloc,loc)
- 6.数据冗余(duplicated,drop_duplicates)
- 7.表与表的连接(merge,concat,append)
- 8.改变数据类型(dtype,astype)
- 9.数据分组聚合(groupby)
- 10.数据抽取与拆分
1.缺失值处理
- 数据缺失主要包括记录缺失和字段信息缺失等情况，其对数据分析会有较大影响，导致结果不确定性更加显著
- 缺失值的处理：删除记录 / 数据插补 / 不处理
import warnings
warnings.filterwarnings('ignore')
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from scipy import stats
s = pd.Series([12,33,45,23,np.nan,np.nan,66,54,np.nan,99])
df = pd.DataFrame({'value1':[12,33,45,23,np.nan,np.nan,66,54,np.nan,99,190],
'value2':['a','b','c','d','e',np.nan,np.nan,'f','g',np.nan,'g']})
print(s.isnull())
print(df.isnull().sum())
1.1删除缺失值dropna
s.dropna(inplace=True)
df1=df[['value1','value2']].dropna()
print(df1.isnull().sum())
1.2填充/替换缺失数据 – fillna、replace
s.fillna(0,inplace=True)
df['value1'].fillna(method='pad',inplace=True)
df['value2'].replace([1,2,3],np.nan,inplace=True)
1.3缺失值插补(mean,median,mode,ffill,lagrange)
u = s.mean()
me = s.median()
mod = s.mode()
print('均值为：%.2f, 中位数为：%.2f' % (u,me))
print('众数为：', mod.tolist())
print('------')
s.fillna(u,inplace = True)
print(s)
s.fillna(me,inplace = True)
print(s)
s.fillna(mod,inplace = True)
print(s)
s.fillna(method='ffill',inplace=True)
from scipy.interpolate import lagrange
data = pd.Series(np.random.rand(100)*100)
data[3,6,33,56,45,66,67,80,90] = np.nan
print(data.head())
print('总数据量:%i' % len(data))
print('------')
data_na = data[data.isnull()]
print('缺失值数据量:%i' % len(data_na))
print('缺失数据占比:%.2f%%' % (len(data_na) / len(data) * 100))
data_c = data.fillna(data.median())
fig,axes = plt.subplots(1,4,figsize = (20,5))
data.plot.box(ax = axes[0],grid = True,title = '数据分布')
data.plot(kind = 'kde',style = '--r',ax = axes[1],grid = True,title = '删除缺失值',xlim = [-50,150])
data_c.plot(kind = 'kde',style = '--b',ax = axes[2],grid = True,title = '缺失值填充中位数',xlim = [-50,150])
def na_c(s,n,k=5):
y = s[list(range(n-k,n+1+k))]
y = y[y.notnull()]
return(lagrange(y.index,list(y))(n))
na_re = []
for i in range(len(data)):
if data.isnull()[i]:
data[i] = na_c(data,i)
print(na_c(data,i))
na_re.append(data[i])
data.dropna(inplace=True)
data.plot(kind = 'kde',style = '--k',ax = axes[3],grid = True,title = '拉格朗日插值后',xlim = [-50,150])
print('finished!')
2.异常值处理
- 异常值是指样本中的个别值，其数值明显偏离其余的观测值
- 异常值也称离群点，异常值的分析也称为离群点的分析
- 异常值分析 → 3σ原则 / 箱型图分析
- 异常值处理方法 → 删除 / 修正填补
2.1 3σ原则
import statsmodels as stats
data = pd.Series(np.random.randn(10000)*100)
u = data.mean()
std = data.std()
print('均值为：%.3f，标准差为：%.3f' % (u,std))
fig=plt.figure(figsize=(10,6))
ax1=fig.add_subplot(2,1,1)
data.plot(kind = 'kde',grid = True,style = '-k',title = '密度曲线')
ax2=fig.add_subplot(2,1,2)
error=data[np.abs(data-u)>3*std]
data_c=data[np.abs(data-u)3*std]
print("异常值共%d条"%(len(error)))
plt.scatter(data_c.index,data_c,color = 'k',marker='.',alpha = 0.3)
plt.scatter(error.index,error,color = 'r',marker='.',alpha = 0.5)
plt.xlim([-10,10010])
plt.grid()
均值为：0.840，标准差为：99.366
异常值共27条
2.2箱型图分析
fig = plt.figure(figsize = (10,6))
ax1 = fig.add_subplot(2,1,1)
color = dict(boxes='DarkGreen', whiskers='DarkOrange', medians='DarkBlue', caps='Gray')
data.plot.box(vert=False, grid = True,color = color,ax = ax1,label = '样本数据')
s = data.describe()
print(s)
print('------')
q1 = s['25%']
q3 = s['75%']
iqr = q3 - q1
mi = q1 - 1.5*iqr
ma = q3 + 1.5*iqr
print('分位差为：%.3f，下限为：%.3f，上限为：%.3f' % (iqr,mi,ma))
print('------')
ax2 = fig.add_subplot(2,1,2)
error = data[(data < mi) | (data > ma)]
data_c = data[(data >= mi) & (data ma)]
print('异常值共%i条' % len(error))
plt.scatter(data_c.index,data_c,color = 'k',marker='.',alpha = 0.3)
plt.scatter(error.index,error,color = 'r',marker='.',alpha = 0.5)
plt.xlim([-10,10010])
plt.grid()
3.数据归一化和标准化
数据的标准化（normalization）是将数据按比例缩放，使之落入一个小的特定区间。
在某些比较和评价的指标处理中经常会用到，去除数据的单位限制，将其转化为无量纲的纯数值，便于不同单位或量级的指标能够进行比较和加权.
最典型的就是数据的归一化处理，即将数据统一映射到[0,1]区间上
3.1 0-1标准化
df = pd.DataFrame({"value1":np.random.rand(10)*20,
'value2':np.random.rand(10)*100})
def data_norm(df,*cols):
df_n = df.copy()
for col in cols:
ma = df_n[col].max()
mi = df_n[col].min()
df_n[col + '_n'] = (df_n[col] - mi) / (ma - mi)
return(df_n)
df_n = data_norm(df,'value1','value2')
print(df_n.head())
3.2 Z-score标准化
from sklearn import preprocessing
df = pd.DataFrame({"value1":np.random.rand(10) * 100,
'value2':np.random.rand(10) * 100})
def data_Znorm(df, *cols):
df_n = df.copy()
for col in cols:
u = df_n[col].mean()
std = df_n[col].std()
df_n[col + '_Zn'] = (df_n[col] - u) / std
return(df_n)
dd=preprocessing.scale(df)
print(dd)
df_z = data_Znorm(df,'value1','value2')
u_z = df_z['value1_Zn'].mean()
std_z = df_z['value1_Zn'].std()
print(df_z)
print('标准化后value1的均值为:%.2f, 标准差为：%.2f' % (u_z, std_z))
4.数据连续属性离散化(cut,qcut)
连续属性变换成分类属性，即连续属性离散化
在数值的取值范围内设定若干个离散划分点，将取值范围划分为一些离散化的区间，最后用不同的符号或整数值代表每个子区间中的数据值。
4.1等宽法(cut)
ages=[20,22,25,27,21,23,37,31,61,45,41,32]
df=pd.DataFrame({'ages':ages})
bins = [18,25,35,60,100]
group_names=['Youth','YoungAdult','MiddleAged','Senior']
cats=pd.cut(ages,bins=bins,labels=group_names)
cut_counts = s.value_counts(sort=False)
print(cats)
plt.scatter(df.index,df.values)
4.2等频法(qcut)
data = np.random.randn(1000)
s = pd.Series(data)
cats=pd.qcut(s,4)
print(pd.value_counts(cats))
plt.scatter(s.index,s,cmap = 'Greens',c = pd.qcut(data,4).codes)
plt.xlim([0,1000])
plt.grid()
5.查看数据(info,describle,enumerate,iloc,loc)
import pandas as pd
import numpy as np
test_dict = {'id':[1,2,3,4,5,6],'name':['Alice','Bob','Cindy','Eric','Helen','Grace '],'math':[90,89,99,78,97,93],'english':[89,94,80,94,94,90]}
df = pd.DataFrame(test_dict)
df.info()
df.describe()
for i, v in enumerate(df.columns):
print(i, v)
df_means = df.loc[:,'id':'math']
df_means.head(3)
6.数据冗余(duplicated,drop_duplicates)
import pandas as pd
import numpy as np
test_dict = {'id':[1,2,3,4,5,6,6],'name':['Alice','Bob','Cindy','Eric','Helen','Grace','Grace'],'math':[90,89,99,78,97,93,93],'english':[89,94,80,94,94,90,90]}
df = pd.DataFrame(test_dict)
print(df.duplicated())
print(df.drop_duplicates(inplace=True))
7.表与表的连接(merge,concat,append)
import pandas as pd
import numpy as np
test_dict1 = {'id':[1,2,3,4,5,6],'name':['Alice','Bob','Cindy','Eric','Helen','Grace '],'math':[88,89,99,78,97,93],'english':[89,94,80,94,94,90]}
df1 = pd.DataFrame(test_dict1)
test_dict2 = {'id':[1,2,3,4,5,6],'name':['Alice','Bob','Cindy','Eric','Helen','Grace '],'sex':['female','male','female','female','female','female']}
df2 = pd.DataFrame(test_dict2)
merge函数,默认情况下，会按照相同字段的进行连接,其他参数一般用不到，主要只能两两拼接
df1.merge(df2)
concat()函数
pd.concat(objs,
axis=0,
join='outer',
join_axes=None,
ignore_index=False,
keys=None,
levels=None,
names=None,
verify_integrity=False,
copy=True
)
pd.concat([df1,df2],axis=1)
pd.concat([df1,df2],axis=0)
append函数将被 append 的对象添加到调用者的末尾（类似 list 的方法）
DataFrame.append(other,
ignore_index=False,
verify_integrity=False,
sort=None
)
df1.append(df2)
8.改变数据类型(dtype,astype)
def downcast_dtypes(df):
cols_float = [c for c in df if df[c].dtype == 'float66']
cols_object = [c for c in df if df[c].dtype == 'object']
cols_int64_32 = [c for c in df if df[c].dtype in ['int64', 'int32']]
df[cols_float] = df[cols_object].astype(np.float32)
df[cols_object] = df[cols_object].astype(np.float32)
df[cols_int64_32] = df[cols_int64_32].astype(np.int16)
return df
9.数据分组聚合(groupby)
data.groupby(by='列名').mean()
聚合函数：将一组数据进行计算返回一个值agg()是进行聚合操作
data.groupby(by='月份')['最高温度'].max()
agg_dict={'最高温度':['max','mean'],'最低温度':'min'}
data.groupby(by='月份').agg(agg_dict)
def top(month):
return month.sort_values(by='最高温度')[-2:]
df.groupby(by='月份',sort = False).apply(top)
10.数据抽取与拆分
df[df.comments>10000]
between(left,right)
df[df.comments.between(1000,10000)]
pandas.isnull(column)
df[pandas.isnull(df.title)
str.contains(patten,na=False)
如:df[df.title.str.contains("台电",na=False)]
如:df[(df.comments>=1000)&(df.comments10000)]
等价于df[df.comments.between(1000,10000)]
Original: https://blog.csdn.net/m0_49263811/article/details/121750232
Author: CHRN晨
Title: 【数据分析系列】Python数据预处理总结篇
原创文章受到原创版权保护。转载请注明出处：https://www.johngo689.com/680090/
转载文章受原作者版权保护。转载请注明原作者出处！