1.介绍几个DataFrame的小功能
检查数据类型 df.dtypes
df=DataFrame({'name':['zs,','ls','ww','zl'],'score':[19,29,39,20],'date':['-10-10','-10-11','-10-12','-10-13']})df.dtypesname objectscore int64date objectdtype: object
将'date'这行数据转换为时间数据类型
df.date=pd.to_datetime(df.date)df.dtypesname objectscore int64date datetime64[ns]dtype: object
将'date'设置为行索引
df.set_index('date',inplace=True)dfdate name score -10-10 zs 19-10-11 ls 29-10-12 ww 39-10-13 zl 20
可视化
import matplotlib.pyplot as plot%matplotlib inlinedf['score'].plot()
2.pandas数据处理
(1)删除重复元素
df.drop_duplicates(subset=None, keep='first', inplace=False)df = DataFrame({'color':['white','white','red','red','white'],'value':[2,1,3,3,2]})display(df,df.duplicated(),df.drop_duplicates())输出:colorvalue0white21white12red33red34white20 False1 False2 False3 True4 Truedtype: boolcolorvalue0white21white12red3
(2)映射
映射的含义:创建一个映射关系列表,把values元素和一个特定的标签或者字符串绑定
需要使用字典:
map = {'label1':'value1','label2':'value2', ... }
包含三种操作:
a. replace()函数:替换元素
df = DataFrame({'item':['ball','mug','pen'],'color':['white','rosso','verde'], 'price':[5.56,4.20,1.30]})newcolors = {'rosso':'red','verde':'green','pen':'pi'}df.replace(newcolors)itemcolorprice0ballwhite5.561mugred4.202pigreen1.30
b. 最重要:map()函数:新建一列
!!!map中返回的数据是一个具体值,不能迭代
使用map()函数,由已有的列生成一个新列,适合处理某一单独的列df3 = DataFrame({'color':['red','green','blue'],'project':['math','english','chemistry']})price = {'red':5.56,'green':3.14,'chemistry':2.79}df3['price'] = df3['color'].map(price)display(df3)输出:colorprojectprice0redmath5.561greenenglish3.142bluechemistryNaN
c. rename()函数:替换索引
df4 = DataFrame({'color':['white','gray','purple','blue','green'],'value':np.random.randint(10,size = 5)})new_index = {0:'first',1:'two',2:'three',3:'four',4:'five'}display(df4,df4.rename(new_index))输出:colorvalue0white21gray02purple93blue24green0colorvaluefirstwhite2twogray0threepurple9fourblue2fivegreen0
(3)异常值检查和过滤
使用describe()函数查看每一列的描述性统计量
np.random.seed(0)df = DataFrame(np.random.randint(10,size = 10))display(df.head(10),df.describe())输出:005102333475963758294 0count10.000000mean4.100000std2.558211min0.00000025%3.00000050%3.50000075%5.000000max9.000000
使用std()函数可以求得DataFrame对象每一列的标准差
df2=DataFrame(np.random.randint(10,100,size=(8,8)))df2.std()0 26.7177711 20.3118542 20.2608003 26.0463194 28.6602645 36.4025126 18.0158667 22.646349dtype: float64
根据每一列的标准差,对DataFrame元素进行过滤。借助any()函数,对每一列应用筛选条件
display(df.std(),np.abs(df)>(3*df.std()),df[(np.abs(df)>df.std()*3).any(axis = 1)])0 2.558211dtype: float64 00False1False2False3False4False5True6False7False8False9False059DataFrame.any功能说明DataFrame.any(self,axis=0,bool_only=None, skipna=True, level=None,**kwargs,)Return whether any element is True, potentially over an axis.
(4)排序
使用.take()函数排序,可以借助np.random.permutation()函数随机排序
df5 = DataFrame(np.arange(25).reshape(5,5))new_order = np.random.permutation(5)display(df5,new_order,df5.take(new_order))输出:0123400123415678921011121314315161718194222324array([4, 2, 3, 1, 0]) 0 1 2 344222324210111213143151617181915 6 7 8900 1 2 34
随机抽样
当DataFrame规模足够大时,直接使用np.random.randint()函数,就配合take()函数实现随机抽样
sample = np.random.randint(0,len(df5),size = 3)df.take(sample)输出:01234001234210111213144222324
(5)数据聚合
df = DataFrame({'color':['white','red','green','red'],'item':['ball','mug','pen','pencil'], 'price1':np.random.rand(4),'price2':np.random.rand(4)})g = df.groupby('color')['price1']display(df,g,g.groups,type(g))display(g.sum(),g.mean(),g.max())输出:coloritemprice1price20whiteball0.6527900.4143691redmug0.6350590.4746982greenpen0.9953000.6235103redpencil0.5818500.338008{'green': Int64Index([2], dtype='int64'), 'red': Int64Index([1, 3], dtype='int64'), 'white': Int64Index([0], dtype='int64')}pandas.core.groupby.SeriesGroupBycolorgreen 0.995300red 1.216909white 0.652790Name: price1, dtype: float64colorgreen 0.995300red 0.608455white 0.652790Name: price1, dtype: float64colorgreen 0.995300red 0.635059white 0.652790Name: price1, dtype: float64
(6)高级数据聚合
可以使用pd.merge()函数包聚合操作的计算结果添加到df的每一行
d1={'item':['luobo','baicai','lajiao','donggua','luobo','baicai','lajiao','donggua'], 'color':['white','white','red','green','white','white','red','green'], 'weight':np.random.randint(10,size = 8), 'price':np.random.randint(10,size = 8)}df = DataFrame(d1)sums = df.groupby('color').sum().add_prefix('total_')items = df.groupby('item')['price','weight'].sum()means = items['price']/items['weight']means = DataFrame(means,columns=['means_price'])df2 = pd.merge(df,sums,left_on = 'color',right_index = True)df3 = pd.merge(df2,means,left_on = 'item',right_index = True)display(df2,df3)输出:coloritempriceweight0whiteluobo921whitebaicai592redlajiao583greendonggua114whiteluobo745whitebaicai806redlajiao687greendonggua43total_pricetotal_weightcolorgreen54red1116white2915pandas.core.frame.DataFramepandas.core.frame.DataFrameOut[141]:coloritempriceweighttotal_pricetotal_weight0whiteluobo9229151whitebaicai5929154whiteluobo7429155whitebaicai8029152redlajiao5811166redlajiao6811163greendonggua11547greendonggua4354
可以使用transform和apply实现相同功能
transform
d1={'item':['luobo','baicai','lajiao','donggua','luobo','baicai','lajiao','donggua'], 'color':['white','white','red','green','white','white','red','green'], 'weight':np.random.randint(10,size = 8), 'price':np.random.randint(10,size = 8)}df = DataFrame(d1)sum1 = df.groupby('color')['price','weight'].sum().add_prefix("total_")sums2 = df.groupby('color')['price','weight'].transform(lambda x:x.sum()).add_prefix('total_')sums3 = df.groupby('color')['price','weight'].transform(sum).add_prefix('total_')display(sum,df,sum1,sums2,sums3)输出:coloritempriceweight0whiteluobo771whitebaicai772redlajiao273greendonggua664whiteluobo125whitebaicai366red lajiao707greendonggua02total_pricetotal_weightcolorgreen68red97white1822total_pricetotal_weight01822118222973684182251822697768total_pricetotal_weight01822118222973684182251822697768
apply
def sum_price(x): return x.sum()sums3 = df.groupby('color')['price','weight'].apply(lambda x:x.sum()).add_prefix('total_')sums4 = df.groupby('color')['price','weight'].apply(sum_price).add_prefix('total_')display(df,sums3,sums4)输出:coloritempriceweight0whiteluobo441whitebaicai032redlajiao043greendonggua754whiteluobo315whitebaicai336redlajiao067greendonggua07colortotal_pricetotal_weightgreen712red010white1011colortotals_pricetotals_weightgreen712red010white1011