前面有文章数据分析之探索性数据分析，里面详细阐述了何为EDA，以及一些常用的分析方法，感兴趣的小伙伴们可以参考下。
本文分析数据科学家更换工作情况数据集，运用常见EDA方法分析每个特征情况及他们与目标变量之间的关系。使用seaborn进行数据可视化辅助分析数据科学家们更换工作都有哪些特征。
因篇幅过长，将分为上下两篇展开此次探索性数据分析案例精讲。
导包
importnumpyasnpimportpandasaspdimportmatplotlib.pyplotaspltimportseabornassnsplt.rcParams['font.sans-serif']=['SimHei']plt.rcParams['axes.unicode_minus']=False
数据导入与查看
path_train='../input/hr-analytics-job-change-of-data-scientists/aug_train.csv'path_test='../input/hr-analytics-job-change-of-data-scientists/aug_test.csv'path_submission='../input/hr-analytics-job-change-of-data-scientists/sample_submission.csv'
train=pd.read_csv(path_train)test=pd.read_csv(path_test)
train.head()
test.sample(5)#表示返回随机5行数据。
探索性数据分析
一般性分析
train.shape
(19158, 14)
test.shape
(2129, 13)
train.dtypes
enrollee_id int64city objectcity_development_index float64gender objectrelevent_experience objectenrolled_university objecteducation_level objectmajor_discipline objectexperience objectcompany_sizeobjectcompany_typeobjectlast_new_jobobjecttraining_hours int64targetfloat64dtype: object
test.info()
<class 'pandas.core.frame.DataFrame'>RangeIndex: 2129 entries, 0 to 2128Data columns (total 13 columns):# Column Non-Null Count Dtype --- ------ -------------- ----- 0 enrollee_id 2129 non-null int64 1 city2129 non-null object 2 city_development_index 2129 non-null float643 gender 1621 non-null object 4 relevent_experience2129 non-null object 5 enrolled_university2098 non-null object 6 education_level 2077 non-null object 7 major_discipline 1817 non-null object 8 experience 2124 non-null object 9 company_size 1507 non-null object 10 company_type 1495 non-null object 11 last_new_job 2089 non-null object 12 training_hours2129 non-null int64 dtypes: float64(1), int64(2), object(10)memory usage: 216.4+ KB
list(train.columns)
['enrollee_id','city','city_development_index','gender','relevent_experience','enrolled_university','education_level','major_discipline','experience','company_size','company_type','last_new_job','training_hours','target']
describe方法用于生成DataFramem描述统计信息。可以很方便的查看数据集的分布情况。注意，这里的统计分布包含NaN值。
count表示计数。
mean表示平均值。
std表示标准差。
min表示最小值。
25%表示四分之一分位数
50%表示二分之一分位数。
75%表示四分之三分位数。
max表示最大值。
train.describe(include='all').T
缺失值分析
defpercentage_nulls(df):number_nulls=pd.DataFrame(df.isnull().sum(),columns=['Total'])number_nulls['%nulls']=round((number_nulls['Total']/df.shape[0])*100,1)returnnumber_nulls
训练集的缺失值。
nulls_train=percentage_nulls(train)nulls_train
#选择缺失数量最大的5行数据。nulls_train.nlargest(5,'Total')
percentage_nulls(test)
缺失值的可视化
importmissingnoasmsno
msno.matrix(train)
测试集的缺失值。
msno.matrix(test)
为了后面分析方便，将缺失值用字符串'Unknown'填充。
df_train=train.fillna("Unknown")#填充完后检查下是否处理完毕。df_train.isnull().sum()
enrollee_id0city 0city_development_index 0gender0relevent_experience 0enrolled_university 0education_level 0major_discipline0experience0company_size 0company_type 0last_new_job 0training_hours 0target0dtype: int64缺失值分析可以参见缺失值处理，你真的会了吗？
分析每个特征
特征 'City' -- 城市
#切片索引出'City'city_train=df_train['city']city_train.value_counts()
city_103 4355city_212702city_161533city_114 1336city_160845... city_129 3city_111 3city_121 3city_140 1city_171 1Name: city, Length: 123, dtype: int64
city_train.value_counts().plot()
city_test=df_test['city']city_test.value_counts()
city_103 473city_21318city_16168city_114 155city_160 113... city_84 1city_1711city_25 1city_93 1city_1411Name: city, Length: 108, dtype: int64
观察到，城市是用数字编码的，训练集用123个不同的城市，测试集用108个不同的城市。
'city_ development _index' 城市发展指数
可以查看维基百科中的名词解释/wiki/City_development_index
#查看特征分布状况sns.displot(data=df_train,x='city_development_index',height=6,aspect=2,color='lightblue')sns.set_context(font_scale=1)
withsns.axes_style():g=sns.displot(data=df_test,x='city_development_index',height=5,aspect=2,color='coral')g.set_ylabels(fontsize=15)g.set_xlabels(fontsize=15)g.set_xticklabels(fontsize=15)g.set_yticklabels(fontsize=15)
注意这种写法，设置x,y轴标签字体大小。
'gender'性别
性别及后面几个特征为分类型变量，因此分别统计每个变量的总数及占比。
gender_train=df_train['gender']gender_test=df_test['gender']
后面需要多次用到缺失值统计，因此将其定义为函数。
defpercent_nan(df):number=pd.DataFrame(df.value_counts())number.columns=['Total']number['%']=round((number['Total']/df.notnull().sum())*100,1)returnnumber
percent_nan(gender_train)
percent_nan(gender_test)
两个数据集的缺失值差不多，因此对两者可视化，以更加方便看出其差异。
同样，为后续分析方便，将其定义为函数。
defdraw_countplot(feature,palette,order=None):fig,ax=plt.subplots(nrows=1,ncols=2,figsize=(10,5))sns.countplot(x=feature,data=df_train,palette=palette,ax=ax[0],order=order).set_title('Train')sns.countplot(x=feature,data=df_test,palette=palette,ax=ax[1],order=order).set_title('Test')fig.tight_layout()
draw_countplot("gender","Set1")
'relevent_experience' 相关经验
relevent_experience_train=df_train['relevent_experience']relevent_experience_test=df_test['relevent_experience']
percent_nan(relevent_experience_train)
percent_nan(relevent_experience_test)
draw_countplot("relevent_experience","Set2")
'enrolled_university' 专业课
已注册的大学课程类型(如有)。
enrolled_university_train=df_train['enrolled_university']enrolled_university_test=df_test['enrolled_university']
percent_nan(enrolled_university_train)
percent_nan(enrolled_university_test)
order_enrolled_university=percent_nan(enrolled_university_train).indexorder_enrolled_university
Index(['no_enrollment', 'Full time course', 'Part time course', 'Unknown'], dtype='object')
draw_countplot('enrolled_university',"Set3",order_enrolled_university)
'education_level' 学历
education_level_train=df_train['education_level']education_level_test=df_test['education_level']
percent_nan(education_level_train)
percent_nan(education_level_test)
order_education_level=percent_nan(education_level_train).index
draw_countplot('education_level',"Set1",order_education_level)
'major_discipline' 专业学科
major_discipline_train=df_train['major_discipline']major_discipline_test=df_test['major_discipline']
percent_nan(major_discipline_train)
percent_nan(major_discipline_test)
order_major_discipline=percent_nan(major_discipline_train).index
draw_countplot('major_discipline',"Set2",order_major_discipline)
'experience' 工作年限
experience_train=df_train['experience']experience_test=df_test['experience']
percent_nan(experience_train)
percent_nan(experience_test)
order_experience=percent_nan(experience_train).index
draw_countplot('experience',"Set3",order_experience)
'company_size'公司规模
company_size_train=df_train['company_size']company_size_test=df_test['company_size']
percent_nan(company_size_train)
percent_nan(company_size_test)
order_company_size=percent_nan(company_size_train).index
draw_countplot('company_size',"Set1",order_company_size)
'company_type'公司类型
company_type_train=df_train['company_type']company_type_test=df_test['company_type']
percent_nan(company_type_train)
percent_nan(company_type_test)
order_company_type=percent_nan(company_type_train).index
draw_countplot('company_type',"Set2",order_company_type)
'lastnewjob' 以前的工作与现在的工作的年差
last_new_job_train=df_train['last_new_job']last_new_job_test=df_test['last_new_job']
percent_nan(last_new_job_train)
percent_nan(last_new_job_test)
order_last_new_job=percent_nan(last_new_job_train).index
draw_countplot('last_new_job',"Set3",order_last_new_job)
'training_hours'已完成的培训时长
sns.displot(data=df_train,x='training_hours',height=5,aspect=1.5,color='lightblue')
sns.displot(data=df_test,x='training_hours',height=5,aspect=1.5,color='coral')
'Target'
0 - Not looking for job change
1 - Looking for a job change
target=df_train['target']percent_nan(target)
sns.countplot(x='target',data=df_train,palette="Set1").set_title('Train')
本文数据使用 kaggle数据集，关注公众号，并回复
【 hr 】获取数据。
推荐阅读
Python数据分析之探索性数据分析(EDA)
缺失值处理，你真的会了吗？
-- 数据STUDIO --
