机器学习训练营——机器学习爱好者的自由交流空间（入群联系qq：2279055353）
特征工程
特征工程，是根据数据集已有的特征，删除、加工产生新特征，从而改善模型的预测效果。著名的计算机科学家、人工智能与机器学习领域的领军人物吴恩达(Andrew Ng)有一句名言：“applied machine learning is basically feature engineering.” 具体上说，特征工程包括：
特征构造：从数据集产生新特征。
特征选择：仅选择最重要的特征或降维方法。
下面开始我们的特征工程！
Polynomial Features
在这个方法里，我们产生存在的特征的幂次或交互项作为新特征。例如，
EXT_SOURCE12EXT\_SOURCE_1^2EXT_SOURCE12, EXT_SOURCE1×EXT_SOURCE2EXT\_SOURCE_1\times EXT\_SOURCE_2EXT_SOURCE1×EXT_SOURCE2.
现在，我们使用EXT_SOURCEandDAYS_BIRTH产生新的多项特征。为了避免过度拟合，我们选择的阶数不超过3.
# Make a new dataframe for polynomial featurespoly_features = app_train[['EXT_SOURCE_1', 'EXT_SOURCE_2', 'EXT_SOURCE_3', 'DAYS_BIRTH', 'TARGET']]poly_features_test = app_test[['EXT_SOURCE_1', 'EXT_SOURCE_2', 'EXT_SOURCE_3', 'DAYS_BIRTH']]# imputer for handling missing valuesfrom sklearn.preprocessing import Imputerimputer = Imputer(strategy = 'median')poly_target = poly_features['TARGET']poly_features = poly_features.drop(columns = ['TARGET'])# Need to impute missing valuespoly_features = imputer.fit_transform(poly_features)poly_features_test = imputer.transform(poly_features_test)from sklearn.preprocessing import PolynomialFeatures# Create the polynomial object with specified degreepoly_transformer = PolynomialFeatures(degree = 3)
# Train the polynomial featurespoly_transformer.fit(poly_features)# Transform the featurespoly_features = poly_transformer.transform(poly_features)poly_features_test = poly_transformer.transform(poly_features_test)print('Polynomial Features shape: ', poly_features.shape)
Polynomial Features shape: (307511, 35)
产生了35个多项特征，为了取得名字我们必须使用多项特征方法get_feature_names.
print(poly_transformer.get_feature_names(input_features = ['EXT_SOURCE_1', 'EXT_SOURCE_2', 'EXT_SOURCE_3', 'DAYS_BIRTH']))
我们接着看看这些新特征是否与target有相关性。
# Create a dataframe of the features poly_features = pd.DataFrame(poly_features, columns = poly_transformer.get_feature_names(['EXT_SOURCE_1', 'EXT_SOURCE_2', 'EXT_SOURCE_3', 'DAYS_BIRTH']))# Add in the targetpoly_features['TARGET'] = poly_target# Find the correlations with the targetpoly_corrs = poly_features.corr()['TARGET'].sort_values()# Display most negative and most positiveprint(poly_corrs.head(10))print(poly_corrs.tail(5))
有几个新特征的相关性比原始特征大，当建立机器学习模型时，我们尝试包括或不包括这些特征，看看它们是否对模型学习有帮助。
# Put test features into dataframepoly_features_test = pd.DataFrame(poly_features_test, columns = poly_transformer.get_feature_names(['EXT_SOURCE_1', 'EXT_SOURCE_2', 'EXT_SOURCE_3', 'DAYS_BIRTH']))# Merge polynomial features into training dataframepoly_features['SK_ID_CURR'] = app_train['SK_ID_CURR']app_train_poly = app_train.merge(poly_features, on = 'SK_ID_CURR', how = 'left')# Merge polnomial features into testing dataframepoly_features_test['SK_ID_CURR'] = app_test['SK_ID_CURR']app_test_poly = app_test.merge(poly_features_test, on = 'SK_ID_CURR', how = 'left')# Align the dataframesapp_train_poly, app_test_poly = app_train_poly.align(app_test_poly, join = 'inner', axis = 1)# Print out the new shapesprint('Training data with polynomial features shape: ', app_train_poly.shape)print('Testing data with polynomial features shape: ', app_test_poly.shape)
Training data with polynomial features shape: (307511, 275)
Testing data with polynomial features shape: (48744, 275)
Domain Knowledge Features
除了从模型角度出发产生的多项特征，我们还可以从问题出发加工特征。在这里，我们根据金融知识，想使用这5个特征：
CREDIT_INCOME_PERCENT: 相对于客户收入的贷款比例。
ANNUITY_INCOME_PERCENT: 相对于客户收入的年贷款比例。
CREDIT_TERM: 月付款长度。
DAYS_EMPLOYED_PERCENT: 相对于客户年龄的工作日比例。
app_train_domain = app_train.copy()app_test_domain = app_test.copy()app_train_domain['CREDIT_INCOME_PERCENT'] = app_train_domain['AMT_CREDIT'] / app_train_domain['AMT_INCOME_TOTAL']app_train_domain['ANNUITY_INCOME_PERCENT'] = app_train_domain['AMT_ANNUITY'] / app_train_domain['AMT_INCOME_TOTAL']app_train_domain['CREDIT_TERM'] = app_train_domain['AMT_ANNUITY'] / app_train_domain['AMT_CREDIT']app_train_domain['DAYS_EMPLOYED_PERCENT'] = app_train_domain['DAYS_EMPLOYED'] / app_train_domain['DAYS_BIRTH']app_test_domain['CREDIT_INCOME_PERCENT'] = app_test_domain['AMT_CREDIT'] / app_test_domain['AMT_INCOME_TOTAL']app_test_domain['ANNUITY_INCOME_PERCENT'] = app_test_domain['AMT_ANNUITY'] / app_test_domain['AMT_INCOME_TOTAL']app_test_domain['CREDIT_TERM'] = app_test_domain['AMT_ANNUITY'] / app_test_domain['AMT_CREDIT']app_test_domain['DAYS_EMPLOYED_PERCENT'] = app_test_domain['DAYS_EMPLOYED'] / app_test_domain['DAYS_BIRTH']
可视化新特征
plt.figure(figsize = (12, 20))# iterate through the new featuresfor i, feature in enumerate(['CREDIT_INCOME_PERCENT', 'ANNUITY_INCOME_PERCENT', 'CREDIT_TERM', 'DAYS_EMPLOYED_PERCENT']):# create a new subplot for each sourceplt.subplot(4, 1, i + 1)# plot repaid loanssns.kdeplot(app_train_domain.loc[app_train_domain['TARGET'] == 0, feature], label = 'target == 0')# plot loans that were not repaidsns.kdeplot(app_train_domain.loc[app_train_domain['TARGET'] == 1, feature], label = 'target == 1')# Label the plotsplt.title('Distribution of %s by Target Value' % feature)plt.xlabel('%s' % feature); plt.ylabel('Density');plt.tight_layout(h_pad = 2.5)
从图上很难判断哪些特征有用，只能在建模时尝试加入它们。
未完待续
