MH8111 Project:
Titanic: Machine Learning from Disaster
Submitted by:
Pua Ming Xiu G1902258C
Li FengZhi G1901809H
Leong Yew Wing G1902405D
Jiang Lei G1902335G
| Names Work Distribution |
| Pua Ming Xiu § Data Exploration, Feature Engineering, missing data imputation report and slides § Logistic Regression, K Nearest Neighbour for report and slides § Report compilation and review of report |
| Li FengZhi • Feature Engineering, Missing Data Imputation, Feature Importance • Naive Bayes Classification, Neural Network report and slides • R Code aggregation and compilation |
| Leong Yew Wing • Feature Engineering ,Missing Data Imputation, Feature Importance • RandomForest, Neural Network report and slides • Slides compilation |
| Jiang Lei • Feature Engineering, Missing Data Imputation, Feature Importance • Feature Dropping and Selection, Decision Tree Modelling report and slides • Slides compilation |
Table of Contents
1. PROBLEM STATEMENT ……………………………………………………………………………………………………………. 3
2. DATA DESCRIPTION …………………………………………………………………………………………………………………. 3
3.1 DATA EXPLORATION AND FEATURE ENGINEERING…………………………………………………………. 4
3.1.1 HOW DOES FAMILY SIZE AFFECT SURVIVAL? ……………………………………………………………………………… 4
3.1.2A HOW DOES AGE AFFECTS SURVIVAL?………………………………………………………………………………………. 4
3.1.2B HOW DOES ONE’S TITLE RELATE TO AGE?…………………………………………………………………………………. 5
3.1.3 HOW DOES PASSENGER CLASS AND SOCIAL CLASS AFFECT SURVIVAL? …………………………………………. 7
3.1.4A HOW DOES PASSENGER FARE AFFECTS SURVIVAL?……………………………………………………………………. 8
3.1.5B HOW DOES THE ASSIGNMENT OF CABIN AFFECTS SURVIVAL? …………………………………………………….. 9
3.1.6 HOW DOES LOCATION OF EMBARKATION AFFECTS SURVIVAL? …………………………………………………… 10
3.2 MISSING DATA IMPUTATION ………………………………………………………………………………………………. 11
3.2.1 IMPUTATION OF MISSING EMBARKED VALUES…………………………………………………………………………. 11
3.2.2 IMPUTATION OF MISSING FARE ………………………………………………………………………………………………. 11
3.2.3B DISCRETIZATION OF AGE……………………………………………………………………………………………………… 12
3.3.1 RECAP AND INITIAL FEATURE DROPPING…………………………………………………………………………………. 12
3.3.2 FEATURE IMPORTANCE………………………………………………………………………………………………………….. 12
3.3.3 CHECK FOR CORRELATIONS …………………………………………………………………………………………………… 13
3.4.1 LOGISTIC EGRESSION …………………………………………………………………………………………………………….. 14
3.4.2 RANDOM FOREST………………………………………………………………………………………………………………….. 14
3.4.3 NAÏVE BAYES……………………………………………………………………………………………………………………….. 15
3.4.4 NEURAL NETWORK……………………………………………………………………………………………………………….. 15
3.4.6 K NEAREST NEIGHBOUR (KNN)…………………………………………………………………………………………….. 17
4 MODEL COMPARISON……………………………………………………………………………………………………………… 18
5 CONCLUSION ……………………………………………………………………………………………………………………………. 18
6.FUTURE DIRECTIONS FOR IMPROVEMENT…………………………………………………………………………. 19
1. Problem Statement
Background
The Titanic sank after its collision with an iceberg on April 15, 1912. The tragedy led to the deaths of
1,502 out of 2,224 passengers and crew, calling for better safety regulations for ships. The large
number of casualties was partly caused by the insufficient number of lifeboats for the passengers and
crew. Besides luck, it seemed that certain groups of people such as women, children, and the upperclass were more likely to survive than others.
Goal
This report will explore and feature engineer the data to select features that have higher predictive
power of the class label “Survived”. The selected features will be used to predict the survival of the
passengers in the Test set, therefore 0 (Died) and 1 (Survived) via machine learning methods using R.
The performance of the various machine learning models will be analysed and compared. Finally, we
will look at the actual performance of our models via our Kaggle submission scores.
2. Data Description
There are 2 datasets namely “training set (train.csv)” and “test set (test.csv)”. The training set and test
set each contains 891 observations (12 variables) and 418 observations (11 variables) respectively.
The training set contains one more variable Survived which depicts the binary predictions, therefore,
1 (Survived) and 0 (Died). Description of the variables are as follows:
We note that there are various missing data in the Train and Test datasets. In particular, Age and
Cabin has large number of missing data. In Section 3.2, we will be imputing these missing data
3.1 Data Exploration and Feature Engineering
In this section, we will be exploring the data to form interesting insights and to create relevant
features that will enhance the predictive power of our models. To do so, we will first combine the
training and test sets to have a clearer picture of the passengers for the purpose of data exploration
and cleansing.
3.1.1 How does family size affect survival?
We will create a new family size variable (FamilySize), which is the sum of one’s siblings & spouses
(SibSp), parents & children (Parch) and the passenger himself/herself. From the chart, we can see
that singles and bigger family with a size of 4 or more have a lower chance of survival as compared
to smaller families with a size of 2 to 4. As the number of family members in one’s family affects the
survival rate, we will discretize our FamilySize variable to 3 categories namely, “single”, “small” and
“large”.
3.1.2a How does age affects survival?
Age may seem to be a strong predictor of survivability. From the chart, it is apparent that older
women passengers and younger child passengers (in particular male) had a higher chance of survival.
Survival rate is also lower for adult passengers especially those passengers in their late twenties to
thirties as compared to other age ranges. It is therefore meaningful to perform discretization to create
a new AgeGroup variable. As there is a significant number of missing Age values, we will impute
these missing Age values first before performing discretization of Age in section 3.2.3.
3.1.2b How does one’s title relate to age?
As there are no missing data in Name, Name may be a fairly robust predictor for Age. The structure
of Names is “surname, title. given name (maiden name)”. We can see that some of the titles are
underrepresented while some are abundant. As we note that Titles with small number of occurrences
has limited predictive power, we will feature engineer the titles to ensure the predictive power of our
imputation model for the missing Age data.
Interestingly, maiden name presented in parentheses indicates that the passenger is a married woman.
As the 2 instances of “Ms” do not have maiden name in parentheses, we will assign it together with
“Miss”. We have also assigned the French designations Madamoiselle (“Mlle.”) and Madame
(“Mme.”) to their corresponding English designations of “Miss” and “Mrs” respectively.
We will categorize the rest of the Title that had few occurrences namely “Capt”, “Col”, “Don”,
“Dona”, “Dr”,”Jonkheer”, “Lady”, “Major”,”Rev”,”Sir”, “the Countess” into RareTitle as these titles
indicate an adult age due to honorifics and professional terms used. By plotting the Age by Title
using boxplot, we can see that passengers with titles ‘Master’ and ‘Miss’ are of a younger age. On the
other hand, passengers with ‘RareTitle’ are of a higher age. It therefore shows that Title is a fairly
robust predictor for Age. We can also see that all the younger male passengers who survived are with
the ‘Master’ title and all the women passengers with ‘RareTitle’ all survived.
3.1.3 How does passenger class and social class affect survival?
Generally, passengers from Class 1 have a higher chance of survival as compared to the other classes,
for both men and women. Among those who are from passenger class 1, passengers with high social
status may have a higher rate of survival. We will discretize the Names variable to depict the
different social class namely Commoner, Royal and Rank.
We can see that passengers from the Royal social status have higher rate of survival than other social
classes. In particular, women from the social class of Rank and Royal in passenger class 1 all
survived. On the other hand, men of social class Rank has a lower survival rate as they are likely to
be the heroes who helped others to escape.
3.1.4a How does passenger fare affects survival?
We can see that passengers who paid a lower fare have a lower chance of survival. It is interesting to
note that all the male passengers who paid 0 fare in the train dataset all died. All of them embarked
from Southampton and were all singles! These men were mostly likely the crew who sacrificed
themselves and helped the other passengers to escape. We therefore created a PayNoFare variable
which we will consider including or dropping during feature selection.
3.1.4b How does family size relates to passenger fare?
From the boxplots, we can see that bigger family has higher median fare. It is likely that passenger
fare is the total amount paid for the entire family. We also note that the fare of singles is more
extreme as can be seen from the outliers. In this regard, we will create a new feature
“PerPassengerFare”.
3.1.5a How does cabin affects survival?
The Cabin feature could be a strong predictor for survivability. Passengers who are located in cabins
that are nearer to exit and/or access to lifeboats will have higher chance of survival. However, there
are many missing Cabin values. The first letter of the Cabin feature denotes the deck. As seen from
the chart below, the relationship between the Deck and survival is not clear. However, it seems that
survival rate is relatively high for passengers where Cabin value is not missing. The missing Cabin
values may not be missing data but an indication that these passengers were not assigned a Cabin!
3.1.5b How does the assignment of cabin affects survival?
From the chart below, we can see that passengers who were being assigned a cabin have a higher
survival rate than those who were not assigned a cabin, for all passenger classes. Being assigned a
cabin beforehand may denote one’s social status and wealth. We replaced NA with Unassigned and
those that are not NA as Assigned. As such, there is no need to fill in the missing Cabin values. We
will include our imputed Cabin as one of the independent variables to build our models as it seems to
have a relatively high predictive power of Survived.
3.1.6 How does location of embarkation affects survival?
Passengers embarked Titanic from 3 different location namely Cherbourg (=C), Queenstown (=Q)
and Southampton (=S). Near 7 in 10 passengers boarded the Titanic from Southanpton. As compared
to S and Q, passengers who embarked from C has higher survival rate. Furthermore, a passenger who
embarked from C has higher survival rate if he was from passenger class 1 as compared to passengers
who were from other passenger classes whom he/she had embarked the ship with.
3.2 Missing Data Imputation
3.2.1 Imputation of Missing Embarked Values
We note that passenger with ID 62 and 830 have missing Embarked values. We will impute these
missing values by S as it is where most of the passengers embarked as seen from section 3.1.6.
3.2.2 Imputation of Missing Fare
The missing fare is from a Pclass 3 passenger who boarded the Titanic from S. We will look at the
fares of the other passengers who were also from Pclass 3 whom had boarded the Titanic from S.
From the boxplot, we can see that there are a few passengers who paid higher prices (outliers), we
will impute the missing fare with median instead of mean fare of the Pclass 3 passengers who
boarded the Titanic from S.
3.2.3a Predictive Imputation of Missing Age Values
In section 3.1.2b, we saw that Title may be a robust predictor of the missing Age values. We will
perform the MICE imputation using the mice package which will exclude the less than useful
variables to impute the missing Age. As the Age distribution of the MICE output is in line with the
original data, we will replace the original Age with our MICE output.
3.2.3b Discretization of Age
In section 3.1.2a, we note that survival rate differs for different age range. Now that we have all the
imputed missing Age values, we will perform discretization to create a new AgeGroup variable,
taking note to ensure none of the groups is overrepresented. We will categorize the Age variable into
a new AgeGroup variable consisting 5 categories as follows:
– YoungChildren: less than 12 years old
– ChildrenAndYoungAdult: more than or equal to 12 and less than 21 year old
– Adults: more than or equal to 21 and less than 30 year old
– MiddleAgedAdult: more than or equal to 30 and less than 50 year old
– SeniorAdult: more than or equal to 50 years old
The frequency of each age group is as shown below. If we do not split the adults into different
category, the adult class will be overrepresented and may result in imbalance issue.
3.3 Feature Selections and Importance
3.3.1 Recap and Initial Feature Dropping
We have created 6 new features namely FamilySize; PerPassengerFare; Title; SocialClass;
AgeGroup and PayNoFare in Section 3.2 on Data Exploration and Feature Engineering. To avoid
redundancy, variables that were used to create these new features namely SibSp, Parch; Fare, Name
and Age were dropped. Two variables, Passenger ID and Ticket which are of low relevance were also
dropped. We have also backed up our Passenger ID for our final prediction before dropping it from
our model. This left us with 10 remaining variables.
3.3.2 Feature Importance
We will now look at the variable importance to determine the predictive power of the variables in
their prediction of the class label “Survived”. In particular, we will use the Boruta and RandomForest
Algorithm to determine the variable importance.
The results from both Boruta and RandomForest algorithm are consistent in showing the top 5
important variables (in no particular order, Title; Sex; Pclass; PerPassengerFare; FamilySize) and
bottom 2 important variables (AgeGroup and PayNoFare). We will drop PayNoFare as it has the
least variable importance and has the lowest decrease in mean Gini if it were kept. Although
AgeGroup is the second least important variable, it will lead to relatively high mean Gini decrease if
it were to be dropped. Hence, AgeGroup was kept.
Boruta Variable Importance
RandomForest Variable Importance
3.3.3 Check for Correlations
With PayNoFare dropped, we are left with 9 variables. Features that are highly correlated would lead
to overfitting of the machine learning model. While the accuracy on the training dataset might be
high, accuracy of the test dataset might be lowered due to overfitting. To avoid overfitting our model
as far as possible, we will look at the correlation of the remaining 9 variables. From the Pearson
Correlation result visualization below, 4 pairs of features with correlation that are more than 0.5 were
observed (i.e. FamilySize & SocialClass; Pclass & Cabin; PerPassengerFare & Pclass; Title &
SocialClass).
As SocialClass has high correlation with FamilySize and Title which are the top 3 important features,
we will drop SocialClass which has relatively low importance. On the other hand, we will keep both
Cabin and Pclass as they have relatively high importance and has relatively high mean decrease in
gini if it were kept. Hence, we dropped SocialClass and were left with 8 remaining variables namely
Pclass; Sex; Cabin; Embarked; FamilySize; PerpassegerFare; Title and AgeGroup
3.4 Building Machine Learning Models
In this section we will train different machine learning models based on the selected features in
section 3.3. We will then evaluate the performance of the models and predict the Survived variable in
Kaggle’s test data.
3.4.1 Logistic egression
Logistic regression is a predictive modeling technique for binary categorical variable. To run the
model, the classes of the selected independent variables were changed to numeric, as required by
logistic regression. As the independent variable (Survived) is missing in the test set, evaluation of
model performance using the test set is not possible. In view of this, the train set was randomly split
into a train (train_train) and test (train_test) subsets using the “caTools” package with a split ratio of
0.7. The logistic regression model was trained using the train_train data based on all the independent
variables to predict “Survived” in the train_test data. The model performed relatively well with an
accuracy of 56.9% in which 169 out of 297 train_test subset was predicted correctly. A 5 times
repeated 5-fold Cross Validation (CV) was performed using the “caret” package and the prediction
accuracy improved to 82.5%. As the CV trained model has a higher performance, it was used to
predict the Survived variable in Kaggle’s test data. The Kaggle submission result was 75.1%.
3.4.2 Random Forest
Random Forest is a ensemble learning method for classification and regression. A large number of
individual decision trees will be constructed at training. Each decision tree will come out with a
prediction, i.e. mode of the classes for classification or mean prediction (regression). The tree with
the most votes will become the model prediction. The R package “randomForest” is used to train the
randomForest model using Train data that consists of the selected variables with ntree=2000.
Variables Title, Sex and PerPassengerFare are the top 3 important features that have high predictive
power of the class label “Survived”. Given 16.72% out-of-bag error, our random forest model has an
accuracy of 83.28% in which 742 out of 891 of the Train set was predicted correctly. The Kaggle
submission result was 79.9%.
Confusion matrix:
0 1 class. error
0 497 52 0.09471767
1 97 245 0.28362573
OOB estimate of error rate: 16.72%
3.4.3 Naïve Bayes
Naïve Bayes classification model is simple to build, runs fast and is effective when dealing with
categorical features. The main drawback is that it assumes feature independence and feature equal
importance, which, in most of the cases, are not true. In our Titanic disaster survival prediction
project, most of our engineered and selected features happen to be categorical, this section covers our
survival prediction with Naïve Bayes classification model.
The 891 training records were split into 2/3 for training and 1/3 for validation. After the split,
proportion of the class feature is verified to ensure a balance split.
| prop.table Results | ||
| Survived = 0 | Survived = 1 | |
| Training | 0.6076795 | 0. 3923205 |
| Validation | 0.6335616 | 0.3664384 |
For the training setup, we applied Laplace smoothing to the model. The following table presents our
prediction results’ confusion matrix with Naïve Bayes classification with a total of 292 records in the
validation set.
Total Observations in Table: 292
| Naïve Bayes Confusion Matrix | |||
| Actual | |||
| Prediction | 0 | 1 | Row Total |
| 0 | 185 0.634 |
0 0.000 |
185 |
| 1 | 0 0.000 |
107 0.366 |
107 |
| Column Total | 185 | 107 | 292 |
We achieved an excellent prediction accuracy of 100% with the validation data. Despite adding or
removing single feature from our model, we did not observe any reduction in prediction. With the
trained Naïve Bayes classifier, we predicted the class label “Survived” using Kaggle’s Test dataset
and received a score of 75.6%.
3.4.4 Neural Network
As one of the Black Box methods, Neural Network borrows the concept from an understanding of
human brains to model arbitrary functions. Recently, it’s become really popular in the Artificial
Intelligence / Machine Learning fields thanks to the massive adoption of parallel computing with
GPGPU.
Different from some other prediction models, Neural Network requires all features to be numeric,
and unlike the Naïve Bayes classification we discussed above, it makes few assumptions about the
relationships among features. Potential drawback is that it’s a Black Box model, which in a way
limits its usage in many industries that has regulatory requirements.
We have adopted the same data splitting method as the Naïve Bayes classification model in Section
3.4.3. When building the model, we have tried the following adjustments to the different parameters,
(i) with no explicit hidden layer and smoothing settings; and (ii) with different explicit number of
hidden layer and StepMax settings. The optimal configuration with the highest accuracy is a hidden
layer of 5 with logistic smoothing.
We achieved a prediction accuracy of 83.2% with the Validation dataset. We then predicted the class
label “Survived” using Kaggle’s Test dataset and received a score of 77.5%.
Total Observations in Table: 292
| Neural Network Confusion Matrix | |||
| Actual | |||
| Prediction | 0 | 1 | Row Total |
| 0 | 169 0.579 |
33 0.113 |
202 |
| 1 | 16 0.055 |
74 0.253 |
107 |
| Column Total | 185 | 107 | 292 |
3.4.5 Decision Tree
As the independent variable (Survived) is missing in the test set, evaluation of model performance
using the test set is not possible. In view of this, the train set was split into a new train set (train_train,
599 observations) and new test set(train_test, 291 observations). The classifier model was trained
using the train_train data and validated against the train_test set. The best decision tree that was built
as shown below, has only 3 features namely Title, Pclass and FamilySize. The decision tree model
has an accuracy of 83.14%.
Confusion matrix:
0 1
0 322 59
1 42 176
Accuracy = 0.8314
To reduce the possibility of overfitting by the model, we will perform cross validation using ‘10 fold
technique’. A 10 times repeated 10-fold Cross Validation (CV) was performed and the best model
with the highest accuracy of 82.4% has a cp= 0.0325752. We note that there is a decrease in the
accuracy when cross validation is performed, from 83.1% to 82.4%. Hence, there might be some
overfitting in the previous model. The CV model was used predict the class label “Survived” using
Kaggle’s Test dataset and received a score of 78.9%.
3.4.6 K Nearest Neighbor (KNN)
KNN is a non-parametric method used for classification and regression. Important parameters are the
k(number of neighbor observations that contribute to the output predictions) and distance metric. As
the output depends on the distance calculation for KNN, normalization was performed on all the
selected independent variables before we run our model. We also ensured that the ordinal variables
AgeGroup and FamilySize were ordered before normalization. As the independent variable
(Survived) is missing in the test set, evaluation of model performance using the test set is not
possible. In view of this, the train set was further randomly split into a train (train_train) using the
“caTools” package with a split ratio of 0.7. The KNN model was trained using the train_train data
based on all the selected variables using k=24 (square root of the number of 594 observations in the
train_train set) to predict “Survived” in the train_test data. The model performed relatively well with
an accuracy of 60.3% in which 179 out of 297 train_test subset was predicted correctly. A 5 times
repeated 5-fold Cross Validation (CV) was performed to obtain the optimal k, k=3 which has the
highest accuracy of 78.0%. We note that when we retrained the model using optimal k=3, accuracy
remains the same at 60.3%. The “Survived” variable in Kaggle’s test data were then predicted using
the optimal k and submission result was higher than expected at 74.6%. The model has better than
expected performance despite being a distance metric, which works better with numeric features and
does not handle categorical features well.
4 Model Comparison
Let us look at our expected model performance versus the actual performance in predicting the class
label “Survived” in Test set (i.e. Kaggle submission score). All the models have relatively high
predictive power of the class label “Survived” ranging from as low as 74.6% to 79.9%. The expected
performances were higher than actual prediction except for K Nearest Neighbors, which performed
better, in the actual context despite it being the worst model in terms of expected and actual
performance. Among all the models, Random Forest and decision tree has the smallest difference
between expected and actual prediction and can be said to perform relatively well. Although Naïve
Bayes has the highest expected accuracy, it does not fare as well as predicted in actual context and
can said to be the most unstable model. Only Random Forest and Neural Networks remains as top 3
models in terms of expected and actual performance. In this regard, Random Forest seems to be the
most consistent and accurate model out of all the 6 models tested.
| Types of Model |
Expected Performance | Kaggle Submission Score(s) – Actual |
Difference (expected actual prediction) |
||
| Score | Rank | Score | Rank | ||
| Logistic Regression |
82.5% | 4 | 75.1% | 5 | 7.4% |
| Random Forest | 83.3% | 2 | 79.9% | 1 | 3.4% |
| Neural Network | 83.2% | 3 | 77.5% | 3 | 5.7% |
| Naïve Bayes | 100% | 1 | 75.6% | 4 | 24.4% |
| Decision Tree | 82.4% | 5 | 78.9% | 2 | 3.5% |
| K Nearest Neighbors |
60.3% | 6 | 74.6% | 6 | -14.3% |
5 Conclusion
We have explored the data to feature engineered 6 new features namely FamilySize;
PerPassengerFare;Title; SocialClass; AgeGroup and PayNoFare. During feature selection process,
we dropped (i) 2 variables, Passenger ID and Ticket which are of low relevance; (ii) PayNoFare as it
has the least variable importance and has the lowest decrease in mean Gini if it were kept from the
variable importance chart; (iii) SocialClass to avoid overfitting of the model as it has high correlation
with FamilySize and Title which are the top important features. 8 variables namely Pclass; Sex;
Cabin; Embarked; FamilySize; PerpassegerFare; Title and Age were selected as they have higher
predictive power of the class label “Survived”.
From the variable importance charts, Boruta and RandomForest, it is consistent that Title, Sex,
Pclass, PerPassengerFare and FamilySize are the top most important features that determined the
survival of a passenger. For instance, a male passenger with the title of “Master” from passenger
class 1 who paid a high fare and from a small family will most likely survive!!
We then compared the expected (accuracy of the trained model) and actual performance (Kaggle
submission) of various machine learning methods namely Logistic Regression, Random Forest,
Neural Network, Naïve Bayes, Decision Tree and K Nearest Neighbors in predicting the survival of
the passengers. We found out that Random Forest is the most consistent and accurate model out of all
the 6 models tested as it performs well in both expected and actual prediction.
6.Future Directions for Improvement
We have come to the end of our report and would like to propose the following to further enhance
our models’ prediction.
Feature Engineering and feature selection:
1. Explore possibility of other features creation. For instance, from the Age , Title and Parch, we
can deduce if the passenger is a mother with kids and see if it affects her survival.
2. Removal of Cabin feature as it has over 70% of missing data instead of categorising it as
“Assigned” and “Unassigned”. We have kept “Assigned” and “Unassigned” as the chart
shows that it affects survival and that it is something interesting to present. Propose to run all
the models again with Cabin removed and compare if it improve our model performance.
Missing data handling:
1. Explore the use of other missing data imputation packages like Amelia, Hmisc and GUIDE
instead of using MICE
2. Explore the use of catboost package for the generation of feature importance instead of
Boruta and Random Forest
Enhancement of our machine learning models:
1. Explore the use of caretEnsemble, which allows us to train a list of models all at the same
time. Using Caret, we can run the 5 or 10 fold cross-validation for each model. This will
allow parallel processing to boost speed. We can further boost our models by using Caret for
parameter tuning before comparing the performance of all the models.
2. As Random Forest was found to be the most stable and accurate model, more efforts can be
put into parameter tuning, therefore, mtry (no of splits) and ntrees (number of trees) etc.
3. Explore the usage of other types of Machine Learning Algorithms like Linear SVM, Nonlinear Radial SVM & Extreme Gradient Boosting
The post MH8111 Project Machine Learning from Disaster appeared first on My Assignment Online.