Titanic: Machine Learning from Disaster
Overview
• Background & Objectives
• Train-Test Split
• Feature Engineering
• Data exploration and Feature engineering
• Data Preparation
o Handling Missing Data
o Missing Values Imputation
o Features generation
o Features dropping
o Features importance
o Check for correlation
o Features selection
• Modeling
o Logistic Regression
o K Nearest Neighbour (KNN)
o RandomForest
o Naïve Bayes
o Neural Networks
o Decision Tree
• Compare models
• Kaggle Performance
• Conclusion
• Future improvements
Titanic…
Background
• The ship sank after its collision with an iceberg on April 15, 1912. The tragedy
led to the deaths of 1,502 out of 2,224 passengers and crew, calling for better
safety regulations for ships. The large number of causalities was partly caused
by the insufficient number of lifeboats for the passengers and crew. Besides
luck, it seemed that certain groups of people such as women, children, and
the upper-class were more likely to survive than others.
Goal:
• Explore and feature engine the data to select features that result in higher
survival rate. The selected features will be used to predict the survival of the
passengers provided in the Test set, therefore 0 for *Died* and 1 for *Survived*
using R.
| Datasets | Observations | Variables |
| training set (train.csv) |
891 | 12 |
| test set (test.csv) | 418 | 11 (Survived to be predicted!) |
Data Description
Data Dictionary
Missing Data!!
Data Exploration and Feature Engineering
Insights
◆ Create a family size variable
(= SibSp + Parch + 1)
◆ Singles and bigger family with a size
of 4 or more have a lower chance of
survival.
◆ Smaller families with a size of 2-4 are
more likely to survive.
Feature Creation
◆ Create a new FamSize variable
with 3 categories
single (FamSize =1)
small (1 <FamSize < 5)
large (FamSize > 4)
How does family size affect survival?
Data Exploration and Feature Engineering
Insights
◆ Age seems to be a a strong predictor
of survivability.
◆ Older women passengers and
younger child passengers (in
particular male) had a higher chance
of survival.
◆ Survival rate is lower for adult
passengers and higher for senior
passengers.
Feature Creation
◆ Create a new AgeGroup variable
after we impute missing Age
How does age affects survival?
Data Exploration and Feature Engineering
Insights
◆ No missing data in Name -> robust
predictor for Age?
◆ Create a new feature on Title from
Name e.g. “surname, title. given
name (maiden name)”
e.g. Allen, Mr. William Henry
◆ Maiden name present in
parentheses indicate that the
passenger is a married woman
◆ French designations Madamoiselle
(“Mlle.”) and Madame (“Mme.”) =
designations of “Miss” and “Mrs”
in English respectively
Feature Creation
◆ Title variable consists of Master,
Miss, Mr, Mrs and Rare Title.
◆ Master and Miss are younger and
Rare Title are older
How does one’s title relates to age?
Data Exploration and Feature Engineering
Insights
◆ Older women with Rare Title all
survived
◆ Younger men with the Master title
has a higher chance of survival than
other male
◆ Women have a higher chance of
survival than men
How does one’s title relates to age?
Data Exploration and Feature Engineering
Insights
◆ Passengers from Class 1 have a
higher chance of survival
◆ Among those who are from
passenger class 1, passengers with
higher social status may have a
higher rate of survival?
Feature Creation
◆ New feature SocialClass created
from raw Title
Commoner
Royal
Rank
How does passenger class affects survival?
Data Exploration and Feature Engineering
Insights
◆ Passengers from a “Royal” social
status have higher rate of survival
◆ Women with the social class
“Royal” and “Rank” all survived
◆ Men of social class ”Rank” have a
lower survival rate -> they are
likely to be the heroes who helped
others to escape
How does social class affects survival?
Data Exploration and Feature Engineering
Insights
◆ Passengers who paid a lower fare
has a lower chance of survival.
◆ Male passengers who paid 0 fare in
the train dataset all died.
– embarked from Southampton (S)
– singles
– mostly likely the crew who
sacrificed to help others
Feature Creation
◆ New feature ”PayNoFare”?
How does no passenger fare affects survival?
Data Exploration and Feature Engineering
Insights
◆ Bigger in the family size, the
higher the fare
◆ More outliers for singles
Feature Creation
◆ New feature ”PerPassengerFare”
How does number of family member relates to Fare?
Data Exploration and Feature Engineering
Insights
◆ Cabin could be a strong predictor for
survivability. Passengers who are located
in cabins that are nearer to exit and/or
access to lifeboats will have higher chance
of survival
◆ 78% of the Cabin data are missing
◆ There is no clear relationship between
Deck and survival
Feature Creation
◆ Replace values ”Cabin” with assigned
and unassigned -> imputed missing
values with unassigned
How does cabin affects survival?
Data Exploration and Feature Engineering
Insights
◆ Passengers with Cabin assignment has
a higher survival rate than those
unassigned
◆ Cabin assignment may denote one’s social
status and wealth -> most assigned Cabin
are from the first passenger class!!
How does cabin affects survival?
Data Exploration and Feature Engineering
Insights
◆ Passengers who embarked from C has
higher survival rate
◆ Passenger who embarked from C has
higher survival rate if he was from
passenger class 1 as compared to
passengers who were from the other
passenger classes
How does location of embarked affects survival?
| Missing Cabin in both Train & Test |
| • 77.46% missing • Argument 1: Drop it since majority is missing • Argument 2: Convert it into Assigned or Unassigned • We voted for the 2nd approach |
| Age |
| • MICE |
| Fare |
| • Only 1 record missing • Filled based on median of PerPassengerFar e |
| Embarked |
| • Only 2 records missing • Filled with the most frequent value |
Handing Missing Data
Missing Data Imputation
Imputed with the mode of Embarked (~70%), i.e. S
| Missing values | Train | Test |
| Embarked | 2 | 0 |
| Fare | 0 | 1 |
| Age | 177 | 86 |
Missing fare is from a Pclass 3, Embarked = S.
Impute missing value with the median fares of the
other passengers from Pclass 3 and Embarked =S.
We did not use mean as it is more prone to outliers.
Missing age are imputed using MICE. Age
Distribution of the MICE output is in line with
the original data.
With the missing Age value imputed, we created
a new AgeGroup variable, ensuring that each
group is not overrepresented.
– young children: < 12 yo
– children and young adults: >= 12 and < 21 yo
– adults: >= 21 & < 30 yo
– middle aged adults: >= 30 & < 50 yo
– older adults: >= 50 yo
• PerPassengerFare
• The original ‘Fare’ feature does not refer to the fare paid by individual.
’PerPassengerFare’ is derived from ‘Fare’ feature:
o PerPassengerFare = Fare / (Parch + SibSp + 1)
• PayNoFare
• We noticed that for some passengers, the ‘Fare’ feature is assigned with 0.0
instead of NA. ‘PayNoFare’ is generated based on the following logic:
o Fare > 0.1: PayNoFare = FALSE
o Else: PayNoFare = TRUE
• SocialClass
• SocialClass derived from ‘Title’ feature is based on the following logic:
| o common: ‘Mr’, ‘Ms’, ’Mrs’} |
title with ‘Dona’, ‘Don’, ‘Miss’, ‘Mile’, ‘Mme’, |
| o royal: ‘Jonkheer’, ‘Master’ |
title with {‘Lady’, ‘the Countess’, ‘Sir’, |
o rank: title with ‘Capt’, ‘Col’, ‘Dr’, ‘Major’, ‘Rev’
Feature Generation
At this stage, we realized that some of the features have been covered by those newly
generated features. Therefore, the following list of features are the ones which we
decided to drop at this point:
• SibSp, as we have the FamilySize
• Parch, as we have the FamilySize
• Fare, as we have the PerPassengerFare
• Name, as we have the Title and SocialClass
• Age, as we have the AgeGroup
Also, we will drop 2 features which are irrelevant to the result:
• Passenger ID
• Ticket
Feature Dropping
To Evaluate the feature importance, we have adopted the following two approaches
to rank the features,
1) Boruta Algorithm
2) RandomForest Algorithm
RandomForest
(top 5)
• Pclass
• Title
• PerPassengerFare
• Sex
• FamilySize
Boruta (top 6)
• Sex
• Pclass
• Title
• PerPassengerFare
• FamilySize
• Cabin
Feature Importance
From the Boruta shown above we can conclude that the Sex, Pclass, Title and
PerPassengerFare are the important factors while from the random forest can
conclude that the Title, Sex and PerPassengerFare are the important factors deciding
if someone is a survivor or not. Since the PayNoFare is the least important, we had
decided to drop it.
VARIABLE IMPORTANCE PLOT USING BORUTA & RANDOMFOREST
Feature Importance
We can assess the correlation using Pearson Correlation among the remaining
features.
SocialClass is decided to be dropped at this point because it’s derived on a newly
generated feature ‘Title’ and with strong correlation with it.
Cabin was voted to be kept, we believe that it has correlation with chance of survival.
Correlation Matrix:
• FamilySize & SocialClass
• Pclass & Cabin
• PerpassengerFare & Pclass
• Title & SocialClass
Correlation
• Here are all the remaining features that prepared for modeling:
• Pclass
• Sex
• Cabin
• Embarked
• FamilySize
• PerpassegerFare
• Title
• AgeGroup
Feature Selection
Modeling – Logistic Regression
Logistic regression is a predictive modeling technique for binary categorical
variable.
Change class of
selected variables to
numeric
Split train data into
train_test and
Train_train
Train model using
train_train and
predict train_test
Perform repeated 5-
fold CV 5 times to
enhance model
performance
Use the CV model to
predict Survived in
Kaggle’s Test data
Accuracy=56.9%
Actual : Predicted (169:297)
Submission result: 75.12% Improved to 82.5%
Accuracy : 0.8249
95% CI : (0.7768, 0.8664)
No Information Rate : 0.569
P-Value [Acc > NIR] : < 2e-16
| Predicted | Actual | ||
| 0 | 1 | Row Total |
|
| 0 | 151 50.8 % |
34 11.4% |
185 |
| 1 | 18 6.06% |
94 31.6% |
112 |
| Column Total |
169 | 128 | 297 |
Modeling – K Nearest Neighbour (KNN)
KNN is a distance
metric -> perform
normalisation
Split train data
into train_test and
Train_train
Train model using
train_train and
predict train_test
(k=sqrt(594)=24)
Parameter tuning
using repeated 5-
fold CV 5 times
Use the CV model to
predict Survived in
Kaggle’s Test data
(k=4)
Accuracy=60.3%
Actual : Predicted (179:297)
Highest accuracy when k=3
(optimal k). Retrain our model
using k=3, accuracy remains at
60.3%.
Submission result: 74.64%
KNN is a non-parametric method used for classification and regression
– k : number of neighbor observations that contribute to the output predictions
– distance metric
Although KNN is a
distance metric which
works better with
numeric features, the
model performed better
than expected!
Change AgeGroup and FamSize to
ordinal factor before normalisation
Modeling – Random Forest
• From the random forest shown above we can
conclude that the Title, Sex and
PerPassengerFare are the important factors
deciding if someone is a survivor or not.
• The model has only 17.20% error which
means we can predict with 82.80% accuracy.
• Gained an accuracy of 0.79904 in Kaggle
submission.
• Confusion matrix consists of 71 false positives
and 32 false negatives.
Modeling – Random Forest
| Training |
| • Training 2/3 and Test 1/3, roughly Train_Train <- Train_Data[1:599,] Train_Test <- Train_Data[600:891,] • rf Model • fit <- randomForest(as.factor(Survive d) ~ Pclass + Sex + Cabin+ AgeGroup + Embarked + PerPassengerFare + Title + FamilySize, data=Train_Train, importance=TRUE, ntree=2000) |
| Features |
| • All 9 features |
OOB estimate of error rate: 17.2%
| Predicted | Actual | ||
| 0 | 1 | Row Total | |
| 0 | 332 55.4 % |
32 5.34% |
364 |
| 1 | 71 11.8% |
164 27.4% |
235 |
| Column Total | 403 | 196 | 599 |
| Features |
| • All 9 features |
| Training |
| • Training 2/3 and Test 1/3, roughly • Train_Train <- Train_Data[1:599,] • Train_Test <- Train_Data[600:891,] • Model trained with Laplace smoothing • survival_classifier <- naiveBayes(Train_Train, Train_Train$Survived, laplace = 1) |
• Prediction and Evaluation
o Train Validation: 100%
o Kaggle submission: 0.75598
Modeling – Naïve Bayes
| Predicted | Actual | ||
| 0 | 1 | Row Total | |
| 0 | 185 63.3 % |
0 | 185 |
| 1 | 0 | 107 36.6% |
107 |
| Column Total | 185 | 107 | 292 |
| Features |
| • All 9 features |
| Training |
| • Training 2/3 and Test 1/3, roughly • Train_Train <- Train_Data[1:599,] • Train_Test <- Train_Data[600:891,] • NN Model • survival_NNet <-neuralnet( Survived ~ ., data = Train_Train, hidden=5, act.fct = “logistic”, stepmax=1e6) |
• Prediction and Evaluation
• Train Validation: 83.2%
• Kaggle submission: 0.77511
Modeling – Neural Networks
AgeGroup
Title
PerPassengerFare
FamilySize
Embarked
Cabin
Sex
Pclass
Survived
1 1
| Predicted | Actual | ||
| 0 | 1 | Row Total | |
| 0 | 172 58.9 % |
35 12.0% |
207 |
| 1 | 13 4.5% |
72 24.7% |
85 |
| Column Total | 185 | 107 | 292 |
From the decision tree graph, there are only 3
features (Title, Pclass, FamilySize).
• Confusion matrix consists of 21 false
positives and 74 false negatives.
• The accuracy is 84.20% according to the test
data.
• Overfitting might exists.
Modeling – Decision Tree
| Features |
| •All 9 features |
| Training |
| •Training 2/3 and Test 1/3, roughly •Train_Train <- Train_Data[1:599,] •Train_Test <- Train_Data[600:891,] •DecisionTree Model •Model_CDT <- train(x = Train_Train_Except_Survival, y = Train_Train$Survived, method = “rpart”, tuneLength = 30, trControl = ctrl) •rpart.plot(Model_CDT$finalModel) |
| Predicted | Actual | ||
| 0 | 1 | Row Total | |
| 0 | 343 57.3 % |
74 12.4% |
417 |
| 1 | 21 3.5% |
161 26.9% |
182 |
| Column Total | 364 | 235 | 599 |
According to the above result, we can see that the accuracy is decreased a little bit,
84.2% to 83.2%. Hence, there might be some overfitting in the previous model.
Gained an accuracy of 0.78947 in Kaggle submission.
We try to build classifier model by using ‘10 fold technique’ and below
are the results obtained:
Modeling – Decision Tree
| Predicted | Actual | ||
| 0 | 1 | Row Total | |
| 0 | 170 58.2 % |
34 11.6% |
204 |
| 1 | 15 5.1% |
73 25% |
88 |
| Column Total | 185 | 107 | 292 |
1) Logistic Regression
Gained an accuracy of 0.75119 from Kaggle submission.
Four features, Sex, PerPassengerFare, Pclass and FamilySize, significantly contribute
to the model in predicting survival.
Confusion matrix consists of 18 false positives and 34 false negatives.
2) Random Forest
Gained an accuracy of 0.79904 from Kaggle submission.
The first five important features based on their importance (highest to lowest) are as
follows: Title, Sex, PerPassengerFare, Pclass and FamilySize.
Confusion matrix consists of 71 false positives and 32 false negatives.
3) Neural Network
Gained an accuracy of 0.77511 from Kaggle submission.
The first five important features based on their importance (highest to lowest) are as
follows: Sex, Pclass, PerPassengerFare and FamilySize.
Confusion matrix consists of 16 false positives and 33 false negatives.
Modeling Comparisons
4) Naïve Bayes
Gained an accuracy of 0.75598 from Kaggle submission.
Train Validation: 100%
5) Decision Tree
Gained an accuracy of 0.78947 from Kaggle submission.
The improved model uses three features, Title, Pclass and FamilySize for
classification.
Confusion matrix consists of 15 false positives and 34 false negatives
6) K Nearest Neighbour
Gained an accuracy of 0.74641 from Kaggle submission.
Highest accuracy when k=3 (optimal k). Retrain our model using k=3,
accuracy remains at 60.3%.
Modeling Comparisons
Kaggle Submission Scores
Among all the models, Random Forest and decision tree has the smallest difference
between expected and actual prediction and can be said to perform relatively well.
Although Naïve Bayes has the highest expected accuracy, it does not fare as well as
predicted in actual context and can said to be the most unstable model.
Only Random Forest and Neural Networks remains as top 3 model in terms of
expected and actual performance. In this regard, Random Forest seems to be the most
consistent and accurate model out of all the 6 models tested.
| Types of Model | Expected Performance | Kaggle Submission Score(s) – Actual |
Difference (expected actual prediction) |
||
| Score | Rank | Score | Rank | ||
| Logistic Regression | 82.5% | 4 | 75.1% | 5 | 7.4% |
| Random Forest | 83.3% | 2 | 79.9% | 1 | 3.4% |
| Neural Network | 83.2% | 3 | 77.5% | 3 | 5.7% |
| Naïve Bayes | 100% | 1 | 75.6% | 4 | 24.4% |
| Decision Tree | 82.4% | 5 | 78.9% | 2 | 3.5% |
| K Nearest Neighbors | 60.3% | 6 | 74.6% | 6 | -14.3% |
Feature Engineering and Selection
Dropping Cabin feature. There were two proposals,
1. To remove it completely as over 70% of the feature data are
missing data (What we didn’t try).
2. To categorize it based on whether the value as “ASSIGNED” or
“UNASSIGNED” (What we voted for)
• Missing Data Handling
Explore the use of other data imputation packages like Amelia,
Hmisc and GUIDE, we used MICE only.
Explore other feature importance packages.
• Model Enhancement and Boosting
Some of our models weren’t boosted, due to time constraint, some
weren’t even adopting 5/10 fold cross validation. We could explore
the other libraries like XGBoost or Caret.
Parameter Tuning of our models
Explore other models, SVM models, Multi-Regression etc.
Possible Improvements
• Feature engineering
o New features
• FamilySize, PerPassengerFare, Title
• SocialClass, AgeGroup, PayNoFare
o Dropped features:
• PassengerId, Ticket, PayNoFare, SocialClass
• Feature Importance
o Title, Sex, Pclass, PerPassengerFare and FamilySize
o Consistent from both Boruta and RandomForest
• Models
o Logistic Regression, Random Forest, Neural Network, Naïve Bayes, Decision Tree and
K Nearest Neighbors
• Result
o Winner: Random Forest ~ 79.9% on Kaggle Submission
Conclusion
We have developed skills on handling missing data, feature engineering
and exploratory data analysis.
Overall, it’s a very fun project, we had gained in-depth understanding of
the different type of Machine Learning Algorithms models which can be
applied against a data set to find a pattern in the data.
We came to know better on how models differ from each other in terms of
feature requirements and problem suitability, computational performance
and model evaluation. Given that all our members are part-time students,
we are proud to pull through it.
What we have learnt
Thank You!
The post Titanic: Machine Learning from Disaster appeared first on My Assignment Online.