Limited Offer Get 25% off — use code BESTW25
No AI No Plagiarism On-Time Delivery Free Revisions
Claim Now
Prices start at $5 per page · plagiarism-free papers · free revisions for 14 days 24/7 support: support@bestwriters.com
Uncategorized

Titanic: Machine Learning from Disaster

  • Posted
  • Authoradmin
  • Length2,550 words
  • Reading time12 min
Write My Assignment FREE

Titanic: Machine Learning from Disaster
Overview
• Background & Objectives
• Train-Test Split
• Feature Engineering
• Data exploration and Feature engineering
• Data Preparation
o Handling Missing Data
o Missing Values Imputation
o Features generation
o Features dropping
o Features importance
o Check for correlation
o Features selection
• Modeling
o Logistic Regression
o K Nearest Neighbour (KNN)
o RandomForest
o Naïve Bayes
o Neural Networks
o Decision Tree
• Compare models
• Kaggle Performance
• Conclusion
• Future improvements
Titanic…
Background
• The ship sank after its collision with an iceberg on April 15, 1912. The tragedy
led to the deaths of 1,502 out of 2,224 passengers and crew, calling for better
safety regulations for ships. The large number of causalities was partly caused
by the insufficient number of lifeboats for the passengers and crew. Besides
luck, it seemed that certain groups of people such as women, children, and
the upper-class were more likely to survive than others.
Goal:
• Explore and feature engine the data to select features that result in higher
survival rate. The selected features will be used to predict the survival of the
passengers provided in the Test set, therefore 0 for *Died* and 1 for *Survived*
using R.

Datasets Observations Variables
training set
(train.csv)
891 12
test set (test.csv) 418 11 (Survived to be predicted!)

Data Description
Data Dictionary
Missing Data!!
Data Exploration and Feature Engineering
Insights
◆ Create a family size variable
(= SibSp + Parch + 1)
◆ Singles and bigger family with a size
of 4 or more have a lower chance of
survival.
◆ Smaller families with a size of 2-4 are
more likely to survive.
Feature Creation
◆ Create a new FamSize variable
with 3 categories
▪ single (FamSize =1)
▪ small (1 <FamSize < 5)
▪ large (FamSize > 4)
How does family size affect survival?
Data Exploration and Feature Engineering
Insights
◆ Age seems to be a a strong predictor
of survivability.
◆ Older women passengers and
younger child passengers (in
particular male) had a higher chance
of survival.
◆ Survival rate is lower for adult
passengers and higher for senior
passengers.
Feature Creation
◆ Create a new AgeGroup variable
after we impute missing Age
How does age affects survival?
Data Exploration and Feature Engineering
Insights
◆ No missing data in Name -> robust
predictor for Age?
◆ Create a new feature on Title from
Name e.g. “surname, title. given
name (maiden name)”
e.g. Allen, Mr. William Henry
◆ Maiden name present in
parentheses indicate that the
passenger is a married woman
◆ French designations Madamoiselle
(“Mlle.”) and Madame (“Mme.”) =
designations of “Miss” and “Mrs”
in English respectively
Feature Creation
◆ Title variable consists of Master,
Miss, Mr, Mrs and Rare Title.
◆ Master and Miss are younger and
Rare Title are older
How does one’s title relates to age?
Data Exploration and Feature Engineering
Insights
◆ Older women with Rare Title all
survived
◆ Younger men with the Master title
has a higher chance of survival than
other male
◆ Women have a higher chance of
survival than men
How does one’s title relates to age?
Data Exploration and Feature Engineering
Insights
◆ Passengers from Class 1 have a
higher chance of survival
◆ Among those who are from
passenger class 1, passengers with
higher social status may have a
higher rate of survival?
Feature Creation
◆ New feature SocialClass created
from raw Title
▪ Commoner
▪ Royal
▪ Rank
How does passenger class affects survival?
Data Exploration and Feature Engineering
Insights
◆ Passengers from a “Royal” social
status have higher rate of survival
◆ Women with the social class
“Royal” and “Rank” all survived
◆ Men of social class ”Rank” have a
lower survival rate -> they are
likely to be the heroes who helped
others to escape
How does social class affects survival?
Data Exploration and Feature Engineering
Insights
◆ Passengers who paid a lower fare
has a lower chance of survival.
◆ Male passengers who paid 0 fare in
the train dataset all died.
– embarked from Southampton (S)
– singles
– mostly likely the crew who
sacrificed to help others
Feature Creation
◆ New feature ”PayNoFare”?
How does no passenger fare affects survival?
Data Exploration and Feature Engineering
Insights
◆ Bigger in the family size, the
higher the fare
◆ More outliers for singles
Feature Creation
◆ New feature ”PerPassengerFare”
How does number of family member relates to Fare?
Data Exploration and Feature Engineering
Insights
◆ Cabin could be a strong predictor for
survivability. Passengers who are located
in cabins that are nearer to exit and/or
access to lifeboats will have higher chance
of survival
◆ 78% of the Cabin data are missing
◆ There is no clear relationship between
Deck and survival
Feature Creation
◆ Replace values ”Cabin” with assigned
and unassigned -> imputed missing
values with unassigned
How does cabin affects survival?
Data Exploration and Feature Engineering
Insights
◆ Passengers with Cabin assignment has
a higher survival rate than those
unassigned
◆ Cabin assignment may denote one’s social
status and wealth -> most assigned Cabin
are from the first passenger class!!
How does cabin affects survival?
Data Exploration and Feature Engineering
Insights
◆ Passengers who embarked from C has
higher survival rate
◆ Passenger who embarked from C has
higher survival rate if he was from
passenger class 1 as compared to
passengers who were from the other
passenger classes
How does location of embarked affects survival?

Missing Cabin in
both Train & Test
• 77.46% missing
• Argument 1:
Drop it since
majority is
missing
• Argument 2:
Convert it into
Assigned or
Unassigned
• We voted for the
2nd approach
Age
• MICE
Fare
• Only 1 record
missing
• Filled based on
median of
PerPassengerFar
e
Embarked
• Only 2 records
missing
• Filled with the
most frequent
value

Handing Missing Data
Missing Data Imputation
Imputed with the mode of Embarked (~70%), i.e. S

Missing values Train Test
Embarked 2 0
Fare 0 1
Age 177 86

Missing fare is from a Pclass 3, Embarked = S.
Impute missing value with the median fares of the
other passengers from Pclass 3 and Embarked =S.
We did not use mean as it is more prone to outliers.
Missing age are imputed using MICE. Age
Distribution of the MICE output is in line with
the original data.
With the missing Age value imputed, we created
a new AgeGroup variable, ensuring that each
group is not overrepresented.
– young children: < 12 yo
– children and young adults: >= 12 and < 21 yo
– adults: >= 21 & < 30 yo
– middle aged adults: >= 30 & < 50 yo
– older adults: >= 50 yo
• PerPassengerFare
• The original ‘Fare’ feature does not refer to the fare paid by individual.
’PerPassengerFare’ is derived from ‘Fare’ feature:
o PerPassengerFare = Fare / (Parch + SibSp + 1)
• PayNoFare
• We noticed that for some passengers, the ‘Fare’ feature is assigned with 0.0
instead of NA. ‘PayNoFare’ is generated based on the following logic:
o Fare > 0.1: PayNoFare = FALSE
o Else: PayNoFare = TRUE
• SocialClass
• SocialClass derived from ‘Title’ feature is based on the following logic:

o common:
‘Mr’, ‘Ms’, ’Mrs’}
title with ‘Dona’, ‘Don’, ‘Miss’, ‘Mile’, ‘Mme’,
o royal:
‘Jonkheer’, ‘Master’
title with {‘Lady’, ‘the Countess’, ‘Sir’,

o rank: title with ‘Capt’, ‘Col’, ‘Dr’, ‘Major’, ‘Rev’
Feature Generation
At this stage, we realized that some of the features have been covered by those newly
generated features. Therefore, the following list of features are the ones which we
decided to drop at this point:
• SibSp, as we have the FamilySize
• Parch, as we have the FamilySize
• Fare, as we have the PerPassengerFare
• Name, as we have the Title and SocialClass
• Age, as we have the AgeGroup
Also, we will drop 2 features which are irrelevant to the result:
• Passenger ID
• Ticket
Feature Dropping
To Evaluate the feature importance, we have adopted the following two approaches
to rank the features,
1) Boruta Algorithm
2) RandomForest Algorithm
RandomForest
(top 5)
• Pclass
• Title
• PerPassengerFare
• Sex
• FamilySize
Boruta (top 6)
• Sex
• Pclass
• Title
• PerPassengerFare
• FamilySize
• Cabin
Feature Importance
From the Boruta shown above we can conclude that the Sex, Pclass, Title and
PerPassengerFare are the important factors while from the random forest can
conclude that the Title, Sex and PerPassengerFare are the important factors deciding
if someone is a survivor or not. Since the PayNoFare is the least important, we had
decided to drop it.
VARIABLE IMPORTANCE PLOT USING BORUTA & RANDOMFOREST
Feature Importance
We can assess the correlation using Pearson Correlation among the remaining
features.
SocialClass is decided to be dropped at this point because it’s derived on a newly
generated feature ‘Title’ and with strong correlation with it.
Cabin was voted to be kept, we believe that it has correlation with chance of survival.
Correlation Matrix:
• FamilySize & SocialClass
• Pclass & Cabin
• PerpassengerFare & Pclass
• Title & SocialClass
Correlation
• Here are all the remaining features that prepared for modeling:
• Pclass
• Sex
• Cabin
• Embarked
• FamilySize
• PerpassegerFare
• Title
• AgeGroup
Feature Selection
Modeling – Logistic Regression
Logistic regression is a predictive modeling technique for binary categorical
variable.
Change class of
selected variables to
numeric
Split train data into
train_test and
Train_train
Train model using
train_train and
predict train_test
Perform repeated 5-
fold CV 5 times to
enhance model
performance
Use the CV model to
predict Survived in
Kaggle’s Test data
Accuracy=56.9%
Actual : Predicted (169:297)
Submission result: 75.12% Improved to 82.5%
Accuracy : 0.8249
95% CI : (0.7768, 0.8664)
No Information Rate : 0.569
P-Value [Acc > NIR] : < 2e-16

Predicted Actual
0 1 Row
Total
0 151
50.8 %
34
11.4%
185
1 18
6.06%
94
31.6%
112
Column
Total
169 128 297

Modeling – K Nearest Neighbour (KNN)
KNN is a distance
metric -> perform
normalisation
Split train data
into train_test and
Train_train
Train model using
train_train and
predict train_test
(k=sqrt(594)=24)
Parameter tuning
using repeated 5-
fold CV 5 times
Use the CV model to
predict Survived in
Kaggle’s Test data
(k=4)
Accuracy=60.3%
Actual : Predicted (179:297)
Highest accuracy when k=3
(optimal k). Retrain our model
using k=3, accuracy remains at
60.3%.
Submission result: 74.64%
KNN is a non-parametric method used for classification and regression
– k : number of neighbor observations that contribute to the output predictions
– distance metric
Although KNN is a
distance metric which
works better with
numeric features, the
model performed better
than expected!
Change AgeGroup and FamSize to
ordinal factor before normalisation
Modeling – Random Forest
• From the random forest shown above we can
conclude that the Title, Sex and
PerPassengerFare are the important factors
deciding if someone is a survivor or not.
• The model has only 17.20% error which
means we can predict with 82.80% accuracy.
• Gained an accuracy of 0.79904 in Kaggle
submission.
• Confusion matrix consists of 71 false positives
and 32 false negatives.
Modeling – Random Forest

Training
• Training 2/3 and Test 1/3, roughly
Train_Train <- Train_Data[1:599,]
Train_Test <- Train_Data[600:891,]
• rf Model
• fit <-
randomForest(as.factor(Survive
d) ~ Pclass + Sex + Cabin+
AgeGroup + Embarked +
PerPassengerFare + Title +
FamilySize,
data=Train_Train,
importance=TRUE, ntree=2000)
Features
• All 9 features

OOB estimate of error rate: 17.2%

Predicted Actual
0 1 Row Total
0 332
55.4 %
32
5.34%
364
1 71
11.8%
164
27.4%
235
Column Total 403 196 599
Features
• All 9 features
Training
• Training 2/3 and Test 1/3, roughly
• Train_Train <- Train_Data[1:599,]
• Train_Test <- Train_Data[600:891,]
• Model trained with Laplace smoothing
• survival_classifier <-
naiveBayes(Train_Train,
Train_Train$Survived, laplace = 1)

• Prediction and Evaluation
o Train Validation: 100%
o Kaggle submission: 0.75598
Modeling – Naïve Bayes

Predicted Actual
0 1 Row Total
0 185
63.3 %
0 185
1 0 107
36.6%
107
Column Total 185 107 292
Features
• All 9 features
Training
• Training 2/3 and Test 1/3, roughly
• Train_Train <- Train_Data[1:599,]
• Train_Test <- Train_Data[600:891,]
• NN Model
• survival_NNet <-neuralnet(
Survived ~ ., data = Train_Train,
hidden=5, act.fct = “logistic”,
stepmax=1e6)

• Prediction and Evaluation
• Train Validation: 83.2%
• Kaggle submission: 0.77511
Modeling – Neural Networks
AgeGroup
Title
PerPassengerFare
FamilySize
Embarked
Cabin
Sex
Pclass
Survived
1 1

Predicted Actual
0 1 Row Total
0 172
58.9 %
35
12.0%
207
1 13
4.5%
72
24.7%
85
Column Total 185 107 292

From the decision tree graph, there are only 3
features (Title, Pclass, FamilySize).
• Confusion matrix consists of 21 false
positives and 74 false negatives.
• The accuracy is 84.20% according to the test
data.
• Overfitting might exists.
Modeling – Decision Tree

Features
•All 9 features
Training
•Training 2/3 and Test 1/3, roughly
•Train_Train <- Train_Data[1:599,]
•Train_Test <- Train_Data[600:891,]
•DecisionTree Model
•Model_CDT <- train(x =
Train_Train_Except_Survival, y =
Train_Train$Survived, method = “rpart”,
tuneLength = 30, trControl = ctrl)
•rpart.plot(Model_CDT$finalModel)
Predicted Actual
0 1 Row Total
0 343
57.3 %
74
12.4%
417
1 21
3.5%
161
26.9%
182
Column Total 364 235 599

According to the above result, we can see that the accuracy is decreased a little bit,
84.2% to 83.2%. Hence, there might be some overfitting in the previous model.
Gained an accuracy of 0.78947 in Kaggle submission.
We try to build classifier model by using ‘10 fold technique’ and below
are the results obtained:
Modeling – Decision Tree

Predicted Actual
0 1 Row Total
0 170
58.2 %
34
11.6%
204
1 15
5.1%
73
25%
88
Column Total 185 107 292

1) Logistic Regression
▪ Gained an accuracy of 0.75119 from Kaggle submission.
▪ Four features, Sex, PerPassengerFare, Pclass and FamilySize, significantly contribute
to the model in predicting survival.
▪ Confusion matrix consists of 18 false positives and 34 false negatives.
2) Random Forest
▪ Gained an accuracy of 0.79904 from Kaggle submission.
▪ The first five important features based on their importance (highest to lowest) are as
follows: Title, Sex, PerPassengerFare, Pclass and FamilySize.
▪ Confusion matrix consists of 71 false positives and 32 false negatives.
3) Neural Network
▪ Gained an accuracy of 0.77511 from Kaggle submission.
▪ The first five important features based on their importance (highest to lowest) are as
follows: Sex, Pclass, PerPassengerFare and FamilySize.
▪ Confusion matrix consists of 16 false positives and 33 false negatives.
Modeling Comparisons
4) Naïve Bayes
▪ Gained an accuracy of 0.75598 from Kaggle submission.
▪ Train Validation: 100%
5) Decision Tree
▪ Gained an accuracy of 0.78947 from Kaggle submission.
▪ The improved model uses three features, Title, Pclass and FamilySize for
classification.
▪ Confusion matrix consists of 15 false positives and 34 false negatives
6) K Nearest Neighbour
▪ Gained an accuracy of 0.74641 from Kaggle submission.
▪ Highest accuracy when k=3 (optimal k). Retrain our model using k=3,
accuracy remains at 60.3%.
Modeling Comparisons
Kaggle Submission Scores
▪ Among all the models, Random Forest and decision tree has the smallest difference
between expected and actual prediction and can be said to perform relatively well.
▪ Although Naïve Bayes has the highest expected accuracy, it does not fare as well as
predicted in actual context and can said to be the most unstable model.
▪ Only Random Forest and Neural Networks remains as top 3 model in terms of
expected and actual performance. In this regard, Random Forest seems to be the most
consistent and accurate model out of all the 6 models tested.

Types of Model Expected Performance Kaggle Submission
Score(s) – Actual
Difference
(expected
actual
prediction)
Score Rank Score Rank
Logistic Regression 82.5% 4 75.1% 5 7.4%
Random Forest 83.3% 2 79.9% 1 3.4%
Neural Network 83.2% 3 77.5% 3 5.7%
Naïve Bayes 100% 1 75.6% 4 24.4%
Decision Tree 82.4% 5 78.9% 2 3.5%
K Nearest Neighbors 60.3% 6 74.6% 6 -14.3%

▪ Feature Engineering and Selection
▪ Dropping Cabin feature. There were two proposals,
1. To remove it completely as over 70% of the feature data are
missing data (What we didn’t try).
2. To categorize it based on whether the value as “ASSIGNED” or
“UNASSIGNED” (What we voted for)
• Missing Data Handling
▪ Explore the use of other data imputation packages like Amelia,
Hmisc and GUIDE, we used MICE only.
▪ Explore other feature importance packages.
• Model Enhancement and Boosting
▪ Some of our models weren’t boosted, due to time constraint, some
weren’t even adopting 5/10 fold cross validation. We could explore
the other libraries like XGBoost or Caret.
▪ Parameter Tuning of our models
▪ Explore other models, SVM models, Multi-Regression etc.
Possible Improvements
• Feature engineering
o New features
• FamilySize, PerPassengerFare, Title
• SocialClass, AgeGroup, PayNoFare
o Dropped features:
• PassengerId, Ticket, PayNoFare, SocialClass
• Feature Importance
o Title, Sex, Pclass, PerPassengerFare and FamilySize
o Consistent from both Boruta and RandomForest
• Models
o Logistic Regression, Random Forest, Neural Network, Naïve Bayes, Decision Tree and
K Nearest Neighbors
• Result
o Winner: Random Forest ~ 79.9% on Kaggle Submission
Conclusion
▪ We have developed skills on handling missing data, feature engineering
and exploratory data analysis.
▪ Overall, it’s a very fun project, we had gained in-depth understanding of
the different type of Machine Learning Algorithms models which can be
applied against a data set to find a pattern in the data.
▪ We came to know better on how models differ from each other in terms of
feature requirements and problem suitability, computational performance
and model evaluation. Given that all our members are part-time students,
we are proud to pull through it.
What we have learnt
Thank You!

The post Titanic: Machine Learning from Disaster appeared first on My Assignment Online.

Plagiarism Free Assignment Help

Expert Help With This Assignment — On Your Terms

✓ Native UK, USA & Australia writers ✓ Deadline from 3 hours ✓ 100% Plagiarism-Free — Turnitin included ✓ Unlimited free revisions ✓ Free to submit — compare quotes

admin

Academic writer and editor at BestWriters.

Get your paper off your plate tonight

Post your instructions for free. Prices start at $5 per page.

Place an order