Data leakage occurs when a predictive model uses data in the training phase that are unavailable when the model is in production. Consider the example below (source: Mostafa Saad Ibrahim): The main concern about the data is related to splitting it, where images of the same animal might have occurred in both train and test …
Imbalanced data is when classes do not have similar proportions and this can occur in multiple common problems like predicting mortality where the vast majority of data points will have “Survived” instead of “Died” as the target variable, so imbalanced data occurs and trained models will probably have a bias towards the majority class. Oversampling …
There are many classification metrics that we can use to evaluate a model. Such as precision, recall, accuracy –which is one of the most used metrics-, and many others.Let’s consider the following confusion matrix: We can use confusion matrix to calculate different metrics.Accuracy: The number of correct predictions our model made divided by the total …
Suppose we have collected students’ grades at a certain test, and we have found that the mean of grades for male students is 70.2 and the mean for female students is 72.59.We can see that the two means are different but, is this difference statistically significant? T-Test can determine if there is a significant difference …
Machine Learning is the set of tools that we can use to answer questions with data. The type of asked questions may vary from predicting the price of a house based on certain features, to classifying a tumor to find out whether it is benign or malignant. The image above shows the 3 types of …
Pearson’s correlation coefficient is the test that measures the statistical linear relationship or association between two variables.Consider an example of a company with over 100 employees that wants to answer the following question:Are employees with more years of experience more likely to get paid more? We can see in the above scatter plot, as the …
Data Science is one of the best-paying jobs around the world, but what does it really mean? Data science involves studying the data to conclude useful insights by using different methods such as visualizing the data, building machine learning models, and applying statistical tests. or in simple words according to Cassie Kozyrkov: Data science is …