Detecting multicollinearity in data sets is a crucial step but additionally difficult. I’ll show the way to detect variables with similar behavior in mixed data sets and the way to deeper examine the relationships with interactive charts.
Understanding the strength of relationships between variables in an information set is very important because variables with statistically similar behavior can affect the reliability of models. To remove the so-called multicollinearity we will use correlation measures for continuous variables. Nevertheless, once we even have categorical variables and thus mixed data sets, it becomes even more difficult to check for multicollinearity. Statistical tests, reminiscent of Hypergeometric testing and the Mann-Whitney U test could be used to check for associations across variables in mixed data sets. Although that is great, it requires various intermediate steps reminiscent of the typing of variables, one-hot encoding, and multiple test corrections, amongst others. This whole pipeline is quickly implemented in a way named HNet. On this blog, I’ll show the way to detect variables with similar behavior in order that multicollinearity could be easily detected.
Real-world data often incorporates measurements with each continuous and discrete values. We’d like to have a look at each variable and use common sense to find out whether variables could be related to one another. But when there are tens (or more) variables, where each variable can have multiple states per category, it becomes time-consuming and error-prone to manually check all of the variables. We are able to automate this task by performing intensive pre-processing steps, along with statistical testing methods. Here comes HNet [1, 2] into play which uses statistical tests to find out the numerous relationships across all variables in a dataset. It permits you to input your raw unstructured data into the model after which outputs a network that sheds light on the complex relationships across variables. Let’s go to the subsequent section where I’ll explain the way to detect variables with similar behavior using statistical…