Where to get data from?

Looking for interesting data sets to train your models on or create hypothesis?

Here some well organized lists of free (most of them) and public datasets in good quality:

• http://rs.io/100-interesting-data-sets-for-statistics/ 
• https://github.com/caesar0301/awesome-public-datasets (mainly data from America)
• https://github.com/bayandin/awesome-awesomeness

This list will be updated. Enjoy!

Data Scientists - Skills

It is obvious that a data scientist should have an interest in data, a strong analytical, mathematical background and a good intuition on numbers. A profund basic of statistics (the more the better), an insight into the tools (Excel is a very powerful and useful tool!) and a good part of endurance to handle complex processes and work on large data sets.
However these skills are not the essential competencies of a data scientist, moreover it is important that a data scientist understands the details on the algorithms applied in order to decide how to further optimize models, performance and processes. Experience with performance optimization algorithms, distributed calculation and programming are of great help. A good data scientist is able to structure and prioritize, distinguish between useful and unuseful information and also be able to work with people from different departments. Further imprescindible abilities are good communication and presentation skills to a wide audience, the task to help business in decisions on complex business processes is only fulfilled if the business actually understands and gets to "see" the results.
Last but not least a data scientist should always be open to new ideas, as the models and the requirements change regularily.

The Data Mining Process

The Cross Industry Standard Process for Data Mining (CRISP-DM) is a process model that describes the different steps data scientists use for analyzing data. Here some more details on the different steps:

In the Business Understanding Phase, you describe the current situation and the requirements from a business point of view, here the business defines, what it wants to accomplish with the data mining process. This included not only a business project plan, a list of risks and measures, costs and benefits, limitations and terminologies, but also a business success criteria.
In the Data Understanding Phase, the business requirements are translated into a data science problem. Here you describe the problem in technical terms, define inputs and outputs, detect dependencies and create a dynamic technical project plan. Here you also describe the data mining success criteria in a simple and easily understandable way. This coulds be a goal like the creation of BI-reports to learn from the historical data, cluster models, segmentations for unsupervized/undirected learning or a prediction, risk estimation or a classification for supervised/directed learning.
In both upper mentioned steps, the requirements should be described as concrete as possible.

The next phase is the Data Preparation Phase, in which an initial data set is being collected from documented tables in documented locations, described (missing fields, amounts of data, relations,...), explored (first findings, simple associations, initial hypothesis,...)  and verified (is the collected data correct/useful/intuitive?). Here you also transform class categories into numerical values, aggregate suitable datasets, clean and format data (eg. unsort data for neural networks).

The Modeling Phase stands in close contact to the Data Preparation Phase. In it the available data is cut into training, test and cross validation set. Here starting with the initially retrieved data you define new records (e.g. areas from witdh and length), datasets from different tables are collected to create an analytical record (or DNA) of a target object (also called entity, e.g. a customer), that hold the whole amount of data needed. Here you also identify outliers, select and sample data to be used in a model and analyze groupings of data. You decide on the model to use and the features (delete unrelevant features, include new features).

In the Evaluation Phase you analyze the results of the model, the accurancy and robustness of the model, the error and optimization opportunities. With these results a discussion round with the business is needed in order to verify if the results from the evaluated data are meeting the business requirements and that all important insights and goals have been included into the analysis. Usually more than one round is needed.

In the Deployment Phase the model is deployed, so that the functionality can be handed over and taken care of by the business. This could be in form of a BI-report or in the implementation of programs that allow the regular update of data.

As the models are created for a certain point in time, a certain business goal and on historic data, the data mining process has to be adapted and updated with new data.

Machine Learning

What does "machine learning" mean, how can a computer "learn"?

The term "learning" is used in this context to refer to the fact that a machine uses new data in order to better up previous results. For sure the machine will not develop a brain or a physically similar organ. But for optimization or minimizing tasks, especially tasks including complex calculations and transformations, a machine controlled adaption is a useful and often indispensable tool. The basics for such tasks are often machine learning algorithms like artificial neural networks or clustering algorithms which run iterative trying to minimize errors in each calculation step.

Machine learning algorithms usually can be devided in two main groups:

• Supervised learning algorithms, in which the output is known and the rules are being trained by using the input and trying to minimize the error between the predicted output and the given result. Neuronal networks are here the most promising examples, they are called like that as they reflect the way our brains work: given some input the human brain learns by trying out and correct until it finds the perfect rule to explain the result.

• Unsupervised learning algorithms in which data is given to an algorithm which then tries to find pattern in the data. As an example think about astronomical data: if you can cluster the stellar data you could find a structure in it and learn about the past and future.

In addition modern algorithms can also be active learning algorithms, in which the input can be completed by additional requests for input trying to minimize the numbers of these additional inputs.

What are these algorithms needed for and why are they considered promising?

Machines have the capacity to calculate fast and storage is cheap nowadays, also in presicion they are unbeatable and often parallelize their tasks. As nearly unlimited data is available and often more data leads to better results (not always, an intelligent way to sort out data is one of the reasons for a data scientist!) machines and computer visualizations are essential reasons for machine supported analysis. In addition cloud computing and distributed systems improve the way data is collected, loaded and analyzed.
Internet of Things scenarios are considered the modern way to improve business processes and drive Industry 4.0 by collecting huge amounts of (sensor) data. To analyze those datasets machines are not only helpful, but necessary.