Showing posts with label Data Science. Show all posts
Showing posts with label Data Science. Show all posts

Thursday, May 27, 2021

Data Science Life Cycle

Today in many organizations especially where data projects are being rolled out or data is being sold as product , one of the important aspects to be considered is the data science life cycle. This becomes very important when one is trying to use insights to drive the business forward. This could be in the following areas, I just picked a sample.

1. Revenue Generation
2. Customer Experience
3. Customer attrition
4. Increase Engagement

Here are some of the key Steps in a DS Life cycle:

1. Identification of Data Sources. Data (Structured, Unstructured)
2. Data Extraction
3. Data Storage (Persistence of Data), what type of Data storage - On Prem, Public Cloud, Private Cloud or Hybrid Cloud.
4. Data Engineering - Data Profiling,Data Quality Checks, Cleansing of Data, Data Transformation
Handling of bad data from Source.
5. Data Science Layer - This includes the following:
    a. Data Transformation - Addition of Categorical Variables
    b. Feature Engineering - Enhance the data set to make it more meaningful - requires domain expertise.
    c. Model Training, Testing and Validation
    d. Deployment of Models to Production
    e. Documentation of Models
    f. All of the above Steps could be iterative s new data keeps coming in
6. Data Delivery - How data is going to be sent to end users, stakeholders
   a. Method of Data Delivery - Web Services, Visualizations, Reports
   b. Actionable insights - There needs to be action taken on the insights provided. Today in organizations there is an issue of these insights not even being looked at. How does an organization go about this is very important.

Data Lineage/Governance - This step would cover all aspects of the steps mentioned here. This includes Data catalog, Metadata repository, Data Value Chains, Capturing important data assets. How data is being represented cross the enterprise.




Tuesday, August 2, 2016

Data Science - Education

With the rapid growth of Big data technologies , there has been an exponential growth of data science and its related technologies. This has has also led to the demand for data scientists and also the jobs related to data science are very lucrative. Microsoft has been steadily expanding it s cloud based offerings and also getting into big Data related technologies and efforts. Since there is tremendous need for Data science skills, Microsoft has come forward to offer a curriculum  totally devoted to Data Science. This curriculum is offered via edx.org.  There are a total of 9 courses and the price per course ranges from $49-$99. There is also a final project for which around 6-10 hours is required. One can check the link below for all the details:
https://www.edx.org/microsoft-data-science-curriculum
The courses cover from Use Microsoft Excel to explore data to Iimplement a machine learning solution for a given data problem. Each of the course can be done a auditing course or one can upgrade to get a validity certificate on passing the course. Each of the courses have Labs ,Quizzes and discussion forums, the discussion forums can be use to get questions answered related to the concepts being discussed. I hope the courses provide the much needed insights into Data Science.


Tuesday, March 22, 2016

Data Science Resource/Blog...

One of the rapidly advancing areas today is Data Analytics/Data Science in the Business Intelligence space. There are lot of resources/tools that are coming explaining the capabilities of Data Science. Data Science has seen tremendous growth alongside the adoption of Big Data technologies. It is a very challenging space to keep up with, one of the blogs that i regularly visit to understand the concepts in Data Science and keep up with the trends in Data Science/Analytics, is http://www.analyticsvidhya.com/. The blog/website is very nicely laid out with content rich articles and tutorials. They cover a variety of tools and also encourage folks interested in data science to participate in challenges/contests. Please do visit the site, it provides amazing content and coverage.

Thursday, December 31, 2015

Big Data Trends in 2016...

First, I would like to wish the readers of my blog a very happy and prosperous new year 2016. Due to personal challenges i could not post much in 2015, i hope i can do better in 2016. Looking forward to how the year 2016 shapes up for analytics, if there are newer trends or consolidation of big and small technology firms in the analytics space. There were lot of strides made in data science, the programming language R has become increasingly popular across various industry domains, More business folks are beginning to understand the power of good models and how it can be pursued for predictive analytics. There were also strides made in bringing R and SQL Server together, this will be seen more in 2016 as we begin to see more valid business used cases.

Here is a link for the Top Big 3 data trends in 2016.

http://insidebigdata.com/2015/12/29/the-top-3-big-data-trends-of-2016/

Once again wishing everyone a very happy 2016.

Monday, November 16, 2015

Using Data Science Techniques with R- Zillow.com

R language has been gaining increasing popularity as Machine Learning, Predictive analytics take more and more importance within business. The applications of R has been steadily increasing in different industry domains. One of the areas where the R language is being very strongly used is in the are of Real Estate to predict selling prices of Homes Please read the link to blog post below related to how the company Zillow uses R for predicting estimates of prices for houses.
http://www.datanami.com/2015/08/12/inside-the-zestimate-data-science-at-zillow/
As the article indicates they use a mix of tools at different stages of the data value chain. The more cleaner the data is, the better the prediction is going to be.

Friday, August 7, 2015

Azure ML Studio - Part 1

Data Science has been experiencing tremendous growth in the business world today, there is a tremendous scope/job opportunities for people with Data Science Experience. One of the challenges has been to learn the different components of Data Science since most of them involve lot of Statistical, Math, Data Mining Algorithms knowledge. Microsoft on its part has been working steadily expose data science for the programming public. Initially Azure was slow to take off, but now with growing cloud implementations, Azure has been experiencing a lot of growth. So Microsoft decided to use the Azure platform and provide Data Science tools for the programmers. One of the very effective tools that is offered is called the Azure ML Studio, this is a development environment for Machine Learning Model Development. The interface of this tool is similar to some of the Visual Studio tools provided earlier by Microsoft. In order to start using a the Azure ML Studio one needs to have a Azure account. The whole concept of Azure ML works on the concept of Software as a service. One can use the following link to learn more about the Azure ML capabilities: https://studio.azureml.net/ Once you login to the azure studio, the first that will happen is that the workspace will be set up. There will be a + symbol at the bottom of the workspace, click on that to create your first experiment. You have a couple of choices here 1) You can create a blank experiment 2) You Can create a experiment based on the templates provided. The option 2 would help one to set up an experiment quickly and understand the various components of the experiment. When you choose from the samples, you can either open it in ML Studio and view it in the gallery. I feel tools like Azure ML studio provide a great first step in exploring the power of Machine Learning/Data Science.

 One of the components in the above image is the Enter Data component. This component is primarily used for defining column headings, these column headings can be used to assign to the data sets that are read through the Reader component. In this case in the Reader component, we are downloading a file from a website. Since in this example the headers of the file downloaded by Reader component were not user friendly, we use the Enter Data component to provide meaningful column_names. In this example we have used the column_names to be in the csv format. For example please see the image below for Enter Data component:
In the image above the column_name is the header in the csv file and the other below it are actual column names which would be used to assign it to the data set ready by reader component.

Monday, May 11, 2015

SQL Server and R Integration...

One of the most popular languages in the Data Science Paradigm is the R language, there are plenty of resources and online classes being offered with respect to the R language. Once of the main advantages with the R language is that it is open source and can be downloaded on multiple platforms such as windows/Mac/Linux, also there is a very popular IDE for R called the R-Studio. SQL Server has been expanding to get into different domains one of them now being is Data Science. in SQL Sever 2016 there are lot more integration promised between R and SQL Server. In the mean time there are ways to integrate between R and SQL Server, here is a blog article on Simple talk by SQL Server Expert Feodor Georgiev, the article is titled: Making Data Analytics Simpler: SQL Server and R.
https://www.simple-talk.com/sql/reporting-services/making-data-analytics-simpler-sql-server-and-r/.
The article also provides a brief overview of the R language.