Showing posts with label Big Data. Show all posts
Showing posts with label Big Data. Show all posts

Sunday, August 29, 2021

Exploratory Data Analysis - EDA

 One of the key aspects of getting into AI,ML is to understand the data, what the data is trying to represent, is there data a god sample for the business problem one is trying to solve. In order to answer these questions, there needs to be scientific way to understand the data, so this is where Exploratory data analysis come into picture. In order to do EDA once can use tools or it is also possible to start getting to know the patterns with tools such as Excel. One need not feel the pressure of not knowing tools such s Python or R. You can gradually learn and getting into those techniques available in the tools.

The first step is to get the data from a trusted source, make sure it is a source that is relevant to the business problem we are trying to solve.  Now the data can be in different forms such as text files, csv, excel spreadsheets, logs or relational databases. Once you identify the source, choose a platform you want to bring the data into. For example let us assume spreadsheet is how you want to analyse the data.

1. Get the data formatted into a spreadsheet, if you are using a tool like python, read the data into a dataframe using pandas.
2. Understand the Business problem you are trying to solve, also know the domain area of the business.
3. Try as much as possible not to be biased, review the data points.
4. Discard data points that are redundant, would not offer any value. In case you have doubts ask the SME or folks who know the business/data.
5. Identify how many missing observations are there. if there is lot of data missing, discuss with the data source and get valid data.
6. A certain margin rate of missing values is acceptable, some say 5%. Identify those data attributes.
7. Begin to classify data, one can use pivot tables/charts in excel, use pandas in python to start grouping/categorizing data.
8.Document the trends, there are techniques like First Principles thinking (where one does not make any biases ,starts to analyze data from scratch).
9. Review if the data is balanced, for example if you are doing a batch prediction model, based on success/failures of daily jobs. In this case if there are more than 80% success or 80% failures then the data is skewed. Imbalance in data can cause models to generate bad predictions.
10. Add features to data, meaning add additional data points to enhance the data like adding categorical variables.
11. Once you have documented all your findings, you can proceed to the next step in determining the type of model needed.
Here is a link to First Principles thinking


Saturday, July 31, 2021

Kafka - Event Processing

In a traditional data world which started to come into picture in the 1980s and become very popular during the 2000's is the concept of storing data based on entities/structured schema.
The field of Business Intelligence rose exponentially during this period and was the main area driving analytics. There was reporting, dashboards which started growing due to the flourish of BI. The infrastructure driving this was mainly servers, data centers, relational databases and in order to scale the operations there was a concept of web farms, clustering. Also with this type of  data structure, in case we had to track changes in data or entities, we relied on triggers, writing stored procedures. This was fine initially to provide such information to business and stakeholders. As the amount of data grew, requirements became complex and more time sensitive, there was a need to move to a scalable architecture. With the advent of big data technologies, there was one technology that is grown in popularity and usage, it is KAFKA.
Here are some basic concepts in Kafka, there lot of online tutorials on more in depth explanation of Kafka
Producer - Sends a Message Record (data), array of bytes
All records in table will be sent as message - Collect the result from query and sending each row as a message
You need to create a Producer application
Consumer - is an application that receives the data.
Producer sends data to Kafka server then requesting data from this is a Consumer
Producer -> Kafka Server -> Sends data to consumer
Broker is also the Kafka Server, it is a broker between Producer and consumer
Cluster - Can contain multiple brokers
Topic is a name given to a data set/stream
For example a topic can be called as Global Orders
Partition - Broker could have challenge in storing large amounts of data. Kafka can break a topic into partitions. How many partitions are needed, we need to make that decision for a topic. Every partition sits on a Single machine
Offset is a sequence number of message in a partition. Offsets starts from 0 for a message, they are local to the partition. To access a message Topic name, Partition Number, Offset number.

Consumer Group - Group of consumers , members of the group share the work
Retail chain
Billing counters
Producer for each Billing location
Sends message
Consumer will get the above messages
Create clusters and also create partition.
Consumer groups can then access a set of partitions


These are some basic concepts in Kafka...


Thursday, April 22, 2021

Data Ingestion-Hive/Impala/HUE

 With the emergence of all the big data, cloud technologies, there is a major push to get data moved from legacy systems to cloud. The reason for the migration is to provide better data access, allow for more advanced analytics, provide better insights and better customer experience. This type of migration is also to provide more self service capabilities. One of the initiative that i have been involved with is to move data from oracle databases to Hadoop. There are lot of pieces involved in moving data from RDBMS to hadoop. Here are some steps involved that i would like to share along with the tools involved.

1. Extraction of data from RDBMS to HDFS - What type of tools available: Sqoop works for pulling data from rdbms to HDFS, now one thing to watch out for is performance, how big are the tables. The data is extracted in the form of text files. So it is good to be clear of what type of data needs to be on HDFS. There are other tools like Zaloni(Arena),Atlan, some of these work well with Public Cloud providers.

2. Design the the structures where the files will be stored - The approach i am using is to utilize HIVE where in we have different schemas depending on the classification of data. Create tables in thee schemas and pull i the data using the text files created from Step 1. There are 2 main types of HIVE tables - Internal and External. HIVE supports SQL, it is very similar to SQL in RDBMS with few variations. There is UI available called HUE which is Hadoop User Experience. This is web based, so with the right permissions, one can connect to HIVE. Once you login, you can view the schemas/tables that have been created in HIVE. You can write SQL type queries and view results. HIVE is Apache based, where Impala is  Cloudera based. The difference is that in HIVE you have metadata and query capability whereas in IMPALA you mainly work with queries

3. Typically when you first load data into HIVE, they are text files, basically in Text format. These text files are loaded into tables. Now when you are ready to make the data available for processing, i decided to create separate schemas where the data is still stored in tables but it is stored in Parquet format. This allows for compression , it is in columnar format. For analytics purposes this works well and other thing is for our installations we use parquet. There are other big data formats that can be used:

Avro, Parquet, ORC, see the link below for comparisons.

https://www.datanami.com/2018/05/16/big-data-file-formats-demystified/

4. Once Step 1 through 3 is completed then you are in a position to start provisioning the data, typically this data can feed into a data lake. The data lake can be on premise or it could be a private or a public cloud based on the business need,

Monday, February 8, 2021

Data Pipelines/Data Transformation

 Data is available in abundance in organizations, more so in bigger companies. How do they make good use of the data to get valuable insights and add business value is the key strategic question in companies today. Early on there were lot of data warehouses, data marts built using complex ETL techniques, then there was this concept of ELT (Extraction, Loan and Transform) which was used to transport data. With the advent of Big data, AI/ML techniques there has been continuous need to improve the data integration in order to have successful data projects. One of the challenges companies have had is how to handle the flow of data in order to gain maximum value. There have been approaches such have one set of folks help with the sourcing of data, another set to transform and finally have the data science folks figure out the value. This approach has caused too many hand off points, and also caused lot of silos of expertise. In order to address this problem, a new concept is being used lot of organizations recently, it is called the Data Pipeline.

What is a Data Pipeline: It is the complete end to end process of getting the data from source and build the complete lifecycle (Source, transform, data quality checks, AI/ML model generation and Data visualization). Using this concept, resources are now being engaged to manage the complete pipeline, Data scientists/Analyst are being encouraged to manage and/own data pipelines. There are different tools available that help you manage the data pipeline. In some cases the data pipeline is also referred as workflows. The tools that are available today that help with these concepts are dbt:getdbt.com (Analytics engineering is the data transformation work that happens between loading data into your warehouse and analyzing it. dbt allows anyone comfortable with SQL to own that workflow.), please refer to https://docs.getdbt.com/docs/introduction.

Matillion,https://www.matillion.com/, Atlan (Data Management Tool), Arena from Zaloni and host of other tools like Collibra. The tools here help provide a complete end to end perspective of the data that is being used for different types of data projects. The tools mentioned above are cloud ready, with companies making the move to the cloud to store all of the data, the adoption of the above tools hopefully is much easier.

When a data scientist and/or analysts has the exposure to the tools mentioned above, they get a complete perspective of the data, understand the lineage which would help in building out better AI/ML Models that can be used by the companies. Lot of AI/Ml efforts fail because of bad data, unable to understand the lineage and dependencies. One of the key aspects to keep in mind while working on data projects is Data Drift, what does this mean? Data is not static, it keeps changing constantly, the structure of the data could change. There can be changes in schema, the granularity of data and the volume of data can keep fluctuating. When you have the tools that have been mentioned in the blog post earlier, they help understanding these changes to a great extent and help tuning the AI/ML Models. There is a lot more research made in the are of data drift and related tools. I will update on those in a different blog post.


Monday, January 11, 2021

Data Changes/Data Quality-AI/ML

 With AI/ML getting increased adoption in organizations, Data is fast becoming the most sought after asset. The flow of data is becoming more continuous, just like opening a tap and water flows.One of the trends i have noticed in data projects is that once the desired goal is achieved, it seems like there is a tendency to think that it is done. This is where the problem starts, data is not static, it is subject to changes or continually changes. When changes in data happens, it can affect the underlying structure or grain of the tables which are used in AI/ML Projects. If these changes are not determined or tracked upfront, it can cause the AI/ML Models generate inaccurate results. So the AI/ML models which were very efficient three months ago , now are not accurate anymore.

Tracking data changes either in terms of format coming from a vendor or schema changes in the source is very time consuming and inefficient/manual. There is very good opportunity to identify this as a problem area and look for if this gap can be addressed by providing automated solutions. With infrastructure and related components we have lots of utilities/dashboards that indicate if certain parameters are reaching a limit nd adjustments are needed. Using the same analogy we would need to have tools that can tell the data teams/business users any changes happening in data. This would help the data teams spend more time on tuning/optimizing the data models and not spend too much time on manual/inefficient tasks. Handling data changes is very critical for product management, with increasing reliance of product management on data. This is a little more of a abstract post but these are high level patterns that i have seen in the data related projects.

Tuesday, December 15, 2020

AI/ML Framework

There is a lot of focus on Artificial Intelligence, Machine Learning in various organizations and they want to get better insights and value out of such efforts. There are significant investments made in terms of time, money to get AI projects started and delivered. One of the key aspects to kept in mind , the success of such projects depends on sourcing the right date, have the right data governance and make sure such efforts align with the business goals. Given such a environment it is imperative that there is a general framework in place as to how AI Projects are executed, so that there are truly value added deliverables being achieved. In a Data world we have the following broad roles trying to make a AI Project a success.

Possible Personas in a data project.

1. Data Engineers
2. Data Analysts
3. Data Scientists
4. AI/ML Developers, Model Creators
5. End users/Reviewers of Model for Compliance and regulations

Each of the above personas are going to be involved in different stages of a data project and would be utilizing different tools to achieve the end goal. When we talk about different stages we could identify that the following steps would be there in any data project, they are:

1. Sourcing data - Sample Tools IBM DataStage, Informatica, Structured Database Oracle, SQL Server, Teradata, Unstructured Data: Infobase
2. Organize data/Metadata Management/Lineage Analysis/Cataloging - Trifacta/Alteryx for Data Wrangling/Prep, Data Catalog/Governance Atlan/Collibra
3. Build Model/Experiments/Analysis - SAS, JUPYTER (Python), R, H2O
4. Quality Check, Deployments of ML Models - Model Frameworks Such as H20, Python, R
5. Consumption of Data By end Users - Tableau, Cognos, Microstrategy

All of the above would require  storage and consumption component, these could be Hadoop/Spark, AWS/SnowFlake, Azure or Google Cloud. 

When you organize the different tool sets and personas it will provide an overview of what is available and how AI projects could be structured to deliver effective value. For example if there is a lot of need to consume unstructured data and derive data from it in a financial institution for example, once could look at leveraging the Instabase Platform : Quoting from the site: The biggest opportunity for efficiency within a business is stifled by document processes that only humans can do. With Instabase, you can build workflows to access information in complex documents and automate business processes.https://about.instabase.com/.  
Providing a platform where all of these tools can be used so that different personas can have access and move data from one point to another would be of great help for any data project. This would allow consistency of operation , help in tracing data, provide better model validation, make sure any audit/compliance requirements are met.
Successful data projects that include AI/ML have the above ingredients in the right mix, well tracked/cataloged and also take care of changes in data over time plus closely align with the business objectives.
Happy Holidays and  a Very Happy, pandemic free New Year 2021, Stay safe everyone.


Monday, December 7, 2020

Snowflake - UI Components

Snowflake is fast becoming a very important component of the cloud migration strategies in the business/technology world today. I have been in discussion with different leaders, also in participating in different conferences there is a lot of interest in Snowflake. One good thing is that there is a trial period for snowflake which one can sign up for. You can go to the Snowflake web site and sign up for the trial. The sign up process for the trial is very straightforward and once you have it set up, you are provided with an option to go through the introductory material related to snowflake and there are some very good documentation related to different areas in Snowflake. Once you sign into snowflake you should see the following interface. There are Different components that are listed in the interface, lot of it looks very similar to the sql server management studio layout especially the object explorer. The main components in the UI are:

1. Database - Lists the Databases in Snowflake Instance.
2. Shares - This is related to Data Sharing within your Organization.
3. Data Marketplace - This option allows one to look at the Snowflake Data Marketplace, which lists the public data sources that are available to different categories like Government, Financial, Health, Sports
4. Warehouses - Lists the warehouses available on the snowflake instance.
5. Worksheets - The option where one can write SQL Queries like below. There is a sample database called DEMO DB which has list of tables that can be used for querying. Also these queries can be used for building out Dashboards that is available in the Data Market place option.


One can also load data into snowflake based on the different data loading strategies that are available which has been discussed in earlier blog post written by me: https://www.blogger.com/blog/post/edit/2437651727370625818/3834832172078982453




Wednesday, November 18, 2020

Data Cloud

 One of the more commonly used buzz words these days is Data Cloud, it has been used as a marketing term mainly in the Cloud Domain across different business/organizations. There is a concept underneath the word Data cloud, it is mainly aimed at having data available in/migrated in the public cloud offerings such as Amazon, azure, Google Cloud. One of the key projects that have been undertaken by lots of organizations across different Business is how does one have data available in the cloud without compromising on the security aspect of data. I had the opportunity to attend listen to the data cloud summit 2020 organized by Snowflake. It was a virtual event organized by Snowflake where features of the snowflake was discussed in different sessions. There were also use cases presented by different Customers, Vendor Partners on how they are utilizing snowflake for the data projects and how much impact has this product had on their Business. There were some interesting points that i had picked up from different Sessions, i am listing them below. They cover a variety of topics related to Data.

1. Compute/Storage: Snowflake Separates compute from Storage, this is one of the main concepts in this product which has been highlighted by lot of customers. How this concept helps them in their daily data operations and business.
2. Scalability - Ability to ingest multiple workloads, this is a common requirement across all customers.
3. Simplify: Simplification of the Data Pipeline. How can one get the raw data and turn them into actionable insights quickly, this is called the lapse time. One of the questions raised is that does all the transformation of the data happen during early hours in the morning? Can this be spread out or done in real time?
4. Data Silos: Breaking down Data silos is a significant effort that is being undertaken by different organizations. Data Silos has a direct/indirect impact on cost and efficiency in a very negative way. One of the reasons for using a product like Snowflake is to break down the data silios, having data data in one place. This would allow better understandability ad searchability for the data in an organization.
5. Proof of Value: Data cloud products or cloud offerings need to provide a proof of value. It has to be tangible for the business, how does the investment in cloud provide better results for the business.
6. Orchestration: Since there is movement to cloud infrastructure taking place at different pace, there needs to be better orchestration across multiple cloud installations. This can lead to better abstraction, this is the challenge lot of companies are facing today.
7. Data is an Asset: Data can be monetized in the following ways: Generating value for the Business, reducing costs.
8. Support: Snowflake provides good support tools, Cost Effective. Some of the customers explained how the uptime of snowflake has been very good in spite of the huge data loads coming into the system.
9. Data: What type of information needs to be sent out/provisioned. One of the guest mentioned there are two important aspects with respect to data: 1. Information that a person needs to know, 2, How the information will affect you.

Overall lot of information in a single day event, i am sure each one of the aspects mentioned above can lead to deeper discussions and/or projects. The event provided a overall perspective of where things are headed in the data space and how companies are planning their work in the coming years.

Saturday, September 19, 2020

Online Transaction History - Database Design Strategies

In todays world of technology one of the common occurrence in financial services is the concept of omni Channel. The basic premise is that customers can access information related to their accounts(checking/savings/Credit/Debit/Mortgage) information through various channels such as:

1. Financial Centers
2. Online Banking
3. Mobile/Phone Applications
4. Statements related to accounts (Mailed)
5. SMS/Email (where applicable)

When information related to accounts is presented via different channels like above, it is critical/obvious to have the customer experience consistent. Now looking at the technologies that are utilized to solve the above problem/create such experiences, API's have made a tremendous amount of penetration. The API layer has succeeded in making the customer request from the client applications/Phone Apps very seamless. Now these API's have to have a very good response time, for example if i am looking at the balance of my account through a phone banking app, the results need to come back quickly. In case response times are slow it will lead to bad customer experience. It is very essential that the Data Services behind these API's are very efficient. This in turn translates to have a very good database design (The databases can be on perm or on the cloud). Lot of times when use the applications and go to financial centers we tend to take these response times for granted. Recently i had the opportunity to work on designing a solution for a online/mobile banking channel to display transaction/statement information.
The data was going to be accessed via calling API/Web services by the client applications. The data resided in a exadata oracle platform.

The information needed for providing transaction information was coming from a vendor which gets ingested into the exadata database. In order to provide the information to the client, a process had to be run on the production database to aggregate the transaction information. Now the challenge was when these processes are running, if a client tries to access his transaction information, how does one make sure there is no distortion or breaking of the service call. Information still needed to be provided to the customer and there cannot be a time lag. In order to achieve this we had 2 options:

1. Perform a SYNONYM SWAP as part of the Oracle Procedure that is aggregating the information. Basically in this scenario, see example below, available in link: https://dba.stackexchange.com/questions/177959/how-do-i-swap-tables-atomically-in-oracle
We went with this option, the data was reloaded everyday, but we started to service call failures only at the time when the synonym swap happened.
2. We used this option, Perform delta processing of records every day and merge the changes into main table, use batch sizes during the final merge so that records are ingested into the main table in small chunks and that should minimize any contention of resources. In this option we processed only changed/new records and we did not perform any synonym swap. In this option, though it took a little longer for the job to run complete, there was no distortion of the service and the sla was well within what the customer expected. In order to get the accounts that have changed, we used a table to maintain the tables that are involved in the processing and capture the accounts that have changed in those tables.

These were couple of options we experimented with and we went with Option 2. It is very critical to design your database according to the expectations of the online/mobile applications. We experimented with multiple options and we narrowed down to the 2 options mentioned above.
In case you happen to read this post in my blog and you have any other suggestions, please leave a comment and i will definitely look into it.


Friday, February 24, 2017

Alteryx - Self-Service Data Analytics

There has been a lot of debate about ETL, how does it stack up with recent advancement in Big data technologies. One of the question being discussed how data can be ingested and how quickly can it be made relevant to the business. For many analysts in line of business  the process to blend and analyze data is slow and painful. It requires different tools and people to gather, cleanse data from different sources, lot of tools to build and publish analytic models. One of the products that handles this challenge is Alteryx. The product offers different capabilities: Prepare & Blend, Predictive Analytics, Spatial Analytics and Publish Insights.
Here is the link for one of their products:
http://www.alteryx.com/products/alteryx-designer

Quoting Alteryx: Alteryx Designer solves this by delivering a repeatable workflow for self-service data analytics that leads to deeper insights in hours, not the weeks typical of traditional approaches.
Please refer to the link to gain more insights into the Product Capabilities.

Sunday, December 25, 2016

2017 - Analytics Trends...

As we wind down year 2016 and move into 2017, there is lot of anticipation as to what will be the analytics like in 2017. 2016 saw a lot of development in the area of analytics with increasing range of applications in different industries/business. Big data in 2016 made significant inroads in 2016 and hope to see it solidify in 2017 with more number of business adopting such technologies. There was also lot of constructive debate around Big data analytics in 2016. There are more companies introducing machine learning technologies, one of the most prominent being Echo, Alexa from Amazon. There are other companies which are introducing machine learning capabilities in their applications.
Now i am providing a link which takes a lot what are the expected trends in 2017. One of the expected trend in 2017 especially for Developers in the BI/Database arena:
Traditional programmers will be required to gain data science skills in order to stay relevant, employable, and effective in their careers.
http://www.ibmbigdatahub.com/blog/big-data-and-analytics-trends-2017-james-kobielus-s-predictions.

Wishing every one a very happy and prosperous new year 2017. Hope to write more articles in year 2017.

Thank you

Tuesday, August 2, 2016

Data Science - Education

With the rapid growth of Big data technologies , there has been an exponential growth of data science and its related technologies. This has has also led to the demand for data scientists and also the jobs related to data science are very lucrative. Microsoft has been steadily expanding it s cloud based offerings and also getting into big Data related technologies and efforts. Since there is tremendous need for Data science skills, Microsoft has come forward to offer a curriculum  totally devoted to Data Science. This curriculum is offered via edx.org.  There are a total of 9 courses and the price per course ranges from $49-$99. There is also a final project for which around 6-10 hours is required. One can check the link below for all the details:
https://www.edx.org/microsoft-data-science-curriculum
The courses cover from Use Microsoft Excel to explore data to Iimplement a machine learning solution for a given data problem. Each of the course can be done a auditing course or one can upgrade to get a validity certificate on passing the course. Each of the courses have Labs ,Quizzes and discussion forums, the discussion forums can be use to get questions answered related to the concepts being discussed. I hope the courses provide the much needed insights into Data Science.


Sunday, June 19, 2016

Hadoop, Big Data...

I was provided an opportunity to write an blog article at analytics vidhya about big data related technologies. Analytics vidhya is very well maintained website about big data technologies. All the articles are well researched and presented. Please see the Link below for my article.

http://www.analyticsvidhya.com/blog/2016/06/started-big-data-integration-hdfs-dmexpress/

Hope the readers find the above article interesting and useful...


Tuesday, March 22, 2016

Data Science Resource/Blog...

One of the rapidly advancing areas today is Data Analytics/Data Science in the Business Intelligence space. There are lot of resources/tools that are coming explaining the capabilities of Data Science. Data Science has seen tremendous growth alongside the adoption of Big Data technologies. It is a very challenging space to keep up with, one of the blogs that i regularly visit to understand the concepts in Data Science and keep up with the trends in Data Science/Analytics, is http://www.analyticsvidhya.com/. The blog/website is very nicely laid out with content rich articles and tutorials. They cover a variety of tools and also encourage folks interested in data science to participate in challenges/contests. Please do visit the site, it provides amazing content and coverage.

Thursday, December 31, 2015

Big Data Trends in 2016...

First, I would like to wish the readers of my blog a very happy and prosperous new year 2016. Due to personal challenges i could not post much in 2015, i hope i can do better in 2016. Looking forward to how the year 2016 shapes up for analytics, if there are newer trends or consolidation of big and small technology firms in the analytics space. There were lot of strides made in data science, the programming language R has become increasingly popular across various industry domains, More business folks are beginning to understand the power of good models and how it can be pursued for predictive analytics. There were also strides made in bringing R and SQL Server together, this will be seen more in 2016 as we begin to see more valid business used cases.

Here is a link for the Top Big 3 data trends in 2016.

http://insidebigdata.com/2015/12/29/the-top-3-big-data-trends-of-2016/

Once again wishing everyone a very happy 2016.

Monday, November 16, 2015

Using Data Science Techniques with R- Zillow.com

R language has been gaining increasing popularity as Machine Learning, Predictive analytics take more and more importance within business. The applications of R has been steadily increasing in different industry domains. One of the areas where the R language is being very strongly used is in the are of Real Estate to predict selling prices of Homes Please read the link to blog post below related to how the company Zillow uses R for predicting estimates of prices for houses.
http://www.datanami.com/2015/08/12/inside-the-zestimate-data-science-at-zillow/
As the article indicates they use a mix of tools at different stages of the data value chain. The more cleaner the data is, the better the prediction is going to be.

Thursday, July 23, 2015

Big Data, Column Store, InMemory...

With the increase in data that is being stored and analyzed , the concepts such as Big Data, Column Store and In Memory databases have been in discussion or being used by different organizations. Also there are multiple vendors at play, the one size fits all doesn't seem to be working. Please see this video where in Michael Stonebraker talks about the different concepts such as Column Store, Big Data and In Memory databases. He specifically mentions not to give in to the marketing hype and use proper proof of concepts on site with the data that needs to be analyzed to arrive at the right solution needed for an Organization. Here is the link for the video, it is a bit old but I felt is very relevant in terms of the questions to be asked.
https://www.youtube.com/watch?v=_00H1cgXeWw. Mike specifically talks about how important columnstore is going to be in Data warehouses.

Saturday, July 18, 2015

SQL Server 2016 and R-Integration...Part 1

Earlier I had blogged about SQL Server and R can be connected to perform data analysis. Also in SQL Server 2016 there is supposed to be better R Integration. In today's market Data scientists/Data Wranglers are in great demand since organizations have piles of data and want to make sense of the data, also if data can provide valuable insights. Given these market trends and the push for aligning the business and data more closely it is important to have the analytic skills. In this blog I would like to provide an article which gives a flavor of sql server 2016 and R-Integration. In the link below:
http://blog.revolutionanalytics.com/2015/05/r-in-sql-server.html. The article describes SQL Server program managers Lindsey Allen and Borko Novakovic demonstrated a prototype of running R within SQL Server. For the R-Integration,Microsoft SQL Server has collaborated with http://www.revolutionanalytics.com/products. I hope you find the article and demo provided in the link above useful.

Monday, January 5, 2015

Happy New Year 2015...

Wishing all the readers of my blog a happy and prosperous New Year 2015. In this year one can see how the big data technologies mature or adopted within organizations. What happens to predictive analytics/machine learning type of technologies, how well they come into mainstream organizations. How is the role of data scientists going to evolve, along side with the traditional data warehouse technologies. There is a very good introduction course on The Data Scientist's toolbox that covers the foundation required for Data analysis and Data Scientists role.www.coursera.org.

Monday, October 13, 2014

Data Science/Data Scientist...

With the advent of Big Data technologies, the increased use of non structured data in business analytics, one of the roles that is beginning to rise in terms of demand is the Data Scientist. The area of study that a Data Scientist specializes is the area of Data Science. The huge amounts of data has called for more predictive analysis of the data instead of just focusing on the operational data. The power of predictive analytics if done right is that it gives the ability to a business to make better decision about future plans. For example how a company should target is products, what type of features does a product need to have to address the customer requirements. The emergence of data science has made Microsoft re prioritize the data mining features they incorporated into SSAS. All of the Data Mining algorithms are being re purposed into the Machine Learning tool in Microsoft Azure. I have explained about the Machine learning topic in my previous blog topic. In addition to Machine learning, to be make more effective usage of the tools available in Azure, it is important to understand the basics behind Machine Learning. Coursera offers a course on R Programming, it is free of cost and one can enroll by going to the https://www.coursera.org/ website. In case one is interested in going beyond learning the R language, Coursera has a complete Capstone course in Data science which includes learning about the algorithms behind data mining, getting introduced to machine learning apart from learning the R language. The Capstone course covers experimental design, statistical inference and regression models. This is one area which is growing very fast, so having a good foundation with courses listed above can provide a step into the predictive analytics world. One can also refer to the following link to understand the different types of algorithms in Data mining.
http://msdn.microsoft.com/en-us/library/ms175595.aspx

Data Science Capstone Course link:
https://www.coursera.org/specialization/jhudatascience/1?utm_medium=courseDescripTop