IP EXPO Manchester 2019 - Artificial intelligence, Microsoft
3-4 April 2019 / Manchester Central

First Access >>

What the Data??? …

Monday 27 March 2017


IP EXPO Manchester speaker and Microsoft, Technical Evangelist Amy Nicholson on how the the world is once again transforming … Technology innovation is moving at an ever faster pace and the expectation of systems is far higher than ever before. Those businesses who are leveraging their data, actively gaining insights from it, and driving their businesses forward, are finding huge reward from their change in thinking...

The world is once again transforming… Technology innovation is moving at an ever faster pace and the expectation of systems is far higher than ever before. Those businesses who are leveraging their data, actively gaining insights from it, and driving their businesses forward, are finding huge reward from their change in thinking.

Is this digital transformation?…

The industry has been talking about digital transformation and the power of Artificial Intelligence for a while now and the main theme for my session at IP Expo Manchester on 26th April will be to show you how to build and develop services today that will set you on the front foot to embrace Artificial Intelligence and progress your digital transformation efforts tomorrow.

In this article, I want to introduce you to Cortana Intelligence Suite  …

Some of you may already know Cortana - either as Master Chief's right hand lady in the videogame series Halo or as Microsoft's Intelligent Personal Assistant . In all forms, Cortana is the brains of the operation and can understand information about you to make you more productive. 

And along with one of Microsoft's bold ambitions, to build the intelligent cloud, Cortana Intelligence was created. A fully managed set of Big Data and Advanced Analytics services in Microsoft Azure. Meaning that you can piece together these services, like a jigsaw, to build out your businesses data platform story.

From information management and big data stores, to advanced analytics and intelligence - these services can be pieced together to create data pipelines in the cloud that help you with traditional BI, predictive analytics, real time analysis and high scale data scenarios


Graph 1


But where do I store my data? ....

As we all know, data is now coming at us in different forms (variety), in huge amounts (volume) and at different speeds (velocity). So, the first part of building out solutions in this space is to pick the correct and suitable data store. With the cloud and Cortana Intelligence, you don't have to worry about the underline infrastructure and looking after machines, you can setup an account and start pushing data into it.  So whether your looking to store relational and transaction type data in an Azure SQL Database or NoSQL, document/JSON based data in Document DB , or looking into Big data stores such as Lakes and Warehouse s; there is a right tool for the right job available in Microsoft Azure.

Check out a great series of technical talks on Channel 9 as part of an event we ran called TechDays Online . Myself and a colleague, David, talked in detail about all the data stores on Azure and how and where to store your data: https://aka.ms/azure-data-stores

Do I just use each service separately? …

The key to building out data architectures in Microsoft Azure, is to scope the aim of your data project and start piecing together the services that best support building out your solution.

The services you choose will often depend on the characteristics of your data, the characteristics of your project and the skill sets your team has that can be leveraged. For example, see some of the services I would use when looking at different data projects:

Traditional Business Intelligence -  Moving data into the cloud on a cadence, applying ETL to tidy up the data and storing in a SQL Data Warehouse for batch calculations and direct reporting from Power BI visualisation tool


Graph 2


Predictive Analytics - Considering a data store, such as a data lake, to store historical data and transactions that might be useful retrospectively to apply machine learning and predict what might happen in the future. Depending on your teams skill set you could use the fully-managed Hadoop distribution on Azure, HD Insight and Apache Spark. Or, if you are getting into this space, look at Azure Machine Learning , a UI based machine learning service that allows you to build out machine learning models and publish them as APIs


Graph 3


High Scale Data - like the characteristics of predictive analytics, choosing between data stores for data types (structured/unstructured) and then choosing a big data analytics service that suits your skill set. Using your Hadoop skills with HDInsight or, if you are from a C#/SQL background, Azure Data Lake Analytics means you can run analytics jobs, not clusters, and query over them using a language called U-SQL (a combination of C# and SQL)


Graph 4


Real Time Analysis - ingesting the data into the cloud in real time and then processing and aggregating over that data as a service. The complexity in the underline infrastructure for real time data analysis is taken away so that you can concentrate on the project at hand and the queries you are executing. Again, the option to use your great Hadoop skills in the Apache Storm space. Or have a look at Azure Stream Analytics, a fully managed service that allows you to query over real time data using T-SQL.


Graph 5


What skills do I have in my team? …

As I have eluded to above, sometimes the choice around the service or component you choose is based on skill sets. People with Hadoop based skill sets can hit the ground running on the Azure platform, spinning up Hadoop, HBase, Storm, Spark and R Server clusters, applying your computations and once complete removing the need for those clusters of machines.

But if you are new to this area of technology, the barrier to entry can be lowered slightly by leveraging skills you already have (such as SQL, T-SQL and C#) and choosing other platform-as-a-service offerings such as Azure Stream Analytics , Azure Data Lake Analytics and Azure Data Factory .

If you are looking to get yourself or your team skilled up on these different technologies a great place to start would be with the EDX Online courses Microsoft has provided: https://www.edx.org/school/microsoft

Embracing experimentation…

Embracing the need to experiment and build Proof of Concepts (POC) in this space is how we have seen customers become successful. Sometimes projects that involve data science and machine learning skills can be close to research and therefore experimentation with different approaches may be needed. Even though one experiment may not pay off, often the team and business is a step closer to producing software and services that will help the business and your customers be more productive.

Here are examples of a few customers Microsoft has worked with on POC projects, where Microsoft and customer development teams spend a week together building out possible new solutions to business needs: https://microsoft.github.io/techcasestudies/

Want to know more? Join Amy Nicholson’s session ‘What the data??? ’ at IP EXPO Manchester on 26th April 2017, where she will build on this conversation and show some of the technology in action.