25 - 26 April 2018 / Manchester Central

Register Now

The Convergence of Data Science and Software Development

Thursday 01 March 2018

By Margriet Groenendijk , Developer Advocate at IBM Watson Data Platform


There is no need for data scientists to be outright software experts thanks to emerging tools for using data delivered through the cloud. Making use of the cloud as a shared platform ensures the delivery of the right data to the right people at the right time, without the inefficiency of workflows passing through many hands through hand-overs.

Using cloud services to build a user-friendly and productive collaboration platform means that data scientists can focus on their core skills to derive insights and useful knowledge from data. Using these new tools, data scientists can begin to think of their role as one running parallel to their developer colleagues – both working to deliver business value to end users keen to take advantage of the amount of data being produced.

A marriage between technique and technology

Data scientists and developers work on different parts of the same workflow. Data scientists explore the data for new insights, and developers use these insights to automate the workflow and create apps. Closer collaboration is vital to fully capitalise the potential of the vast quantities of data now available and make the process of app creation as efficient as possible.

A cloud-ready tool encouraging the quicker delivery of data through collaboration is the Jupyter notebook . Notebooks allow users to write and share code in different languages such as Python, R, Scala and Node.js  in one place. Data can be loaded from and saved to any cloud database, cleaned and processed to be used for prediction with machine learning models, and finally, the results can be published directly from a notebook as visualisations and APIs.

PixieDust  is the magic open-source ingredient to add to notebooks to speed up the exploration of data. PixieDust allows both data scientists and developers to quickly create data visualisations without any code and publish these as standalone web apps . This means that data becomes accessible to even non-technical end-users. Data presented visually lends itself more readily to the identification of business opportunities.

Collaboration results in innovation

Cloud continues to be on the rise, and with it comes the power to explore more data, and extract value faster. This cloud-facilitated potential is bringing closer together the roles of data scientists and developers. With this new efficiency, data scientists and developers have the capability to deliver creative, task-focused products faster.