The agile approach relies on fast feedback to drive iterative decision making, but when working on complex or legacy systems, feedback can be difficult to gather and to interpret. Following a release, how confident are you that you haven't introduced a subtle new bug that may not be immediately obvious, but will have a significant impact over time? Also, how sure are you that bug fixes really improve your system? Do you have a clear idea before a release of what constitutes success and failure?
Data science can help us to make better assessments on the impact of releases and better decisions on whether to roll back or not. But do you need to be a maths specialist to use data science? I will be arguing that, with the right tools and a willingness to learn, non-specialists can benefit from adopting a data science approach.
I will be presenting a case study of data science work our team undertook on a complex system involving enterprise integration with a third party, where errors/resends were common and it was difficult and time-consuming to directly monitor the impact of releases on the third party front-end.
The main purpose of the session is to discuss how to take a more rigorous data science approach to measuring the impact of releases, in particular, the importance of having clear definitions of success and failure before a release. I will also be describing the different data science methods which we used to analyse this data pre and post release, specifically hypothesis testing vs Bayesian methods. I will be discussing the differences in the 2 different approaches and the advantages and disadvantages of each.
This session will be suitable for everyone; no specialist knowledge of maths is needed. As well as the presentation of the case study, I will then lead a broader discussion on the use of data science methods to analyse feedback, the challenge of using data analysis techniques when you do not have specialist maths knowledge, and tools and resources that can help.
Karen Pudner is a software engineer at the BBC, with a particular interest in data science and how it can be used to improve decision-making.