AGILE BUSINESS INTELLIGENCEOR HOW TO GIVE MANAGEMENT WHAT THEY NEED WHEN THEY NEED ITEvan LeybournAuthor Directing the Agile OrganisationMelbourne, ness Intelligence is a tricky thing. No matter how much effort is put into the design, planning orarchitecture of a business intelligence project, they rarely go to plan. This is where agile, as a productdelivery approach that is responsive to change, comes in. Agile is a generic term that describes over 50methods and frameworks for working in an adaptable, customer focused, and incremental way. In thecontext of Business Intelligence, these frameworks cover the full development spectrum; including strategy,planning, design, development, and quality control. Well-known Agile frameworks and techniques includeScrum, Extreme Programming (XP), Test-Driven Development (TDD), and Kanban.SOME BACKGROUND TO AGILEFirstly, it is important to understand that Agile is a value system – not a process. Developed in 2001, theAgile Manifesto1 consists of 4 core values and 12 principles and forms the basis of all agile frameworks.126.96.36.199.We value individuals and interactions over processes and toolsWe value working software over comprehensive documentationWe value customer collaboration over contract negotiationWe value responding to change over following a planThe values on the right (processes, documentation, contracts and plans) are still important to successfuldelivery; however, to be Agile, a greater appreciation of the values on the left (individuals, workingsoftware, customer collaboration, and responding to change) is needed.Supporting the 4 core values, are the 12 principles that define the agile mindset. These are the key businessattributes that are most important to Agile Business Intelligence teams.1 Agile Manifesto, Beedle, et al: http://agilemanifesto.org/
188.8.131.52.184.108.40.206.220.127.116.11.Our highest priority is to satisfy the customer through early and continuous delivery of valuablesoftware.Welcome changing requirements, even late in development. Agile processes harness change for thecustomer’s competitive advantage.Deliver working software frequently, from a couple of weeks to a couple of months, with apreference to the shorter time-scale.Business people and developers must work together daily.Build projects around motivated individuals. Give them the environment and support they need,and trust them to get the job done.The most efficient and effective method of conveying information to and within a developmentteam is face-to-face conversation.Working software is the primary measure of progress.Agile processes promote sustainable development. The sponsors, developers, and users should beable to maintain a constant pace indefinitely.Continuous attention to technical excellence and good design enhances agility.Simplicity, the art of maximising the amount of work not done, is essential.The best architectures, requirements, and designs emerge from self-organising teams.At regular intervals, the team reflects on how to become more effective, then tunes and adjusts itsbehaviour accordingly.Understanding these 12 principles is critical to understanding Agile, and Agile Business Intelligence.AGILE FRAMEWORKS FOR BUSINESS INTELLIGENCERegardless of the framework, Agile Business Intelligence projects or teams use Just-In-Time planning, andan incremental, or iterative, delivery process, which allows for rapid change when scope and circumstanceschange. Customers work alongside the team to shape and direct the outcomes, while the Team regularlydelivers partial, though functional, products to the Customer. The product itself will continue to evolve, aseach Iteration builds upon the last.The customer’s responsibilities also change within an Agile Business Intelligence project or team, to take ondirect responsibility for the delivery of their requirements. Teams and customers work closely together,collaborating towards the desired outcomes.Depending on the context, and organisational requirements, Agile Business Intelligence can utilise anynumber of, complimentary, agile frameworks. Common frameworks include Scrum (for its incrementalproduct development processes), Kanban (for its continuous workflow management process), Test-DrivenDevelopment (as a mechanism to embed quality control in the work cycle), and Extreme Programming (toprovide sustainable delivery within teams). Below is a brief background to each of these frameworks.
Scrum2: Primarily used as a projectmanagement and productdevelopment framework, Scrumdescribes a framework (as shown inFigure 1) for the incremental deliveryof complex products. The Scrumframework is primarily team based,and defines associated roles, events,artefacts and rules.Kanban3: Kanban (カンバン), whichFigure 1: Scrum framework – Mountain Goat Software (CC-AT)approximately translates as‘signboard’, is described as a ‘visualprocess management system that tells what to produce, when to produce it, and how much to produce’. Atits simplest, each prioritised Task (or Card) on a Kanban Board passes through a visualisation of the Team’sprocess, or workflow, as they happen. Each primary activity in the Team’s workflow is visualised as columnson the Kanban Board, usually starting at Task definition, and finishing with delivery to the Customer.Test-Driven Development4: Test-Driven Development (TDD) is primarily asoftware engineering process that forces programmers to write small,incremental verification tests prior to writing each function of the software.Each set of verification tests defines the outcome of a single feature orimprovement. This ‘test first’ approach encourages simple design, concisedevelopment, and confidence in the product.Extreme Programming5: Extreme Programming (XP) is an Agile developmentframework which, like all Agile frameworks, advocates incremental deliveryand responding to changing customer requirements. XP’s focus is on themethod and role of the delivery team, and defines basic activities within thesoftware development process.There are other development concepts, not directly associated with agile, thatshould also be considered to ensure the success of a business intelligenceproject. These include Unit Testing, Continuous Integration or CodeGeneration.Figure 2: Test-DrivenDevelopment flowchart2 Scrum Guide, Schwaber and Sutherland (2011).3 Kanban: Successful Evolutionary Change for Your Technology Business, Anderson (2010).4 Test-Driven Development by Example, Beck (2003).5 Extreme Programming Explained: Embrace Change, Beck (1999).
BUSINESS INTELLIGENCE PLANNING AND DESIGNPrior to beginning any Agile Business Intelligence project, and to minimise later rework, an initial productbacklog, or requirements backlog, needs to be defined by the customer. The product backlog describes thecurrent and best-understood requirements, sometimes known as user stories. The highest priorityrequirements, those that will be delivered next, are then expanded with additional detail. This ensures thatthe outcomes at the end of an iteration are always usable, and deliver value to the customer.To contain the impact of change, each business intelligence user story is initially defined at a high level. Thelevel of detail required should contain sufficient information to help prioritise and identify problem areasfor prior to beginning the process, without wasting effort. For example:1.What quantity of data is being dealt with? - e.g. a CMS with 1,000 records or a financial system with100,000,000 records2. How quickly does the data grow? - A CMS which adds 100 new pages a day will be easier to managethan a financial management system which add 1,000,000 records3. How frequently does this data change? - Compare the financial system in which the data neverchanges (and so there is no need to manage data revisions) to a CRM in which the data changesdaily4. When does this data change? - We can optimise the ETL process if we know the data is updated atthe end of the month, rather than try and ETL the same data every day.5. How much of this data is duplicated elsewhere? - If the data is duplicated in another system, andthe other system is authoritative, we can simplify the ETL process.6. How much of this data is obsolete or irrelevant? - This comes down to GIGO. If the data is obsoleteor not relevant to current decision making, including it in the data warehouse would be counterproductive7. How is this data used? - This question is used to inform the necessity/benefit of the data andshould be used as part of the cost-benefit analysis.8. What reports currently use this data, and are they satisfactory? - Often the transactional system(e.g. finance system) provides its own reports. If these are satisfactory and do not need to strengthof a BI system behind it, it de-prioritises the data source for extraction.9. What access is available to the data, and how can we extract it? - This is a purely technical questionthat informs the effort involved in the ETL process. Is it a database, can we connect to it directly,via ODBC, do we need software engineers to write extraction scripts etc?AGILE TEAMSThe key distinction between Agile Business Intelligence and traditional business intelligence teams is theuse of a cross-functional team structure. A cross-function team is one where individuals with different, butcomplementary, skills, work together as a team and are empowered with personal authority andaccountability. Each team should contain all the key skills required to deliver on the business intelligencerequirements. Each cross-functional agile team is typically between 5-9 full-time staff, where the wholeteam works towards a single, specific outcome.Benefits to this integration include faster delivery times, rapid response to new issues, and improvedinformation sharing across the organisation. However, to realise these benefits requires that all team
members are committed to the outcome, and adaptable in their role. This is different to the traditionalhierarchical or matrix management structures, where one team would start the process, and at predetermined stages, request input from, or handover to, another team. By passing work between silos, strictmatrix structures lack consistent ownership of work, cause poor communication between departments, andincrease delays in the overall process.Each cross-functional team also consists of both technical and business people. By utilising business andtechnical experts working together, outcomes can generally be met sooner and more accurately than if theteams were separate. Roles may include: Business Analysts - the interface between the BI teams and external business teams Business Specialists - often a subset of the business analyst, business specialists (or subject matterexperts) are sourced from the business teams responsible for creating or consuming theorganisational data, e.g. call centre staff, program / policy staff, or retail staff Report / Technical Writers - only required where high quality and complex reports for externalbusiness teams is needed. It is also common for the BI team to provide a “self-serve” tool Statisticians / Demographer - only required if when dealing with complex data and there is a needto do in-depth statistical analysis BI / ETL Specialists - specialist team members who understand the organisational BI and ETL tools,be it Pentaho, Cognos, Oracle BIEE, etc Data Analysts - responsible for reverse engineering data sources, maintaining the databases andcubes and interpreting business need into information required Systems / Database Administrators - responsible for maintaining the BI servers and DBMS'Please note that these are roles and responsibilities, not necessarily individual people. e.g. an ETL specialistand data analyst can be the same person, as can a data analyst and DBA. As for organisational structure, anAgile Business Intelligence team would ideally report to a business manager, rather than a technicalmanager or CTO. The technical manager will generally be focused on the systems and technology, whereasthe business manager will be focused on the information outcomes.TEST-DRIVEN DEVELOPMENTFormal quality control and testing frameworks, such as Test-Driven Development, are a critical componentof Agile Business Intelligence. The consequence of poor testing is relatively well understood and includes;inefficient design, significant data integrity and validation issues, and defects within ETL and reportingscripts.Dedicated testers are embedded within each team to undertake quality control while still ensuring aseparation between the doers and testers. One dedicated tester per team (between 5-9 people) should beable run complete quality control tests at roughly the same rate as requirements are completed. High-riskor high-complexity requirements may require additional testers per team (or smaller teams); play with theteam numbers until equilibrium is reached. That is, where everyone is busy, but no one is being abottleneck.Test-driven development (TDD) aims to improve business intelligence quality by focusing on early andregular testing. The core premise is that automated or “unit” tests are written before each softwarecomponent. Therefore, the development process should look something like this:
18.104.22.168.5.6.Write the (failing) testRun all tests (ensuring that the new test fails)Write the codeRun all testsRepeat steps 3-4 until the test passesRefactor the new code to acceptable standardsTo make this work within a BI context, it is important to define user stories as discrete components. Thefirst step is to define our test scope, which at its simplest is an ETL script that, given a known input, willoutput data that is reasonable and within expected norms. This scope can include: The number of results - e.g. we know there are 1,000 records in the data source which change daily,so our ETL should extract approximately 1000 records each day The scale of results - e.g. financial transactions must be between 1 and 1000 The relationship of results - e.g. the average or standard deviation should be similar each day Logical tests - e.g. the sum of all transactions for a customer should equal the customers balance Known good tests - e.g. customer 123 exists and has a known form Relationship tests - e.g. that each transaction has an associated customerUnder the TDD framework, these tests should be written before any ETL scripts (though usually after themodel/schema has been designed). It has been shown that this encourages developers to create simplerdesigns and inspires confidence in their work.To get the greatest benefit out of TDD and to ensure a robust testing process, all tests should be setup andrun through good continuous integration environment. This will automatically run the tests on a scheduled(or triggered) basis and alert the BI administrators of any failing tests, which can forewarn of issues in thedata sources and data warehouse.RISKSAs with everything, Agile Business Intelligence teams and projects are not without their risks. Indeed, whileAgile brings many benefits, the impact of some risks are increased. Some risks to be aware of include: Lack of customer engagement: Where projects stalled while waiting for clarification or decisionsfrom the customer Competing priorities: Where assigned team members also have conflicting BAU responsibilities Low team morale: Low morale (for example, due to lack of delivery or a low understanding ofproject value) can affect team productivity and, in turn, can lead to even worse morale Insufficient allocated time for delivery: Where more work has been allocated than can be deliveredin a given deadline. It can be helpful to visualise a project teams as a pipeline. The width of the pipeis the team size and the length is the time available to deliver. If an estimate is incorrect or a newrequirement comes into the pipe, the lowest priority requirement will fall out the end. In Agileterms, the velocity of each team doesn’t change just because they are given more work Unfocused development: Where developers go off on tangents building “frameworks” that add nodirect value for the customer Poor quality (or inconsistent) test data: Poor test data causes inaccurate tests results (both falsepositives and negatives) resulting in rework
SUCCESS MEASURESFinally, it is important to measure the success of an Agile Business Intelligence project. A set of specificsuccess criteria, with measurable targets, need to be defined to validate when the Agile BusinessIntelligence goals have been achieved. Success criteria should be concise, realistic, directly measurable andinclude both quantitative measures, based on facts & figures, and qualitative measures, based on feedback &opinion. Agile specific maturity measures include: Staff are trained & experienced in Agile Business Intelligence and associated methods.Staff have an understanding of the underlying reasons for moving to Agile Business IntelligenceStaff are directly empowered to engage with & deliver to the customersStaff are conversant in the work, QA, and release proceduresCustomers have been trained in their new responsibilitiesCustomers (or their representatives) are involved in the teams daily activitiesCustomers actively define & prioritise requirements at least once per IterationClearly defined business processes exist for all domainsAs a final note, not all BI projects are suitable for an agile approach. Projects and teams with low morale, nostaff or executive buy-in, high staff turnover, or a lack of trust between themselves & the customer, need toaddress these issues before beginning any Agile Business Intelligence project.- Evan Leybourn, 2013
SOME BACKGROUND TO AGILE Firstly, it is important to understand that Agile is a value system – not a process. Developed in 2001, the Agile Manifesto1 consists of 4 core values and 12 principles and forms the basis of all agile frameworks. 1. We value