Monday, February 08, 2016

Specification by Example in Actuarial Modelling

Specification by example is a collaborative approach to defining requirements and business-oriented functional tests for software products based on capturing and illustrating requirements using realistic examples instead of abstract statements. Specification by example is also known as example-driven development, executable requirements, acceptance test-driven development (A-TDD), Agile Acceptance Testing or Test-Driven Requirements. I prefer A-TDD but will use both specification by example and A-TDD.

Examples as a single source of truth

A key aspect of specification by example is creating a single source of truth about required changes or new functionality from all perspectives. When business analysts work on their own documents, software developers maintain their own documentation and testers maintain a separate set of functional tests, software delivery effectiveness is significantly reduced by the need to constantly coordinate and synchronise those different versions of truth.

With Specification by example, different roles participate in creating a single source of truth that captures everyone's understanding. Examples are used to provide clarity and precision, so that the same information can be used both as a specification, documentation and a business-oriented functional test. Any additional information discovered during development or delivery, such as clarification of functional gaps, missing or incomplete requirements or additional tests, is added to this single source of truth. As there is only one source of truth about the functionality, there is no need for coordination, translation and interpretation of knowledge inside the delivery cycle.

When applied to required changes, a refined set of examples is effectively a specification and a business-oriented test for acceptance of software functionality. After the change is implemented, specification with examples becomes a document explaining existing functionality. As the validation of such documents is automated, when they are validated frequently, such documents are a reliable source of information on business functionality of underlying software.

Specification by example is very usefull to projects with sufficient organisational and domain complexity to cause problems in understanding or communicating requirements from a business domain perspective. It does not apply to purely technical problems or where the key complexity is not in understanding or communicating knowledge. There are documented usages of this approach in domains including investment banking, financial trading, insurance, airline ticket reservation, online gaming and price comparison.

This brings me to my very succesfull experience with A-TDD within an actuarial modelling project. To be specific the creation of a new MCEV model for a  life insurance company.

MCEV Modelling

The Embedded Value (EV) of a life insurance company is the present value of future profits plus adjusted net asset value. It is a construct from the field of actuarial science which allows insurance companies to be valued. European embedded value (EEV) is a variation of EV which was set up by the CFO Forum which allows for a more formalised method of choosing the parameters and doing the calculations, to enable greater transparency and comparability. Market Consistent Embedded Value (MCEV) is a more generalised methodology, of which EEV is one example.

Depending on how you implement such a model one part of it is the future cash flow generation component. Input for this component is a coded feature vector of an insured person and his/her contract. Based on this information the component computes the expected cash flows for this insured person for t=0 (now) untill t=40 (40 years from now). Very simplified you could say that the cashflow component computes premiums, benefits and total savings for each person in any given year. Afterwards an ALS component computes Profit/Loss Statements and Balance sheets based on given Assets, Liabilities (the cashflows) and Scenarios (interest rates, market behaviour, company behaviour etc). 

As you might allready have noticed the cashflow component seems to be representable by input data, model, expected output data. This is exactly what we have done. We create one table containing input examples. We expressed business rules and model in tables to make them more comprehensible and assists in finding missing cases. And we defined an expected output table with one row for each input example. Based on this the model could be coded in in the modelling software of choice and tested automatically.

PersonIDDate of BirthGenderProductStart DateCurrent Savings
1
2
3
4
5
6
Table with Input Example

PersonIDPremium t0Benefits t0Savings t0...Premium t1
1
2
3
4
5
6
Table with Expected Output

Implementation

So how did we got those tables? Since we used Scrum at this project we mapped the A-TDD steps on a Scrum iteration as described by the guys from LeSS.

Discuss in workshop - Before the detailed Sprint Planning, the team, Product Owner, and other stakeholders clarify the requirements collaboratively in a workshop.

Develop in concurrence - Tasks for implementing the tests/requirements are created in the detailed Sprint Planning and implemented during the iteration. All activities happen “at about the same time.”

Deliver for acceptance - The working product increment—the passing acceptance tests—are delivered for acceptance to stakeholders and discussed together in the Sprint Review.

Automation

Successful application of A-TDD on larger scale projects requires frequent validation of software functionality against a large set of examples (tests). In practice, this requires tests based on examples to be automated. A common approach is to automate the tests but keep examples in a form readable and accessible to non-technical and technical team members, keeping the examples as a single source of truth. We did this by using Excel for our Examples and SharePoint for versioning.

This process is supported by most test automation tools which work with tests divided into two aspects - the specification and the automation layer. The specification of a test which is in this case a CSV file and contains the examples, their results and auxiliary descriptions. The automation layer connects the example to a software system under test. When you combine this with a Continuous Integration tool that runs all tests with each build of your software/model you will improve the quality and potential delivery speed of your project a lot.  

Conclusion

At the time I moved on to another project we had not automated the whole process 100% but came close. The whole process of A-TDD was received very positively by all team members. We got great cross learning and understanding between model coders, business analysts, actuaries and DB specialists. Besides that the next audit will be much easier since this kind of documentation is up-to-date, clear and detailed. Also regression testing of the model has become far much easier. So I can only recommend starting Specification by Example when you can code (parts of) your business in tables. It is definitely worth the effort.

Read more…

Friday, February 05, 2016

Why are we actually doing this project?

In a number of projects I have been part of, one of the most important questions is asked not at all or far to late. Why are we actually doing this project?

The given answer to this question was often incomplete or even incorrect. The latter sometimes even knowingly incorrect, i.e. an organizational/political/personal lie. I admit my experience is biased because I typically join projects at a point of time the project allready has some serious issues and my role is helping to solve these issues. But even in happy projects the question of why we are doing this project seems rather difficult to answer.

That is why I use a set of 11 questions that together will help you answer the big one. My list is based on the list of the 10 questions from the Silicon Valley Product Group when assessing product opportunities. I then adapted them to a project environment and added one question.

1. Exactly what problem will this project solve? (value proposition)
2. For whom do we solve that problem? (target market or target users)
3. How big is the opportunity? (market size or potential savings)
4. What alternatives are out there? (competitive landscape or alternative solutions/products/suppliers)
5. Why are we best suited to pursue this? (our differentiator as a company or project team)
6. Why now? (market window and urgency)
7. How will we get this project go live? (implementation strategy)
8. How will we measure success/make money from this product? (metrics/revenue strategy)
9. What factors are critical to success? (solution requirements, skill requirements, budget)
10. What are the main cost drivers? (people, licences, hardware)
11. Given the above, what’s the recommendation? (go or no-go, continue or stop)

When I join a project I will try to get answers to these questions. They will guide the project, give the team focus and will help with feature discussions and selling your vision. When things change you will have to rethink your answers. And sometimes you will just have to stop the project. Do not fall in the sunk cost trap.

Read more…