Agile Exercises – The Generation Gap

Agile Exercises – The Generation Gap

As a relative newcomer to the world of Agile – having worked over 15 years in the Data Warehousing Industry – I was more than excited and willing to explore this new methodology in my first Agile project.

To those inhabiting the world of agile development – the buzz words fall off their lips like something that others are supposed to know and understand as soon as they hear it. For example: SCRUM, Stand-ups, Dashboard, User Stories with Story Points, and features and tasks – these were the equivalent of a cultural shock to me. I did not understand them, and few people were willing to help me understand them – they didn’t have the time and if they did, they didn’t have the inclination – despite my having informed them at the start that I had no idea of Agile methodology. I don’t know whether it’s the fact that the people implementing this methodology either did not know what they spoke of, or they were unable to articulate it properly. Either way, it was no help to me.

Point in fact – take the project manager (um… Agile doesn’t do project managers – they’re called Scrum Masters) from one of my projects. She insisted I put everything into business like terms for the User stories. I told her – okay – this is what I want to do – I want to merge data from two tables, de-dup some data and then merge the rest. How would you put it in terms that a business user understood? She looked at me like I had gone mad. From the word “Master” attached to the word “Scrum” you would expect someone who would be a MASTER at the work that scrum was supposed to do! Sadly, that was not the case.

Perhaps this was an aberration – not the reality. So I decided to keep an open mind, and moved onto my next project… This time, I thought, this time I am prepared.

I was not.

I was given a task that under normal circumstances would take one person between 15-20 days. A chunk of design, development with complete unit testing. Despite my protesting that this particular task could not be completed in that period of time – I was given the directive – I had to finish it in 10 days. With all my years of experience, I realized that these so-called scrum “masters” did not have the foggiest clue as to what went into an actual development cycle. And be it agile or waterfall or cascade – they would have failed to understand what each person’s task involved. They failed to understand the complexity of the work, they failed to understand the nuances of estimating based on complexity, they failed to understand a simple concept – that design and development require different elapsed times for task completion.

Small wonder then, that it was with trepidation and a complete absence of expectation, that I approached my third project. I was reasonably sure that this was going to be an equally uphill task for me. To my pleasant surprise, I found a group of people, who not only understood Agile methodology, they also understood the philosophies and hence the history behind it – they practiced as they preached. They provided training for us. It was a welcome change to be able to understand exactly what the meanings of those buzz words were. They made their expectations clear. It was a very, very enlightening time for me.

But I was also a bit disturbed at their willingness to completely denigrate the waterfall methodology. Having worked in that world, and successfully at that, I was piqued when they constantly dismissed the method as a non-working, non-productive, non-profit-making method for software development.

It is this point that I had my epiphany. What was I hearing other than the generation gap between tried and true methods of software development – albeit methods that evolved with decades separating their evolution? Much like children whose parents do not understand their “language”, and who do not understand their parents’ language – this was the difference that I was hearing.

So here I am. Trying to make sense of what makes Agile tick and what made the waterfall method tock, and whether there is a mean between them – where the twain could meet?

My opinions here are solely my own – based on years of experience working with successful Enterprise Data Warehouse Development projects. So here I list what I feel are the pros and cons of either method as applied to EDW development.

The waterfall methodology assumes that each phase of software development – i.e. – requirements definition, analysis, design, development and deployment follow one another after the successful completion of a previous phase. The weakness in this methodology is apparent – results take a long time to reach the business who are paying for those results. The strengths here are also apparent:

–        Development takes the least amount of overall time, as everything is planned out and the work can be handed out to the developers who will develop to specs that are as complete as possible.

–        It allows time to build a framework for development – so that the developers could concern themselves with only the actual build.

–        It leverages each person’s expertise, and each team usually reports to a leader who allocates the tasks and is responsible for the delivery of each piece, thereby ensuring that each person is leveraged to their best ability.

–        There is less oversight on a micro basis on each person allowing them to focus intently on one and only one task at a time.

–        There is a deadline that has to be met, and expectations are clearly understood.

–        The whole team works towards one set of deliverables within each release of the software. 

–        Lulls are incorporated into the plan (if well planned)

–        Corporate politics and bureaucracy is an accepted fact of life – especially in those corporations that are large and already have well-entrenched enterprise architecture standards and processes. And there is usually person on the project team who deals with these issues.

–        In other words, there is a fair level of protection from internal politics for the delivery teams.

–        Big teams for each of the areas.

The agile methodology assumes that

–        Each set of tasks can be bundled up into a “user story” each of which can be completed over a set period of time – usually two weeks.

–        There are no team leads, only a scrum master who leads a team of folks usually about 10 people.

–        Not sure how development tasks are measured vs other tasks

–        Framework building seems to be an on-the-fly effort – which means not much thought is given to either the requirements or the design of this piece of work

–         Each person is supposed to take up tasks that are outstanding within a user-story and work on them.

–        Each day, there is a stand-up – 10-15 minutes where you recite what one has done, and what one will do.

–        There is extreme over-sight on each person’s work, because every single day one has to spend at least 15 mins updating their storyboards with the tasks they’ve completed or not

–        There are no lulls in agile. It is expected that one will be working x # of hours per week – regardless. The pace is non-stop. The burn-out is inevitable.

–        The only insulation against corporate politics and bureaucracy is the scrum master – and then even they might not know how to deal with technical stuff

I can see where Agile would work extremely well with regular software development.

-          With each release of software, newer features can be aligned, pulled in and worked on.

-          Each release can be as big or as small as needed

-          The data model can be as big or as small, as conformed or non-conformed as needed

-          Most software designers write “code” – scripts that are tucked into version control systems that allow any and all people to work on it. Since each piece is constantly evolving over time, this method of development works very well.

-          There is a sense of isolation within each scrum team, in that they work in their bubble and can be successful at it, since it is the same pieces of code they are working with.

-          There are few software systems that incorporate multiple technologies within the system itself. Although they may interact with multiple systems and technologies.

-          There is little demand to have a framework for data movement or integration.

-          There is a lower demand to manipulate terabytes of data on a daily basis as compared to an EDW.

It would be difficult to imagine these positives working in an Enterprise Data Warehouse.

An EDW requires that multiple systems feed into it. It requires that there are multiple software and hardware systems incorporated within the EDW platform – and deployment must account for each of these platforms. There are multiple access modes for an EDW – both canned and dynamic. Therefore, the foundations of an EDW need to be solid, yet flexible. Hence my crib about the foundational work or framework for the ETL portion. There needs time for this activity. Agile methodology cannot afford that.

Most EDW builds work within phases, i.e. they take a set of tasks that can be performed to provide results at the end of a three-to-six month period. The approach may be to take a source system at a time, a subject area (or part thereof) at a time or a report at a time. The results are usually in the form of a full deliverable – or as close to a perfect deliverable as possible – in other words – “done-done”. This is probably as close to a release period in comparison.

Perhaps the biggest difference in both methods lies in the oversight and daily tasks.

My complaint is that Agile is very micro-managed. And it is.

For example, I estimated the build of a complex ETL process to cover three days. My scrum master said – so for three days in scrum your only response will be working on task xyz? Yes, said I. Unacceptable, he said. My mouth dropped open. Can’t you break it down? He asked. No – was my response. Because in all honesty – how can you break down a particular piece of task  - which has already been unitized as far as possible. With waterfall method – I would not need to do this. More importantly, I would not need to explain to my lead why I need the time I do so. In fact, I would have been told that I had 3 days to complete the task.

Then there’s the scrum meetings. 10-15 mins every day for a group of around 6 six people sounds good, and it is. However, for a development team there is almost always the need to meet to discuss issues, logical flaws, business process understanding, design discussions. To me – the 15 minute stand up is a complete waste of time. Besides all our statuses are reflected in the daily dash board – which reflects exactly what we will say. I don’t get the need for redundancy in a supposedly lean method.

There is the end of sprint retrospective. Each team sits down and goes over what was good and what wasn’t. Lessons learnt etc. This takes the better part of half a day. Then there’s the sprint planning session which takes about a day. Ergo – with each sprint remove a day and a half of work spent in planning!

Here are some numbers for agile:

People on team                                                                                          60

Number of scrums                                                                                     12

People per scrum                                                                                       5

avg burn rate per person per hour                                                      90

Days per release (6 weeks)                                                                    30

Sprints per Release                                                                                    3

                                                                                                                          

Time/Money in Scrum Meetings                                                        

Time per person per release (hours)                                                 5

Per Day burn rate per person                                                                $               450.00

Time taken for 60 people                                                                        300

Dollar amount per release                                                                      $         27,000.00

                                                                                                                          

Time/Money per Sprint Retrospective                                             

Hours per Retrospective (assuming half day)                                 240

Dollar Amount per Sprint                                                                        $         21,600.00

Hours /Retro / Release                                                                            720

Dollar Amount / retro / Release                                                           $         64,800.00

                                                                                                                          

Time/Money per Release Retrospective                                         

Hours per Retrospective (8 hrs *  # of people)                              480

Dollars Per Retro                                                                                         $         43,200.00

                                                                                                                          

Total hrs spent in Planning Per six Weeks                                        1500

Total $$ spent                                                                                              $       135,000.00

                                                                                                                          

Total Releases in a Year                                                                            9

Total Hours in I Year                                                                                   13500

Total $$ in one year                                                                                   $   1,215,000.00

Perhaps the most telling difference between Agile and Waterfall methodologies is the concept of “done-done “.

Within Waterfall, the concept does not exist. When a task is done-done – it means that a task is perfectly done and ready for release – and it conforms or fulfills all requirements that it has to address. Any changes to be incorporated – will require a change request process.

Within Agile this phrase means that a task is for that particular sprint only. Agile methodology requires keeping tasks in backlogs. In itself a very descriptive way of saying – whatever is not done or cannot be done at the current sprint, can be pushed on the back burner and we’ll work on it if and when the requirement presents itself. In the meantime, it can stay tucked away, swept under a rug, you get the picture. When it is pulled out of the backlog – then no change request process is required – because, in all fairness, it is non-existent and therefore net-new.