This blog addresses two questions: (a) How to determine if a methodology is better for the situation? (b) Can this methodology be further improved with cost and benefits well balanced?
A methodology without effectiveness measurement cannot be optimized with any certainty. Ambiguity will only lead to argument, ineffective spending, vendor exploitation, room for politics, and a whole slew of inefficiencies. A very simple example will be that of the traditional traveling salesman's problem in the industrial engineering discipline. The objective is to minimize traveling cost, which is linearly related to the distance. Without quantification, evaluation of effectiveness of the problem will be impossible. The keyword is linearity. However, there are apparent non-linear factors in software development. For example, risks, randomness in the delivery team's skills, and other human factors that affect the effectiveness of producing the results. As in engineering, when things are non-linear, we linearize them using approximation, which is accurate within a known range. So, this is very different from human guestimation which one cannot even quote a range with any true confidence. Of course, providers of agile service tend to be wary of measurability, because the results might not justifiably reflect the methodology itself. For instance, human factors can be muffling the benefits of a methodology. However, there is no excuse to this. Methodology and human factors need to be coupled with each other properly for any process to be successful.
Most businesses are already doing some agile practices without labeling it as such. Quarterly budgeting is a good example of short iteration. By retrospecting on the profit, loss, expenditure, etc., in quarterly cycles, better decisions can be made. This resembles integrate early and often agile principle. In this case, the integration being the external market and operations of the company. Another example will be the Stage Gate Management method in the product management field (http://www.prod-dev.com/stage-gate.php). It is not hard to see that Stage Gate and Lean's value stream mapping are compatible. The latter tends to be a bit single lined, while Stage Gate takes in the program/portfolio level view as well. The gist is that external factors like the Porter Five Forces aside, by examining the development gate, indirect measurement of overall success/failure of a software methodology can be analyzed. Again, there is no excuse. Methodology and human factors are flip sides of the same coin in the hands of management. How many coins hit the target bin is quantifiable. Methodology and how it is applied must be considered together.
Certainly, finer grain measurement helps visualization and hence improvement. Number of lines of code produced per period, or number of bugs are simple but insufficient indicators of quality or performance because these numbers change when the technology or the experience of the delivery team change. Imagine that the team now switched from low level language like C to Python. Then the number of lines of code decreases naturally. On the other hand, if there are turnover in a team, then perfomance will change but not directly reflective of the methodology itself. Also, maturity of the software also affects the "meaning" of these numbers.
In the finance world, ratios are often used to evaluate health of a company. It is definitely not my intention to go too deep on this subject. My point is that by applying ratios, or ratios of ratios, and so on in a Stage Gate fashion on Lean's value stream of software development, we can get quantitative measurement approximation linear/good enough for relative comparison. This makes continuous improvement possible. Otherwise, a change might achieve the opposite effect and get lost in the noises. In a top-down fashion, high level effectiveness measurements can be fine tuned and scoped in to gain more detailed insights into process strengths and weaknesses.
As an example, instead of using defects per line of code, we can use defects per effort hours. This prevents programming language/tools/technology from clouding the meaning of "process" or "methodology" quality. For example, if we use C language to implement a dynamic web application, then no matter what libraries or API we use, the number of lines will bound to be larger than those using frameworks in Python, PHP, or Ruby on Rails. This is similar to earned value analysis technique in project management, because effort hours is probably the highest common denominator of human intellectual factors.
As far as optimization goes, with some consistent measurement framework in place, scenario analysis using expected value techniques which take into account the risk, cost and benefits will aid better decision making. Certainly, the measurement framework might need to change too. In that case, lower scoped measurements will become obsolete.
Last but definitely not the least, measurement must be taking a global view. Optimizing just one stream and hurt ten others is unwise unless that single stream worths that much. Program/portfolio analysis must be considered at the same time. So often, agile or lean proponents talk in one dimension. For example, the classic cost of defects vs time curve is really a one dimensional view of the whole deal. On the other hand, maintaining flow is great, but if the overall cost does not justify the benefits, then we will need to find a better balance point. This is certainly no simple matter, and the number of scenarios will get exponentially larger as size of the company or number of initiatives increases. Simulations like Monte Carlo analysis, and the likes should be used together with expected value techniques in statistics to help guide the decision process.
In conclusion, to answer question (a) and (b), it is possible and necessary to use proper metrics instead of BS to measure effectiveness. A global portfolio/program level consideration must be taken into account as well. Isolated success on a stream or two does not guarantee overall success of the company.
Tuesday, November 4, 2008
Friday, August 29, 2008
Agile Database Development
Is agile database development beneficial? Yes. The values and principles of agile methodology are generic, and they aim at reducing process wastes, mis-communication, and risks in software development where human intellectual involvment is high, requirement changes are often, and time to market is aggressive. Synergy in the team, sustainable high productivity, and customer satisfaction are frequent benefits of agile development. All these benefits are desirable to database development as well, because it often involves volatile requirements, and highly human intellectual interactions just like application development.
Can we make database development agile then? Yes. Trivial? No. There are some key differences between database development and application development that call for our attention. Most traditional application development agile practices apply, but some do need minor tweaks.
In reality, most significant applications involve data persistence. However, conventionally, data is persisted centrally for different applications. This is particularly the case in bigger organizations. As a result, database development is often somewhat separated from application development. A typical practice is to have a separate database team. When application developers have database needs, they will forward them to the database developers/DBAs. These latter individuals will then design/develop according to the needs of application developers.
In some smaller organizations, application developers sometimes directly implement changes to the database. In this article, the term application development refers strictly to non-database related development work. The differences between the two will be explored first. Then, the significance of such differences in regards to Agile methodology will be examined. Finally, a proposed ideal database architecture and development process will be given as a framework of reference, which can possibly allow visualization and hence easier adaptation to suit different environments and database designs. This suggested solution is by no means universally applicable.
I. UNIQUE DATABASE DEVELOPMENT CHARACTERISTICS
The following traits are characteristic of database development when compared with application development. Please note that these traits are all inter-related.
II. AGILE VALUES AND DATABASE DEVELOPMENT
III. AGILE PRINCIPLES AND DATABASE DEVELOPMENT
IV. SOME AGILE PRACTICES AND DATABASE DEVELOPMENT
There are two objectives in enabling agile database development in an organization. The first one is to make database development agile itself. The second objective is to make database development more supportive of agile application development. The first objective is not a pre-requisite of the second. The two can co-exist as well. Its a matter of the particular situation in the organization.
ARCHITECTURE:
Here is a proposed database architecture (see diagram) that will enable both objectives to be met:
1. Encapsulated Database Interface Access Only.
External access to the data can be done via either stored procedure, persistence frameworks, or web services only. Not only is this more secure. Similar to interface patterns, and SOA, this layer of abstraction allows changes of internal structures without breaking existing contracts to external dependent systems. In other words, informational and behavioral semantics should be cleanly delineated. stored procedures, persistent frameworks, and web services can be used. However, only stick with one. This will mitigate the coupling of having multiple applications accessing the same database since it will be easier to locate and hence control the dependencies. It will also be easier to refactor.
2. Testable Operations Only.
Data should be operated or accessed via only testable means. If a test case/script cannot or does not exist for a DML, query, or access interface, then such access should not be permitted. This will ensure regression coverage to support shorter and iterative development cycles. Analogous to this will be test driven development in application development. It will also make automated regression testing possible.
3. Centralized Logic Layer.
Only the encapsulated layer should be responsible for cascading changes, or making any side effects for any operations. This separation of database access logic and database. Such locality will also make dependencies analysis manageable. Without such measure, one database developer will not be able to find out if his/her work will impact other applications easily.
For mission critical referential integrity, triggers or constraints can be applied as a redundant check in case the encapsulating layer has missed the checking. However, this should be minimized and conventionalized. This will improve visibility across multiple teams in case there are multiple external dependencies to the database.
On the other hand, let's say stored procedures are used for implementing this layer. The database team can benefit readily from existing good agile application development practices, because developing stored procedures is just like developing other application languages.
4. Traceability/Recoverability.
No hard delete (when size becomes an issue then rotating to tapes or a secondary database will be an option). Audit trail for all DMLs. Backup all data loss due to DDL related changes. Version control of all artifacts, database code, access interface code, DLL, DML, scripts, and even data itself will be required. Obviously, in the case of data, it might not be realistic to backup and version label too frequently. An optimal frequency will need to be figured out. Data will need to be handled more discretely, because it is much harder to guarantee state recovery. From ground up, a database state should be recoverable to any point in time.
5. Normalization.
Peformance optimization should not be done via breaking normalization. Creation of extra views, or temporary tables should be used. These view or temporary tables should be read only. Buffer tables can be used too but the states should be centrally recorded in a normalized table. If there is no choice but to break normalization to fulfill performance requirement, then a central scheme of such approach will be needed across all applications.
6. Strict Convention.
With convention, less documentation, and less mis-understanding. Convenstion should be enforced with automated test scripts, which should ideally be invoked automatically during check-ins. In other words, version control tied with integration system will facilitate enforcement of convention preservation. Certainly, it might not be possible or efficient to always do this, but attempting such either automatically, or manually through defined workflow should be encouraged.
7. Impact Analysis Facilitation.
The database code should be designed with good convention and structure so that impact can be easily analyzed and visualized. This should be taken into account as a design goal. There are also tools to identify side effects or database objects affected. For example, some stored procedure IDE tool can determine what tables are touched by a stored procedure with the click of a button.
DEVELOPMENT PROCESS
1. Scheduled Pull Requests.
External users of the data should pull their demands at defined time. This will help synchronization, and hence traceability. The intervals between pulls should be minimized as much as possible so that rapid response to changes can be obtained.
2. Small Steps.
This makes it easier to trace, recover, and minimize damage. This applies within the database team.
3. Full Regression.
Analogous to frequent integration, full regression test essentially achieve the same purpose, albeit less ideal, as long as all the rules here are followed. It is an alternative to the end-to-end integration in application development. More precisely, database full regression involves testing of all database interfaces including those from other applications. As number of applications dependent on the same database increases,
4. Test Automation.
In order for fast response to changes and quick turnaround to support short release cycles of agile application development, test needs to be automated, especially with the highly coupled nature of database. This will reduce the overhead of manual testing, and errors from human mistakes.
5. Iteration Size.
The architecture and process suggested here support quicker turnaround. Ideally, database development iteration should be much shorter than that of application development so that database development will not be a bottleneck to application development. Otherwise, end to end production ready releases will not be ready unless the database work is done. However, the highly couple, critical, and many other traits of database development can make it difficult to achieve quick turnaround.
Alternatively, during application development iteration planning, database needs, can be looked ahead one iteration. For example, during iteration i planning, database needs for iteration i+1 will be looked at already. Doing it this will will allow database development iteration to be of the same length as that of application development. Of course, this might result in a synchronization of iteration among different applications depending on the same database. In a sense, it is like the CPU clock inside the microprocessor.
6. Flow.
So far, we have talked bits and pieces of processes, and practices. One approach, if economically feasible, is to have a DBA/database developer directly assigned to an application project, and at the same time keeping communication with the rest of the database team from other projects for cross-application impact resolution.
A more realistic approach will be explained below. The following diagram (TBD) might be helpful. Essentially, database development will be done in iterations. During each iteration, applications from different projects will send their requests for database work to the database team. The database team will then examine the requests for their impacts. If an impact is localized to the requesting application only, and the bandwidth allows fulfillment of the request, then it can be done within the iteration. Otherwise, if either the impact is cross-application, or the effort is large, then such request story will be analyzed and prioritized with other request stories. Application team representative, preferrably include the business, should get involved in prioritizing the stories together during database development iteration planning.
Obviously, if the database team continuously become the bottleneck for other application development teams, then resource increase will be justifiable. Another option will then be having separate database schemas. But the problems of master data management, synchronization, reconciliation, etc., will arise.
Application development team should be aware of the potential of database work delivery delay, one iteration lookahead should be considered to minimize impact. If the database is architected properly, then such instances should be minimal.
7. Roles and Responsibilities.
While the architecture and process suggested so far delineate the database team as the sole implementor of the interface access layer. However, this is not a requirement. As a matter of fact, for data intensive applications, it might be better for either the application development team to directly implement this layer, or the database team to assign someone to be devoted to the project working directly with the application team. Of course, whoever works in this layer should have good understanding of database development practices. Moreover, if some core database objects affected is/will be depended by other applications, for example the common helper library or some shared tables, then the database team must be involved as they are the final gate keepers and maintainers of the database.
The layer I am proposing here is more like onion layers. The closer to the core, the more central the control should be. In other words, it is a matter of how we layer the onion. An optimal solution will maximize both turnaround time and data wholesomeness. I have deliberately used "wholesomeness" instead of "integrity" because the latter seems to imply only correctness. For "wholesomeness", I am also referring to a state of the database that future access, and manipulation of the data are not hampered. For example, if a modification of the database makes it difficult or inefficient to retreive certain reports, then the "wholesomeness" of the database is affected.
Can we make database development agile then? Yes. Trivial? No. There are some key differences between database development and application development that call for our attention. Most traditional application development agile practices apply, but some do need minor tweaks.
In reality, most significant applications involve data persistence. However, conventionally, data is persisted centrally for different applications. This is particularly the case in bigger organizations. As a result, database development is often somewhat separated from application development. A typical practice is to have a separate database team. When application developers have database needs, they will forward them to the database developers/DBAs. These latter individuals will then design/develop according to the needs of application developers.
In some smaller organizations, application developers sometimes directly implement changes to the database. In this article, the term application development refers strictly to non-database related development work. The differences between the two will be explored first. Then, the significance of such differences in regards to Agile methodology will be examined. Finally, a proposed ideal database architecture and development process will be given as a framework of reference, which can possibly allow visualization and hence easier adaptation to suit different environments and database designs. This suggested solution is by no means universally applicable.
I. UNIQUE DATABASE DEVELOPMENT CHARACTERISTICS
The following traits are characteristic of database development when compared with application development. Please note that these traits are all inter-related.
- Stateful. The main artifacts of application development code are programming language specific code files, and sometimes configuration files. Business data is often not considered as a product of application development. This is not the case for database development. Business production data is a "living" entity that retains the current state of the business entities. The production data also changes with time. Maintaining the healthy state of such data is definitely part of the deliverables of any database development.
- High Coupling. Applications are usually developed for different business initiatives at different times in an organization. However, most applications share the same fundamental business entities which are mapped to the data. Redundant/duplicated code is bad, and it is worst for data for obvious reasons. Synchronization and security are two of them. So, naturally, having the same database serving multiple applications is common place. The consequence is a central dependency on the database and development work around it.
- Mission Critical. There is probably no error more critical than data loss and data corruption. So quality control will have to be even more stringent than say, GUI design in most cases. This implies defined processes in place to guard against improper modifications to the database.
- Security. Business data can contain personal privacy information, as well as corporate secrets. From the security principle of minimal access, database artifacts are accessible by only a small number of individuals.
- Size. Databases grow in size with time most of the cases. Performance tuning, deployment, backup, and testing are some of the issues that get harder to execute with size. When it comes to the point of doing master slave, or views, or temporary tables, things get complicated.
- Intrinsic Meaning. A lot of applications are driven by the content of the data. If the database is not 1NF normalized, the situation can be more hairy. On the other hand, when and how this data is changed is also very significant. Unfortunately, most organizations do not have good means to document by text or by integrated regression test script to preserve the data life cycle across applications sharing the same database.
- Testability. As a database gains size, it loses testability as well. A database table with tens of thousands of rows and a dozen columns make testing manually too costly, and too time consuming. The accuracy of manual testing is also diminished due to human errors. Typical database testing strategies are extreme cases, manual sampling, and aggregate examination. However, the temporal nature of data states makes coverage hard to determine, especially when there are multiple applications depending on the data.
- SDLC Flow. As a result of the above traits, DBAs and/or database developers are often the final gate keeper of database activities. Staging, migration, and conversion are often part of the SDLC, which are not part of non-database application development.
- Longevity. Application comes and goes. Newer ones replace older ones. Not the case for data. Data might get migrated/converted, but they stay. As time goes, the history of the database gets richer, and development/maintenance grows more complicated, especially if the design and process are not done properly.
II. AGILE VALUES AND DATABASE DEVELOPMENT
- Individuals and interactions over processes and tools. Because of the multi-application coupled nature of database, the database team usually serves many projects concurrently. Some sort of process will need to be in place to orchestrate these changes. However, emphasizing individuals and interactions is not in anyway in conflict with having some processes or tools in place. It is a matter of emphasis, and mentality.
- Working software over comprehensive documentation. Because of intrinsic meaning and testability traits of database development, working software is harder to define. But it is possible as will be pointed out by the suggested ideal database design and process later. To help with communication, documentation on data shared among applications can be documented. At least a documentation map pointing to test scripts (if the test scripts comprehensively describe the functionality of the database objects) should be maintained.
- Customer collaboration over contract negotiation. Since most database development deal with application development teams directly instead of the customers, the customers now become the application developers, while the contract will be the design specification. Since database development indirectly, and sometimes directly, fulfill customers' interests, this Agile value should still be pursued. How? The ideal database development process will illustrate some feasible approaches.
- Responding to change over following a plan. The nature of typical/traditional database development clearly makes it harder to respond to change because the ramification can impact many facets of the organization, from applications, projects, to departments. Again, I hope the ideal database design and process can address this.
III. AGILE PRINCIPLES AND DATABASE DEVELOPMENT
- Our highest priority is to satisfy the customer through early and continuous delivery of valuable software. This can be difficult because the same database can serve multiple applications. However, there are ways to mitigate this as will be explained later.
- Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage. This can be difficult because the same database can serve multiple applications. However, there are ways to mitigate this as will be explained later.
- Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale. This can get more difficult as the number of dependent projects on the same database increases.
- Business people and developers must work together daily throughout the project. Since database developers are sometimes shielded by application developers when it comes to communication with business people, This might not be very applicable. However, communication should still be encouraged to understand the business reasons behind the data requirements.
- Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done. No conflict.
- The most efficient and effective method of conveying information to and within a development team is face-to-face conversation. There can be face-to-face conversation, however, with a single DBA/database developer team supporting multiple application, the amount of time for face-to-face conversation will be limited.
- Working software is the primary measure of progress. Since the data is stateful with a history, it is difficult to say whether a database is working using the conventional database development and management.
- Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. No conflict.
- Continuous attention to technical excellence and good design enhances agility. No conflict.
- Simplicity--the art of maximizing the amount of work not done--is essential. No conflict.
- The best architectures, requirements, and designs emerge from self-organizing teams. No conflict.
- At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly. No conflict.
IV. SOME AGILE PRACTICES AND DATABASE DEVELOPMENT
- Pairing. No conflict. Pairing can still be very advantageous be it DBA/DBA, DBA/application developer, DBA/BA, DBA/QA.
- Integration Early and Often. Often and Early are relative terms. These should still be emphasized even though an iteration in database development might be longer than that of application development. However, this is not cast in stone.
- Test Driven Development. This can still be done. And it is even more important since multiple applications are dependent on the integrity of the database.
- Short Release Cycles. Short is also a relative term. Cycles should be as short as possible.
- Story Based Process. This is also desirable, however, dependencies must be identified early.
- JIT Design. This is also applicable because requirements from different applications increase the volatility exponentially. It will be even harder to predict good design. Design incrementally will be a better option. Simpler design will also aid refactoring later on if called for.
- Co-location. This can be difficult because the same DBA/database developer can potentially work on multiple projects. But this is not a problem that cannot be fixed.
- Refactoring. Due to the coupling and critical nature, the cost of refactoring will be higher. So, automation will be called for to reduce the impact. The book "Refactoring Databases" by Scott Ambler and Pramod Sadalage talks about categories of database refactoring, and good practices that help with the process.
There are two objectives in enabling agile database development in an organization. The first one is to make database development agile itself. The second objective is to make database development more supportive of agile application development. The first objective is not a pre-requisite of the second. The two can co-exist as well. Its a matter of the particular situation in the organization.
ARCHITECTURE:
Here is a proposed database architecture (see diagram) that will enable both objectives to be met:
1. Encapsulated Database Interface Access Only.
External access to the data can be done via either stored procedure, persistence frameworks, or web services only. Not only is this more secure. Similar to interface patterns, and SOA, this layer of abstraction allows changes of internal structures without breaking existing contracts to external dependent systems. In other words, informational and behavioral semantics should be cleanly delineated. stored procedures, persistent frameworks, and web services can be used. However, only stick with one. This will mitigate the coupling of having multiple applications accessing the same database since it will be easier to locate and hence control the dependencies. It will also be easier to refactor.
2. Testable Operations Only.
Data should be operated or accessed via only testable means. If a test case/script cannot or does not exist for a DML, query, or access interface, then such access should not be permitted. This will ensure regression coverage to support shorter and iterative development cycles. Analogous to this will be test driven development in application development. It will also make automated regression testing possible.
3. Centralized Logic Layer.
Only the encapsulated layer should be responsible for cascading changes, or making any side effects for any operations. This separation of database access logic and database. Such locality will also make dependencies analysis manageable. Without such measure, one database developer will not be able to find out if his/her work will impact other applications easily.
For mission critical referential integrity, triggers or constraints can be applied as a redundant check in case the encapsulating layer has missed the checking. However, this should be minimized and conventionalized. This will improve visibility across multiple teams in case there are multiple external dependencies to the database.
On the other hand, let's say stored procedures are used for implementing this layer. The database team can benefit readily from existing good agile application development practices, because developing stored procedures is just like developing other application languages.
4. Traceability/Recoverability.
No hard delete (when size becomes an issue then rotating to tapes or a secondary database will be an option). Audit trail for all DMLs. Backup all data loss due to DDL related changes. Version control of all artifacts, database code, access interface code, DLL, DML, scripts, and even data itself will be required. Obviously, in the case of data, it might not be realistic to backup and version label too frequently. An optimal frequency will need to be figured out. Data will need to be handled more discretely, because it is much harder to guarantee state recovery. From ground up, a database state should be recoverable to any point in time.
5. Normalization.
Peformance optimization should not be done via breaking normalization. Creation of extra views, or temporary tables should be used. These view or temporary tables should be read only. Buffer tables can be used too but the states should be centrally recorded in a normalized table. If there is no choice but to break normalization to fulfill performance requirement, then a central scheme of such approach will be needed across all applications.
6. Strict Convention.
With convention, less documentation, and less mis-understanding. Convenstion should be enforced with automated test scripts, which should ideally be invoked automatically during check-ins. In other words, version control tied with integration system will facilitate enforcement of convention preservation. Certainly, it might not be possible or efficient to always do this, but attempting such either automatically, or manually through defined workflow should be encouraged.
7. Impact Analysis Facilitation.
The database code should be designed with good convention and structure so that impact can be easily analyzed and visualized. This should be taken into account as a design goal. There are also tools to identify side effects or database objects affected. For example, some stored procedure IDE tool can determine what tables are touched by a stored procedure with the click of a button.
DEVELOPMENT PROCESS
1. Scheduled Pull Requests.
External users of the data should pull their demands at defined time. This will help synchronization, and hence traceability. The intervals between pulls should be minimized as much as possible so that rapid response to changes can be obtained.
2. Small Steps.
This makes it easier to trace, recover, and minimize damage. This applies within the database team.
3. Full Regression.
Analogous to frequent integration, full regression test essentially achieve the same purpose, albeit less ideal, as long as all the rules here are followed. It is an alternative to the end-to-end integration in application development. More precisely, database full regression involves testing of all database interfaces including those from other applications. As number of applications dependent on the same database increases,
4. Test Automation.
In order for fast response to changes and quick turnaround to support short release cycles of agile application development, test needs to be automated, especially with the highly coupled nature of database. This will reduce the overhead of manual testing, and errors from human mistakes.
5. Iteration Size.
The architecture and process suggested here support quicker turnaround. Ideally, database development iteration should be much shorter than that of application development so that database development will not be a bottleneck to application development. Otherwise, end to end production ready releases will not be ready unless the database work is done. However, the highly couple, critical, and many other traits of database development can make it difficult to achieve quick turnaround.
Alternatively, during application development iteration planning, database needs, can be looked ahead one iteration. For example, during iteration i planning, database needs for iteration i+1 will be looked at already. Doing it this will will allow database development iteration to be of the same length as that of application development. Of course, this might result in a synchronization of iteration among different applications depending on the same database. In a sense, it is like the CPU clock inside the microprocessor.
6. Flow.
So far, we have talked bits and pieces of processes, and practices. One approach, if economically feasible, is to have a DBA/database developer directly assigned to an application project, and at the same time keeping communication with the rest of the database team from other projects for cross-application impact resolution.
A more realistic approach will be explained below. The following diagram (TBD) might be helpful. Essentially, database development will be done in iterations. During each iteration, applications from different projects will send their requests for database work to the database team. The database team will then examine the requests for their impacts. If an impact is localized to the requesting application only, and the bandwidth allows fulfillment of the request, then it can be done within the iteration. Otherwise, if either the impact is cross-application, or the effort is large, then such request story will be analyzed and prioritized with other request stories. Application team representative, preferrably include the business, should get involved in prioritizing the stories together during database development iteration planning.
Obviously, if the database team continuously become the bottleneck for other application development teams, then resource increase will be justifiable. Another option will then be having separate database schemas. But the problems of master data management, synchronization, reconciliation, etc., will arise.
Application development team should be aware of the potential of database work delivery delay, one iteration lookahead should be considered to minimize impact. If the database is architected properly, then such instances should be minimal.
7. Roles and Responsibilities.
While the architecture and process suggested so far delineate the database team as the sole implementor of the interface access layer. However, this is not a requirement. As a matter of fact, for data intensive applications, it might be better for either the application development team to directly implement this layer, or the database team to assign someone to be devoted to the project working directly with the application team. Of course, whoever works in this layer should have good understanding of database development practices. Moreover, if some core database objects affected is/will be depended by other applications, for example the common helper library or some shared tables, then the database team must be involved as they are the final gate keepers and maintainers of the database.
The layer I am proposing here is more like onion layers. The closer to the core, the more central the control should be. In other words, it is a matter of how we layer the onion. An optimal solution will maximize both turnaround time and data wholesomeness. I have deliberately used "wholesomeness" instead of "integrity" because the latter seems to imply only correctness. For "wholesomeness", I am also referring to a state of the database that future access, and manipulation of the data are not hampered. For example, if a modification of the database makes it difficult or inefficient to retreive certain reports, then the "wholesomeness" of the database is affected.
Monday, July 7, 2008
Agile Story Points - Scope and Effort Management
Abstract: Using story points to manage both scope and effort is convenient but not ideal. Story points should be split for scope and effort management. Such separation will ease project analysis, planning, and communication. Features or business values should be the reference for scope story points. Efforts should be tracked separately. Having scope story points separately tracked, better product and financial management of the project can be achieved. By tracking effort story points independently, re-estimation for better progress management can be done without impacting scope measures. Appropriate periodic re-estimation of effort story points empirically basing on actual efforts instead of stale and somewhat arbitrarily assigned values at the beginning improves predictability of delivery schedule and collaboration coordination. Because of planned re-estimation, there should be no fear of using ideal work days as measures, which reduces confusion and difficulties in planning and estimating. Visualization of the correlation between the scope and effort story point velocities provides another means to assist with work prioritization.
5. Conclusion
In this article, the cost and benefits of separating story points into scope and effort points are explained. It is not difficult to separate them. The simplest way is to assign one point per business value delivering story, and then track effort points just like we do with story points nowadays. More sophistication management of scope points is to associate business values. There are many ways to do this. How much efforts should be spent in sophisticated scope point management is subject to the ROI of such endeavors.
1. Introduction
In one of the Agile practices, business value delivering functionalities/features are organized into units called stories. These stories are often relatively sized and recorded using a parameter called story point. With consistent and proper story sizing, story points are good indicators of scope and effort. Total story points are often interpreted as the total scope as well as the total effort for the project. However, scope and effort are very different concepts and have different quantitative and qualitative variations in the course of a project.
As more knowledge is gained about the nature of the stories, previous effort estimations often need revision. In other words, story points will need to be changed to reflect the new effort estimation. As a result, scope measure will inevitably and inappropriately change as well since both effort and scope are tracked with story points. Depending on whether the new total story point is higher or lower than before, scope inflation or deflation will occur respectively. By having an effort story point attached to each story instead, we can safely do re-estimation without causing inaccurate scope change. With re-estimation as part of the process, the fear and worry of estimating in ideal man days also disappear (see http://agilesanity.blogspot.com for details )
On the other hand, a highly valuable and complicated feature/functionality might take very little effort to complete. For example, the shopping cart of a web application can be easily implemented by purchasing and integrating existing tool kits. On the contrary, a low value feature/functionality might take a lot of effort to deliver. For instance, to add contextual help for an air ticket booking web application might sound like a simple one liner feature, but it actually entails a lot of work behind the scene. The gist of these examples illustrate that feature size and effort size are not necessarily positively correlated with business value.
But the most immediate benefit of scope and effort story point differentiation is reduction of abstraction. Clarity is gold.
However, if we make story point management too complicated, then the ROI might not be justifiable. As in anything, balance is the key. There is a Confucius saying, "The fools never worry about balance, but the smarts try too hard to balance and losing it." So being agile in spirit is the true way.
2. Scope Management with Scope Story Points
Scope story points, or simply scope points, can be allocated as simple as one point per identified story. So, if we have 19 stories, then we will have 19 scope points, which represent the total scope at the moment. This is acceptable when stories are supplemented with effort story points.
However, there are more useful ways of assigning scope points. For example, we can associate scope points with business values. Let’s say the project is about producing a PC car racing game, which is expected to make a net profit of 5 million dollars in 3 years. Then, the upper bound for total scope points can be assigned 5 million (the lower bound can be the cost of the project). In other words, completing 5 million scope points will mean that all essential features of the game are developed. With this lump sum figure as the reference point, consideration of addition or removal of features/functionalities can be made by taking into account whether such action will increase enough sales/profit to justify the extra cost based on effort story points (to be discussed later). So, better decisions can be made towards scope changes, and resource allocation.
A few more words on scope point assignment with the PC car racing game example. Sometimes, there are critical inter-dependent stories which aggregate completion is required for a product to be shippable. If the majority of stories are under this category, then the burn up or burn down chart of scope points will be useful. However, if, say, less than half the stories are critical and inter-dependent, then, these charts will not be as useful because the scope velocity of non-critical independent tasks does not depict a true picture of how far the project is progressing towards completion if the critical path is lagging behind. This is the case because the risk and unknown are not explored in the critical path. The simplest way is to track critical inter-dependent stories separately.
With scope point come scope velocity, and scope burn up chart. The former is not as useful as effort velocity (to be addressed later) in terms of completion prediction. The latter, on the other hand, is the better indicator of earned values than effort burn up. For more information on earned value management, please see "Project Management Body of Knowledge - 3rd Edition".
3. Effort Management with Effort Story Points
See "Adaptive Agile Effort Management" post in http://agilesanity.blogspot.com for details.
4. Relationship between Scope and Effort Story Points
5. Conclusion
In this article, the cost and benefits of separating story points into scope and effort points are explained. It is not difficult to separate them. The simplest way is to assign one point per business value delivering story, and then track effort points just like we do with story points nowadays. More sophistication management of scope points is to associate business values. There are many ways to do this. How much efforts should be spent in sophisticated scope point management is subject to the ROI of such endeavors.
Thursday, July 3, 2008
Adaptive Agile Effort Management
Prelude:
Most business see software endeavors as projects. Resources need to be set aside for projects at defined times. And timing of deliverables often tied to marketing, budgeting, and external collaboration initiatives. The day when developers will become perfect technically, know everything about the business, be trusted by business ,and be given the freedom to do whatever they feel best has not come, and might never come any time soon. So, visibility and predictability are still required throughout a project to fit in existing business practices.
Most projects out there aren't quantum physics or sci-fi creativity work. After completing 20-40% of the stories, we should pretty much know enough for more accurate estimate. This is especially the case when proper spiking and critical path analysis are done early on. Cone of uncertainty diminishes with progress most of the time. So, re-estimation should provide more accurate adjustment.
Effort Re-estimation:
In the course of a project, there can be order of magnitude differences between original effort estimates and actual efforts spent. This points to the need of re-estimation. As Mike Cohn put it in his latest edition of "Agile Estimation and Planning (2006) : page 61-76", as long as the relative sizing of effort estimates among stories/tasks is preserved, this should be a safe thing to do. Certainly, we should re-estimate in such scenarios to maintain more accurate predictability and manageability of progress.
The question then is when and how often we should re-estimate. Should re-estimation only happen when some stories/tasks are order of magnitude out of whack? Or, should we re-estimate every iteration so that we won't forget about fine tuning our stale estimates? Should all unfinished stories be re-estimated?
Obviously, for a large project with many stories, many developers and QAs of different skill levels, ROI of such re-estimation should be considered. To make re-estimation even easier and more effective (i.e. not planning too much details too far ahead), David Leigh-Fellowes, an experienced Agile practitioner, suggested re-estimating stories in the next 1.5 iterations during IPM. The 0.5 iteration look ahead is a great idea since external dependencies can be sorted out earlier. This is especially benefitial in big organizations where collaboration is often distributed across departments and countries. Moreover, environments/code base can be shared. He prefers re-estimation by tasking out certain interesting stories in details over using actual efforts on similar stories completed in past iterations. Since re-estimation by tasking out is based on actual experience in the project then, this is actually superior to merely taking actual story points from the past and apply by analogy. He also thinks re-estimation by the whole team will make more sense in a big project setting since different developers/QAs can work on a story which has a similar counterpart with actual effort tracked, and hence different performance.
Does it mean that actual story effort points are not useful? Well, not quite since re-estimation by tasking out by the whole team is labor intensive, and can only be practically applied for 1.5 iterations look ahead, stories with wrong estimates beyond 1.5 iterations will still potentially skew overall progress projection significantly. So, a happy balance will be applying both re-estimation by tasking out for the next 1.5 iterations based on actual experience gained, and re-estimation based on actual efforts on similar stories completed for stories beyond 1.5 iterations. Needless to say that relative sizing of all the stories should be maintained in the process. This way, we achieve more accurate prediction for both short term and long term work without spending too much effort.
Effort Estimation Units:
Two well known figures in the agile community, namely Kent Beck and James Shore have reverted to the use of ideal developer days as story point measuring unit. James thinks that making estimating units more abstract makes estimating and planning more confusing and difficult. Two common opposing views are: (a) one's ideal day estimate is not necessarily the same as another's [ideal day individuality], and (b) people can be hold responsible to ideal days or be judged by whether they finish within the ideal days since it is much easier to track [ideal day fixation/accusation]. Let's dive into these opposing views one by one.
The same story implemented by different developers can take different amount of time since they have different skill sets, skill levels, experiences and approaches. This problem is actually independent of whether abstract story point or ideal developer day is used. It is a matter of "how" the estimate is done. As per the previous section on effort re-estimation, ideal day individuality should no longer be an issue because for the next 1.5 iteration, re-estimation is done by the "whole" team during, say, iteration planning meeting, and that should take into account the differences in speed between the old and new implementers. For the rest of the iterations, story effort revision from actual efforts on similar stories should still provide better estimate than not revising the story at all. After all, in an agile setting, pairing, rotation or pairs, and close communication should reduce the effort variation. And with sufficient time/scale, and mixing of stories/tasks, the aggregate estimate should still be accurate due to averaging.
If estimation is done in ideal days, it can be tempting for management or co-workers to accuse a developer's inability to finish in the estimated time frame. Or, a developer can be judged to be incompetent by not being able to finish in time. This is not desirable because estimate is still estimate, and there are often factors not fully comprehended at the time of estimation. The only thing that will come out of this is defensive behavior, damaged morale, and ineffectiveness from the commotion. Doing estimate in abstract story points might seem like a solution, however, in reality, we are shifting the problem from ideal day to velocity fixation/accusation. Developers can still be accused of not maintaining or improving on velocity. The culprit of this is not the process, but wrong mentality. Understanding the nature of the degree of unpredictability of software development is really the key to solve this. So, the good solution is not to cloud the issue, but to make it more concrete, more measurable, and hence more visible. If we do re-estimation as per described previously on a pre-defined regular basis, and repeatedly make it very clear that estimated efforts are not final, and will undergo further revisions, then we have effectively reduced the fixation on estimates. At the same time, we will have a reference point (not measurement) of performance. Variance in performance can then be managed for improvement.
Either approach requires proper communication with the clients. And periodic reminders will be needed. Yes, even ideal days can be confusing because a client might be confused about ideal versus elapsed, and whether ideal day is for one man or one pair.
Conclusion:
From the discussion above, re-estimating is a good practice for providing better prediction and progress management by adapting on better knowledge gained. By only re-estimating the next 1.5 iterations with the whole team, and then re-estimate the rest by applying actual efforts on similar stories while preserving relative sizing of the rest of the stories also reduce the problem associated with ideal day individuality. Moreover, with such standard practice and proper communication, we can also take care of the ideal day accusation/fixation issue, and making the team more productive in a sustainable manner.
Most business see software endeavors as projects. Resources need to be set aside for projects at defined times. And timing of deliverables often tied to marketing, budgeting, and external collaboration initiatives. The day when developers will become perfect technically, know everything about the business, be trusted by business ,and be given the freedom to do whatever they feel best has not come, and might never come any time soon. So, visibility and predictability are still required throughout a project to fit in existing business practices.
Most projects out there aren't quantum physics or sci-fi creativity work. After completing 20-40% of the stories, we should pretty much know enough for more accurate estimate. This is especially the case when proper spiking and critical path analysis are done early on. Cone of uncertainty diminishes with progress most of the time. So, re-estimation should provide more accurate adjustment.
Effort Re-estimation:
In the course of a project, there can be order of magnitude differences between original effort estimates and actual efforts spent. This points to the need of re-estimation. As Mike Cohn put it in his latest edition of "Agile Estimation and Planning (2006) : page 61-76", as long as the relative sizing of effort estimates among stories/tasks is preserved, this should be a safe thing to do. Certainly, we should re-estimate in such scenarios to maintain more accurate predictability and manageability of progress.
The question then is when and how often we should re-estimate. Should re-estimation only happen when some stories/tasks are order of magnitude out of whack? Or, should we re-estimate every iteration so that we won't forget about fine tuning our stale estimates? Should all unfinished stories be re-estimated?
Obviously, for a large project with many stories, many developers and QAs of different skill levels, ROI of such re-estimation should be considered. To make re-estimation even easier and more effective (i.e. not planning too much details too far ahead), David Leigh-Fellowes, an experienced Agile practitioner, suggested re-estimating stories in the next 1.5 iterations during IPM. The 0.5 iteration look ahead is a great idea since external dependencies can be sorted out earlier. This is especially benefitial in big organizations where collaboration is often distributed across departments and countries. Moreover, environments/code base can be shared. He prefers re-estimation by tasking out certain interesting stories in details over using actual efforts on similar stories completed in past iterations. Since re-estimation by tasking out is based on actual experience in the project then, this is actually superior to merely taking actual story points from the past and apply by analogy. He also thinks re-estimation by the whole team will make more sense in a big project setting since different developers/QAs can work on a story which has a similar counterpart with actual effort tracked, and hence different performance.
Does it mean that actual story effort points are not useful? Well, not quite since re-estimation by tasking out by the whole team is labor intensive, and can only be practically applied for 1.5 iterations look ahead, stories with wrong estimates beyond 1.5 iterations will still potentially skew overall progress projection significantly. So, a happy balance will be applying both re-estimation by tasking out for the next 1.5 iterations based on actual experience gained, and re-estimation based on actual efforts on similar stories completed for stories beyond 1.5 iterations. Needless to say that relative sizing of all the stories should be maintained in the process. This way, we achieve more accurate prediction for both short term and long term work without spending too much effort.
Effort Estimation Units:
Two well known figures in the agile community, namely Kent Beck and James Shore have reverted to the use of ideal developer days as story point measuring unit. James thinks that making estimating units more abstract makes estimating and planning more confusing and difficult. Two common opposing views are: (a) one's ideal day estimate is not necessarily the same as another's [ideal day individuality], and (b) people can be hold responsible to ideal days or be judged by whether they finish within the ideal days since it is much easier to track [ideal day fixation/accusation]. Let's dive into these opposing views one by one.
The same story implemented by different developers can take different amount of time since they have different skill sets, skill levels, experiences and approaches. This problem is actually independent of whether abstract story point or ideal developer day is used. It is a matter of "how" the estimate is done. As per the previous section on effort re-estimation, ideal day individuality should no longer be an issue because for the next 1.5 iteration, re-estimation is done by the "whole" team during, say, iteration planning meeting, and that should take into account the differences in speed between the old and new implementers. For the rest of the iterations, story effort revision from actual efforts on similar stories should still provide better estimate than not revising the story at all. After all, in an agile setting, pairing, rotation or pairs, and close communication should reduce the effort variation. And with sufficient time/scale, and mixing of stories/tasks, the aggregate estimate should still be accurate due to averaging.
If estimation is done in ideal days, it can be tempting for management or co-workers to accuse a developer's inability to finish in the estimated time frame. Or, a developer can be judged to be incompetent by not being able to finish in time. This is not desirable because estimate is still estimate, and there are often factors not fully comprehended at the time of estimation. The only thing that will come out of this is defensive behavior, damaged morale, and ineffectiveness from the commotion. Doing estimate in abstract story points might seem like a solution, however, in reality, we are shifting the problem from ideal day to velocity fixation/accusation. Developers can still be accused of not maintaining or improving on velocity. The culprit of this is not the process, but wrong mentality. Understanding the nature of the degree of unpredictability of software development is really the key to solve this. So, the good solution is not to cloud the issue, but to make it more concrete, more measurable, and hence more visible. If we do re-estimation as per described previously on a pre-defined regular basis, and repeatedly make it very clear that estimated efforts are not final, and will undergo further revisions, then we have effectively reduced the fixation on estimates. At the same time, we will have a reference point (not measurement) of performance. Variance in performance can then be managed for improvement.
Either approach requires proper communication with the clients. And periodic reminders will be needed. Yes, even ideal days can be confusing because a client might be confused about ideal versus elapsed, and whether ideal day is for one man or one pair.
Conclusion:
From the discussion above, re-estimating is a good practice for providing better prediction and progress management by adapting on better knowledge gained. By only re-estimating the next 1.5 iterations with the whole team, and then re-estimate the rest by applying actual efforts on similar stories while preserving relative sizing of the rest of the stories also reduce the problem associated with ideal day individuality. Moreover, with such standard practice and proper communication, we can also take care of the ideal day accusation/fixation issue, and making the team more productive in a sustainable manner.
Subscribe to:
Posts (Atom)
