Introduction
In today's IT world, organizations increasingly work with external partners. In addition, more and more organizations are adopting an Agile-at-scale approach in which cooperation throughout the cross-organization IT delivery process is essential. In these types of environments, control over end-to-end quality is becoming increasingly important.
This section considers the management of end-to-end quality in a scaled environment, where control on quality is key when implementing changes in the solution. To control end-to-end quality, organizing an end-to-end test is relevant, but at least as important is to organize prerequisites such as Test environments, Test data, Stakeholder management and Release management.
This section elaborates on the integration of the end-to-end Quality process within a scaled environment, considering:
- Focus areas that apply where the greatest end-to-end quality benefits can be achieved.
- The teams that play a role in organizing end-to-end Quality Engineering.
- Specific roles and responsibilities in those teams.
- Artefacts (activities and means) that are supportive to this integrated end-to-end Quality process.
End-to-end areas of interest
There are 6 areas of interest where management of end-to-end quality can achieve the greatest benefits. Once you control these areas, you can shift your attention and look further. Some additional areas, such as Test Automation, are implicitly affected.
End-to-end parties
As far as the end-to-end parties are concerned, you can achieve a recognizable and controlled delivery process for the entire application chain, where it is clear to each party what the expectations are and where individual commitment is in place.
People
Of course, the people are key to deliver quality. They do the actual work, everyone from their own role(s). For the people, the focus is shifting to delivering quality and seeking collaboration in an open, safe and challenging work environment.
Change dynamics
Change dynamics are about the complex interplay of IT delivery models (sequential, high-performance and hybrid) by all parties involved, but also the frequency and way of delivering work packages. In the complex playing field of the change dynamics, it is possible to shorten the time to market, through coordinated release moments and through well-organized collaboration.
Test environments
Depending on choices within the organization, one or more test environments are available per application / component, each with its own goal. A test environment of an application is also often related to another test environment, in order to create a complete Solution. With the use of modern techniques like containerization, infrastructure as code and cloud environments setting up environments becomes more consistent and manageable. The delivery process will be much more optimized and controlled and you can grow to predictable releases. Investing in improving these environments will benefit the end-to-end testing.
Test data
Test data is used within test environments. For the test data, too, a higher maturity level in the management of test data will make the delivery process much more optimal and a reliable prediction can be made about the behavior of the functionality to be delivered. For more detail on test data management refer to chapter 31 of the book.
End-to-end QA process
Finally, we also recognize the end-to-end QA process as an area of attention. This of course includes the well-known end-to-end test process, but also the orchestration of all these areas of attention. In fact, it indicates how you work as an orchestrator. The entire end-to-end quality and end-to-end test process will become recognizable, transparent, efficient and effective, with controlled release moments and a continuous improvement process will arise.
In the end, a learning and transparent chain of “processes, data, applications and people” will arise in which the agility and dynamics to deal with changes, are in people's genes. Completely in accordance with the Agile way of working.
Organizing end-to-end Quality
End-to-end QA at scale may require expert knowledge and experience that is not available in the agile teams. In that case, this knowledge and experience can be made available through teams outside the value stream. In a scaled environment we distinguish two types of teams that are involved in achieving the required end-to-end quality:
A Support Team E2E Quality (support team) is a dedicated team that consist of specialists that support the Continuous Delivery process of one or more Value Streams from an end-to-end Quality perspective. The support team also organizes the orchestration of end-to-end quality. Having a team facilitating agile teams with CI/CD and overall release is advised. End-to-end Quality can be part of their scope. In this way this Support Team end-to-end Quality does not have to be a completely new team.
The support team includes at least:
- End-to-end Quality Orchestrator for organizing end-to-end quality,
- System Architect who describes the high-over solution for a Feature,
- Agile Quality Coach who supports the agile teams on improving QA,
- Test Automation architect to support the agile teams on TA level,
- Operations (or Ops)-person for support on operations activities.
Virtual Teams (VT) are created with members from the agile teams within a Value Stream. These virtual teams bring skills & knowledge together from different parts of the chain. In some organizations these virtual teams are called communities or guilds.
Benefits of Virtual teams:
- Direct input and sharing of existing and actual specific knowledge & expertise on both content and technique.
- Up- and Downscale team resourcing based on actual Business needs.
- Test specialist’s mindset grows from delivering business value on a team level to solution level which benefits team quality.
- Process integration benefit: end-to-end activities receive the priority based on actual business needs.
In addition to these two types of teams, specialized and scarce expert knowledge can be made available through Shared Services. They represent the special roles, people, and services required for the success of a value stream but are not full-time dedicated to one value stream. Since these shared services resources are specialized – often single-sourced and typically quite busy – each value stream must plan in advance at what time which shared service is needed. The Shared service team has the autonomy of helping only the teams that want to be helped. They likely work closely with the Support Team end-to-end Quality to support them. For example, there can be Shared service for system-level integrations and build environments, Financial solutions, DMS, etc.
Scope of Virtual teams
We distinguish two types of activities to be organized with the use of Virtual Teams.
A team that consists of quality engineering specialists and acceptors to refine end-to-end User stories and for other standard activities such as sprint planning and sprint retro. This Virtual Team meets for a limited number of hours every week for these activities.
The Virtual team includes a delegation from Business and Operations as acceptors. By involving them at an early stage, not only their involvement increases, but also their confidence regarding the end result increases at an early stage. They enter their additional criteria per user story and are taken by hand in the intermediate and end results. This way, optimal results can be achieved with minimal effort.
Secondly a team for organizing end-to-end improvements on the areas of interest. This is a flexible team as each Feature or User Story may require different expertise (e.g. GDPR knowledge). For these areas mainly the Shared Services are used. The delegation of Operations has an important content role from the support team, as most of the improvement areas are his/her area of attention.
Since the quality risk analysis & test strategy affect the quality of the solution but also the quality of the tests, it must be considered by both Virtual Teams.
Velocity
The most important point of discussion will be about the impact a virtual team has on the velocity of the teams.
To stay in IT terms, the answer is binary:
- Yes, a part of the specialist hours of the own Product team will be spent on end-to-end activities, which means that it has impact on the team velocity.
- No, it doesn’t have impact on the team velocity because teams work in Scaled Agile and are part of a larger solution. It’s all about delivering Business value. This is not only done within the teams, but also on a higher solution level. So, it is valid to use a part of the team velocity to use for solution activities to deliver Business value. It is all about mindset.
To visualize this the PO’s of the individual teams will define end-to-end User Stories, based on information the support team will provide before a planning session. Hours will be written within the individual teams, so the velocity won’t be impacted.
Roles & Responsibilities
This section explains the roles, the teams and their responsibilities that are key in organizing end-to-end quality in a scaled environment and deviate from the standard roles that the most common “@scale frameworks” already provide.
Roles
End-to-end quality orchestrator
To organize end-to-end quality of a business process supported by multiple IT components and/or systems, we identify the role of end-to-end quality orchestrator (often indicated as “Orchestrator”). The Orchestrator will be responsible for organizing end-to-end Quality. A central position via the support team and with the mandate of a PO, are the keys to success of this orchestrator role. The most important stakeholder for the Orchestrator is the Business Owner(s).
Test Automation architect
As the automation of end-to-end Testing is a specialist’s area regarding tooling and expertise, it is wise to have a Test Automation (TA) architect in the support team. This specialist will have the responsibility to visualize in which areas TA adds value and organize TA for the end-to-end Test. This specialist can also support the individual Product teams in evolving TA on a smaller scale, which also results in more control on end-to-end quality.
Agile Quality Coach
An Agile Quality Coach helps the organization improve their Quality Engineering activities. This is done on both team and organization level. An Agile Quality Coach helps teams improve their skills and knowledge on QA and testing and on organization level helps with aspects like drawing up and monitoring vision, policy and guidelines. But also leads the communities of practice in the field of QA and testing, that discuss and develop the needs in the field.
Responsibilities
Support Team E2E Quality
Responsibilities of the support team are included in the arguments why the support team must be given the responsibility to organize orchestration of end-to-end quality, for example
- Take care of a QA infrastructure, including toolkit.
- Support System- and Solution integration and releases.
- Support System- and Solution demos.
- Organize and manage end-to-end tests.
Not only in terms of ensuring that a framework for continuous integration is established for individual systems, but also that at least once per sprint an integrated version of the complete solution across systems is established. It is also the support team that is responsible for specifying and organizing end-to-end tests. Test execution (of parts) of the end-to-end test are covered as much as possible by the Product Teams trough the virtual team. This way, end-to-end testing can be limited to some final checks. The remaining is already organized, arranged, and tested by the team.
End-to-end Quality Orchestrator
Responsibilities of the Orchestrator are:
- Create control on quality when implementing changes in the chain of applications.
- Create and secure transparency and communicate and report on this.
- Support the continuous improvement process and organize an end-to-end test.
Furthermore, the Orchestrator is concerned with, for example:
- Perform stakeholder analyses.
- Estimate structural efforts of virtual team members and arrange these efforts.
- Prepare available tooling regarding backlog items and test-ware.
- Define and prioritize backlogs for end-to-end Quality, end-to-end Test (manual and automated) together with Business Owner(s).
- Organize process-wise alignment of Features with Value Stream external parties.
- Organize Virtual Team sessions (Refinement, Sprint planning, Sprint review, Sprint retro, …) First meeting including the BO!
- Complement DoD on end-to-end level.
The Orchestrator also participates in (amongst others):
- The PO/Backlog Sync, to discuss the progress and priorities of Features,
- High-level refinement, to identify end-to-end impact on feature level,
- Planning sessions to provide insight on the end-to-end impact.
Artefacts
Backlog
The End-to-end Quality orchestrator is the spider in the web for organizing end-to-end quality and for organizing the end-to-end test. Organizing the underlying activities is being done via the commonly used model of the backlog of Epics, Features and User Stories.
As for the end-to-end test, a Feature is the first possibility to define whether there is an end-to-end impact. The System architect describes the high over solution for a Feature and points out the teams involved. More than two teams result in a possible end-to-end impact.
The end-to-end Quality orchestrator translates these Features in End-to-end Test User Stories and refines them with the relevant virtual team. The priority of the end-to-end test user stories is reflected from the related Feature.
The input for possible improvement areas comes for example via the individual Product teams (retrospective), the value stream (Inspect & Adapt/Big Retro) or via audit results. This input results in new Improvement Features. The priority of the improvement Features is determined together with the Business Owner(s).
These improvements are reflected in so-called Improvement Stories (that is not relevant for a Team but for the value stream to ensure quality of the chain). The Orchestrator keeps in touch with the external end-to-end parties and shared services to discuss priorities and monitor progress of the Improvement Stories.
Consequently, this results into two types of backlog items. One for the End-to-end Test User Stories. One for Improvement Stories, which are captured in the support team backlog. This last area results in a fully integrated continuous improvement process.
Quality risk analysis & test strategy
As generally known, at Team level, a quality risk analysis is required to give focus to the team for their quality engineering activities and is input for determining the test strategy. For the end-to-end business process it should be defined in collaboration with the other involved teams, because the teams together own quality of what they create, so they need to own the strategy for how to test too.
The test strategy will often contain test activities outside the team(s) as well: how to do end-to-end testing, what to do about non-functional testing on solution level (if this cannot be handled by the teams) how to support the teams in getting the skillset needed to ensure quality of their own work, and of the solution as a whole.
On Solution level, there will be also focus on identifying and mitigating cross-team quality risks which might come from feature dependencies. Therefore, it is recommended to define and implement a (overall) quality strategy that defines what is needed in order to have an integrated and tested solution delivered to customers. Of course, its implementation will also have an impact to the strategy on team level.
As a result, you need to do the quality risk analysis both for the increment just as well as for the individual sprints.
The quality risk analysis could very well be part of the Feature definition during the PI planning, including their acceptance criteria and priority. Criticality from the business perspective (impact) and the complexity of the Feature (probability) complete the analysis.
More in depth information is to be found in the building block: Quality risk analysis & test strategy.
End-to-end regression testing
In high-performance IT delivery teams, the work is done incrementally. Changes to existing software or fixing an anomaly, together with the CI/CD principles, brings a great need for regression testing. A regression test is aimed at verifying that all unchanged parts of a system still function correctly after the implementation of a change.
The focus in end-to-end testing is on multiple interacting IT processes or IT systems. These regularly are indicated as a “chain of IT systems”. The term chain easily leads people to think this is a one-way flow of data through the chain. Often however the test object of end-to-end testing is far more complex, a so-called “mesh of IT systems”. The figure below indicates the difference. People involved in organizing end-to-end regression tests should very carefully determine the exact scope of their efforts and align their activities accordingly.
In many cases, it will be hard for an individual team to perform an end-to-end regression test, there the virtual team comes in play. Autonomous teams check whether the adaptions they make don’t break the existing processes. In a scaled environment, teams are working on a shared code base. This implies that other teams make adaptions that may trigger regression and have an impact on the costumer journey.
In a Scaled environment, when looking at regression testing, we still aim for Shift left and applying the test automation pyramid. In other words: End-to-end quality orchestration is focusing on dealing with dependencies within the chain as early as possible in the process. Resulting in minimizing the effort needed for a full end-to-end regression test.
Automation provides the ability to quickly regression test the system, enhancing continuous integration, refactoring, and maintenance. Needless to say, that an analysis of costs/benefits applies.
Related End-to-End Building Blocks