It is the responsibility of data professionals to protect business data. Changes to the structure and coding of software essential to organizational operations must be executed with little down time or data loss. Consequently, database administrators work tirelessly to prevent system crashes and data failures. However, the risk of failure in deployment, while possibly minimized, can still exist.
Release Management tends to anticipate positive results. However, real world technology at times brings about undesirable results when the release doesn’t go precisely as anticipated or planned. When you have tested software changes to the database before confirming changes into the system, and you have test automation integration; QA has performed analytics with refined testing metrics; scripts have been tested in staging against a copy of the database, and still red error messages cascade down your testing screen indicating a crash and burn of your hard work and concentrated efforts, what to do now?
What are the options for getting back on track when operations respond to software releases with system crashes? Do you use a backup to restore system functions? Do you idealize the system with abstractions then switch back to the real-world issue? Or do you execute a software rollback to revert the deficiencies while preserving the database?
One way to carry on with software functionalities is to fix or patch specific software issues and execute another release. However, fixing the software only addresses the application deficiencies. Software fixes do not approach the possibilities of system or interface errors.
Therefore, when things go awry, software rollbacks are the all-inclusive manner of return to a point before the act-up occurred. Rollbacks return the software database to a previous operational state.
To maintain database integrity, rollbacks are performed when system operations become erroneous. In worse case scenarios, rollback strategies are used for recovery when the structure crashes. A clean copy of the database is reinstated to a state of consistency and operational stability.
The transaction log, or history of computer actions, is one way to utilize the rollback feature. Another is to rollback through multiversion currency control, an instantaneous method of concurrently restoring all databases.
At times rollbacks are cascading, meaning a failed transaction causes many to fail. All transactions which the failed transaction supports must also be rollbacked. Practical database recovery strategies avert cascading rollbacks. Enterprise, development, and QA departments therefore consistently seek to devise the best strategies for software rollbacks.
Best practice strategies are to avert the need for software rollbacks through incremental and automated testing within development and QA environments. However, even with iterative testing, software/system failures do happen.
A Sound Release Management Plan
Sound development, testing, and deployment focused on the end user and system performance are the fundamentals of release management. The release management team of developers, quality assurance teams, and IT system administrators perform activities geared towards successfully completing deployment of software applications. Release management teams must ensure that there is a plan to smoothly recover from deployment calamities. Planning mechanisms focused on documented rollback procedures which effectively enable recovery from deficiencies in deployment:
Build Servers Over Backups
Backups require sizable computer resources with uncertain success. Backups themselves and backup recoveries are slow and tedious. Backup strategies are also inconsistently verified with source code fundamentals and reliability, as well as with raw data. Recovery with backups mean longer user restricted mode to prevent data being added. Rollbacks require running transaction log backups in conjunction with recovery rather than quicker and more reliable rollbacks from a build server repository. The delays in recovery with transaction log rollbacks only increase as the size of the database increases.
Automated scripts can be created with build servers, while with transaction log backups scripts must be manually created and tested. Rollback scripts are some of the most difficult aspects of application development and deployment to create and maintain, especially when data is updated. Maintaining rollback scripts with structural data updates will likely require complex data migration scripts. Finally, when errors are discovered during the release, and new transactions are in place, backup rollbacks will cause the loss of data. Rollbacks from build server repositories providing automated updates are much more reliable for system recovery.
More on Abstraction Layers
Branching by abstraction to gradually change a software system to allow releases which are concurrent with changes, is a fairly common coding practice. However, the method of introducing change to a system through the use of abstraction layers for the purpose of enabling supporting services to concurrently undergo substantial changes has not always been universally recognized as a rollback facilitator.
Abstraction occurs without requiring change to front-end coding, unless desired. The built-in background database abstraction layer of stored procedures can work on top of the underlying system architecture to accommodate additional parameters, such as version numbers or flags within the rollback routine, to fulfill coding commands while both new and old software versions continue to function. Updates to the structure and the abstraction layer which supports structural change allow you to introduce new coding alterations while leaving existing code untouched. Abstraction layers not only smooth deployment, but also simplify rollbacks. With abstraction layers, rollbacks merely need to consider how the code path relates to the original functionality, which abstraction layers have left in place.
Abstraction layers do however require a thorough, and preferably automated, test procedure to prevent errors when older code travels through newer abstraction paths. Fundamental database functionalities are retained, requiring consideration of original coding stability.
Blue/Green deployments are automated. Automating deployment reduces system resistance and delays. Automated deployments implement continuous delivery and continuous integration with no downtime and very little risk. From two copies of the database, Blue/Green deployments deploy to one final copy. With two copies in blue mode, if one copy does not deploy well, there is the option to switch to the second copy, retaining all data. Once you have validated that deployment is fully implemented the application can be switched in green mode to the database.
Blue/Green deployments simplify rollbacks in that the old database and old coding are never accessed. Even when transactions have occurred after release, immediate switch backs to the old system can occur, avoiding the restore process or the need for rollback scripts. One of the safest and most efficient deployment and rollback mechanisms are Blue/Green deployments.
Image Source: Figment Engine
A Blue/Green deployment however requires a database that is small enough for two copies of the database to be accommodated on one server. In addition, reliable synchronization or reliable data migration creation and maintenance are required to sync data between databases.
Inherent issues with database deployment is retaining data when failures occur. Combining rollback strategies with Blue/Green deployment can further reduce risk of failed deployment, as well as better ensure complete and efficient rollback recoveries. Combining strategies allows for more agility and flexibility for stable deployment or required rollbacks.
Rollbacks and the Enterprise
Enterprise requirements for cost efficiency and ROI dictate that software and system business functions are reliable. The length of time required for rollbacks necessarily becomes an enterprise issue. Processing scenarios dealing with database updates, rollbacks, and recompiles are extremely efficient in avoiding and reducing the time required for rollbacks.
Decentralized development even further dictate that the deployment pipeline be standardized per enterprise priorities. Transparency in process and performance, as well as collaboration and enteprise test management are crucial to communication with management concerns. Diffused independent responsibilities rather than centralized protocol, place reporting responsibilities and further obligations on developers, QA, and IT administrators to continuously collaborate with business management and stakeholders.
Rollback procedures must be executed within organizational boundaries with a discreet and defensive stance towards possibilities of risk. The process of rollbacks and recovery are direct in process and results. Rollback operators therefore bear responsibility for directly communicating processes and collaborating on enterprise priorities in relation to executions. To assure that the rollback process only positively impacts business priorities, thorough and consistent collaboration is imperative between operational concerns and enterprise priorities.
Theoretical or abstract considerations can commonly be overlooked as important for communication to management. However, theoretical rollback considerations can change the import or direction of deployment outcomes, which can pivot the objective away from business goals and priorities. Reliable messaging and reporting, as well as face-to-face collaboration, is crucial within rollback activities.
The spectrum of rollback protocols in free and specialized domains can breathe new life into the stability of software/system interfaces. Efficient rollback strategies reduce costs to the enterprise, enhance deployment time to market, and engage customer loyalty through efficient operations.