The complexity of software development

A recurring and common theme in conversations with several of my friends and colleagues is the increasing challenge they face in developing and delivering successful enterprise software applications. The pressure is high as was evident on the face of a friend and neighbor whom I met walking into our apartment complex late one night with a completely dazed expression on his face. All he could mumble about was the issues of performance they were trying to iron out and his struggle to estimate if he needed additional project time. This guy is a project manager at a fairly well known technology services organization based in Bangalore.

Why is developing software for the enterprise so hard? Surely with the advances in tools and technology, with established processes and experience life should become a little easier. However this is not the case and probably because there has been significant changes to the domain in which software is developed and used at a rate which has outpaced the rate at which we have adapted to these changes. In this post, I have described some of these changes and their implications. In future posts I will explore how these challenges themselves could be rephrased as engineering problems and possibly solved using design. Design not of software but of the method of making software. But first the challenges:

There has been a fundamental shift in human society over the last couple of decades that has touched and impacted all of us. Access to information, automation and global communication has, to borrow a phrase – made “geography” history. We live today in a world where computing is taken for granted, where it is becoming increasingly difficult (and unnecessary) to tell the difference between a cell phone and a personal workstation. The average young person today, who incidentally is the target customer for the products and services offered by several competing enterprises, is completely comfortable with the notion of an online existence and with services and products delivered online. This young person is not just comfortable but is extremely knowledgeable, very demanding of quality of service and has very little patience before switching to a better set of options. This young person is now an integral part of the existence of many enterprises. Her tastes and expectations have a significant impact on the development of enterprise application systems. Paralleling the evolution of this young person has been the evolution of the internet itself. The net 2.0 and 3.0 as some people refer to it is all about collaboration. The net has progressed from an information dissemination mechanism, to a viable commerce mechanism to now being truly a collaborative network that brings people and organizations together. Collaborative services are where the true power of the net lies. The same evolution is also happening within the business strategies of enterprises. Collaborate to compete is the mantra and the provision and consumption of services is the environment businesses live in. An interesting description of the service oriented economy is the statement that businesses are comfortable using services provided by resources they neither own nor control.

This shift in market and social dynamics has had a significant impact on the nature of enterprise information systems and how they are developed today. The fundamental changes that are happening include:

1. Software today is business critical – Automation, centralization of critical information and enterprise application integration have created environments where manual processing of information has been almost completely eliminated. While this has brought great efficiency to the enterprise – the dependence on enterprise systems to process information is total.
2. Enterprise information systems are not built and operated in silos any more. Integration between applications across business functions is almost a necessity.
3. For several enterprises whose products are services such as banks and insurance companies, the notion of fixed and silo products is disappearing. Products are built and rebuilt all the time by repurposing a set of underlying services and functions
4. Software systems are often the direct interface to the business’s customers and stakeholders. Unlike the earlier days of terminal and back office processing where the customer was separated from the IT systems of an enterprise by an effective wall of human interaction, the customer today is a key user of many enterprise IT systems.
5. The distinction between classes of systems such as content management, transactional and analytical systems is vanishing in the eyes of the user. The same user often uses all three and expects consistency across the systems
6. Increasing software complexity – every generation of software is functionally more complex and has a richer set of features. Every set caters to a larger and wider span of audience with a more diverse set of needs.
7. The growing importance of standards – the business, operational and political environment in which software is used today mandates compliance with a plethora of standards such as SOX and BASEL2. Adherence to a standard is often not an option for businesses today.
8. The growing cost of software and the increasing pressure to reel this in – software systems today have astronomical Total Cost of Ownership figures. Organizations are adopting multiple strategies to bring these costs down including global outsourcing of maintenance and management, centralization of systems, leveraging pay-per-use and hosted service models and increasing adoption of open source technologies
9. The management of enterprise systems is no longer considered to be something that must be retained within the enterprise. Outsourcing the management and delivery of the service to managed service providers is the preferred option. Enterprise applications are today expected to be built to integrate with larger enterprise management contexts
10. The service oriented universe – enterprise systems are not built as isolated silos any more. Well defined service interfaces form the basis of their integration and interoperability and make them essential to every part of the business. Expectations of quality of service of enterprise systems are significantly high. The cost of unavailability or degraded service is in direct monetary terms and of significant business impact
11. Maturing of open sources – the success of open source technologies such as Linux and the JBOSS application server coupled with innovative licensing models and lower cost of ownership has led to a significant acceptance of open source within enterprises. The collaborative development approach and sourcing of specific components from small vendors and the open source environment is the way to go in many enterprises
12. Pressure to reduce the software development times – software is moving from development to configuration. The expected time to deploy a new software system has gone down from 2 years to nine months to 3 months over the last few years.
13. Pressure to adopt methodologies such as agile development – the success of agile development methodologies and collaborative, iterative development is accepted and is becoming increasingly the approach development teams adopt. The biggest challenge teams face is integrating these methods into the context of overall organization level processes such as CMM and ISO
14. The growth of enterprise architecture as a strategic part of an enterprise’s business strategy – for several enterprises whose products are services such as banks and insurance companies, the notion of fixed and silo products is disappearing. Products are built and rebuilt all the time by repurposing a set of underlying services and functions. The need for a common enterprise architecture across these services and the way they are used is critical and enterprises are increasingly investing in developing such architectures
15. Developer turnover, rapid promotions and astronomical growth – software development is an increasingly competitive and expensive venture. Small companies I meet often describe themselves as training grounds for fresh developers who spend a couple of years gathering skills and experience and then move into the race for higher pay by jumping to larger and multi-national development organizations. 2 years seems to be the magic number for stability in this space – after which the itch to switch kicks in. To combat this organizations adopt multiple strategies including astronomical salary growth (the % increase in Indian salaries this year is amongst the highest in the world) and bizarre promotions quite out of perspective with number of years of experience (entry level developer for 2 years, senior developer for another 2 years, tech lead for 2 more, manager and senior manager in another 4 and president of the country in 24 years?) Then there is the issue of growth in software organizations – you have around 10 to 15 thousand new employees joining one of the larger services organizations every year – imagine the challenge of settling them in and getting them to be productive parts of teams
16. Increasing salaries vis-à-vis decreasing cost of production – the fact that the cost of ownership of software is already out of proportion is well known and there is a universal push to rein this in. Development teams in this space face the significant issue of trying to balance increasing salary and operating costs against the pressure from their customers to deliver more complex software at lower costs

So what do these changes imply?

1. Architecture and design are super-complex requiring a team of super specialists. Take for example the architecture of the virtual university that I am currently working on, we need expertise on distributed application design, large scale data modeling, semantic networks, interaction techniques and learning management, multi-media technologies in the context of virtual classrooms, networking expertise, interfaces with broadcast media, standards such as SCORM, setting up collaboration systems and many more.
2. Architecture and design needs more than technology skills – systems thinking, problem solving, abstraction and generalization, communication using different languages to different stakeholders, integration skills and not the least project planning skills are all equally important.
3. There is an increasing need to move away from single system engineering to the development of platforms that evolve over time using progressive improvements in functionality are offered to users.
4. The productivity challenge of building more complex teams with cheaper resources at a faster pace is severe – several approaches are practiced including COTS, reusability, agile programming and so on. These approaches in turn add to the quality engineering problem of building systems whose stability and performance can be assured.
5. For a long time now, design Gurus have been advocating simplicity as one of the key factors to the stability and quality of a system. The environment is not going to get simpler – nor are the challenges of development going away. Simplicity has to be arrived at through conscious effort to improve engineering quality
6. Simplification of the development universe for a development team has in itself to be considered as a design problem and solutions such as those based on product lines and platform oriented development have to be engineered and invested into
7. NFRs should probably be renamed as Important Stability Requirements – somehow I have a sneaky feeling that by calling requirements that specify the stability, performance and such characteristics of a system Non Functional Requirements, we have done systems a big disservice. It seems to me that the “Functional” part is often silent and we read this as “Non Requirements” either consciously or unconsciously. Why else do we ignore these for the most part in our design and deal with them only at the end of a project after we have built most of the functionality? The stability of a system that needs to operate with the challenges I listed earlier is as important if not even more so than the functionality it provides. I propose that we start thinking of NFRs as Important Stability Requirements – change the name, drop the “NON” and see the difference it makes
8. Manageability is a the new key ISR – performance, availability and even reliability have traditionally been stated and defined as expectations of stability of a system. The critical addition to these in today’s business environment is manageability. Manageability is the index of what is built into an application to be able to guarantee that it will operate in its environment with predictable quality of service

So what?

Life is tough for organizations developing software to be used in the enterprise space. They face pressure on all fronts – what they need to build is more complex, the knowledge they need is more intense and specialized, they face severe resource pressures and are driven to improve efficiency of production and the effectiveness of what is produced. There are not many magic bullets available – there are however some medicines that one may try but these call for patience and a little investment into engineering that is not in the context of what is required for a project. Will these organizations avail of these medicines – your guess is as good as mine, but I do believe that most of them will.
Mañana las medicinas. Some mañana that is

Advertisement