ISSN 1566-6379 57 ©Academic Conferences Ltd
Reference this paper as:
Hallikainen P and Chen L (2005) “A Holistic Framework on Information Systems Evaluation with a Case Analysis” The
Electronic Journal Information Systems Evaluation Volume 9 Issue 2, pp 57 - 64, available online at www.ejise.com
A Holistic Framework on Information Systems Evaluation with a Case Analysis
Petri Hallikainen1 and Lena Chen2,
1Information Systems Science, Helsinki School of Economics, Finland
2General Education Center, National University of Arts, Taiwan
petri.hallikainen@hse.fi
lenachen@info.ntua.edu.tw
Abstract: This paper presents a framework for understanding IS evaluation in its broader context. The role of IS
is applied to analyze a single IS project in details. The results show that sometimes formal IS evaluation might not be
important or necessary, but rather it may be more important, with an informal and flexible evaluation process, to quickly
gain experience of a new kind of business and system to maintain a leading position in the competitive market.
Keywords: information systems projects, IS evaluation, organisational context, holistic framework on IS evaluation
1. Introduction
It has been widely noticed in the literature that information system (IS) evaluation is a very difficult task involving a variety of dimensions (Smithson and Hirschheim, 1998; Irani, 2002; Peffers and Saarinen, 2002) and various stakeholders (McAulay et al., 2002). IS investments often include intangible benefits (Powell, 1992) and the benefits are often realised during a long period of time (Saarinen and Wijnhoven, 1994). Ad hoc practices for IS evaluation are frequently reported (Irani and Love, 2001) and only simple methods, like payback period, are used in evaluation (Lederer and Mendelow, 1993). This is in sharp contrast to management theory suggestions that IS investments should be evaluated using a covering set of criteria. For example, Peffers and Saarinen (2002) stated that evaluation of IT in financial terms may be biased toward the most easily measured benefits and prone to manipulation to justify predetermined investment decisions,
resulting in systematic over or under-investment in IT. Some contingency models for selecting evaluation methods for IS investments have been presented in the literature. At organisational level, contingency factors may include, for example, the industry situation (stable or changing), and the leadership role of the organisation (pioneer or follow) (Farbey et al., 1992). At IS project level, contingency factors may include, for example, project types, project sises, the type of expected benefits (qualitative vs. quantifiable), the stages of
the system’s life cycle, and development and procurement strategy (see e.g. Farbey et al., 1992; Hochstrasser, 1990).
As a result of their study on IS evaluation methods, Peffers and Saarinen (2002) divided evaluation criteria into five broad categories as follows: Strategic value; Profitability; Risk; Successful Development and Procurement; and Successful Use and Operations. The above categories might be seen as universal for evaluation in any particular case. Moreover, Irani and Love (2002) presented a comprehensive frame of reference for ex-ante IS investment evaluation. However, the need for a better understanding of in-context IS evaluation still exists.
This paper aims to provide an instrument for understanding IS evaluation in its broader context. The instrument is expected to be of value to both researchers and practitioners. In the next section, a holistic framework on IS evaluation is presented. Then the empirical research methodology is described and the presented framework is applied in a case analysis. Finally, the paper concludes with discussions and
conclusions.
2. A holistic framework for analysing IS evaluation in context
The outset situation of the IS investment project consists of the organisational norms and values, project specific contextual factors and the resources given to the project. The actual outcomes of the IS investment project are produced in conjunction with the business development process, the IS development and
procurement processes. The outcomes of the IS project are defined by the success of the system with respect to the investment perspective, the success of the IS project implementation and the success of the desired IS functionality. The basic idea in our analytical framework, presented in Figure 1, is the inclusion and integration of the IS evaluation process into business development and IS development processes. The evaluation
categories defined by Peffers and Saarinen (2002) are suggested to be applied in all IS projects, and in projects with great uncertainty they should be applied frequently, to iteratively
assess the system investment throughout its lifecycle. Furthermore, the IS evaluation process should also adapt to the possible changes in the assumptions that the IS investment is based on, thus in our conceptual framework there is a two way relationship between the IS evaluation process and, business development and IS development processes. The components of the framework are discussed in more details.
2.1 Outset situation
2.1.1 Organisational norms and values
The investment characteristics and the organisational environment affect the way in which the evaluation is conducted (Huerta and Sánchez, 1999). Organisations operate and survive through organisationally accepted rules that are justified by goals or a hierarchical goal system. Within organisations, there are individual goals, objectives, desires, wishes, intentions, etc, as well as organisational goals, objectives, missions, etc. (Kivijärvi, 2004). As pointed out by Hallikainen et al. (2002), any strategic investment process employs individual and organisational values and preferences, goals and objectives as an input. At best, evaluation would help the
organisation to understand its processes, problems and opportunities, thus facilitating organisational learning (Barrow and Mayhew, 2000).
2.1.2 Project contingencies
An information system can, on the one hand, be a small application supporting only one single activity, but on the other hand, it can be a wider system supporting the whole company, or it can even be an inter-organisational system. There is one additional type of information system that deserves special attention, namely infrastructure investments. Infrastructure investments are of high importance because they create the platform on which future applications can be built (see e.g. Dos Santos, 1991). Moreover, why the information
system is actually built, depends on several factors. In some situations a company may be forced to build a new information system, e.g. because of legislation changes. Additionally, the senior management may perceive that the system needs to be built, for example, to support a business strategy. Finally, the arguments for building the system can be from the expected and clear quantitative or qualitative benefits from the
investment perspective.
The nature of the investment varies according to the novelty of the system. An investment can deal with improving an existing system, replacing an old system or developing an entirely new system. The nature of the investment differs according to how common this type of system is in the field of industry where the company operates (see e.g. Saarinen and Vepsalainen, 1994). For example, investment in a routine system is different from an investment in an innovative system.
2.1.3 IS project resources
Both the material and the immaterial resources are crucial while developing information systems. The integrating role of evaluation includes detecting possible problems and as a result of evaluation it may be noticed, for example, that the project needs more system development resources.
2.2 Business development process
As information technology can make alternative operational designs possible, it in many cases lays a central role when developing the company's business strategy. Furthermore, IT enables new kinds of flexible inter-organisational arrangements (Venkatraman, 1994). Moreover, information technology can support the development of new business, or new products and services. Thus, IS projects are often connected to larger strategic business development programs and the role of IS evaluation would be to ensure that the IS project could deliver the required technological capabilities for achieving the strategic business objectives. The degree that an IS project is involved in business development can range from a system that supports the current business strategy to a system that creates competitive advantages and new business opportunities to the company. An information system investment is in any cases an important part of a business process re-egineering project. The actual aim of the system investment would be cutting costs, improving products or services, or serving a certain customer group better.
2.3 IS development process
Read the rest of the article at:A Holistic Framework on Information Systems Evaluation with a ...
No comments:
Post a Comment