https://docs.google.com/a/staff.uksw.edu/file/d/0BznUozOdnQRUZVJNZFdHYlFENTA/edit
EISSN 2231-279X – ISSN 2249-0280 16
INDIAN JOURNAL OF MANAGEMENT SCIENCE (IJMS)
THE USED OF IT BALANCED SCORECARD TO BUILD THE
PERFORMANCE MEASUREMENT MODEL OF ACADEMIC
INFORMATION SYSTEMS (CASE STUDY ACADEMIC
INFORMATION SYSTEM OF SATYA WACANA)
Paskah Ika Nugroho,
Faculty of Economics and Business
Satya Wacana Christian University, Indonesia.
Prihanto Ngesti Basuki,
Evi Maria,
Faculty of Information Technology
Satya Wacana Christian University
Indonesia.
Faculty of Information Technology
Satya Wacana Christian University
Indonesia.
ABSTRACT
The aim of this research is to make a model of performance measurement of academic information
system to facilitate the auditors in conducting a periodically performance measurement of Satya
Wacana Academic Information System using IT Balanced Scorecard. SI performance measurement
model was developed through systematic measures in the form of the action process, reflection,
evaluation, and innovation by applying the method of survey research, development, experiments ,
and evaluation. Performance measurement modeling of Academic Information Systems (SIASAT) in
SWCU has been done by making a framework model which was developed by considering the
following parameters: (a) the duties and functions of the university, (b) the aspects of university
management, (c) the duties and functions of the IT organization in university, (d) the need of
information system for academic activities, and (e) the methodology of IT basic framework used,
which is the IT Balanced Scorecard (IT-BSC).
Keywords: IT Balanced Scorecard, Academic Information System, Performance Measurement.
www.scholarshub.net
Vol.– III, Issue – 3, July 2013
INDIAN JOURNAL OF MANAGEMENT SCIENCE (IJMS)
EISSN 2231-279X – ISSN 2249-0280 17
Introduction:
The use of Information Technology (IT) in Higher Education institutions especially for the use of information systems and the
Internet can not be separated due to the demands of the stakeholders (Indrajit, 2006). IT Management in Higher education
institution is a Critical Success Factor (CSF) for leaders and partners of Higher education institutions (Henderi, 2010).
However, the complexity of IT implementation makes the leaders of the various levels in the Higher education institutions and
stakeholders have difficulty in managing the IT. The complexity of IT implementation in higher education institutions in
Indonesia happens because the higher education institution does not have a specific framework model when establishing the
information system (Mutyarini and Sembiring, 2006). As a result, the benefits of using IT is not comparable to the investments
value which has already been incurred.
Satya Wacana Christian University is one of the universities, which has already used IT as an infrastructure and facility to
provide services for students, lecturers and all the staff, and also assists the running of the activities around the work units. In
carrying out its main activity, that is to provide educational services, SWCU has supported by IT of Satya Wacana Academic
Information Systems (SIASAT). IT management has been applied in SWCU, but it has not been applied using a well-structured
method and approach. On the other hand, IT implementation must be controlled because the control provides reasonable
assurance to management that the implementation process has been done in accordance with the plans and goals of the
organization (Maria, 2011).
Each IT process requires a controlled IT measurement to indicate the performance of IT in achieving the control objectives and
facilitate the management to make improvements to the performance of IT. IT performance measurement can be performed by
using IT Balanced Scorecard IT where the IT performance is measured from 4 perspectives: corporate contribution, user
orientation, operational excellence, and future orientation (Van Grembergen, 2000). IT Balanced Scorecard is an effective
method of managing IT organizations as well as evaluating the success and development of the system/application, the
development of computer and network investment, quality of products and IT services, as well as improving the quality of
human resources, even though, most universities in Indonesia have not been using this method (Prabowo, 2007).
In addition, monitoring and evaluating towards SIASAT performance has not been done periodically, but only if there are
complaints from the working units about the SIASAT service (Maria and Haryani, 2011). This condition is not consistent with
the results of Maniah and Surendro’s research (2005) which stated that SI performance measurement must be done
periodically to ensure the sustainability of IT operations used by the organization or company as well as to assess the
sustainability between the planning and implementation of the system. Since the importance of SI performance measurement
should be done periodically, this research will try to make a model of performance measurement of academic information
system to facilitate the auditors in conducting a periodically performance measurement of SIASAT using IT Balanced
Scorecard.
Literature Review:
IT Balanced Scorecard:
The balanced scorecard can be applied to the IT function and its processes as Gold (1994) and Willcocks (1995) have conceptually
described and has been further developed by Van Grembergen and Van Bruggen (1997) and Van Grembergen and Timmerman (1998).
IT-BSC has four perspectives: (1) Corporate Contribution, contains a measure which indicates how the management (the manager)
evaluates/views the IT organization; (2) User orientation contains a measure which indicates how users evaluates/sees the results of
the IT organization, (3) Operational Excellence contains a measure of the effectiveness and efficiency of the IT process, and (4)
Future orientation contains a measure which describes how IT position within the next challenge.
Performance Measurement:
Mulyadi (2001) defines performance measurement as a process of assessment on the company operational activities in a
particular period, whether it has been done based on the defined goals or not. The main purpose of the performance
measurement is that the leader of the company has an objective basis in giving the compensation in accordance with the
achievement which has been done by each department as a whole. It is expected that all of these will give motivation and
stimulation in each section to work more effectively and efficiently.
Previous Researches:
IT model development and performance measurement can adopt IT standards such as ITIL, ISO/IEC 17799, COSO dan
COBIT (O’Donnell, E, 2004). Van Grembergen’s research (2000) discussed about how the IT balanced scorecard (IT-BSC) can
be linked to the business balanced scorecard (BU-BSC) and in this way support the IT/business governance and alignment
processes. The considered aspects in the IT application in IT-BSC method are corporate contribution, customer (user)
orientation, operational excellence and future orientation. IT-BSC method used to measure the performance of the
implementation system of Enterprise Resource Planning systems (ERP) at the University. The method was continuously
developed to make a strategic plan which is in accordance with the mission of educational institutions to continue in surviving
www.scholarshub.net
Vol.– III, Issue – 3, July 2013
INDIAN JOURNAL OF MANAGEMENT SCIENCE (IJMS)
EISSN 2231-279X – ISSN 2249-0280 18
in the business competition (Sa'adi and Suhardi, 2006).
IT Balanced Scorecard has not been widely used to measure the performance of information systems in universities in
Indonesia, but actually this method is very effective for managing the IT organization and evaluating its success (Prabowo,
2007). This is because the Universities in Indonesia did not have a specific model of the framework when they build their
academic information system (SI), so Mutyarini and Sembiring (2006) created an academic architecture Information system
model by adapting the architecture of Monash University which used TOGAF in order to achieve the mission of Tri Dharma
higher education.
The previous research on IT in SWCU as the research object, including the study of Maria and Haryani (2011) who found that the
supervision and the assessment towards the IT performance in SWCU has not been carried out periodically, just only if there are
complaints from the users (the working units) about the IT service. This research produced a model of information audit system
which is developed using the COBIT framework especially for delivery and support (DS) domain. Maria‘s research (2011) also
found that so far the IT management in SWCU has been done, but it has not been done using the structured method and approach.
This research also choose to use COBIT framework in doing the comparison among the academic information systems because
COBIT can notice the link between the business goals without neglecting the IT process as the focus. Maria’s research, et al
(2012) found that IT in SWCU has been well managed where IT processes to support business goals has been standardized,
documented and communicated well. There should be a continuous monitoring and evaluation of the IT in SWCU, so the quality
of IT services in SWCU can be improved day by day in accordance with what is expected.
Research Methodology:
This type of research is a combination of descriptive studies that describes the phenomenon that actually occurs in an event or
population and exploratory research that found a model of the SI Academic performance measurement done by doing an
approach on the "Research and Development", that was a research program which was followed up by doing some
development programs. SI performance measurement model was developed through systematic measures in the form of the
action process, reflection, evaluation, and innovation by applying the method of survey research, development, experiments ,
and evaluation.
The location of this study, Satya Wacana Christian University Salatiga Indonesia, was chosen on purpose. Primary data of this
study was the results of guided interviews and observation. While secondary data such as documents, reports and policies are
taken based on the SIASAT
The steps of this study are as follow:
a. Preliminary studies
In the initial study, there were prelimanary research on previous studies, literature and standards that support the research
topic, guided questionnaire drafting, and SIASAT understanding.
b. Data collection
At this stage, the data was obtained by interview, observation, and questionnaires given to the relevant units and users of
SIASAT. The secondary data is also collected from related units of SIASAT.
c. Development of performance measurement model of IS
At this stage, development of performance measurement of IS was managed by interviews, observation and related
documents to state parameters and Critical Success Factor (CSF), which will be used as constraints to determine criteria of
performance measurement of SIASAT based on IT BSC perspective. Then we mapped the steps to measure the
performance of acedemic information systems.
d. Conclusions
In the final stage of this research, a conclusion from all research processes was stated.
Result and Discussion:
Brief description of the Satya Wacana Academic Information Systems:
The internet-based of SWCU Academic Information System (SI), known as SIASAT , is an application used to record data
from each student's academic administration from the entry (admission) to the exit (graduation). This application can be
accessed easily via the SWCU homepage, http://www.uksw.edu address, then go to the SIASAT menu in the ACADEMIC
group, or directly go to http://siasat.uksw.edu. These applications provide an online and a real time of academic information.
All students who are listed as SWCU students, have the right to access the application via the homepage institution. It is
important for SWCU students to know and master this application and its operations in order to see the financial obligations
that they have to pay, course registration, and see the results of their study for each semester.
SIASAT Performance Measurement Model by Using IT Balanced Scorecard:
Pyle (2003) stated that the development of performance measurement model will be based on one of modelings, i.e. the model
which is developed by its constituent components, such as business processes and their correct data components. Performance
measurement modeling of Academic Information Systems (SIASAT) in SWCU has been done by making a framework model
www.scholarshub.net
Vol.– III, Issue – 3, July 2013
INDIAN JOURNAL OF MANAGEMENT SCIENCE (IJMS)
EISSN 2231-279X – ISSN 2249-0280 19
which was developed by considering the following parameters: (a) the duties and functions of the university, (b) the aspects of
university management, (c) the duties and functions of the IT organization in university, (d) the need of information system for
academic activities, and (e) the methodology of IT basic framework used, which is the IT Balanced Scorecard (IT-BSC). These
parameters are expected to be the factors that determines the performance of academic information systems which were
observed, and how these parameters can be controlled and regulated, in order to obtain a desired performance system. The
relationship between the 5 parameters in creating the measurement model of academic information system performance by
using IT-BSC is presented in Figure 1.
Figure 1. The relationship between the parameters in creating the performance measurement model
The main parameters of duties and functions of university in this research is the implementation Tridharma High Education,
such as lectures, working in laboratory, practical work, the implementation of the final project, research and training, and the
implementation of community service. Those are the things that encourage the chief of SWCU to formulate its business goals
by using four perspectives of Balance Scorecard (BSC). The business goals of SWCU are presented in Table 1. To achieve
those business goals, IT infrastructure is provided in the form of the use of computers, information systems implementation,
and the use of internet technology. The information system was built in accordance with the internal business processes of
SWCU starting from prospective new students since they enroll, be accepted, join the lectures, until graduate.
Table 1. Business goals SWCU
BSC Perspective
Financial
Customer
Internal
1
2
3
1
2
3
4
5
1
2
3
4
5
6
7
Learning &Growth
SWCU Business Goals
Provide a good return on investment of IT-enabled business investments
Manage IT-related business risk
Improve corporate governance and transparency.
Improve customer orientation and service
Offer competitive products and services
Establish service continuity and availability
Create agility in responding to changing business requirements
Achieve cost optimisation of service delivery
Improve and maintain business process functionality
Lower process costs
Provide compliance with external laws, regulations and contracts
Provide compliance with internal policies
Improve corporate governance and transparency
Manage business change
Improve and maintain operational and staff productivity
Manage product and business innovation
SWCU leaders also formulate the main aspects that need to be considered in the management of a university. University management
should pay attention to the availability of resources, the process aspects and the content aspects. These parameters need to be
formulated, since universities in Indonesia do not have a standard framework for building and managing academic information
system (Mutyarini and Sembiring, 2006). Those aspects will be managed by the organization's culture, values and work ethic and are
www.scholarshub.net
Vol.– III, Issue – 3, July 2013
INDIAN JOURNAL OF MANAGEMENT SCIENCE (IJMS)
EISSN 2231-279X – ISSN 2249-0280 20
manifested in the form of organization structure and management systems in universities as presented in Table 2.
The parameters of IT management organization in SWCU are handled by the Bureau of Information Systems Technology
(BTSI). IT developments are constantly increasing, so that SWCU must make arrangement in the IT management organization.
BTSI plays an important role for the success of the implementation process. This is because BTSI not only manage the
technical aspects of IT but also play a role in determining the organizational culture in SWCU in using IT. BTSI consists of 2
parts: (1) IT section which is in charge of audio-visual section, parts of the communication network and the Internet, and parts
of computer, (2) information system section which is in charge of software parts, parts of information systems management,
parts of flexible learning & web, and parts of documentation and training. The functions which are managed by BTSI are: (1)
the function of technology development and the application of information systems, (2) the function of maintenance of
information systems applications, database, digital documentation of information system, the content of learning resources,
networks and computers, (3) the functions of settings and monitoring of IT implementations in the form of change management
and user relationship, release system and audit.
Table 2. The key aspects of Universitites Management
Organizational Culture, Values and Work Ethics
Process
Content
The key process is to run Tri Dharma University,
which consists of:
1. Education and Teaching
The curriculum and its management,
Lecturers and Non2. Researches
which consist of instructional materials,
lecturers resources
3. Community Service
the results of the study, the results of
Funds
Supporting processes, which include the community service, scientific forums.
Facilities and
processes of: academic administration, students
Infrastructure
and alumni, financial administration, cooperation
IT Infrastructure
and external relations, and promotion
Knowledge Management both tacit
Information Systems
knowledge and explicit knowledge.
Organizational Structure and Management Systems in Universities
On the other hand, the parameters of the IT framework methodology used in this study use IT Balanced Scorecard (IT BSC).
Based on those IT organization functions in SWCU, it is necessary to establish IT Balanced Scorecard for each of those
functions which is essentially a derivative of the IT Balanced Scorecard for universities. The process of identifying Critical
Success Factor (CSF) is needed to determine the limits of performance measurement criteria in each IT process. The CSF of
Academic Information Systems is presented in Table 3.
Table 3. The Critical Success Factors of Academic Information System based on IT-BSC Perspective
Resources
IT-BSC Perspective
Business Contribution
CSF Academic Information System
Control costs, increase revenue and improve service coverage
Customer value proposition that includes the rates, quality, service provided, service
User Orientation
and partnerships
Improvement of internal processes by implementing the operations management,
Operational Excellence
customer management and innovation.
Enhanced capabilities and skills through the strengthening of human capital,
Future Orientation
strengthening of information capital, and strengthening of organization capital
For the parameters of the IS (Information System) requirements related to academic activities of universities which consist of
information about admissions, student registration, course registration, grades, and graduation. Admission information consists of
information regarding enrollment of prospective new students which covers registration process, the data inputting, photo-taking,
selection, selection, announcement of selection result, printing of Rector Decree about new admission and information regarding resigning up includes taking an acceptance letter, registration payment, informing a bank payment receipt, filling out a registration form,
and obtaining student’s number. The importance of IS related to student registration such as providing information about types of
registration, information of procedures, requirements, and student registration deadline. Student registration is an activity of
registration or recording of active-status as the University's student and must be done by students each semester.
Course registration is a subject registration process as a participant of a course in the current semester. Subject registration
process includes academic supervision, financial (dispensation), internet, course registration schedule of study
program/department. IS ought to provide information related to grade of subject for each student and each semester which will
be presented either in a study result card or an academic transcript. As for graduation activities, IS should provide information
includes registration, graduation ceremony, and diploma delivery.
After determination of the CSF and know the needs of IS for the academic activities, it should begin mapping the steps what
needs to be done in order to measure the performance of academic information system. The methods of academic information
www.scholarshub.net
Vol.– III, Issue – 3, July 2013
EISSN 2231-279X – ISSN 2249-0280 21
INDIAN JOURNAL OF MANAGEMENT SCIENCE (IJMS)
system performance measurement based on IT BSC Perspective are presented in Table 4.
Table 4. The Methods of Academic Information System Performance measurement based on IT BSC Perspective.
IT-BSC Perspective
Academic Information System Measurement
1. Performing control towards IS, for example by comparing with the actual budget, analysing
the use of budgets, calculating the cost of IS per number of staff.
2. Calculating the financial benefits derived from selling products and services.
3. Doing business assessment of the projected new IS when the university will create and
develop IS, for example by evaluating business based on economical information and
Business Contribution
performing financial evaluation based on Return on Investment (ROI), Net Present Value
(NPV), Internal Rate of Return (IRR), Payback Time (PB).
4. Doing business assessment of IS functions such as calculating the percentage of capacity
related to IT strategy projects, analyzing the relationship between development/new
infrastructure and the investment/investment displacement.
1. Assessment towards BTSI associated with applications that have been addressed, the
percentage of applications that have been completed, etc..
2. Cooperation with the users of information systems when it will carry out the functions of IT
organization, for example by calculating the number of users involved in the manufacturing
User Orientation
process and the development of IS application.
3. Analyzing the IS’s user satisfaction by measuring the level of user friendliness on the
application, calculating the index of user satisfaction, counting the number of applications
and system availability.
1. Conducting an analysis towards the efficiency of software development, for example in
terms of the average increase of unexpected budget, maintenance activities, the average
number of delayed response of the application.
2. Conducting an analysis towards the efficiency of such operations by computing network
availability, response time per category per person, the percentage of work which is done
ontime, the ratio of operating costs of the system used.
Operational
3. Conducting an analysis towards the acquisition and application of personal computers if
Excellence
there is any upgrade.
4. Conducting an analysis towards problem sloving if the system is on trouble, for example by
calculating the average charge time, troubleshooting time, the percentage of problems
answered in a timely manner.
5. Conducting an analysis towards the training of the IT users.
1. Conducting an analysis towards the training and expertise of IT staff both in terms of the
budget which is owned by institutions and trained individuals based on the age, expertise,
etc.
Future Orientation
2. Conducting an analysis towards the age of the applications and opportunities for investment
in new technologies.
SWCU Academic Information System Measurement Model by using the IT-BSC is illustrated in Figure 2:
Universities’ Business Goals by
using BSC Perspective
Resources, Process, and Content
Aspects of IT management in
University
IT Management organization
The Methodology of IT Framework
(IT BSC Perspective)
Critical Success Factor
The need of IS for academic
activities in Universities
The methods of Academic IS
performance measurement
Business Contribution
User Orientation
Operational Excellence
Future Orientation
Figure 2. Academic SWCU SI measurement model using IT BSC.
www.scholarshub.net
Vol.– III, Issue – 3, July 2013
INDIAN JOURNAL OF MANAGEMENT SCIENCE (IJMS)
EISSN 2231-279X – ISSN 2249-0280 22
Conclusion:
Based on those IT organization functions in SWCU, it is necessary to establish IT Balanced Scorecard for each of those
functions which is essentially a derivative of the IT Balanced Scorecard for universities. The process of identifying Critical
Success Factor (CSF) is needed to determine the limits of performance measurement criteria in each IT process. After
determination of the CSF and know the needs of IS for the academic activities, it should begin mapping the steps what needs to
be done in order to measure the performance of academic information system.
References:
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
Dorian Pyle. (2003). Business Modeling and Data Mining. Morgan Kaufmann Publishers, ISBN:155860653X.
Gold, C. (1994). US measures — a balancing act. Boston: Ernst & Young Center for Business Innovation.
Henderi. (2010). Good IT Governance: Framework and Prototype for Higher Eduation. Creative Communication and
Inovative Technology Journal vol 3, no.2 ISSN: 1978-8282.
Indrajit, Eko. (2006). Mengukur Tingkat Kematangan Pemanfaatan Teknologi Informasi untuk Institusi Pendidikan
(Suatu Pendekatan Kesiapan Pemegang Kepentingan/Stakeholder). Conference proceeding of ICT for Indonesia,
Bandung: 3-4 May 2006, 116-120.
ISACA. (2004). COBIT Student Book. IT Governance Institute.
Maniah and Surendro. (2005). Usulan Model Sistem Informasi (Studi Kasus: Sistem Informasi Perawatan Pesawat
Terbang). National Seminar of Information Technology Application, Yogyakarta: 18th June 2005.
Maria. (2011). Perbandingan Sistem Informasi Akademik Universitas Kristen Satya Wacana Menggunakan COBIT
Framework. Journal of Eonomic Foccus, Vol X, Issue-2, 140-149.
Maria, dan Haryani. (2011). Audit Model Development of Academic Information System: Case Studi on Academic
Information System of Satya Wacana. Journal of Art, Science & Commerce, Researchers World, Vol II, Issue-2, 12-24.
Maria, et al. (2012). The Measurement of Information Technology Performance In Indonesian Higher Education Institutions in
The Context of Achieving Institutional Business Goals Using COBIT Framework Version 4.1: Case Studi Satya Wacana
Christian University Salatiga. Journal of Art, Science&Commerce, Researchers World, Vol III, Issue-3(3), 9-19.
Mulyadi. (2001). Balanced Scorecard. Jakarta: Salemba Press: p 416-420.
Mutyarini and Sembiring. (2006). Arsitektur Sistem Informasi Untuk Institusi Perguruan Tinggi Di Indonesia.
Conference proceeding of ICT for Indonesia, Bandung: 3-4 May 2006, 102-107.
O’donnell, E. (2004). Discussion of Director Responsibility for IT Governance: A Perspective on Strategy. International
Journal of Accounting Information Systems 5: p 101-04.
Prabowo, Harjanto. (2007). Implementasi IT Balance Scorecard di Perguruan Tinggi. National Seminar of Information
Technology Application, Yogyakarta: 16st June 2007.
Sa’adi and Suhardi. (2006). Pengukuran Kinerja Penerapan Sistem Enterprise Resource Planning (ERP) di Universitas
dengan Metode IT-Balaced Scorecard (IT-BSC). Conference proceeding of ICT for Indonesia, Bandung: 3-4 May 2006.
Van Grembergen, W. and Van Bruggen, R. (1997). Measuring and improving corporate information technology through
the balanced scorecard technique. Proceedings of the Fourth European Conference on the Evaluation of Information
Technology, Delft: October 1997, 163-171.
Van Grembergen, W. and Timmerman, D. (1998). Monitoring the IT process through the balanced scorecard. Proceedings
of the 9th Information Resources Management (IRMA) International Conference, Boston: May 1998, 105-116.
Van Grembergen, W. (2000). The Balanced Scorecard Technique and IT Governance. Accessed in
http://www.isaca.org/Certification/CGEIT-Certified-in-the-Governance-of-Enterprise-IT/Prepare-for-the-Exam/StudyMaterials/Documents/The-Balanced-Scorecard-and-IT-Governance.pdf
Willcocks, L. (1995). Information Management. The Evaluation of Information Systems Investments. London: Chapman & Hall.
****
www.scholarshub.net
Vol.– III, Issue – 3, July 2013
INDIAN JOURNAL OF MANAGEMENT SCIENCE (IJMS)
THE USED OF IT BALANCED SCORECARD TO BUILD THE
PERFORMANCE MEASUREMENT MODEL OF ACADEMIC
INFORMATION SYSTEMS (CASE STUDY ACADEMIC
INFORMATION SYSTEM OF SATYA WACANA)
Paskah Ika Nugroho,
Faculty of Economics and Business
Satya Wacana Christian University, Indonesia.
Prihanto Ngesti Basuki,
Evi Maria,
Faculty of Information Technology
Satya Wacana Christian University
Indonesia.
Faculty of Information Technology
Satya Wacana Christian University
Indonesia.
ABSTRACT
The aim of this research is to make a model of performance measurement of academic information
system to facilitate the auditors in conducting a periodically performance measurement of Satya
Wacana Academic Information System using IT Balanced Scorecard. SI performance measurement
model was developed through systematic measures in the form of the action process, reflection,
evaluation, and innovation by applying the method of survey research, development, experiments ,
and evaluation. Performance measurement modeling of Academic Information Systems (SIASAT) in
SWCU has been done by making a framework model which was developed by considering the
following parameters: (a) the duties and functions of the university, (b) the aspects of university
management, (c) the duties and functions of the IT organization in university, (d) the need of
information system for academic activities, and (e) the methodology of IT basic framework used,
which is the IT Balanced Scorecard (IT-BSC).
Keywords: IT Balanced Scorecard, Academic Information System, Performance Measurement.
www.scholarshub.net
Vol.– III, Issue – 3, July 2013
INDIAN JOURNAL OF MANAGEMENT SCIENCE (IJMS)
EISSN 2231-279X – ISSN 2249-0280 17
Introduction:
The use of Information Technology (IT) in Higher Education institutions especially for the use of information systems and the
Internet can not be separated due to the demands of the stakeholders (Indrajit, 2006). IT Management in Higher education
institution is a Critical Success Factor (CSF) for leaders and partners of Higher education institutions (Henderi, 2010).
However, the complexity of IT implementation makes the leaders of the various levels in the Higher education institutions and
stakeholders have difficulty in managing the IT. The complexity of IT implementation in higher education institutions in
Indonesia happens because the higher education institution does not have a specific framework model when establishing the
information system (Mutyarini and Sembiring, 2006). As a result, the benefits of using IT is not comparable to the investments
value which has already been incurred.
Satya Wacana Christian University is one of the universities, which has already used IT as an infrastructure and facility to
provide services for students, lecturers and all the staff, and also assists the running of the activities around the work units. In
carrying out its main activity, that is to provide educational services, SWCU has supported by IT of Satya Wacana Academic
Information Systems (SIASAT). IT management has been applied in SWCU, but it has not been applied using a well-structured
method and approach. On the other hand, IT implementation must be controlled because the control provides reasonable
assurance to management that the implementation process has been done in accordance with the plans and goals of the
organization (Maria, 2011).
Each IT process requires a controlled IT measurement to indicate the performance of IT in achieving the control objectives and
facilitate the management to make improvements to the performance of IT. IT performance measurement can be performed by
using IT Balanced Scorecard IT where the IT performance is measured from 4 perspectives: corporate contribution, user
orientation, operational excellence, and future orientation (Van Grembergen, 2000). IT Balanced Scorecard is an effective
method of managing IT organizations as well as evaluating the success and development of the system/application, the
development of computer and network investment, quality of products and IT services, as well as improving the quality of
human resources, even though, most universities in Indonesia have not been using this method (Prabowo, 2007).
In addition, monitoring and evaluating towards SIASAT performance has not been done periodically, but only if there are
complaints from the working units about the SIASAT service (Maria and Haryani, 2011). This condition is not consistent with
the results of Maniah and Surendro’s research (2005) which stated that SI performance measurement must be done
periodically to ensure the sustainability of IT operations used by the organization or company as well as to assess the
sustainability between the planning and implementation of the system. Since the importance of SI performance measurement
should be done periodically, this research will try to make a model of performance measurement of academic information
system to facilitate the auditors in conducting a periodically performance measurement of SIASAT using IT Balanced
Scorecard.
Literature Review:
IT Balanced Scorecard:
The balanced scorecard can be applied to the IT function and its processes as Gold (1994) and Willcocks (1995) have conceptually
described and has been further developed by Van Grembergen and Van Bruggen (1997) and Van Grembergen and Timmerman (1998).
IT-BSC has four perspectives: (1) Corporate Contribution, contains a measure which indicates how the management (the manager)
evaluates/views the IT organization; (2) User orientation contains a measure which indicates how users evaluates/sees the results of
the IT organization, (3) Operational Excellence contains a measure of the effectiveness and efficiency of the IT process, and (4)
Future orientation contains a measure which describes how IT position within the next challenge.
Performance Measurement:
Mulyadi (2001) defines performance measurement as a process of assessment on the company operational activities in a
particular period, whether it has been done based on the defined goals or not. The main purpose of the performance
measurement is that the leader of the company has an objective basis in giving the compensation in accordance with the
achievement which has been done by each department as a whole. It is expected that all of these will give motivation and
stimulation in each section to work more effectively and efficiently.
Previous Researches:
IT model development and performance measurement can adopt IT standards such as ITIL, ISO/IEC 17799, COSO dan
COBIT (O’Donnell, E, 2004). Van Grembergen’s research (2000) discussed about how the IT balanced scorecard (IT-BSC) can
be linked to the business balanced scorecard (BU-BSC) and in this way support the IT/business governance and alignment
processes. The considered aspects in the IT application in IT-BSC method are corporate contribution, customer (user)
orientation, operational excellence and future orientation. IT-BSC method used to measure the performance of the
implementation system of Enterprise Resource Planning systems (ERP) at the University. The method was continuously
developed to make a strategic plan which is in accordance with the mission of educational institutions to continue in surviving
www.scholarshub.net
Vol.– III, Issue – 3, July 2013
INDIAN JOURNAL OF MANAGEMENT SCIENCE (IJMS)
EISSN 2231-279X – ISSN 2249-0280 18
in the business competition (Sa'adi and Suhardi, 2006).
IT Balanced Scorecard has not been widely used to measure the performance of information systems in universities in
Indonesia, but actually this method is very effective for managing the IT organization and evaluating its success (Prabowo,
2007). This is because the Universities in Indonesia did not have a specific model of the framework when they build their
academic information system (SI), so Mutyarini and Sembiring (2006) created an academic architecture Information system
model by adapting the architecture of Monash University which used TOGAF in order to achieve the mission of Tri Dharma
higher education.
The previous research on IT in SWCU as the research object, including the study of Maria and Haryani (2011) who found that the
supervision and the assessment towards the IT performance in SWCU has not been carried out periodically, just only if there are
complaints from the users (the working units) about the IT service. This research produced a model of information audit system
which is developed using the COBIT framework especially for delivery and support (DS) domain. Maria‘s research (2011) also
found that so far the IT management in SWCU has been done, but it has not been done using the structured method and approach.
This research also choose to use COBIT framework in doing the comparison among the academic information systems because
COBIT can notice the link between the business goals without neglecting the IT process as the focus. Maria’s research, et al
(2012) found that IT in SWCU has been well managed where IT processes to support business goals has been standardized,
documented and communicated well. There should be a continuous monitoring and evaluation of the IT in SWCU, so the quality
of IT services in SWCU can be improved day by day in accordance with what is expected.
Research Methodology:
This type of research is a combination of descriptive studies that describes the phenomenon that actually occurs in an event or
population and exploratory research that found a model of the SI Academic performance measurement done by doing an
approach on the "Research and Development", that was a research program which was followed up by doing some
development programs. SI performance measurement model was developed through systematic measures in the form of the
action process, reflection, evaluation, and innovation by applying the method of survey research, development, experiments ,
and evaluation.
The location of this study, Satya Wacana Christian University Salatiga Indonesia, was chosen on purpose. Primary data of this
study was the results of guided interviews and observation. While secondary data such as documents, reports and policies are
taken based on the SIASAT
The steps of this study are as follow:
a. Preliminary studies
In the initial study, there were prelimanary research on previous studies, literature and standards that support the research
topic, guided questionnaire drafting, and SIASAT understanding.
b. Data collection
At this stage, the data was obtained by interview, observation, and questionnaires given to the relevant units and users of
SIASAT. The secondary data is also collected from related units of SIASAT.
c. Development of performance measurement model of IS
At this stage, development of performance measurement of IS was managed by interviews, observation and related
documents to state parameters and Critical Success Factor (CSF), which will be used as constraints to determine criteria of
performance measurement of SIASAT based on IT BSC perspective. Then we mapped the steps to measure the
performance of acedemic information systems.
d. Conclusions
In the final stage of this research, a conclusion from all research processes was stated.
Result and Discussion:
Brief description of the Satya Wacana Academic Information Systems:
The internet-based of SWCU Academic Information System (SI), known as SIASAT , is an application used to record data
from each student's academic administration from the entry (admission) to the exit (graduation). This application can be
accessed easily via the SWCU homepage, http://www.uksw.edu address, then go to the SIASAT menu in the ACADEMIC
group, or directly go to http://siasat.uksw.edu. These applications provide an online and a real time of academic information.
All students who are listed as SWCU students, have the right to access the application via the homepage institution. It is
important for SWCU students to know and master this application and its operations in order to see the financial obligations
that they have to pay, course registration, and see the results of their study for each semester.
SIASAT Performance Measurement Model by Using IT Balanced Scorecard:
Pyle (2003) stated that the development of performance measurement model will be based on one of modelings, i.e. the model
which is developed by its constituent components, such as business processes and their correct data components. Performance
measurement modeling of Academic Information Systems (SIASAT) in SWCU has been done by making a framework model
www.scholarshub.net
Vol.– III, Issue – 3, July 2013
INDIAN JOURNAL OF MANAGEMENT SCIENCE (IJMS)
EISSN 2231-279X – ISSN 2249-0280 19
which was developed by considering the following parameters: (a) the duties and functions of the university, (b) the aspects of
university management, (c) the duties and functions of the IT organization in university, (d) the need of information system for
academic activities, and (e) the methodology of IT basic framework used, which is the IT Balanced Scorecard (IT-BSC). These
parameters are expected to be the factors that determines the performance of academic information systems which were
observed, and how these parameters can be controlled and regulated, in order to obtain a desired performance system. The
relationship between the 5 parameters in creating the measurement model of academic information system performance by
using IT-BSC is presented in Figure 1.
Figure 1. The relationship between the parameters in creating the performance measurement model
The main parameters of duties and functions of university in this research is the implementation Tridharma High Education,
such as lectures, working in laboratory, practical work, the implementation of the final project, research and training, and the
implementation of community service. Those are the things that encourage the chief of SWCU to formulate its business goals
by using four perspectives of Balance Scorecard (BSC). The business goals of SWCU are presented in Table 1. To achieve
those business goals, IT infrastructure is provided in the form of the use of computers, information systems implementation,
and the use of internet technology. The information system was built in accordance with the internal business processes of
SWCU starting from prospective new students since they enroll, be accepted, join the lectures, until graduate.
Table 1. Business goals SWCU
BSC Perspective
Financial
Customer
Internal
1
2
3
1
2
3
4
5
1
2
3
4
5
6
7
Learning &Growth
SWCU Business Goals
Provide a good return on investment of IT-enabled business investments
Manage IT-related business risk
Improve corporate governance and transparency.
Improve customer orientation and service
Offer competitive products and services
Establish service continuity and availability
Create agility in responding to changing business requirements
Achieve cost optimisation of service delivery
Improve and maintain business process functionality
Lower process costs
Provide compliance with external laws, regulations and contracts
Provide compliance with internal policies
Improve corporate governance and transparency
Manage business change
Improve and maintain operational and staff productivity
Manage product and business innovation
SWCU leaders also formulate the main aspects that need to be considered in the management of a university. University management
should pay attention to the availability of resources, the process aspects and the content aspects. These parameters need to be
formulated, since universities in Indonesia do not have a standard framework for building and managing academic information
system (Mutyarini and Sembiring, 2006). Those aspects will be managed by the organization's culture, values and work ethic and are
www.scholarshub.net
Vol.– III, Issue – 3, July 2013
INDIAN JOURNAL OF MANAGEMENT SCIENCE (IJMS)
EISSN 2231-279X – ISSN 2249-0280 20
manifested in the form of organization structure and management systems in universities as presented in Table 2.
The parameters of IT management organization in SWCU are handled by the Bureau of Information Systems Technology
(BTSI). IT developments are constantly increasing, so that SWCU must make arrangement in the IT management organization.
BTSI plays an important role for the success of the implementation process. This is because BTSI not only manage the
technical aspects of IT but also play a role in determining the organizational culture in SWCU in using IT. BTSI consists of 2
parts: (1) IT section which is in charge of audio-visual section, parts of the communication network and the Internet, and parts
of computer, (2) information system section which is in charge of software parts, parts of information systems management,
parts of flexible learning & web, and parts of documentation and training. The functions which are managed by BTSI are: (1)
the function of technology development and the application of information systems, (2) the function of maintenance of
information systems applications, database, digital documentation of information system, the content of learning resources,
networks and computers, (3) the functions of settings and monitoring of IT implementations in the form of change management
and user relationship, release system and audit.
Table 2. The key aspects of Universitites Management
Organizational Culture, Values and Work Ethics
Process
Content
The key process is to run Tri Dharma University,
which consists of:
1. Education and Teaching
The curriculum and its management,
Lecturers and Non2. Researches
which consist of instructional materials,
lecturers resources
3. Community Service
the results of the study, the results of
Funds
Supporting processes, which include the community service, scientific forums.
Facilities and
processes of: academic administration, students
Infrastructure
and alumni, financial administration, cooperation
IT Infrastructure
and external relations, and promotion
Knowledge Management both tacit
Information Systems
knowledge and explicit knowledge.
Organizational Structure and Management Systems in Universities
On the other hand, the parameters of the IT framework methodology used in this study use IT Balanced Scorecard (IT BSC).
Based on those IT organization functions in SWCU, it is necessary to establish IT Balanced Scorecard for each of those
functions which is essentially a derivative of the IT Balanced Scorecard for universities. The process of identifying Critical
Success Factor (CSF) is needed to determine the limits of performance measurement criteria in each IT process. The CSF of
Academic Information Systems is presented in Table 3.
Table 3. The Critical Success Factors of Academic Information System based on IT-BSC Perspective
Resources
IT-BSC Perspective
Business Contribution
CSF Academic Information System
Control costs, increase revenue and improve service coverage
Customer value proposition that includes the rates, quality, service provided, service
User Orientation
and partnerships
Improvement of internal processes by implementing the operations management,
Operational Excellence
customer management and innovation.
Enhanced capabilities and skills through the strengthening of human capital,
Future Orientation
strengthening of information capital, and strengthening of organization capital
For the parameters of the IS (Information System) requirements related to academic activities of universities which consist of
information about admissions, student registration, course registration, grades, and graduation. Admission information consists of
information regarding enrollment of prospective new students which covers registration process, the data inputting, photo-taking,
selection, selection, announcement of selection result, printing of Rector Decree about new admission and information regarding resigning up includes taking an acceptance letter, registration payment, informing a bank payment receipt, filling out a registration form,
and obtaining student’s number. The importance of IS related to student registration such as providing information about types of
registration, information of procedures, requirements, and student registration deadline. Student registration is an activity of
registration or recording of active-status as the University's student and must be done by students each semester.
Course registration is a subject registration process as a participant of a course in the current semester. Subject registration
process includes academic supervision, financial (dispensation), internet, course registration schedule of study
program/department. IS ought to provide information related to grade of subject for each student and each semester which will
be presented either in a study result card or an academic transcript. As for graduation activities, IS should provide information
includes registration, graduation ceremony, and diploma delivery.
After determination of the CSF and know the needs of IS for the academic activities, it should begin mapping the steps what
needs to be done in order to measure the performance of academic information system. The methods of academic information
www.scholarshub.net
Vol.– III, Issue – 3, July 2013
EISSN 2231-279X – ISSN 2249-0280 21
INDIAN JOURNAL OF MANAGEMENT SCIENCE (IJMS)
system performance measurement based on IT BSC Perspective are presented in Table 4.
Table 4. The Methods of Academic Information System Performance measurement based on IT BSC Perspective.
IT-BSC Perspective
Academic Information System Measurement
1. Performing control towards IS, for example by comparing with the actual budget, analysing
the use of budgets, calculating the cost of IS per number of staff.
2. Calculating the financial benefits derived from selling products and services.
3. Doing business assessment of the projected new IS when the university will create and
develop IS, for example by evaluating business based on economical information and
Business Contribution
performing financial evaluation based on Return on Investment (ROI), Net Present Value
(NPV), Internal Rate of Return (IRR), Payback Time (PB).
4. Doing business assessment of IS functions such as calculating the percentage of capacity
related to IT strategy projects, analyzing the relationship between development/new
infrastructure and the investment/investment displacement.
1. Assessment towards BTSI associated with applications that have been addressed, the
percentage of applications that have been completed, etc..
2. Cooperation with the users of information systems when it will carry out the functions of IT
organization, for example by calculating the number of users involved in the manufacturing
User Orientation
process and the development of IS application.
3. Analyzing the IS’s user satisfaction by measuring the level of user friendliness on the
application, calculating the index of user satisfaction, counting the number of applications
and system availability.
1. Conducting an analysis towards the efficiency of software development, for example in
terms of the average increase of unexpected budget, maintenance activities, the average
number of delayed response of the application.
2. Conducting an analysis towards the efficiency of such operations by computing network
availability, response time per category per person, the percentage of work which is done
ontime, the ratio of operating costs of the system used.
Operational
3. Conducting an analysis towards the acquisition and application of personal computers if
Excellence
there is any upgrade.
4. Conducting an analysis towards problem sloving if the system is on trouble, for example by
calculating the average charge time, troubleshooting time, the percentage of problems
answered in a timely manner.
5. Conducting an analysis towards the training of the IT users.
1. Conducting an analysis towards the training and expertise of IT staff both in terms of the
budget which is owned by institutions and trained individuals based on the age, expertise,
etc.
Future Orientation
2. Conducting an analysis towards the age of the applications and opportunities for investment
in new technologies.
SWCU Academic Information System Measurement Model by using the IT-BSC is illustrated in Figure 2:
Universities’ Business Goals by
using BSC Perspective
Resources, Process, and Content
Aspects of IT management in
University
IT Management organization
The Methodology of IT Framework
(IT BSC Perspective)
Critical Success Factor
The need of IS for academic
activities in Universities
The methods of Academic IS
performance measurement
Business Contribution
User Orientation
Operational Excellence
Future Orientation
Figure 2. Academic SWCU SI measurement model using IT BSC.
www.scholarshub.net
Vol.– III, Issue – 3, July 2013
INDIAN JOURNAL OF MANAGEMENT SCIENCE (IJMS)
EISSN 2231-279X – ISSN 2249-0280 22
Conclusion:
Based on those IT organization functions in SWCU, it is necessary to establish IT Balanced Scorecard for each of those
functions which is essentially a derivative of the IT Balanced Scorecard for universities. The process of identifying Critical
Success Factor (CSF) is needed to determine the limits of performance measurement criteria in each IT process. After
determination of the CSF and know the needs of IS for the academic activities, it should begin mapping the steps what needs to
be done in order to measure the performance of academic information system.
References:
[1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
Dorian Pyle. (2003). Business Modeling and Data Mining. Morgan Kaufmann Publishers, ISBN:155860653X.
Gold, C. (1994). US measures — a balancing act. Boston: Ernst & Young Center for Business Innovation.
Henderi. (2010). Good IT Governance: Framework and Prototype for Higher Eduation. Creative Communication and
Inovative Technology Journal vol 3, no.2 ISSN: 1978-8282.
Indrajit, Eko. (2006). Mengukur Tingkat Kematangan Pemanfaatan Teknologi Informasi untuk Institusi Pendidikan
(Suatu Pendekatan Kesiapan Pemegang Kepentingan/Stakeholder). Conference proceeding of ICT for Indonesia,
Bandung: 3-4 May 2006, 116-120.
ISACA. (2004). COBIT Student Book. IT Governance Institute.
Maniah and Surendro. (2005). Usulan Model Sistem Informasi (Studi Kasus: Sistem Informasi Perawatan Pesawat
Terbang). National Seminar of Information Technology Application, Yogyakarta: 18th June 2005.
Maria. (2011). Perbandingan Sistem Informasi Akademik Universitas Kristen Satya Wacana Menggunakan COBIT
Framework. Journal of Eonomic Foccus, Vol X, Issue-2, 140-149.
Maria, dan Haryani. (2011). Audit Model Development of Academic Information System: Case Studi on Academic
Information System of Satya Wacana. Journal of Art, Science & Commerce, Researchers World, Vol II, Issue-2, 12-24.
Maria, et al. (2012). The Measurement of Information Technology Performance In Indonesian Higher Education Institutions in
The Context of Achieving Institutional Business Goals Using COBIT Framework Version 4.1: Case Studi Satya Wacana
Christian University Salatiga. Journal of Art, Science&Commerce, Researchers World, Vol III, Issue-3(3), 9-19.
Mulyadi. (2001). Balanced Scorecard. Jakarta: Salemba Press: p 416-420.
Mutyarini and Sembiring. (2006). Arsitektur Sistem Informasi Untuk Institusi Perguruan Tinggi Di Indonesia.
Conference proceeding of ICT for Indonesia, Bandung: 3-4 May 2006, 102-107.
O’donnell, E. (2004). Discussion of Director Responsibility for IT Governance: A Perspective on Strategy. International
Journal of Accounting Information Systems 5: p 101-04.
Prabowo, Harjanto. (2007). Implementasi IT Balance Scorecard di Perguruan Tinggi. National Seminar of Information
Technology Application, Yogyakarta: 16st June 2007.
Sa’adi and Suhardi. (2006). Pengukuran Kinerja Penerapan Sistem Enterprise Resource Planning (ERP) di Universitas
dengan Metode IT-Balaced Scorecard (IT-BSC). Conference proceeding of ICT for Indonesia, Bandung: 3-4 May 2006.
Van Grembergen, W. and Van Bruggen, R. (1997). Measuring and improving corporate information technology through
the balanced scorecard technique. Proceedings of the Fourth European Conference on the Evaluation of Information
Technology, Delft: October 1997, 163-171.
Van Grembergen, W. and Timmerman, D. (1998). Monitoring the IT process through the balanced scorecard. Proceedings
of the 9th Information Resources Management (IRMA) International Conference, Boston: May 1998, 105-116.
Van Grembergen, W. (2000). The Balanced Scorecard Technique and IT Governance. Accessed in
http://www.isaca.org/Certification/CGEIT-Certified-in-the-Governance-of-Enterprise-IT/Prepare-for-the-Exam/StudyMaterials/Documents/The-Balanced-Scorecard-and-IT-Governance.pdf
Willcocks, L. (1995). Information Management. The Evaluation of Information Systems Investments. London: Chapman & Hall.
****
www.scholarshub.net
Vol.– III, Issue – 3, July 2013