2 Jurnal Ilmu Komputer dan Informasi (Journal of Computer Science and Information) , Volume 8, Issue 1,

  Jurnal Ilmu Komputer dan Informasi (Journal of Computer Science and Information). 8/1 (2015), 1-10 DOI: http://dx.doi.org/10.21609/jiki.v8i1.278

  

COVERAGE, DIVERSITY, AND COHERENCE OPTIMIZATION FOR MULTI-DOCUMENT

SUMMARIZATION

Khoirul Umam, Fidi Wincoko Putro, Gulpi Qorik Oktagalu Pratamasunu,

Agus Zainal Arifin, and Diana Purwitasari

  Department of Informatics, Faculty of Information Technology, Institut Teknologi Sepuluh Nopember, Kampus ITS Sukolilo, Surabaya, 60111, Indonesia

  E-mail: khoirul.umam35@gmail.com, agusza@cs.its.ac.id

  

Abstract

A great summarization on multi-document with similar topics can help users to get useful informa-

tion. A good summary must have an extensive coverage, minimum redundancy (high diversity), and

smooth connection among sentences (high coherence). Therefore, multi-document summarization that

considers the coverage, diversity, and coherence of summary is needed. In this paper we propose a

novel method on multi-document summarization that optimizes the coverage, diversity, and coher-

ence among the summary's sentences simultaneously. It integrates self-adaptive differential evolution

(SaDE) algorithm to solve the optimization problem. Sentences ordering algorithm based on topical

closeness approach is performed in SaDE iterations to improve coherences among the summary's sen-

tences. Experiments have been performed on Text Analysis Conference (TAC) 2008 data sets. The

experimental results showed that the proposed method generates summaries with average coherence

and ROUGE scores 29-41.2 times and 46.97-64.71% better than any other method that only consider

coverage and diversity, respectively.

  

Keywords: multi-document summarization, optimization, self-adaptive differential evolution, sen-

tences ordering, topical closeness

  

Abstrak

Peringkasan yang baik terhadap dokumen-dokumen dengan topik yang seragam dapat membantu

pembaca dalam memperoleh informasi secara cepat. Ringkasan yang baik merupakan ringkasan de-

ngan cakupan pembahasan (coverage) yang luas dan dengan tingkat keberagaman (diversity) serta ke-

terhubungan antarkalimat (coherence) yang tinggi. Oleh karena itu dibutuhkan metode peringkasan

multi-dokumen yang mempertimbangkan tingkat coverage, diversity, dan coherence pada hasil ring-

kasan. Pada paper ini dikembangkan sebuah metode baru dalam peringkasan multi-dokumen dengan

mengoptimasi tingkat coverage, diversity, dan coherence antarkalimat hasil ringkasan secara simul-

tan. Optimasi hasil ringkasan dilakukan dengan menggunakan algoritma self-adaptive differential

evolution (SaDE). Algoritma pengurutan kalimat yang menggunakan pendekatan topical closeness ju-

ga diintegrasikan ke dalam tiap iterasi algoritma SaDE untuk meningkatkan koherensi antarkalimat

hasil ringkasan. Uji coba dilakukan pada 15 topik dataset Text Analysis Conference (TAC) 2008. Ha-

sil uji coba menunjukkan bahwa metode yang diusulkan dapat menghasilkan ringkasan dengan rata-

rata koherensi 29-41,2 kali lebih tinggi serta skor ROUGE 46,97-64,71% lebih besar dibandingkan

dengan metode yang hanya mempertimbangkan coverage dan diversity hasil ringkasan.

  

Kata Kunci: optimasi, pengurutan kalimat, peringkasan multi-dokumen, self-adaptive differential

evolution, topical closeness 1.

  The massive quantity of data available in the

   Introduction

  Internet today has reached such a huge volume. It The contents of a document can be long. It pres- becomes humanly unfeasible to get efficiently us- ents several information with specified topic. Cur- eful information from the Internet [1]. Thus, auto- rent technological developments makes people matic methods are needed in order to get useful can find related documents with similar topic easi- information from the documents efficiently. er than before. The other documents can be had a Document summarization is one of methods long contents too. It means there is a massive qua- to process information automatically. It creates ntity of data or information with similar obtain- compressed version of documents that provides

2 Jurnal Ilmu Komputer dan Informasi (Journal of Computer Science and Information) , Volume 8, Issue 1,

  February 2015

  the original documents relevantly. Document su- mmarization can be classified based on the num- ber of document processed simultaneously, i.e. si- ngle-document and multi-document summariza- tion. Single-document summarization processes only one document into a summary, whereas mul- ti-document summarization processes more than one document with similar topic into a summary.

  Various kinds of algorithms are proposed on multi-document summarization problem. These algorithms include ontology-based, clustering, and heuristic approach. The example of document summarization method that uses ontology-based approach is the proposed method in [2]. It can perform multi-document summarization by utili- zing Yago ontology to capture the intent and con- text of sentences in documents. It can choose the exact meaning of sentences that has ambiguous word based on Yago ontology scores.

  Multi-document summarization methods ba- sed on clustering approach have been also propo- sed. For example, the method proposed in [3]. It generates a summary from sentences set that ha- ve been clustered based on similarity between sentences. In the other multi-document summa- rization method that has been proposed in [1] also there is a clustering stage.

  Whereas the multi-document summarization methods based on heuristic approach are methods that utilize optimization algorithm in order to se- lect the summary's sentences properly. One of multi-document summarization methods that use this approach is Optimization of Coverage and Di- versity for Summarization using Self-adaptive Di- fferential Evolution (OCDsum-SaDE) method that proposed in [4]. In the method, an optimal sum- mary is searched by considering the coverage and diversity of summary's sentences.

  Multi-document summarization cannot be separated from sentences ordering process. The process is needed to be performed in order to ob- tain the composition of the summary's sentences that allows users to get information easily. Several summary's sentences ordering methods had been proposed in [5-7]. It considers a variety of appro- aches, i.e. chronological, probabilistic, topical clo- seness, precedence, succession, semantic, and text entailment approaches. The process is generally carried out after the document summarization pro- cess completes. Thus, the results of sentences or- dering depend on the summary.

  A good summary is expected to meet three factors. These factors are: 1) an extensive cover- age; 2) high diversity or minimum redundancy; 3) high coherences among summary's sentences [4]. Summary that have an extensive coverage indica- tes it has summarized all information from origin- nal documents. Summary's sentences with high di- versity or minimum redundancy indicate the sum- mary able to presents information without any convoluted. On other hand, the smooth connecti- vity between summary's sentences may help the users to understand and absorb information from summary easily.

  Process to obtain the best summary can be considered as an optimization problem [8]. There- fore, the process to generate a summary with high level of coverage, diversity, and coherences amo- ng the sentences also can be considered as an opti- mization problem. Thus, a multi-document sum- marization method that considers optimizing those factors simultaneously is needed to study in order to generate a good summary.

  In this paper, we propose a novel method for multi-document summarization that considers the coverage, diversity, and coherence of the summa- ry. This method is inspired by self-adaptive differ- ential evolution (SaDE) algorithm from [4] and sentences ordering algorithm using topical close- ness approach in [6]. SaDE algorithm is used to solve the coverage, diversity, and coherence opti- mization problem. Whereas the topical closeness approach that integrated to SaDE iterations helps to find the solution of summary with optimal co- herences. Thus, this method can generates sum- mary with an extensive coverage, minimum re- dundancy, and high coherence among the summa- ry's sentences.

  2. Methods Summary's Quality Factors

  In this section, we describe three factors of sum- mary's quality (i.e. coverage, diversity, and cohe- rence) that optimized in our proposed method.

  Coverage

  Let N denotes the number of sentences from docu- ments that will be summarized, M denotes the nu- mber of distinct terms in documents, denotes the nth sentence from documents which has nor- malized form , denotes the m-th di- stinct term from documents, denotes the nu- mber of occurrences of in , denotes inverse sentence frequency of , and denotes the number of sentences containing

  . Term's weight of in ( ) can be calculated using term frequency in- verse sentence frequency (TF-ISF) scheme in equ- ation(1) and equation(2):

  = × , (1)

  Khoirul Umam, et al., Coverage, Diversity, and Coherence 3 = ( ).

  In this paper, the summary's diversity value is defined as total value of its sentences similarity. Therefore, its diversity value is related with diver- sity of its sentences inversely. The lower its diver- sity value reflects the more diversity in its senten- ces and also the better summary.

  ( ) denotes binary form of vector solution for the pth candidate summary on t-th generation and

  , ( )

  denotes the nth component of ( ). Process to generate this vector will be described in the next section.

  Diversity

  S ummary’s diversity value reflects the diversity of summary’s sentences. It can be considered by calculating similar ity between each summary’s sentences. If the summary has high total value of sentences similarity, then it has low diversity.

  Otherwise, if the summary has low total value of sentences similarity, then it has high diversity be- tween its sentences [4].

  Summary with low diversity between its sen- tences tends to present a poor summary because its sentences tend to discuss redundant informa- tion. Therefore, in order to get a good summary, the combination of summary

  ’s sentences that has high diversity have to be found. In other words, the combination of summary’s sentences with low total value of its sentences similarity have to be found, because it can present the information with minimum redundancy.

  The formulation to calculate summary ’s di- versity value is shown in equation(7).

  .

  Equation(7) only sums similarity between summa- ry’s sentences and ignores sentences which not in the summary [4].

  Coherence

  S ummary’s coherence value reflects the summa- ry’s sentences coherences degree. It corresponds with smooth connectivity between summary's sen- tences. Thus, it also corresponds with readability of information in summary by readers. A summa- ry with higher coherences degree is expected to be easier for reader in order to understand the infor- mation which presents by the summary.

  Generally a summary can simplify the read-

  ( ( )) = ( , ( )) ⋅ ∑ ( , ) , =1 .

  (6)

  (

( )) = ∑ ∑ (

, ) , ,

  

= +1

−1 =1 .

  (5) Alguliev et al. [4] also describes that by con- sidering the similarity between main content of original documents and main content of summary, we will know the importance of summary towards original documents. Moreover, by considering the similarity between main content of original docu- ments with each summary’s sentence, we will kn- ow the importance of each summary’s sentence towards its original documents. The greater simi- larity between main content of original documents with a summary’s sentence reflects the more im- portance of the sentence towards original docu- ments. Therefore, greater summary’s coverage va- lue reflects better summary. The formulation to cal culate summary’s coverage value is shown in equation(6). In the equation(6),

  ∈ ( )

  (2) The is represented as a vector which has M components such that

  ( ) denotes set of sentences in

  = [ 1 , … , ] .

  The similarity between sentences can be calcula- ted using cosine measure formulation in equa- tion(3): ( , ) =

  ∑ ( , ) =1

  √∑

  2 =1 ⋅∑

  2 =1

  . (3) S ummary’s coverage value reflects the cove- r age of summary’s contents towards contents in original documents. It can be calculated by con- sidering similarity between main content in ori- ginal documents with main content in candidate summary [4]. Radev et al. [9] describes that main content of documents set is reflected by its cen- troid or its term’s weight means.

  Centroid of original documents and candi- date summary are represented as a vector with M components. Let

  p- th candidate summary on t-th generation and ( ) denotes the number of sentences in ( ).

  ∑

  Each component of the original documents’s centroid and each component , ( ) of the p th candidate sum mary’s centroid on current gener- ation

  ( ) can be calculated using equation(4) and equation(5), respectively: =

  1 ∑

  =1

  , (4)

  ,

  ( ) =

  1 ( )

  (7)

4 Jurnal Ilmu Komputer dan Informasi (Journal of Computer Science and Information) , Volume 8, Issue 1,

  ℎ ( ( )) =

  in this paper is formulated as mean value of adjacent sum mary’s sentences similarities as shown in equ- ation(8). The

  ( ) in equation(8) denotes the ordered form of vector solution for the p-th candi- date summary on t-th generation and

  ,

  ( ) den- otes the n-th component of ( ). Process to generate this vector will be described in the next section.

  In order to improve the coherences among cess is performed. In this paper we proposed two types of sentences ordering algorithm as described in Algorithm 1 and 2. The proposed algorithms are inspired from topical closeness approach that had been presented in [6]. The first type (Type A) is an algorithm that maximizes similarity between adjacent sentences. Whereas the second type (Ty- pe B) is an algorithm which emphasizes two sen- tences with most similar topic should be at the be- gin ning of summary’s paragraph.

  Example. Let S1, S2, S3, S4, and S5 as five summary's sentences which will be ordered by sentences ordering algorithm type A and B. Assume that they have similarities as shown in Figure 1. Their ordering processes using sentences ordering algorithm type A and B are shown in Table 1.

  Based on the algorithms, pair of sentences which have highest similarity are chosen as the initial of sentences ordering result. Therefore pair of S3 and S5 which has the highest similarity (0.9) is chosen on the first iteration in each algorithm. In algorithm type B, after the initial sentences are chosen, each sentence is labeled as head and tail. Therefore on this iteration S3 and S5 are labeled as head and tail, respectively.

  On the second iteration, S1 is chosen to pair

  ∑ (

  Based on the description we can make con- clusion that a good summary is a summary with high coherences degree between its adjacent sen- tences. However, a good summary have to pre- sents the information about its original documents contents to readers in simple form (i.e. the sum- mary has a little number of sentences). Therefore, the summa ry’s coherence value

  ( )

  ,

  ( +1)

  )

  ( )−1 =1

  ( ) − 1 , ( ) =

  , ( ).

  (8)

  ℎ

  February 2015 Algorithm 1. Sentences Ordering Type A Algorithm

  

1. From sentences in the candidate summary, choose two sentences ( and ) which has highest similarity

( ( , )) and then make it as initialization of ordering result → = [ , ].

  Algorithm 2. Sentences Ordering Type B Algorithm

  2. Change and status to be head and tail, respectively.

  

3. For each sentence which has not in ordering result, choose a sentence ( ) which has highest similarity if paired with head

or tail.

  4. Do one of following conditional:

  a. If (ℎ , ) ≥ ( , ), then put in front of the head and change status to be head → =

  [ , , ].

  b. If (ℎ , ) < ( , ), then put behind the tail and change the status to be tail → =

  [ , , ].

  5. Repeat steps 3-4 until the entire sentences are in the ordering result.

  

1. From sentences in the candidate summary, choose two sentences ( and ) which has highest similarity

( ( , )) and then make it as initialization of ordering result → = [ , ].

  6. Repeat steps 4-5 until the entire sentences are in the ordering result.

  

2. Choose the other sentence ( ) which has highest similarity if paired with one of sentence in ordering result ( or ).

  3. Do one of following conditional:

  a. If ( , ) ≥ ( , ) , then put beside and set and status as head and tail, respectively → = [

  , , ].

  b. If ( , ) < ( , ) , then put beside and set and status as head and tail, respectively → = [

  , , ].

  

4. For each sentence which has not in ordering result, choose a sentence ( ) which has highest similarity if paired with tail.

  5. Put behind the tail and change the status as tail.

  ers to understand the information if its sentences are ordered such as two adjacent sentences discuss similar content or topic. It has same principle with topical closeness approach that has been presented in [6]. The close ness between sentence’s topics can be considered using similarity value between the sentences. The greater similarity between ad- jacent sentences reflects that they have similar contents or topics.

  Khoirul Umam, et al., Coverage, Diversity, and Coherence 5 TABLE 1 E

  The coverage, diversity, and coherence optimiza- tion process in our proposed method consists of preprocessing and main process phase. The main process implements self-adaptive differential evo- lution (SaDE) algorithm inspired from [4] with the additions of the sentences ordering phase. For the convenience, we denote our proposed method as CoDiCo method which stands for three factors that we would be optimized i.e. coverage, di- versity, and coherence. Figure 2 depicts the flow- chart of CoDiCo method.

  0.9

  0.4

  with S5 because it has higher similarity (0.8) than the pair of S3 with S1 or S2 (0.4). Using the same rule in algorithm type A, S2 and S4 are chosen to put in front of S3 on the next iterations. But in al- gorithm type B, S4 and S2 are put behind S1 since

  S1 become the tail and the algorithm only pairs re-

  maining sentences with tail after the second iter- ation by considers their similarities.

  Summary's Quality Factors Optimization

  Preprocessing Phase

  0.3

  Preprocessing phase is a step to prepare the data which would be used in main process. In this step there are some processes, i.e.: 1) sentences extrac- tion; 2) sentences normalization; 3) distinct terms extraction; 4) term weights matrix preparation; 5) sentences similarity matrix preparation.

  Sentences extraction is a process to take each sentence from documents that have same topic in dataset. The process will produce N sentences. Ea- ch extracted sentence is represented as a single line of data in sentences list D such that

  = [

  1 , . . . , ]

  . After the extraction process, each sentence is normalized into using stop-word removal, punctuation removal, and stemming pro- cess. We use 571 stop-words from Journal of Ma- chine Learning Research stop-word list

  1

  for the ₁

  0.1

  0.7

  XAMPLE OF O RDERING P ROCESS U SING P ROPOSED S ENTENCES O RDERING A LGORITHM Iteration Ordering Process

  

Criterion

Selection

Ordering (trial vector)

  Type A Type B

  1 S3-S5 S3-S5

  2 S3-S5-S1 S3-S5-S1

  3 S2-S3-S5-S1 S3-S5-S1-S4

  4 S4-S2-S3-S5-S1 S3-S5-S1-S4-S2 Sentences and terms extraction Initialization

  

Binarization

(target vector)

Ordering (target vector)

  Solutions Evaluation Mutation Stoping

  Binarization (trial vector) Crossover Output

  0.4

  Preprocessing Main Process Sentences Ordering

Figure 2. CoDiCo method flowchart.

  Figure 1. Example of sentences similarities.

  S1 S2 S5 S4 S3

  0.8

  0.2

  0.3

  0.4

  http://jmlr.org/papers/volume5/lewis04a/a11-smart-stop-

6 Jurnal Ilmu Komputer dan Informasi (Journal of Computer Science and Information) , Volume 8, Issue 1,

  (11) The

  derived from

  ( )

  sentences for each

  ( )

  In this step,

  Ordering

  in this step has same value with the one which have been used in initialization step.

  ,

  1 1 + − .

  solution are ordered using sentences ordering algorithm which described in Subsection

  ( ) =

  (10)

  , ( )) 0, otherwise ,

  , < (

  , ( ) = {1, if

  ( ). The formula- tion for this process is shown in equation(10) and equation(11):

  ,

  value with sigmoid value of

  ,

  ( )

  2.3. Ordered form of

  , ( )

  The

  is conducted by considering the three factors of summary's quality, i.e. the its coverage, diversity, and coherence values. The formulation is shown in equation(12).

  ( )( ( ( )))

  ( ( )). Based on our purpose in this paper, calculation for fitness value of

  ( ) which has been en- coded to binary form ( ( )) and ordered form

  The evaluation step is used to calculate fitness va- lue for each summarization solution. Evaluations are performed for each

  Solutions Evaluation

  , ( ) = [1, … , ]).

  which include as summary's sentences in ( ) (i.e.

  , ( ) component stores sentences index

  ( )]

  ( )

  ( ), . . . , ,

  ( ) = [ ,1

  such that,

  , ( )

  com- ponents

  ( )

  which has

  ( )

  is stores in ordering- solution vector

  can be performed by comparing

  February 2015

  stop-word removal process. For the stemming pr- ocess, we use Porter Stemmer algorithm

  × dimensions.

  1 ( ), . . . , ( )]

  ( ) = [

  of summarization. Let P and t denote the number of generated solutions and the current generation, respectively, such that

  U that would be used to find the optimal solution

  Initialization is a step to provide a set of solutions

  Initialization

  The main steps in main process phase of CoDiCo method as shown in Figure 2 consist of initializa- tion, binarization, ordering, evaluation, mutation, crossover, stopping criterion, and output steps. Binarization and ordering steps can be divided in- to two phases, i.e. binarization and ordering for target vectors (i.e. solution vectors which genera- ted by initialization and selection steps) and bina-rization and ordering for trial vectors (i.e. solution vectors which generated by cross- over step). The brief descriptions for each step is describes in the next subsection.

  Main Process Phase

  , = [1, . . . , ] can be calculated using cosine measure scheme in equa- tion(3). This process will produce a sentences si- milarity matrix that has

  ( ) for = [1, . . . , ] is represented as a vector which has N compone- nts such that

  Each term’s weight then used to calculate the sentences similarity. Similarity value between and for

  stores the term’s weight of in norma- lized sentence ( ). The weights calcu- lation is conducted using TF-ISF scheme in equa- tion(1) and equation(2).

  W

  × dimensions. Each component in

  Based on N normalized sentences and M dis- tinct terms, we generate a terms weight matrix W which has

  1 , . . . , ] .

  = [

  On the next step we perform distinct terms extraction from each . This process prod- uces M distinct terms. Each extracted term is stored into terms list T such that

  2 .

  for = 0. Each solution in U is referred as a target vector. Each target vector

  ( ) = [ ,1 ( ), . . . , , ( )]

  ,

  Binarization

  Alguliev et.al. [4] describes that encoding pro-cess of real-value

  in D is not a sentence in the ( ).

  , ( ) = 0, then it indicates that the

  ( ). Otherwise, if

  ( ) = 1, then it indicates that the in D is selected as sentence in the

  ,

  into binary-value. The binary-values are used to indicate the sentences from D which used as sentences in the p-th candidate summary ( ). If

  , ( )

  Binarization is a step to encode real-value of

  denotes a uniform random value between 0 and 1 for the nth component in p-th tar- get vector [4].

  where

  ,

  where

  , (0) = + ( − ) ⋅ , , (9)

  shown in equation(9): ₂

  , (0) is

  tween specified lower bound and upper bou- nd . The formulation to initialize

  , (0) is randomly initialized by a real-value be-

  denotes the n-th component in p-th target vector. Each target vector’s component in this step

  , ( )

  ( ) into binary-value

  Khoirul Umam, et al., Coverage, Diversity, and Coherence 7 ( ( ))

  ( ⋅ ( (12) ( )) = ℎ ( )).

  ( ( )) ( ), if ( ( )) > ( ( − 1))

  ( ) = { ( − 1), otherwise

  (13) .

  (14)

  ( ) = ( ) + (1 − ( )) ⋅ ( ( ) − 1 ( )) + ( ) ⋅ ( ( ) − 1 ( )).

  2 − , ( ), if , ( ) <

  (15)

  , ( ) = { 2 − ( ), if ( ) > .

  , ,

  The best and the worst solutions on current Alguliev et.al. [4] describes that to generate generation can be determined using each soluti- the vector, relative distance between

  ( ) ( )

  on's fitness value. The best solution on current ge- vector and vector has to be calcu-

  ( ) ( )

  neration (local best) ( ) is a target vector that lated first. The

  ( ) then used to calculate the has the highest fitness value. Otherwise, the worst crossover rate

  ( ). Equation(17-19) shows the solution on current generation (local

  ( )

  formulation to calculate the ( ) and ( ). worst) is a target vector that has the lowest fitness

  The rule to determine trial vector component value. In this step we also can update the global ( ) value is formulated in equation(20). In eq-

  ,

  best ( ) i.e. the best solution until current uation(20) k is a randomly selected integer value generation using the rule which formulated in for . It ensures that at least one com-

  = [1, … , ] equation(13).

  ponent of trial vector is obtained from the mutant vector. It will ensure that the solutions on the next

  Mutation

  generation have differences with the solutions on Mutation is a step to generate mutant vectors set V current generation [4]. from target vectors set U. Mutation process of is conducted by involving vector,

  ( ) ( ) ( )) − ( ( )) (

  vector, a randomly selected vector (17)

  ( ) ( ) 1 ( ) = .

  ( )) − ( ( )) (

  where and , and a mutation

  1 = [1, . . . , ] 1 ≠

  factor for current generation . The formulati-

  ( )

  on to generate p-th mutant vector on current gene- 2 ℎ (2 ( )) (18) ( ) = . ration is shown in equation(14), whereas the

  ( )

  1 + ℎ (2 ( )) formulation to calculate the value is shown

  ( )

  in equation(16):

  2 −1

  (19) . ℎ( ) =

  2

• 1 −2 /

  (16) ( ) = .

  , ( ), if ,

  ≤ or = In equation(16) denotes maximum generate-

  , ( ) = { , ( ), otherwise ,

  on which specified in initialization step [4].

  (20) One or more components have a

  ( ) ( ) ,

  probability to violate the boundary constraints. Its

  ( ), if ( ( )) ≥ ( ( )) values can be less than or greater than . ( + 1) = {

  , ( )

  Each which its value violates the boundary ( ), otherwise .

  constraints have to been reflected back. The rules (21) to reflect back the value is formulated in

  ( ) ,

  Selection equation(15).

  Selection is a step to generate a novel target vec- tors set for the next generation

  Crossover

  ( + 1). The vec- Crossover is a step to generate trial vectors set Z. tors are derived from

  ( ) vectors and ( ) vec- tors which have the best fitness value [4]. In other Each trial vector has N components ,

  ( ) , ( )

  words only the best solution for each pair is survi- which its value is derived from the value of ved from this operation. It ensures that the search- or [4]. The purpose of this operati-

  , ( ) , ( )

  ing of optimal solution is always approach to the on is to increase the diversity of solution vectors best solution until the last iteration. The rule to in order to expand the search space.

  

8 Jurnal Ilmu Komputer dan Informasi (Journal of Computer Science and Information) , Volume 8, Issue 1,

February 2015

  ( )

  The testing results are evaluated using Re- call-Oriented Understudy of Gisting (ROUGE) method [10]. This method compares candidate su- mmaries (i.e. summaries generated by proposed and compared methods) with reference summaries (i.e. summaries that created manually by human which provided in TAC 2008 dataset). There are 4 type of ROUGE method which is used to evaluate our experiments i.e. ROUGE-1, ROUGE-2, ROU- GE-L, and ROUGE-SU.

  Both of our proposed method (CoDiCo) and the compared method (OCDsum-SaDE) use four specified parameters in the initialization state i.e. population size (P), maximum generation ( ), lower bound ( ), and upper bound ( ) wh- ich sets to 20, 500, -5, and 5, respectively. Those parameters values assign based on heuristic choi- ces. After the multiple-documents summarizations were processed using the tested methods, we get the summarization results as many as selected to- pics. Therefore, we have 15 summaries from 15 selected topics for each method.

  We also test both of our proposed sentences ordering algorithm by involving a threshold in sentences similarity value in order to evaluate the impact of similarity between summary's sen- tences toward the optimal summary's solution. We use three threshold values, i.e. 0.7, 0.8, and 0.9. In this scenario, the sentences ordering process ex- clude every pair of sentences that have similarity value greater than or equal to value. The ex- cluding pair of sentences will not be chosen as ad- jacent sentences in summary's solutions.

  CoDiCo-B, respectively. In order to compare the summarization results from CoDiCo method with another multi-document summarization method that considers coverage and diversity factors only, we use the OCDsum-SaDE method from [4].

  3 We represented these methods as CoDiCo-A and

  The experiments are performed using Matlab R2013a and run on Microsoft Windows platform. We test both the proposed sentences ordering al- gorithm type A and type B using CoDiCo method.

  to test our Co- DiCo method. This dataset provides articles that classified into some topics and coherent summa- ries which created manually by human for each topic. We choose 15 topics for the testing. Each topic contains 10 documents that would be sum- marized.

  3

  In this paper we use Text Analysis Conference (TAC) 2008 dataset from National Institute of Standards and Technology (NIST)

  3. Results and Analysis

  are return- ed as the summary.

  ( )

  stores the order of the su- mma ry’s sentences index. Furthermore a senten- ces set which indicated in

  denotes the sentences index in D which has selected as summa ry’s sentences, whereas its ord- ered form

  

TABLE 2

R OUGE S CORES C OMPARISON OF E ACH T ESTED M ETHODS

Methods ROUGE Score

  ( )

  has been acquired. Its binary form

  ( )

  This step is the final step in main process of Co- DiCo method. In this step, the global best solution of summarization on the last generation

  Output

  In this step, the iteration of optimal solution sear- ching process is determined to be stopped or not. The stopping criterion in this paper is uses a spe- cified number of generation. If the iteration has reached the maximum generation , then the iteration is stopped. Otherwise, if the iteration has not reached the , then the iteration is conti- nued.

  Stopping Criterion

  choose the p-th target vector on the next genera- tion ( + 1) is formulated in equation(21).

  0.5296 0.1448 0.4923 0.2434 0.3525

  = 0.7) 0.5620 0.1611 0.5205 0.2674 0.3777 CoDiCo-B ( = 0.7)

  CoDiCo-B ( = 0.8) 0.5464 0.1747 0.5127 0.2640 0.3744 CoDiCo-A (

  0.5163 0.1525 0.4785 0.2304 0.3444 CoDiCo-A ( = 0.8) 0.5322 0.1473 0.4963 0.2310 0.3517

  = 0.9) 0.5118 0.1443 0.4749 0.2172 0.3370 CoDiCo-B ( = 0.9)

  Average ROUGE-1 ROUGE-2 ROUGE-L ROUGE-SU

OCDsum-SaDE 0.3701 0.0794 0.3424 0.1254 0.2293

CoDiCo-A (without threshold) 0.5144 0.1395 0.4820 0.2256 0.3404

CoDiCo-B (without threshold) 0.5279 0.1524 0.4933 0.2351 0.3522

CoDiCo-A (

  ROUGE-1 and ROUGE-2 are variants of ROUGE-N that consider n-gram recall between summarization result from candidate summary and reference summary for n assigned by 1 and 2.

  Khoirul Umam, et al., Coverage, Diversity, and Coherence 9

  = 0.7. Nevertheless, the value is higher than the average coherence value of OCDsum-SaDE. If we compare it with OCDsum- SaDE method, CoDiCo-B without threshold can produce summary with average coherence value about 41.2 times higher than the OCDsum-SaDE method, whereas CoDiCo-B method using

  = 0.8) 0.160 CoDiCo-A ( = 0.7)

  CoDiCo-A ( = 0.8) 0.148 CoDiCo-B (

  0.151 CoDiCo-B ( = 0.9) 0.167

  CoDiCo-A (without threshold) 0.193 CoDiCo-B (without threshold) 0.206 CoDiCo-A ( = 0.9)

  V ALUE C OMPARISON FROM E ACH T ESTED M ETHOD Methods Average of coherences value OCDsum-SaDE 0.005

  VERAGES C OHERENCE

  TABLE 3 A

  = 0.9, and using = 0.8. Whereas CoDi- Co-A only has higher average ROUGE score than

  The comparison of two proposed sentences ordering algorithm with same threshold value usi- ng their ROUGE scores shows that CoDiCo-B is better than CoDiCo-A. As shown in Table 2, Co- DiCo-B has higher average ROUGE score than CoDiCo-A when do not using a threshold, using

  = 0.7 can reach average coherence value about 29 times higher than OCDsum-SaDE method. It sho- ws that the CoDiCo method, which involves orde- ring step in optimization process, can produce summary with better coherences or smoother con- nectivity among sentences than the other method which does not consider the ordering of summa- ry's sentences.

  Based on the evaluation of averages of sum- mary's coherence value that shown in Table 3, we know that CoDiCo-B method without threshold reaches the average coherence value higher than the others do. We also know that the lowest avera- ge coherence value among the variants of CoDiCo method is 0.145 which obtained by CoDiCo-B method using

  It is computed by divides the maximum number of n-grams co-occurring in candidate summary and set of reference summary with total sum of the number of n-grams occurring at the reference summary. ROUGE-L is a ROUGE method that considers about longest common subsequence (LCS) between candidate summary and reference summary. It is computed as the ratio between LCS's length with reference summary's length. In other hand ROUGE-SU considers the unigram va- lue on candidate summary and reference summary as counting unit [4,10]. The formulas and comple- te explanation about usage of ROUGE method can be read in [10].

  By comparing the averages ROUGE score for each CoDiCo method with the average ROU- GE score of OCDsum-SaDE method, we know that CoDiCo methods have averages RO-UGE sc- ore in range 46.97-64.71% higher than the aver- ages ROUGE score of compared method. It shows that the multi-document summarization method that considers coverage, diversity, and coherence simultaneously can produce better summary than summarization method that considers coverage and diversity only.

  It should be noted that in CoDiCo method, by considering the coherences of sentences while selecting the best solution will adjust the coverage and diversity factors simultaneously to find the optimal solution. It will produce different summa- ry compared with method that only considers the coverage and diversity factors. But the summary is more similar with summary that created manu- ally by human. It causes the ROUGE scores of CoDiCo method are greater than ROUGE scores of compared method.

  = 0.7. Whereas the ave- rages ROUGE score of OCDsum-SaDE method only reached 0.2293. It means all of CoDiCo me- thod variants have better performance than the co- mpared method.

  = 0.9 and the highest average ROUGE score is 0.3777 which obtained by CoDiCo-A using

  has higher score than the others. From Table II we also know that the lowest average ROUGE score of CoDiCo method is 0.3370 which obtain- ed by CoDiCo-A using

  0.8

  =

  = 0.7 has higher ROUGE score on ROUGE-1, ROUGE-L, and ROUGE-SU than the other methods. Whereas in ROUGE-2 can be shown that CoDiCo-B method using

  Series of experiment has been conducted to evalu- ate our proposed method (i.e. CoDiCo-A and Co- DiCo-B) in comparison with compared method (OCDsum-SaDE). Based on the evaluation results as shown in Table II, we know that the CoDiCo-A method using

  Discussion

  ROUGE score for CoDiCo-A, CoDiCo-B, and OCDsum-SaDE methods are presented in Table 2. It shows the comparison of ROUGE sco- res among each tested methods. The highest ROUGE scores for each ROUGE type are indi- cated by bolded text. We also evaluate our propo- sed method by considering the averages of sum- mary's coherence value which generated by each tested method. The comparison of averages cohe- rence value from tested methods is shown in Ta- ble 3. The highest value is indicated by bolded text.

  0.140 CoDiCo-B ( = 0.7) 0.145

  the workshop on text summarization bran- ches out , pp. 74

  “Inferring Strategies for Sentence Ordering in Multidocument News Summariza tion”,

  [10] C.Y Lin, “ROUGE: A Package for Automat- ic Evaluation Summaries”, in Proceedings of

  “Centroid-based Summarization of Multiple Documents”, Information Processing and Management , vol. 40, pp. 919-938, 2004.

  2010. [9] D.R Radev, H. Jing, M. Stys, & D. Tam,

  Symposium on Intelligent Information Tech- nology and Security Informatics (IITSI) ,

  Li, “Mo- deling Document Summarization as Multi- objective Optimization”, Third International

  [8] L. Huang, Y. He, F. Wei, & W.

  Journal of Recent Technology and Engineer- ing (IJRTE) , vol. 3, no. 2, pp. 71-76, 2014.

  “Semantic Ba- sed Sentence Ordering Approach for Multi- Document Summarization”, International

  Ishizuka, “A Preference Learning Approach to Sentence Ordering for Multi-Document Summarizati- on”, Information Sciences, vol. 217, pp. 78- 95, 2012. [7] P. Sukumar, & K.S. Gayathri,

  vol. 17, pp. 35 –55, 2002. [6] D. Bollegala, N. Okazaki, & M.

  Journal of Artificial Intelligence Research ,

  Based on Evolutionary Optimization Algo- rithm”, Expert Systems with Applications, vol. 40, pp. 1675-1689, 2013. [5] R. Barzilay, N. Elhadad, & K. McKeown,

  

10 Jurnal Ilmu Komputer dan Informasi (Journal of Computer Science and Information) , Volume 8, Issue

1, February 2015