A constructionist approach to student mo
A constructionist approach
to student modelling
Katrien Beuls
VUB Artificial Intelligence Laboratory
katrien@ai.vub.ac.be
Eurocall 2013, Évora, 11 September 2013
2
State-of-the-art language learning
systems promise “immersive learning”
Rosetta Stone
“Don’t learn a language, absorb it”
LANGUA
“Language learning should be natural”
Terminology of learning theories
(Acquisition-learning distinction,
Natural approach)
3
But they fail to accommodate
for individual student differences
Input is not very interesting
-
matching activities
-
elimination
Every student follows same path
irrespective of his answers
Similar to programmed instruction
of the early days of ITS
Krashen, S. (2013). Rosetta Stone: Does not provide compelling input, research reports at best suggestive, conflicting
reports on users' attitudes. International Journal of Foreign Language Education, 8(1), 1–3.
4
Academic initiatives try to counter
this lack of individualization
German Tutor (Heift & Schulze, 2007)
-
web-based German exercises
-
linguistic analysis for feedback
TAGARELA (Amaral & Meurers, 2007)
-
web-based language tutoring system
for Portuguese
-
annotations of learner input
5
I propose an active student model
That can predict a student’s answers
by simulating the learning task
It is implemented as a student agent
that can process and learn
the target language
6
A constructionist approach
to student modelling
1. Design of an agent-based language tutor
2. Operationalizing processing and learning
3. Ideas about tutoring
7
A constructionist approach
to student modelling
1. Design of an agent-based language tutor
A competent language user
A student model
Tutoring strategies
8
A language agent simulates
a competent language user
language agent
cxn
inventoryi
grammar
enginei
flexibility
strategies
A construction inventory (grammar)
An engine to process constructions
Flexibility strategies
to flexibly process the target language
9
e agent
A student agent
has the same architecture
student agent
cxn
inventoryj
grammar
enginej
learning
strategies
Instead of flexibility strategies,
the agent makes use of
learning strategies that target
specific acquisition problems
Also interacts with language agent
(no real student required)
10
Tutoring strategies mediate
the tutor-student interaction
tutor agent
language agent
student agent
cxn
inventoryi
cxn
inventoryj
grammar
enginei
grammar
enginej
flexibility
strategies
learning
strategies
tutoring
strategies
student
profile
11
The tutor agent’s student model
consists of a dual structure
A runnable student agent
-
predict student’s answers
-
align to student every interaction
A more static student profile
-
store user profile, preferences
-
update interaction logs and scores
12
A constructionist approach
to student modelling
2. Operationalizing processing and learning
Fluid Construction Grammar (FCG)
Spanish verb conjugation
Error correction
13
FCG lends itself well
for tutoring purposes
Construction-grammar formalism
Everything is a construction
(phon, morph, lex, phrasal, pragmatic)
A construction consists of features
Unification-based (HPSG) but less strict
Customizable search process
Steels, L. (2011). Design Patterns in Fluid Construction Grammar. (L. Steels, Ed.). Amsterdam: John Benjamins.
14
The formalism is informative
about failed constructional matches
Processing problems can be detected
when parsing student input
Uninterrupted processing guaranteed
with meta-level architecture
ía-past-imperfect-2/3
initial
jugar
1/3sg-morph
15
Language agent is initialized with
600 most common verbs in Spanish
18 x 6 conjugated forms / verb
+ 2 imperatives
+ 2 gerunds
= 112 forms / verb
Fred Jehle’s database contains
> 11 000 conjugated verb forms
Fred F. Jehle, University of New Mexico, Verb list taken from http://users.ipfw.edu/jehle/VERBLIST.HTM
16
Proficiency evaluated on
learner errors from SPLOCC II corpus
“The emergence and development of the
tense-aspect system in L2 Spanish”
408 errors made by
low intermediate learners
-
*cogue ➔ coge, ‘he takes’
-
*leó ➔ leyó, ‘he read’
-
*escuchían ➔ escuchaban, ‘they heard’
Evaluated by parsing and reproducing
Mitchell, R., Dominguez, L., Maria, A., Myles, F., & Marsden, E. (2008). A new database for Spanish second language
acquisition research. EUROSLA Yearbook, 8(1), 287–304.
17
SPLOCC results
unknown stem
!
43%
!
suffix change
100%
stem change
1%
verb class
unknown suffix
26%
12%
stem change
18%
accuracy
suffix change verb class
80%
unknown stem
unknown suffix
70%
18
The student agent is initialized
with an empty grammar
language agent
student agent
cxn
inventoryi
cxn
inventoryj
grammar
enginei
grammar
enginej
flexibility
strategies
learning
strategies
Empty construction inventory
Default grammar engine
settings
Learning strategies to acquire
target grammar
19
Learning strategies tackle
learning problems instead of
processing problems
diagnostics
D1
D2
repairs
problem-a
1.0
problem-b
0.5
0.2
D3
problem-c
D4
D5
0.3
0.9
R1
R2
R3
R4
0.1
problem-d
20
0.6
R5
Three types of learning strategies
Learning the basics
-
unknown stem, suffix
-
irregular verb
-
new grammatical meaning
Learning verb stem/suffix changes
-
coje < coger; pienso < pensar; ...
Learning verb classes
-
hablamos, comemos, vivimos
21
Learning happens in
discriminative contexts
Rosetta Stone Version 3, Spanish
22
The student agent can be
the speaker or the hearer in a game
student agent
1
cxn
inventoryj
1
parse
grammar
enginej
2
learning
strategiesj
3
1,2,3
find
topic
1
select
topic
consolidate
signal
success/
failure
produce
1,2,3
consolidate
situation
situation
language agent
cxn
inventoryi
grammar
enginei
learning
strategiesi
select
topic
game n
produce
signal
success/
failure
parse
game n+1
23
produce
find
topic
give
feedback
game n+2
t
Example: Three picking events
now
t
1
2
Student agent:
-
topic: event 2
-
utterance: “ha recojado”
Language agent:
-
correction: “ha recogido”
Student agent learns verb class
24
3
Communicative success reaches 100%
700
communicative success
500
80%
400
60%
300
40%
200
20%
100
0
0
0
2000 4000 6000 8000 10000 12000 14000 16000 18000 20000
games
Learning full conjugation of 25 verbs (mixed)
25
cxn inventory size
600
100%
Most learned constructions are suffixes
350
number of cxns
300
250
200
150
100
50
0
lex
suffix
irreg
Learning full conjugation of 25 verbs (mixed)
26
aux
gram
A constructionist approach
to student modelling
3. Ideas about tutoring
The Colour Tutoring Game
Tutoring strategies for learning problems
(work in progress)
27
First tutoring game prototype
for colour word learning
Web interface with colour chips
User can be student or tutor
(teach the system a colour lexicon)
New colour lexicon can be used for
future students
Beuls, K., & Bleys, J. (2011). Game-based Language Tutoring. In B. Knox (Ed.) Proceedings of the IJCAI 2011 Workshop
on Agents Learning Interactively from Human Teachers, Barcelona.
28
The colour tutoring game
is lacking tutoring strategies
Tutoring strategies serve two functions
1. Selecting a new situation
2. Providing constructive feedback
Spanish verb tutor should tackle
learning problems that are diagnosed
29
Alignment is critical
for a predictive student model
The tutor agent needs to align
the linguistic knowledge of the student
with the constructions known by the
student agent after every game
30
Conclusions
The language agent and the student
agent form the basic foundations for an
adaptive tutoring system for language
Using Construction Grammar to
represent linguistic knowledge is
beneficial for understanding the
student’s difficulties
Work in progress...
-
Building a user interface
-
Developing tutoring strategies
and a data structure they work on
-
Experiments with alignment
Possible future extensions lead to
-
other game scenarios
-
larger sentences
-
more languages
Further reading
Beuls, K. (2012). Grammatical error diagnosis in Fluid
Construction Grammar: A case study in L2 Spanish
verb morphology. Computer Assisted Language
Learning. doi:10.1080/09588221.2012.724426.
Beuls, K. (2012). Inflectional patterns as constructions:
Spanish verb morphology in Fluid Construction
Gammar. Constructions and Frames, 4(2). p 231-252.
Steels, L. (Ed.). (2011). Design Patterns in Fluid
Construction Grammar. Amsterdam: John Benjamins.
Steels, L. (Ed.). (2012). Computational Issues in Fluid
Construction Grammar. Berlin: Springer.
Additional slides
Flexibility strategies
are active during linguistic processing
R2
R3
R5
problem-b
R4
R1
problem-a
restart
restart
problem-c
D1
37
D4
no
restart
R3
Constructions relate meaning to form via
semantic and syntactic categorizations
meaning
form
semantic
categorizations
syntactic
categorizations
38
Detect feature mismatch
ía-past-imperfect-2/3
initial
jugar
1/3sg-morph
Second merge fails due to
different verb class feature in stem
ía = 2/3; jugar = 1
Diagnostic returns verb class feature
and its correct value
39
Detect unknown stem
Very frequent problem with beginning
learners
juqaba => jugaba
Repaired with closest match on stems in
grammar
Levenshtein distance with additional
weight on first letter
40
Learning problem priority list
Front
Back
LP1
LP2
LP3
LP4
LP5
LP6
LP7
LP8
FI = 0.7
LET = 0.4
1 > FI > LET
LP9
1
2
LP10
3
...
Back
Front
LP8
LP3
LP2
LP4
41
LP5
LP6
LP7
LP1
to student modelling
Katrien Beuls
VUB Artificial Intelligence Laboratory
katrien@ai.vub.ac.be
Eurocall 2013, Évora, 11 September 2013
2
State-of-the-art language learning
systems promise “immersive learning”
Rosetta Stone
“Don’t learn a language, absorb it”
LANGUA
“Language learning should be natural”
Terminology of learning theories
(Acquisition-learning distinction,
Natural approach)
3
But they fail to accommodate
for individual student differences
Input is not very interesting
-
matching activities
-
elimination
Every student follows same path
irrespective of his answers
Similar to programmed instruction
of the early days of ITS
Krashen, S. (2013). Rosetta Stone: Does not provide compelling input, research reports at best suggestive, conflicting
reports on users' attitudes. International Journal of Foreign Language Education, 8(1), 1–3.
4
Academic initiatives try to counter
this lack of individualization
German Tutor (Heift & Schulze, 2007)
-
web-based German exercises
-
linguistic analysis for feedback
TAGARELA (Amaral & Meurers, 2007)
-
web-based language tutoring system
for Portuguese
-
annotations of learner input
5
I propose an active student model
That can predict a student’s answers
by simulating the learning task
It is implemented as a student agent
that can process and learn
the target language
6
A constructionist approach
to student modelling
1. Design of an agent-based language tutor
2. Operationalizing processing and learning
3. Ideas about tutoring
7
A constructionist approach
to student modelling
1. Design of an agent-based language tutor
A competent language user
A student model
Tutoring strategies
8
A language agent simulates
a competent language user
language agent
cxn
inventoryi
grammar
enginei
flexibility
strategies
A construction inventory (grammar)
An engine to process constructions
Flexibility strategies
to flexibly process the target language
9
e agent
A student agent
has the same architecture
student agent
cxn
inventoryj
grammar
enginej
learning
strategies
Instead of flexibility strategies,
the agent makes use of
learning strategies that target
specific acquisition problems
Also interacts with language agent
(no real student required)
10
Tutoring strategies mediate
the tutor-student interaction
tutor agent
language agent
student agent
cxn
inventoryi
cxn
inventoryj
grammar
enginei
grammar
enginej
flexibility
strategies
learning
strategies
tutoring
strategies
student
profile
11
The tutor agent’s student model
consists of a dual structure
A runnable student agent
-
predict student’s answers
-
align to student every interaction
A more static student profile
-
store user profile, preferences
-
update interaction logs and scores
12
A constructionist approach
to student modelling
2. Operationalizing processing and learning
Fluid Construction Grammar (FCG)
Spanish verb conjugation
Error correction
13
FCG lends itself well
for tutoring purposes
Construction-grammar formalism
Everything is a construction
(phon, morph, lex, phrasal, pragmatic)
A construction consists of features
Unification-based (HPSG) but less strict
Customizable search process
Steels, L. (2011). Design Patterns in Fluid Construction Grammar. (L. Steels, Ed.). Amsterdam: John Benjamins.
14
The formalism is informative
about failed constructional matches
Processing problems can be detected
when parsing student input
Uninterrupted processing guaranteed
with meta-level architecture
ía-past-imperfect-2/3
initial
jugar
1/3sg-morph
15
Language agent is initialized with
600 most common verbs in Spanish
18 x 6 conjugated forms / verb
+ 2 imperatives
+ 2 gerunds
= 112 forms / verb
Fred Jehle’s database contains
> 11 000 conjugated verb forms
Fred F. Jehle, University of New Mexico, Verb list taken from http://users.ipfw.edu/jehle/VERBLIST.HTM
16
Proficiency evaluated on
learner errors from SPLOCC II corpus
“The emergence and development of the
tense-aspect system in L2 Spanish”
408 errors made by
low intermediate learners
-
*cogue ➔ coge, ‘he takes’
-
*leó ➔ leyó, ‘he read’
-
*escuchían ➔ escuchaban, ‘they heard’
Evaluated by parsing and reproducing
Mitchell, R., Dominguez, L., Maria, A., Myles, F., & Marsden, E. (2008). A new database for Spanish second language
acquisition research. EUROSLA Yearbook, 8(1), 287–304.
17
SPLOCC results
unknown stem
!
43%
!
suffix change
100%
stem change
1%
verb class
unknown suffix
26%
12%
stem change
18%
accuracy
suffix change verb class
80%
unknown stem
unknown suffix
70%
18
The student agent is initialized
with an empty grammar
language agent
student agent
cxn
inventoryi
cxn
inventoryj
grammar
enginei
grammar
enginej
flexibility
strategies
learning
strategies
Empty construction inventory
Default grammar engine
settings
Learning strategies to acquire
target grammar
19
Learning strategies tackle
learning problems instead of
processing problems
diagnostics
D1
D2
repairs
problem-a
1.0
problem-b
0.5
0.2
D3
problem-c
D4
D5
0.3
0.9
R1
R2
R3
R4
0.1
problem-d
20
0.6
R5
Three types of learning strategies
Learning the basics
-
unknown stem, suffix
-
irregular verb
-
new grammatical meaning
Learning verb stem/suffix changes
-
coje < coger; pienso < pensar; ...
Learning verb classes
-
hablamos, comemos, vivimos
21
Learning happens in
discriminative contexts
Rosetta Stone Version 3, Spanish
22
The student agent can be
the speaker or the hearer in a game
student agent
1
cxn
inventoryj
1
parse
grammar
enginej
2
learning
strategiesj
3
1,2,3
find
topic
1
select
topic
consolidate
signal
success/
failure
produce
1,2,3
consolidate
situation
situation
language agent
cxn
inventoryi
grammar
enginei
learning
strategiesi
select
topic
game n
produce
signal
success/
failure
parse
game n+1
23
produce
find
topic
give
feedback
game n+2
t
Example: Three picking events
now
t
1
2
Student agent:
-
topic: event 2
-
utterance: “ha recojado”
Language agent:
-
correction: “ha recogido”
Student agent learns verb class
24
3
Communicative success reaches 100%
700
communicative success
500
80%
400
60%
300
40%
200
20%
100
0
0
0
2000 4000 6000 8000 10000 12000 14000 16000 18000 20000
games
Learning full conjugation of 25 verbs (mixed)
25
cxn inventory size
600
100%
Most learned constructions are suffixes
350
number of cxns
300
250
200
150
100
50
0
lex
suffix
irreg
Learning full conjugation of 25 verbs (mixed)
26
aux
gram
A constructionist approach
to student modelling
3. Ideas about tutoring
The Colour Tutoring Game
Tutoring strategies for learning problems
(work in progress)
27
First tutoring game prototype
for colour word learning
Web interface with colour chips
User can be student or tutor
(teach the system a colour lexicon)
New colour lexicon can be used for
future students
Beuls, K., & Bleys, J. (2011). Game-based Language Tutoring. In B. Knox (Ed.) Proceedings of the IJCAI 2011 Workshop
on Agents Learning Interactively from Human Teachers, Barcelona.
28
The colour tutoring game
is lacking tutoring strategies
Tutoring strategies serve two functions
1. Selecting a new situation
2. Providing constructive feedback
Spanish verb tutor should tackle
learning problems that are diagnosed
29
Alignment is critical
for a predictive student model
The tutor agent needs to align
the linguistic knowledge of the student
with the constructions known by the
student agent after every game
30
Conclusions
The language agent and the student
agent form the basic foundations for an
adaptive tutoring system for language
Using Construction Grammar to
represent linguistic knowledge is
beneficial for understanding the
student’s difficulties
Work in progress...
-
Building a user interface
-
Developing tutoring strategies
and a data structure they work on
-
Experiments with alignment
Possible future extensions lead to
-
other game scenarios
-
larger sentences
-
more languages
Further reading
Beuls, K. (2012). Grammatical error diagnosis in Fluid
Construction Grammar: A case study in L2 Spanish
verb morphology. Computer Assisted Language
Learning. doi:10.1080/09588221.2012.724426.
Beuls, K. (2012). Inflectional patterns as constructions:
Spanish verb morphology in Fluid Construction
Gammar. Constructions and Frames, 4(2). p 231-252.
Steels, L. (Ed.). (2011). Design Patterns in Fluid
Construction Grammar. Amsterdam: John Benjamins.
Steels, L. (Ed.). (2012). Computational Issues in Fluid
Construction Grammar. Berlin: Springer.
Additional slides
Flexibility strategies
are active during linguistic processing
R2
R3
R5
problem-b
R4
R1
problem-a
restart
restart
problem-c
D1
37
D4
no
restart
R3
Constructions relate meaning to form via
semantic and syntactic categorizations
meaning
form
semantic
categorizations
syntactic
categorizations
38
Detect feature mismatch
ía-past-imperfect-2/3
initial
jugar
1/3sg-morph
Second merge fails due to
different verb class feature in stem
ía = 2/3; jugar = 1
Diagnostic returns verb class feature
and its correct value
39
Detect unknown stem
Very frequent problem with beginning
learners
juqaba => jugaba
Repaired with closest match on stems in
grammar
Levenshtein distance with additional
weight on first letter
40
Learning problem priority list
Front
Back
LP1
LP2
LP3
LP4
LP5
LP6
LP7
LP8
FI = 0.7
LET = 0.4
1 > FI > LET
LP9
1
2
LP10
3
...
Back
Front
LP8
LP3
LP2
LP4
41
LP5
LP6
LP7
LP1