Tamura, Y., & Kusanagi, K. (2014). Validation of the grammatical carefulness scale using a discourse completion task and a reading and underlining task. The 84th LET Chubu Confernce. Shizuoka University, Japan.

1. Validation of the Grammatical
Carefulness Scale Using a
Discourse Completion Task
and a Reading and
Underlining Task
November 22, 2014
84th LET Chubu
Shizuoka University

2. The handout is available from…

3. The handout is available from…

4. Yu TAMURA
Graduate School, Nagoya Univ.
yutamura@nagoya-u.jp
Kunihiro KUSANAGI
Graduate School, Nagoya Univ.
JSPS Research Fellow
kusanagi@nagoya-u.jp

5. Overview
• Introduction
• Background
• The Present Study
• Results
• Discussion
• Conclusion

6. Overview
• Introduction
• Background
• The Present Study
• Results
• Discussion
• Conclusion

7. Introduction
• What is grammatical carefulness(GC)?
• Why is it needed?
• How can it be used?

8. Introduction
• What is grammatical carefulness (GC)?
• a construct referring to learners’
characteristics
• personal traits reflecting learner’s
behaviour and belief in language use
• explains learners’ variance
• reflects highly controlled, cautious, and
analytical language use

9. Introduction
• Why is it needed?
• To capture the variance of L2 learners’
inconsistent performance
• To get more information about individual
differences focusing on grammatical
performance
• To take into account speed accuracy trade-off
(Dennis & Evans, 1996; Goldhammer &
Kroehne, 2014; van der Linden, 2007, 2009)

10. Introduction
• speed accuracy trade-off
• consistent within a learner
• linked to other learners’ factors (e.g., psychological,
behavioural, and meta-cognitive)

11. Introduction
• How can it be used?
• as covariance in psycholinguistic experimental
research
• as independent variables in interventional
experiments
• in investigating relationship between individual
differences and L2 learners’ performance
• as a diagnostic assessment tool in educational
settings

12. Overview
• Introduction
• Background
• The Present Study
• Results
• Discussion
• Conclusion

13. Overview
• Introduction
• Background
• The Present Study
• Results
• Discussion
• Conclusion

14. Background
• Grammatical Carefulness Scale (GCS)
• A questionnaire developed by Kusanagi et al. (2014)
• 14 items (3 sub-scales)
• 7 point Likert scale
• written in Japanese
• Some validation has been done (Kusanagi et al., 2014)
• factorial validity
• content validity
• criterion-based validity

15. Background
• Grammatical Carefulness Scale (GCS)
• 3 sub-scales
• Lexical-Syntactic Carefulness (LSC) (k = 4)
• e.g., I always notice the wrong use of the words.
• Pragmatic Carefulness (PC) (k = 5)
• e.g., I’m likely to think about inconsistent expressions.
• Phonological Carefulness (PHC) (k = 5)
• e.g., I’m always careful about correct pronunciation.

16. Background
• Grammatical Carefulness Scale (GCS)
• factorial validity
• both exploratory and confirmatory
• high reliability (α = .90)
• content validity
• judgments by linguistic experts
• criterion-based validity
• correlated to analytical belief (Tanaka & Ellis, 2003) more strongly
than empirical belief
• correlated to the accuracy score of GJTs
• correlated to the accuracy score of C-test and its time to completion

17. GCS is validated!

18. Validation never ends.

19. Background
• Grammatical Carefulness Scale (GCS)
• factorial validity
• both exploratory and confirmatory
• high reliability (α = .90)
• content validity
• judgments by linguistic experts
• criterion-based validity
• correlated to analytical belief (Tanaka & Ellis, 2003) more strongly
than empirical belief
• correlated to the accuracy score of GJTs
• correlated to the accuracy score of C-test and its time to completion

20. Background
• Grammatical Carefulness Scale (GCS)
• factorial validity
• both exploratory and confirmatory
• high reliability (α = .90)
• content validity
• judgments by linguistic experts
• criterion-based validity
• correlated to analytical belief (Tanaka & Ellis, 2003) more strongly
than empirical belief
• correlated to the accuracy score of GJTs
• correlated to the accuracy score of C-test and its time to completion

21. Background
• Grammatical Carefulness Scale (GCS)
• 3 sub-scales
• Lexical-Syntactic Carefulness (LSC) (k = 4)
• e.g., I always notice the wrong use of the words.
• Pragmatic Carefulness (PC) (k = 5)
• e.g., I’m likely to think about inconsistent expressions.
• Phonological Carefulness (PHC) (k = 5)
• e.g., I’m always careful about correct pronunciation.

22. Background
• Grammatical Carefulness Scale (GCS)
• 3 sub-scales
• Lexical-Syntactic Carefulness (LSC) (k = 4)
• e.g., I always notice the wrong use of the words.
• Pragmatic Carefulness (PC) (k = 5)
• e.g., I’m likely to think about inconsistent expressions.
• Phonological Carefulness (PHC) (k = 5)
• e.g., I’m always careful about correct pronunciation.

23. Background
• RQ
• Do the scores of the tasks focusing
either on LSC or PC correlate with the
LSC and PC measured by GCS?

24. Overview
• Introduction
• Background
• The Present Study
• Results
• Discussion
• Conclusion

25. Overview
• Introduction
• Background
• The Present Study
• Results
• Discussion
• Conclusion

26. The Present Study
• Participants (N = 63)
• undergraduate students (3 university)
and graduate students (1 university)
• 78% of the participants were female
undergrad grad
A B C D
n 20 27 8 8

27. The Present Study
• Tasks
• LSC
• Reading and Underlining Task (RUT)
• PC
• Discourse Completion Task (DCT)

28. The Present Study
• Reading and Underlining Task (RUT)
• often used as a measure of learner’s
noticing toward formal properties of TL
(e.g., Izumi & Bigelow, 2000; Izumi,
Bigelow, Fujiwara, & Fearnow, 1999;
Uggen, 2012)
• requiring to read a short passage and
underline ungrammatical parts of the
sentences.

29. The Present Study
• Reading and Underlining Task (RUT)
• passage (153 words)
• from internet sources (http://www.eslyes.com)
• Flesch-Kincaid Reading Ease 94.252
• Flesch-Kincaid Grade Level 1.558
• revised and controlled vocabularies
(JACET8000 Level1-3)
• Glossary was given.

30. The Present Study
• Reading and Underlining Task (RUT)
• 9 errors
• inflection errors
• was killing/ *kill her; She cried/ *cryed; She stood/ *standing up
• misspelling
• walked/ *wolked; could/ *cuold; always/ *alweys
• wrong use of words
• Whenever she was stressed out, the stress went straight to her/
*his stomach.
• Almost an hour later/ *ago,….
• I feel much/ *more better.

31. The Present Study
• Tasks
• LSC
• Reading and Underlining Task (RUT)
• PC
• Discourse Completion Task (DCT)

32. The Present Study
• Tasks
• LSC
• Reading and Underlining Task (RUT)
• PC
• Discourse Completion Task (DCT)

33. The Present Study
• Discourse Completion Task (DCT)
• used to tap into learners’ intuition about
how they behave in certain speech
acts (e.g., Bardovi-Harlig & Dornyei,
1998; Olshtain & Blum-Kulka, 1985)
• can be both spoken and written
• can be both multiple-choice or free
production

34. The Present Study
• Discourse Completion Task (DCT)
• based on Blum-Kulka, House, and Kasper (1989)
• 2 speech acts (K = 12)
• request (k =8)
• apology (k = 8)
• various dimensions of social power and social
distance
• multiple-choice
• situation and instructions are given in Japanese

35. The Present Study
• Discourse Completion Task (DCT)
• example
• 見知らぬ街へ出かけたメアリー。駐車禁止場所とは知らずに車を停めてしまいました。する
と警察官が寄ってきて…
Policeman: ( )
Mary: Oh, I’m sorry. I didn’t know that. I’ll move the car right now.
a. Would you please move your car from here?
b. How about moving the car from here?
c. Could you move the car from here?
d. Do you want to move the car from here?
e. Move your car.

36. The Present Study
• Discourse Completion Task (DCT)
• example
• 見知らぬ街へ出かけたメアリー。駐車禁止場所とは知らずに車を停めてしまいました。する
と警察官が寄ってきて…
Policeman: ( )
Mary: Oh, I’m sorry. I didn’t know that. I’ll move the car right now.
a. Would you please move your car from here?
b. How about moving the car from here?
c. Could you move the car from here?
d. Do you want to move the car from here?
e. Move your car.

37. The Present Study
• Reliability issue
• Reading and Underlining Task (RUT)
N K M SD Cronbach α 95%CI
60 9 .55 .21 .50 [.28, .72]
• Some items were excluded

38. The Present Study
• Reliability issue
• Reading and Underlining Task (RUT)
N K M SD Cronbach α 95%CI
60 6 .49 .27 .58 [.28, .72]
• Excluded items: was killing/*kill, her/*his, later/
*ago

39. The Present Study
• Reliability issue
• Discourse Completion Task (DCT)
N K M SD Cronbach α 95%CI
61 12 .56 .18 .48 [.26, .69]
• Some items were excluded.

40. The Present Study
• Reliability issue
• Discourse Completion Task (DCT)
N K M SD Cronbach α 95%CI
61 7 .56 .23 .56 [.35, .77]
• Excluded items: Item2, 9, 11, 5, 8

41. The Present Study
• Procedure
• Two English language teachers
working for a college
• Two tasks were administered as part of
classroom activities
• Graduate students were recruited
individually

42. The Present Study
• Analysis
• 10 students missed one of the two tasks
→ excluded from the analysis
• 3 students showed extremely low
products of deviation
→ excluded from the analysis (outlier)
• In total, 50 students were included in
the final analysis.

43. The Present Study
• Analysis
• Correlation between the score of the
tasks and GCs
→ Pearson product-moment correlation
coefficient
→ Partical Correlation Coefficient

44. Overview
• Introduction
• Background
• The Present Study
• Results
• Discussion
• Conclusion

45. Overview
• Introduction
• Background
• The Present Study
• Results
• Discussion
• Conclusion

46. Results

47. Descriptive Statistics of the Two tasks, LSC, and PC
N M SD skew kurtosis SE α
RUT 50 .50 .32 .56 -1.01 -.04 .58
LSC 50 3.24 1.24 .31 -.20 .18 .87
PC 50 3.18 1.20 .18 -.05 .17 .85
DCT 50 .54 .20 -.10 -.46 .03 .56

51. Partical Correlation Coefficient

52. Interval Estimation of
Population Correlation Coefficient (95%CI)
RUT LSC PC DCT
RUT
LSC [.013, .526]
PC [-.083, .453] [.686, .888]
DCT [-.232, .324] [.002, .518] [-.062, .470]

53. Adjusting Correction for Attenuation of Correlation
RUT LSC PC DCT
RUT
LSC .408
PC .285 .942
DCT .088 .401 .319

54. Overview
• Introduction
• Background
• The Present Study
• Results
• Discussion
• Conclusion

55. Overview
• Introduction
• Background
• The Present Study
• Results
• Discussion
• Conclusion

56. Discussion
• RUT and LSC
• Pearson’s r
→weakly correlated
• Partical Correlation Coefficient
→weakly correlated
• Interval Estimation of Population Correlation Coefficient
→◯
• Adjusting Correction for Attenuation of Correlation
→moderately correlated

57. Discussion
• RUT and LSC
• Pearson’s r
→weakly correlated
• Partical Correlation Coefficient
→weakly correlated
• Interval Estimation of Population Correlation Coefficient
→◯
• Adjusting Correction for Attenuation of Correlation
→moderately correlated
• The students who showed high LSC were likely to notice
lexical and grammatical errors

58. Discussion
• DCT and PC
• Pearson’s r
→weakly correlated
• Partical Correlation Coefficient
→△
• Interval Estimation of Population Correlation Coefficient
→△
• Adjusting Correction for Attenuation of Correlation
→weakly correlated

59. Discussion
• DCT and PC
• Pearson’s r
→weakly correlated
• Partical Correlation Coefficient
→△
because LSC and PC showed high correlation
• Interval Estimation of Population Correlation Coefficient
→△
• Adjusting Correction for Attenuation of Correlation
→weakly correlated

60. Discussion
• The results seemed to confirm the validity of
GCS
• difficult to investigate the correlation
between L2 learners’ performance and the
subscales of GC
→ GCS showed high reliability

61. Partical Correlation Coefficient

62. Limitations
• Reliability of the two tasks
• Underrepresentation
• What about the correlation between
phonological carefulness and learners’
performance?

63. Overview
• Introduction
• Background
• The Present Study
• Results
• Discussion
• Conclusion

64. Overview
• Introduction
• Background
• The Present Study
• Results
• Discussion
• Conclusion

65. Conclusion
• The score of LSC predicted the score of
RUT
• The score of PC partly predicted the score
of DCT
• Validation never ends

66. Reference
Bardovi-Harlig, K., & Dörnyei, Z. (1998). Do language learners recognize pragmatic violations? pragmatic versus grammatical
awareness in instructed L2 Learning. TESOL Quarterly, 32, 233–259. doi:10.2307/3587583
Blum-Kulka, S., House, J., & Kasper, G. (1985). Investigating cross-cultural pragmatics: An Introductory Overview. In S. Blum-Kulka, J.
House, & G. Kasper (Eds.), Cross-cultural pragmatics: Requests and apologies (pp. 1–33). Norwood, NJ: Ablex Publisher
Corporation.
Dennis, I., & Evans, J. St. B. T. (1996). The speed-error trade-off problem in psychometric testing. British Journal of Psychology, 87,
105–129.
Goldhammer, F., & Kroehne, U. (2014). Controlling individuals’ time spent on task in speeded performance measures: Experimental
time limits, posterior time limits, and response time Modelling. Applied Psychological Measurement, 38, 255–267.
Izumi, S., & Bigelow, M. (2000). Does output promote noticing and second language acquisition? TESOL Quarterly, 34, 239–278.
Izumi, S., Bigelow, M., Fujiwara, M., & Fearnow, S. (1999). Testing the output hypothesis: Effects of output on noticing and second
language lcquisition. Studies in Second Language Acquisition, 21, 421–452.
Kusanagi, K., Fukuta, J., Kawaguchi, Y., Tamura, Y., Goto, A., Kurita, A., & Murota, D. (2014). Gaikokugo ni okeru bumpouteki shinchou
sei syakudo no kaihatsu [Development of grammatical carefulness in English as a foreign language]. 40th Annual Conference of
Japan Society of English Language Education. Tokushima University, Japan.
Olshtain, E., & Blum-Kulka, S. (1985). Degree of approximation: Nonnative reactions to native speech act behaviour. In S. M. Gass &
C. G. Madden (Eds.), Input in second language acquisition (pp. 303–325). Newbury House.
Tanaka, K., & Ellis, R. (2003). Study abroad, language proficiency, and learner beliefs about language learning. JALT journal, 25, 63-85.
Uggen, M. S. (2012). Reinvestigating the Noticing Function of Output. Language Learning, 62, 506–540. doi:10.1111/j.
1467-9922.2012.00693.x
van der Linden, W. J. (2007). A hierarchical framework for modelling speed and accuracy on test items. Psychometrika, 73, 287–308.
van der Linden, W. J. (2009). Conceptual issues in response-time modelling. Journal of Educational Measurement, 46, 247–272.