Click the star to add/remove an item to/from your individual schedule.
You need to be logged in to avail of this functionality.
Log in
- Convenors:
-
Mari Tanaka
(Nagoya University of Foreign Studies)
Shin Abe (Tokyo University of Foreign Studies)
Send message to Convenors
- Discussant:
-
Michiyo Fuchs-Shimizu
(Japanese School Basel)
- Stream:
- Japanese Language Education
- Location:
- Torre B, Piso 3, Auditório 2
- Sessions:
- Saturday 2 September, -
Time zone: Europe/Lisbon
Short Abstract:
This panel reports findings from an analysis of essays by European JSL learners, drawn from the I-JAS corpus. We performed holistic and multiple-trait rating using six levels, and three multiple-trait perspectives: "Japanese Proficiency," "Purpose and Content," and "Organization and Cohesion."
Long Abstract:
Nowadays, the corpus of Japanese language learners has been widely opened to the public. However, there have been few corpora with evaluations, and most of those have been limited to data for speakers whose L1s are Chinese, Korean, and English.
In this study, three raters applied a holistic rating of levels 1-6 to essays (covering 611 written samples) from the International Corpus of Japanese as a Second Language (I-JAS). Of these, we found data for 193 samples of European learners, and we applied multiple-trait rating (MT) to 55 of the essays with the highest rates of concordance of holistic rating. The traits of MT are "Japanese Proficiency," "Purpose and Content," and "Organization and Cohesion." This panel reports the findings of this data analysis.
For "Japanese Proficiency," using holistic ratings for all 193 samples, we divide them into three groups: high (scores of 4 & 5), middle (score of 3), and low (scores of 1 & 2). We calculate the average value of text features of each group, and specify statistically significant linguistic features such as particles and verbs, etc. We analyze differences found in writings of each group.
For "Purpose and Content," we provide representative examples of writing for each level, and show a relationship between holistic rating and MT rating. The distribution of writing samples shows that, of the 55 samples we examined, most were at Level 2, followed by Level 4, and then Level 3. Even for writing samples at MT Level 2, we could see high scores for holistic rating. Having done this analysis, we then perform qualitative analysis of the writing.
For "Organization and Cohesion," we analyze macro-organizational essay patterns seen in European learners' essays, along with the meta-language, referring to the levels. The use of meta-language is related with cohesion between paragraphs. Of the 55 writing samples, about half were at Level 3. We compare these to higher-level essays, and perform a qualitative analysis.
This study aims to contribute to future developments in Japanese writing education in Europe. We hope to have a discussion with the floor based on our results.
Accepted papers:
Session 1 Saturday 2 September, 2017, -Shin Abe (Tokyo University of Foreign Studies) Aiko Sasaki (National Institute for Japanese Language and Linguistics)
Paper short abstract:
In the "Japanese Proficiency" part, we divided 193 essays into three groups by evaluation score, and analyzed writing tendencies of each group. We found that the high group has a diversity of written expressions, as well as longer compositions.
Paper long abstract:
This report analyzes Japanese linguistic features of 193 essays from the I-JAS corpus of European learners of Japanese by comparing three groups based on holistic rating: 49 rated high (L4-5); 69 rated middle (L3); and 58 rated low (L1-2). We checked the mean and statistical significance of linguistic feature values of each group; specified statistically significant linguistic features (e.g., particles, verbs); and investigated writing differences. We came to three sets of findings.
First, concerning quantitative measures, we saw statistically significant differences between the low and high groups in the number of characters used for their compositions, but we did not see statistical significance for the number of sentences used. The high group's essays and sentences were both longer. Compared to the low group, the high group also had more variety of and higher total number of morphemes; the high group had greater variety of vocabulary as well.
Second, we performed morphological analysis of the essays, calculating use of particles, and found differences in the number of particles used. Compared to the high group, the lower group used more sentence-ending particles such as "ne" and "yo." The high group stood out for using more sentences that ended with "ka" than the low group. We found that the higher the holistic rating, the lower the use of particles that are more spoken expressions such as "ne" and "yo," while "ka" was used as a form of written expression.
Finally, we analyzed conjugated verbs; we calculated different verb conjugations, and found that the high group had many usages of the predicative verb form. Also, when we calculated the number of usages of "desu" and "masu" at the end of sentences, we found that the high group had few usages of "desu" and "masu." From this we understood that the high group is more accustomed to writing essays, and were using sentence endings that better fit written expression.
From this analysis, we learned that the high group uses expressions fitting the written word, diverse expressions, and has longer compositions. In our presentation, we explain these various characteristics in more detail.
Yoko Kageyama (Yamanashi Gakuin University)
Paper short abstract:
We investigated the relationship between multiple-trait (MT) ratings of "Purpose and Content" and holistic (H) ratings for 55 essays. We found 43.6% concordance for H and MT ratings. The low group had high H ratings, and the high group had high MT ratings.
Paper long abstract:
This report investigates the relationship between multiple-trait ratings (MT) of "Purpose & Content" and holistic (H) ratings for 55 essays of European learners of Japanese as a second language in the I-JAS; essays compared fast food and home cooking, and offered opinions on food in daily life.
For MT rating, we used a flow chart designed around MT rating standards. Points were assigned by Level (L) 1-6 from lowest to highest for presence or absence of the following: main idea (comparison and opinion); support for the main idea; consistency and appropriateness of the main idea; social perspective and objectivity; and originality. The essay levels were distributed as follows: 5 L1s, 20 L2s; 12 L3s; 16 L4s; 2 L5s; and none at L6.
Our results of the rating relationships show:
(1) H and MT ratings for "Purpose & Content" aligned for 24 of 55 essays, for a 43.6% concordance rate.
(2) Results by level show that for the middle group on MT ratings for "Purpose & Content" (L3), there was a high level of concordance between H and MT ratings. For the group that rated low on MT ratings for "Purpose & Content" (L1-2), H ratings were higher than MT ratings; for the group that rated high on MT ratings for "Purpose & Content" (L4-5), MT ratings were higher than H ratings.
(3) There were only four essays that had a point spread of two or more between H and MT ratings. Of these, three essays had lower MT than H ratings, and one essay had a higher MT than H rating. In the cases where MT ratings were lower than H ratings, essays showed problems such as new ideas introduced in the conclusion; difficulty communicating the main idea because of regular use of questions such as "and you?"; and topics departing from the main theme. The one essay that had a higher MT rating effectively supported arguments with clearly-grounded assertions and specific examples, but had low Japanese language ability.
Mari Tanaka (Nagoya University of Foreign Studies) Yukari Tsubone (Osaka University of Tourism)
Paper short abstract:
In the "Organization and Cohesion" part, we analyze the macro-organizational essay patterns seen in European learners' comparison/contrast and opinion essays, along with the meta-language they use, referring to the level of multiple-trait rating for "Organization and Cohesion".
Paper long abstract:
This report analyzes macro-organizational essay patterns and meta-language, referring to the level of multiple-trait (MT) rating for "Organization and Cohesion," using 55 essays of European learners of Japanese as a second language in the I-JAS, where they compared fast food and home cooking, and offered their opinions on eating habits.
We used a flow chart designed around MT rating standards. Points were assigned by Level (L) 1-6 from lowest to highest for awareness of organization; awareness of paragraphs; macro-organization (introduction, body, conclusion); cohesion of paragraphs; and agreement between introduction and conclusion. We found the following distribution of essay levels: 4 L1s, 5 L2s, 26 L3s, 15 L4s, 4 L5s, and 1 L6.
First, we examined organizational patterns such as number of formal paragraphs, macro-organization, method of developing comparisons (Block style: B-s and Point-by-Point style: P-s), and way of expressing their opinions. We found that half lacked awareness of paragraphs and macro-organization; had no introduction or conclusion; or started a new paragraph for each sentence. For L3 and higher, we saw B-s and P-s patterns; overall there were more B-s patterns. For L4 and higher, B-s patterns and opinions were clearly recognizable, and for L5, there was correspondence between formal paragraphs and content, and paragraph organization was easy to follow.
Next, meta-language concerns cohesion between paragraphs, and relates to ratings principally of L3-L5. Most writers at L3 did not use meta-language when changing topic from fast food to home cooking (or vice-versa), or even if they did have meta-language, it was misused. At L4, writers used meta-language at the beginning of paragraphs when they switched topics, and at L5, writers used meta-language at the beginnings of paragraph that announced content to follow, and the essays had clear organization.
In our results, we found that scores were higher for "Organization and Cohesion" than for "Japanese Proficiency." This suggests that even if Japanese proficiency of learners is inadequate, high organizational awareness and use of appropriate meta-language can make learners' essays readily comprehensible.