EFFECT OF THE USE OF COMPUTER-AIDED ASSESSMENT SYSTEM IN THE TEACHING OF MATHEMATICAL ANALYSIS WITH REGARD TO STUDENTS’ APPROACHES TO LEARNING

Computer-aided assessment systems are increasingly used in education for both formative and summative assessment. Example of such a system from the area of mathematics education is the platform Maple T. A. (Testing and Assessment), which was developed specially for assessment in mathematics. The paper is focused on the efficiency of the use of the Maple T. A. platform to students’ performance in classes of mathematical analysis (calculus). For this reason, the pedagogical experiment was conducted on the first and later second year undergraduate students of teaching mathematics taking into consideration their approaches to learning and studying. The research results include cluster analysis of students’ approaches to learning and studying and analysis of students’ performance in their homework assignments and final tests.


•
Use of CAA system in teaching positively affects student´s performance in final tests • Approaches to learning do not affect student´s overall results (only results in some partial tasks) • When practicing in CAA, students with deep approach need fewer attempts to reach the required score A. at the University of Hradec Kralove and observe the results of the utilization of this platform on the process of teaching mathematical analysis in the Czech educational environment.
The area of interest included how the software will suit different types of students.Because of this, in the first part of the research the students were divided into characteristic groups based upon a questionnaire survey that was focused on the students' approaches to learning and studying.Students' approaches to learning and studying describe what students do when they go about learning and why they do it.The basic distinction is between a deep approach to learning, where students are aiming to understanding, and a surface approach to learning, where they are aiming to reproduce material in a test or exam rather than actually understand it (Entwistle, 1988;Ramsden, 1992;Biggs, 1999).For its quality, the ETL (Enhancing Teaching-Learning Environments in Undergraduate Courses) project's questionnaires were chosen to be modified and used in this research (Entwistle, 2005).The author of this paper managed to get approval from author professor Entwistle to use these instruments from the ETL project.
Let us summarize that the research dealing with the utilization of the Maple T. A. platform was conducted at the University of Hradec Kralove.The first year undergraduate students of teaching mathematics were divided into characteristic groups based upon their approaches to learning and studying.The students that were divided into these characteristic groups then underwent a pedagogical experiment.During the observed semesters, students' performance in homework and final tests was analysed to find whether there are differences depending on the forms of teaching (with or without the aid of CAA) and characteristic groups of students (according to approaches to learning).The research questions were thus: (RQ1) Do the achieved students' results differ in context to the form of teaching (with or without the aid of CAA)?

Introduction
CAA (Computer-Aided Assessment/Computer Assisted Assessment) is a common term for the use of computers in the assessment of student learning (Sangwin, 2013).Good CAA platforms can help to consolidate student understanding, to support selfdirected learning and to make it easier for instructors to manage growing class sizes (Technical Whitepaper, 2015).Contrary to the common assessment platforms, the systems suitable for mathematics must have special functions used for testing mathematical knowledge.The specificity of assessment of students in mathematics involves use of mathematical symbols, equalities, numerical series or graphs.An example of such system is the platform Maple T. A. (Testing and Assessment), which was established by integrating computational capabilities of computer algebra system Maple to CAA system (Keady et al, 2006;Jones, 2008;Berkova, 2015).The presented paper is focused on the efficiency of the use of the Maple T. A. platform to students' performance in classes of mathematical analysis (calculus).The effects of CAA systems have been the subject of several studies with diverse results.The CAA systems are used here mainly for student´s homework and final exams.To name a few Hauk, Powers and Segalla (2015) as well as Allain and Williams (2006) did not find any statistically significant differences between final results of the students using CAA and students using paper and pencil for their homework.The differences were not discovered even by Demirci (2006).However, contrary to the previous studies there was a slight difference in favour of those students who were using the traditional pencil and paper for their homework assignments.On the other hand Love, Keinert and Shelley (2006) and also Burch and Kuo (2010) claim that the means of results of students who utilize online homework were higher than the means of students who used paper and pencil for their assigned homework.
Based on the review and the fact that the utilization of such systems in the Czech Republic is not widespread (according to survey in Berkova, 2014;Berkova and Kulicka, 2016) the author has decided to implement the CAA platform Maple T.
Printed ISSN: 2336-2375 (RQ2) Is there a relationship between the students' approaches to learning and their achieved results?(RQ3) Is the efficiency of the forms of teaching (with or without the aid of CAA) the same for students with different approaches to learning?This paper is an extension and continuation of the previous conference proceeding focused on the efficiency of the use of the Computer-Aided Assessment system in mathematics presented on the 13th International Conference on Efficiency and Responsibility in Education (ERIE 2016) (Berkova, 2016).
In the mentioned proceeding, answers to research questions were presented only using the results of the final summative tests.This paper provides more detailed view on the experiment through an analysis of the ongoing activities of students in formative homework assignments.

Materials and Methods
The research sample was composed of first (2013/2014) and later second (2014/2015) year undergraduate students at the University of Hradec Kralove which were attending the classes of calculus (Mathematical analysis 1, 2, 3).A total of 22 students between the ages of 18 and 20 participated in the study.To maintain anonymity, each student was given a number (Student 1, Student 2, etc.).
In the first phase, the Q1 questionnaire focused on students' approaches to learning and studying was created based on instruments from ETL project.The Q1 questionnaire consists of three main parts (see Figure 1a).The first one named Learning orientations is focused on the students' expectations and their goals in their university studies.It contains a total of seven questions. the study of mathematics itself (inequalities, limits etc.) and application of new mathematical theorems (see Figure 1b).Data from the experiment (students' performance in homework and final didactic tests) was evaluated using analysis of variance and other methods of data analysis.

Results and Discussion
Characteristic groups of students (Cluster analysis) Clustering or cluster analysis is the process of grouping individuals with similar variable measurements (in our case with similar responses in Q1 questionnaire).Table 1 shows the results of this cluster analysis that gave rise to characteristic groups of students based on their responses in Q1 questionnaire according to their approaches to learning and studying.The results were analysed in the NCSS statistical software.Intrinsic 3.52 4.27 3.00 3.52 4.44 4.19 3.00 Social_and_personal_reasons 3.18 3.55 2.00 3.18 4.17 3.29 2.00 Career_reasons 4.82 4.20 2.00 4.82 5.00 3.86 2.00 Lack_of_purpose 2.82 2.30 4.00 2.82 3.00 2.00 4.00 Deep_approach 3.13 4.00 2.22 3.13 3.85 4.11 2.22 Surface_approach 3.14 2.18 2.00 3.14 2.83 1.89 2.00 Organised_effort 3.45 3.68 2.50 3.45 3.67 3.68 2.50 Supporting_understanding 2.61 3.83 2.00 2.61 4.08 3.71 2.00 Preferred course and teaching types -they conclusively prefer teachers who are simply Transmitting information to teachers who Support understanding of the studied material.On the other hand, Cluster2 is apparently composed out of highly motivated students (high score in the Learning orientation section with a drop in the Lack of purpose item).These 10 students with high score in a Deep approach and low scores in Surface approach are evidently interested in more in-depth studying.Conversely, the students from Cluster2 do not have significant preferences in regard to the type of course and teaching and go down well with various types of being taught.Because of this, it was decided to divide the students again using cluster analysis into four clusters (see the part with Four Clusters in Table 1).The results were interesting.Student 16 formed again his own single member cluster which was for the sake of clarity named again Cluster3.Students from the first cluster again formed Cluster1.Cluster2 has split into two groups (for the sake of clarity named Cluster2.1 and Cluster2.2).Cluster2.2consists of 7 students, Cluster2.1 then consists of the 3 remaining students.Because Cluster2.1 only has 3 students, it has been decided to use the previous cluster division for further analysis, taking into account this finer division if it was necessary.As it can be seen in Fig. 2b the students in Cluster2.2are more pronounced in regards to their Approaches to learning.Cluster2.1 is strongly motivated by career and furthermore slightly higher scores can be seen in the Transmitting information subscale.In regards to their Preferences for types of course and teaching, students from Cluster2.1 also prefer traditional, paper and pencil, educational methods over using information technologies.On the other hand, Cluster2.2does prefer information technologies.

Students' performance in homework
Students' performance in homework was processed using Correlation analysis and Analysis of variance (respectively t-test).Correlation analysis is used to quantify the linear association between variables.As correlation measure, the Pearson's correlation coefficient and Spearman's rank correlation coefficient were used.Analysis of variance (ANOVA) is a method of statistical induction used to determine the effect of independent variables on the dependent variable by analysing the differences among group means.In order to quantify the idea of statistical significance, the p-value is presented here.Assuming the truth of the null hypothesis (for example about equivalence of means), the p-value is the probability with which the obtained data supports the null hypothesis (Montgomery, 2001).If the p-value is less than the selected level of significance, the observed data are inconsistent with the null hypothesis and the differences are statistically significant.The selected level of significance was always 0.05.Despite the fact that normality of data distribution has not been rejected in this study, to increase validity of testing (Hendl, 2009) and because of the small sample of students, corresponding non-parametric tests comparing medians were also performed.NCSS program calculates parametric and non-parametric tests at the same time in most cases.On the contrary to mentioned quantitative methods, students' results in homework was also processed with Content analysis.This method seemed to be the most appropriate because different forms of submission of homework in traditional (paper) and experimental (electronic) semester cannot be easily compared (same assignment for all students in paper version instead of different assignments among students in electronic version, repetition of submission in electronic version etc.).
During both semesters of experiment, students submitted five homework assignments (70% of points were required in homework).Interestingly, students from Cluster1 had more difficulty getting used to work with the Maple T.A. system.However, this imbalance between the characteristic groups quickly settled and it seemed that all students priced possibilities offered by the system at the end of the semester.Content analysis of homework has also shown that the performance of students did not changed depending on the form of assignment submission.Students who submitted well prepared homework in a paper version usually did not have problem with electronic homework assigned in the Maple T.A. and conversely.This finding was confirmed by calculation the correlation coefficient between mean performance in paper and mean performance in electronic homework for each student.A moderate positive correlation has been shown (Pearson's correlation coefficient was 0.646179 1 , Spearman's rank coefficient 0.621093).
In the second experimental semester, electronic assignments in Maple T.A. allowed to monitor work of students with the Maple T.A. system.In Figure 3, we can see the mean number of attempts performed in five homework per semester for each characteristic group of students (clusters).The differences in the number of attempts among characteristic groups was tested with ANOVA.It is obvious from the figure, that differences among characteristic groups are statistically significant.This finding was confirmed for both division into two clusters (Cluster 1, Cluster 2) and also finer division into three clusters (Cluster 1, Cluster 2.1, Cluster 2.2).Student 16 (Cluster3) could not be included in the report due to abandoning studies during the experiment.Achieved p-value was 0.013842 2 for division into two clusters and 0.024727 3 for three clusters division.The results show that students from Cluster 1 needed significantly more attempts to reach the required score (70% points) to complete their homework.
1 Observed correlation coefficient was subjected to the significance test that showed significant correlation (p-value 0.002).

2
P-value 0.025588 in relevant non-parametric Mann-Whitney test.

Students' performance in final tests
From the described pedagogical experiment with repeated measurements, final data were gathered with the help of didactics test T1 (traditional teaching) and test T2 (experimental teaching).In this section, all research questions (RQ1-RQ3) were examined using two-way repeated measures ANOVA.
As already mentioned, ANOVA is statistical method used to determine the effect of independent variables on the dependent variable.Repeated measures ANOVA is used when as in our case the comparative measurements are conducted on the same individuals (Littell, Henry and Ammerman, 1998).The independent variables are represented here as the teaching form (traditional/experimental) and the belonging to characteristic group (clusters according to student´s approaches to learning).The dependent variable was students' results in the didactic tests.
In the two-way ANOVA approach, firstly the last third research question (RQ3) about the interaction of two independent variables is examined and if the interaction is not proven, then it is possible to continue with the testing of the main effects of these two independent variables -the effect of teaching form (RQ1) and the effect of approaches to learning (RQ2).Let us now look at the two-way repeated measures ANOVA of the tests results (Table 2) for distribution of characteristic groups on the two clusters (Cluster1, Cluster2) and finer division into three clusters (Cluster1, Cluster2.1,Cluster2.2).Table 2 shows the p-values for the main items of didactic tests.As already mentioned, if the p-value is less than the selected level of significance α, then the effect of variable(s) is confirmed (as marked in the Table 2), otherwise it is not confirmed.The report was obtained again from NCSS statistical software4 .At the significance level of 0.05, the interaction between independent variables (p-value 0.858300 or 0.795602 in AB line) has not been proven with the TOTAL results.Equally no differences were found between TOTAL results of the characteristic groups of students.Although, when taking into consideration the finer division of characteristic groups (three clusters), the influence of belonging to this characteristic groups on the TOTAL test results has been disapproved but by a slight margin (p-value 0.098016).On the other hand, the influence of the teaching forms has been shown as statistically significant in TOTAL results.Additionally, subsequently performed paired test confirmed these statistically significant differences for teaching forms in favour of the experimental group.When looking on the subscales (Knowledge in thematic areas, New theorems, Inequalities, Limits and simple Sequences and series), more diverse results can be found.In the Knowledge subscale the results were the same as above, however in the New theorems subscale the influence of the belonging to characteristic group on the results of didactic tests has been proven (p-value 0.039644 for two clusters and 0.001688 for three clusters division).It is obvious that when talking about the TOTAL results, the students' approaches to learning are not showing themselves strongly.However, in regards to understanding of new mathematical theorems, the students from the Cluster2 are doing much better than from Cluster1.An interesting fact about the subscale labelled Inequalities is that if the students are divided into three clusters, there is not difference on the 5% significance level between experimental and traditional teaching form.Similarly, in the Limit subscale there has not been proven influence of teaching form on any of the students cluster divisions.Conversely, when taking into consideration the finer division (three clusters), the belonging to a characteristic group has expressed itself on the students' results in the Limit subscale (p-value 0.037685).Finally, among the simple Sequences and series the influence of the forms of teaching has been proven only with the two cluster division (p-value 0.024607  Let us now summarize the obtained results.It should be noted that the students' approaches to learning and studying do not have much influence on the final results and the knowledge the students' gained in a given semester.On the other hand, when talking about the understanding/application of new mathematical theorems, students interested in a deep understanding of the curriculum have better results.Further, the results of the students after completing the classes taught with the aid of CAA were significantly higher than the result of the students after the classes taught using the traditional approach.Finally, the efficiency of the forms of teaching was not different for students with different approaches to learning (no interaction was shown).The better results of experimental group were seen in all characteristic groups of students.

Conclusion
This paper informs about the research focused on the utilization of CAA when teaching Mathematical analysis (Calculus) at the University of Hradec Kralove (Czech Republic).The undergraduate students of teaching mathematics have undertaken a pedagogical experiment.Repeated measurements were carried out in two successive semesters to minimize the influence of the natural development of students as much as possible.Due Printed ISSN: 2336-2375 to this repeated measures approach contrariwise, the problem with imbalance of parallel groups was eliminated in contrast with other studies.In order to take into account the students' approaches to learning and studying, the students were divided into characteristic groups and when evaluating the results not only the teaching forms but also the characteristic groups of students were taken into consideration.
In response to previous studies mentioned in the introduction, which, however, differ in their conclusions and in which students' approaches to learning are not discussed, the positive effect of the use of CAA to student´s performance was clearly shown in this study.This positive effect was seen in all characteristic groups of students.Further, approaches to learning and studying do not affect student´s overall results but affect their performance in some partial tasks and mainly their way of practicing in the CAA system (students with deep approach to learning need fewer attempts to reach the required score in homework than students with surface approach).
Even though the research is limited by the small number of participating students and the fact that both traditional and experimental teaching forms were conducted by the author of the research, we believe that the conclusion of this paper can help lecturers when considering adding CAA system into teaching mathematics.The use of systems CAA should be taken into account also in connection with the downward trend in the field of mathematical knowledge of the first year undergraduate Czech students (as discussed in Kourilova and Bebcakova, 2015).Similarly, CAA system can be useful also in connected STEM subjects such as physics (Nemec, Berkova and Sramek, 2016a;2016b).Related upcoming research as part of Ph.D. theses of the author is focused mainly on the opinions and experience of Czech students/teachers of mathematics with the CAA platform.
The second part is dedicated to the students' approaches to learning and studying.It has 17 questions.The final, third, part focused on students' preferences for different types of course and teaching has 8 questions.The second and the third part contain the added questions (6 questions) focused on the using of information and communication technology (ICT) or traditional techniques in education.Students answered by checking the answers on a scale 1-5 in all of the items.The Q1 questionnaire survey (Cronbach's Alpha 0.797966) was conducted in fall 2013/2014 in the subject Mathematical analysis 1.The results of the survey were evaluated using cluster analysis (division of students into characteristic groups).For the detection of the efficiency of the CAA platform the experiment with repeated measurements has been selected, since the study group of 22 students was too small to conduct a classic experiment utilizing the parallel groups' technique.During the first semester of the experiment, the subject Mathematical analysis 2 (spring 2013/2014) was taught using the traditional teaching form utilized classical, in regards to homework mainly paper and pencil aids.The following subject Mathematical analysis 3 (fall 2014/2015) was taught with the aid of the new CAA mathematical platform Maple T. A. (experimental teaching).Lessons of the semester were always divided into four thematic areas.The students were in the end of both semesters given objectively scored cognitive didactics tests (T1 and T2) which were created in the Maple T. A. system.The T1 (Cronbach's Alpha 0.825213) and T2 (Cronbach's Alpha 0.853111) tests assessed the level of the students' knowledge in given thematic areas (8 questions) and mathematical apparatus and insight into the studied problems (8 questions).The items called mathematical apparatus and insight into the studied problems are focused on key knowledge from the students' previous studies which the students are not in proper semester primarily studying, but are essential for

Figure 1 :
Figure 1: Scales and items of used research instruments

Figure 2 :
Figure 2: Graphical interpretation of cluster analysisBecause Student 16 from Cluster3 has dropped out after the spring semester 2013/2014, only two clusters remained for further evaluations.Because of this, it was decided to divide the students again using cluster analysis into four clusters (see the part with Four Clusters in Table1).The results were interesting.Student 16 formed again his own single member cluster which was for the sake of clarity named again Cluster3.Students from the first cluster again formed Cluster1.Cluster2 has split into two groups (for the sake of clarity named Cluster2.1 and Cluster2.2).Cluster2.2consists of 7 students, Cluster2.1 then consists of the 3 remaining students.Because Cluster2.1 only has 3 students, it has been decided to use the previous cluster division for further analysis, taking into account this finer division if it was necessary.As it can be seen in Fig.2bthe students in Cluster2.2are more pronounced in regards to their Approaches to learning.Cluster2.1 is strongly motivated by career and furthermore slightly higher scores can be seen in the Transmitting information subscale.In regards to their Preferences for types of course and teaching, students from Cluster2.1 also prefer traditional, paper and pencil, educational methods over using information technologies.On the other hand, Cluster2.2does prefer information technologies.

Figure 3 :
Figure 3: Graphical report (NCSS) of mean for appropriate clusters in numbers of attempts in homework assignments

Table 1 : K-Means Cluster Analysis Report (NCSS)
with lack of motivation (highest score in the Lack of purpose item).An interesting fact is that Student 16 dropped out the university during the time of this research.Cluster1 is comprised of 11 students who have conclusively the highest score in the Career reasons in the Learning orientations section.In the area of the Approaches to learning and studying these students are ambivalent with the best score in the Organized effort subscale.Nonetheless, they are most clear in the area of Printed ISSN: 2336-2375