Grading System Documentation

Introduction:

Some schools right now, especially those with huge numbers of students who are not yet using computerized based grading system has difficulty/problems concerning the records of grades of the students. The grades are the important record to keep even for the longest time for the referral and credentials of the student to enter their next level of attaining their goals. It is the tract record that recognizes of one student, this maybe use for analyzing of your attitude and values. Manual computation is very prompt to risk for any circumstances. It is time consuming in terms of recording grades, computation using of calculator. If some records are lost, they never retrieve it in case of unexpected calamity. Accuracy and security is not been so defined.

Grading System is the most commonly used in computing and analyzing the performance, talent and skills of students. It is designed to provide incentive reward for achievement and assist in identifying problems of the student.

The Computerized Grading System will help both the faculty staffs and the students to have easy access on the records and past records, the easier way. The said system will also lessen the staffs on consuming too much time recording and effort locating records of the students. It will also be connected to the schools’ website (if there is), for the students, so they can check their grades online. The staff in charge will do updating of grades.

The Computerized Grading System will also be a convenience for the professors as well, because by this, delayed passing of grade will be prevented and class cards, certificate of grades, and transcript of records will be processed in a speedy way. By this, students would not wait for a long period just to get their papers processed. In case that a student record is lost, we can easily retrieve it with the help of modern technology. There would be no alternation of grades once the records are inputted and finalized, so the records are secure.

This system will served as a tool for easier accessibility of grades, both for the convenience of the students and the faculty staffs, as well as the professors.

Overview of Automated Grading Systems
Anonymous

If bridges and buildings were made like we make software, then we would have disasters happening daily. I have heard this several times from many people. It is sad but true. Buggy software is the bane of the software industry. One of the ways of increasing software quality is by proper education. Several professionals from the software industry also attest to this. They believe that a greater emphasis should be given to quality and testing in university courses. But simply explaining the principles of software quality is not sufficient. Students tend to forget theoretical principles over time. Practical exposure and experience is equally important. Students should be put in an environment where they can appreciate the importance of quality software and can experience the benefits of processes that enhance quality.

Many universities have a period of internship for the students in which they work in a software company and experience these factors first hand. However because the internship usually is of a duration of 3-6 months, it is not sufficient to instill the importance of quality. Emphasis on code quality should be made a part of the entire software curriculum for it to have proper impact. Every assignment that the students submit should be subjected to the same quality standards that an industrial project would be subjected to.

Having university assignments adhere to industrial standards will result in the faculty having to spend more time grading the assignments. The faculty can no longer just give an assignment, wait for the students to submit it, and grade them. The faculty must be more like a project manager who constantly mentors the students and helps them improve the quality of their work. Along with spending a good amount of time mentoring students off class hours another challenge is timely evaluation of student assignments. Faculty members are already overloaded with the task of teaching, designing projects, grading, and research. Once we incorporate testing and quality into the curricula, each assignment will have to be graded along many more dimensions, such as quality of the tests, coverage of the tests, etc. This can be very time consuming. We need a mechanism which will automatically grade student assignments to the best possible extent, so that students are given a timely feedback, and faculty can focus more on providing feedback on the style, design, and documentation of the project. Such a system will also bring consistency to the grading process and will eliminate discrepancies due to instructors bias and lethargy.

A good automated grading system should be capable of executing the test cases written by students as well as the faculty on the project, determining the coverage of the test cases, and compiling and executing the submitted programs. It should be configurable so that faculty can determine the importance of various factors that make up the final grade.

Several efforts have been made to design and implement automated grading systems in universities. Some existing systems are: 1. WEB-CAT
4. Praktomat

In this article, I will briefly explain two such automated grading systems – WEB-CAT, and the Praktomat systems, and propose a system that contains useful features from them as well as some new features.

WEB-CAT

WEB-CAT was created at Virginia Tech University to address the need for incorporating software testing as an integral part of all programming courses. The creators realized the need for software to automatically grade student assignments to enable faster feedback to students and to balance the working load of faculty members.

Since Test Driven Development (TDD) was to be used for all the assignments, the students had to be graded not only on the quality of code, but also on the quality of their test suite. WEB-CAT grades students on three criteria. It gives each assignment a test validity score, a test correctness score, and a code correctness score. Test validity measures the accuracy of the students’ tests. It determines if the tests are consistent with the problem statement. Test coverage determines how much of the source code the tests cover. It determines if all paths and conditionals are adequately covered. Code correctness measures correctness of the actual code. All three criteria are given a certain weight-age and a final score is determined.

WEB-CAT’s graphical user interface is inspired by the unit testing tool JUnit. Just like JUnit it uses a green bar to show the test results. A text description containing details such as the number of tests that were run, and the number that passed is also provided.

Basic features provided by WEB-CAT are:
Submission of student assignments using a web based wizard interface
Submission of test cases using a web based wizard interface
Setup of assignments by faculty
Download of student scores by the faculty

Automatic grading with immediate feedback for student assignment

WEB-CAT follows a certain sequence of steps to assess a project submission. A submission is assessed only if it compiles successfully. If compilation fails, then a summary of errors is displayed to the user. If the program is compiled successfully then WEB-CAT will assess the project on various parameters. It first tests the correctness of the program by running the student’s tests against the program. Since these tests are submitted by the students, and it is expected that 100% of the tests will pass, because we do not expect students to submit a program that fails their own tests. After this the student’s test cases are validated by running them against a reference implementation of the project created by the instructor.

If a student’s test case fails on the reference implementation then it is deemed to be invalid. Finally, the coverage of the student’s test cases is evaluated. Once the scores are obtained a cumulative score out of 100 is calculated applying a certain formula on the scores from all criteria. The results are displayed immediately to the student on an HTML interface. It was observed that the quality of student assignments increased significantly after using WEB-CAT. It was found that the code developed using WEB-CAT contained 45% fewer defects per 1000 (non commented) lines of code.

Praktomat

Praktomat was created at Universitat Passau in Germany. The purpose of creating Praktomat was to build an environment which would help students enhance the quality of their code. Along with automated grading it also has a focus on peer reviews. The creators of Praktomat felt that reviewing others software and having one’s software reviewed helps in producing better code. This is the reason why Praktomat has a strong focus on peer review and allows users to review as well as annotate code written by other students. Students can resubmit their code any number of times until the deadline. This way they can improve their code by adopting things they learned by reviewing other students code as well as lessons they learned by others feedback of their own code.

Praktomat evaluates student assignments by running them against a test suite provided by the faculty. The faculty creates two test suites – a public suite and a secret suite. The public suite is distributed to the students to help them validate their project. The secret test suite is not made available to the students, but they are aware of its existence. An assignment is evaluated by automatically running both the test suites against it, and also by manual examination by the faculty. Praktomat was developed in Python, and is hosted on SourceForge.

Observations

My contention that student project submissions should be backed by a process to encourage best practices, and a software to automate as well as facilitate the process, has become stronger after reviewing WEB-CAT and Praktomat.

What best practices should we incorporate in the process? What are the features that an automated grading software should contain? WEB-CAT, Praktomat, and several other software give a good starting point. We can learn from their successes and failures, and enhance the offering by adding our own experience.

WEB-CAT and several other sources have shown us that TDD is definitely a good practice. In a university environment TDD will work best if it is complemented by instant feedback to the students. We want to have a process that will encourage students to improve the quality of their code. They should be graded on the best code they can submit till the deadline. Two things are needed for this – instant feedback and the ability to resubmit assignments.

WEB-CAT achieves this by assessing submissions in real time, and displaying the results to the students immediately. WEB-CAT allows students to re-submit assignments any number of time till the due date. Since faculty members are already overloaded with work, the software should take some of the faculties responsibilities. WEB-CAT automatically evaluates and grades the student’s assignments, leaving faculty with time for more meaningful activities.

Praktomat has shown us that there is a definite benefit to peer review. When we review code written by others, we can go beyond the paradigms set in our own mind. Having our code reviewed by others can help us see our shortcomings which we may have earlier overlooked. Praktomat allows students to review code written by others. However the review is hidden from the faculty, to ensure that it does not impact grading.

Praktomat does not rely on 100% automatic evaluation of the assignments. Praktomat evaluates certain aspects automatically and the rest are evaluated manually. Factors like code quality, documentation, etc are reviewed and evaluated manually by the faculty. There may be two reasons for this. Software to support automatic evaluation of these things may not have been available when Praktomat was written, or the creators felt that certain things are best evaluated by the faculty.

A proposed system for automated grading

Based on my observations from reviewing the above software systems and from my own experience, I have defined a process and the functional expectations of a software system that supports TDD and automated grading.

The Process

Every project should have a deadline, just like the real world The project should be defined as a set of use cases and a functional test suite. Both should be made available to the students. Students should start developing their project using the TDD philosophy. They should also be provided a source code repository like CVS or VSS. Once the students have completed their project they should tag the build and should upload the tag number to a web based submission software. It must be clearly defined how the students should submit their unit test suite. They should also provide one file which will trigger the remaining unit tests. The software will pull the source from the repository, and evaluate it.

Failure is reported to the student if the project fails to compile. Failure here does not mean that the student fails in the assignment. Assignments can be corrected and submitted any number of time till the deadline. Once the compilation succeeds, the software will run the unit tests written by the student on their code. After collecting results from the unit tests, the test coverage is measured. Then the functional tests created by the faculty are executed against the software. The software is then run through a source code format checker which evaluates it for adherence to coding standards. The software is then run through a source code quality checker which evaluates the quality of code based on known best practices, and anti patterns. The software is finally channeled to the faculty who evaluates it for design. Results from all the tests are given out of 100%.

After collecting all the results a formula (provided by the faculty) is applied to derive the final score.

The Software

The software should provide an account with a username and password to each student and faculty. The software should be web based so that it can be accessed from anywhere using a standard web browser. After logging in students should be able to browse to the homepage for a particular assignment and view the details, such as specification, due dates, and any other details posted by the faculty. When a student completes her assignment, she should be able to upload the CVS tag number to the server. Once the tag number is uploaded the server should pull the source code from a CVS repository and perform the checks mentioned above. Results from each check is recorded in the database.

The detailed result is then displayed to the student.
Students should be able to resubmit an assignment any number of times till the deadline. Student code should be available for peer review and annotations if the faculty desires. The faculty should be able to create an assignment and upload details and files. The faculty should be able to trigger the final evaluation of all assignments either manually, or at a scheduled time.

An evaluation should take the latest tag numbers provided by the student and perform tests on the respective source code. Results should be made available to the faculty, and students. The faculty should be able to add their own scores for parts that were checked manually. The final result is computed by applying a formula provided by the faculty. The final results should be downloadable as a csv text file.

Several technologies such as Java, Python, PHP, .NET, and Ruby can be used to implement such a system. Each has their pros and cons. We will not cover the implementation technology in this paper. Evaluation of these technologies and a final choice based on the evaluation will be dealt with in a separate paper.

Objectives of the Study
The general objective of the study entitled “A Computerized Grading System” is to develop a system that will replace the current Grading System in the state, which is manual. It can also eliminate lag time between the submissions of grades and will help to ease jobs of professors and that student can rely on having an accurate grade.

Statement of the Problem
The study entitled “A Computerized Grading System” is focused on making the old manual way be an improved one by using a computerized grading system that would make it easier to file records.

The study is sought to answer the following questions:
1. The problems encountered in the manual way of grading system which are the following: Some records are lost.
Incorrect inputting of grades.
As to security, easy alteration of grades happens.
Once they received their class cards, they only see their final grade.

2. The following are solutions recommended to address the problems identified above are the following: The manual grading system will be a computerized one.
Once it is computerized, the grades inputted on the said system will be automatically saved on a database. Once the grades are put in the said system, it cannot be edited anymore. Students will know their grades from prelim to finals.

Significance of the Study
The study of “A Computerized Grading System” will be beneficial to the following:

To the School. This study will give benefits to the school because an automated grading system is feasible, economically viable and as accurate as manually grading system. To the Students. It will help the students to get their grades processed fast and on time. Through this grading system, they could also help the students see the connection between effort and grades that could helps them become more accountable for their progress.

To the Instructors. This will serve as their guide for an easy access on the students’ grades because grading is an exercise in professional judgment on the part of teachers.
To the Researcher. This study will equip them with knowledge about the grading system.
To the Future Researchers. It will serve as their reference for their research activities and will help them to gain knowledge to express their ideas in their chosen study.

Scope and Limitation
This study is focused and will only concentrate on the grading system of the school because it is the important record to keep even for the longest time for the referral and credentials of the student to enter their next level of attaining their goals. It is the tract record that recognizes of one student, this maybe use for analyzing of your attitude and values.

The said system is only for recording and keeping of grades. It is not for enrolment and registration use.

Definition of Terms
The following are terms that the researcher used for better understanding about the concepts in the study. The terms are defined operationally, as how they are used in the study.

Computer. In the study, it is the machine used by the researcher in making
the documentation of the study and the developing the computerized system.

Programmer. As used in the study, this person creates and designs the system.

Database. Is an organized collection of data for one or more purposes and usually in digital form.

Grades. To allow the student’s have an access to their over all grades taken from their prelim, midterm, pre-finals and finals.

Chapter II
METHODS OF RESEARCH AND PROCEDURES

Research Design
The method of research we used in the study “A Proposed Computerized Grading System” is the descriptive type of method research, because this kind of research describes the present system, which is the manual way of grading system based on the impressions of the respondents. We used this kind of research because it is appropriate in knowing the extent to which the different conditions of the present set up of the study is. Methods of Research

The descriptive survey method was used to achieve the purpose of the study. We made a survey regarding about the old system of recording grades, which is the manual way. By that, we had come up with the idea that the manual way should really be change into an automated one. Based on our research, Southdale International School of Science, Arts and Technology is using a manual computation of grades of each student, which is the traditional use when technology is not yet developed.

Since we are used of using the manual way of computing grades, why not, we try something new and easier to make the job done fast.

This was deemed appropriate because the study sought to identify all things happen at the present flow of grading system.

Review of Related Literature and Studies
This part discusses the review of related literature and studies of the study entitled “A Proposed Computerized System”.

Foreign Literature and Study

The Trial at Curtin University of Technology

“During the first semester of 2001 a trial of an automated essay grading system was conducted at Curtin University of Technology in Perth, Western Australia. One subject was chosen, a first year introduction to Information Systems, where we had about 1,000 students available to participate.

Unfortunately the semester had already started by the time we were able to undertake this research. This meant that all assessment had already been determined.

Once assessment has been published, the policy at Curtin University is that it cannot be changed without the consent of the majority of students. In order to gain that consent and ensure a high rate of response to our trial, we announced that an additional voluntary essay-type question would be available for bonus marks. Needless to say, we, had a high rate of response.

The system we were trialling was an American system that required two hundred manually graded essays as input to their grading system. Between the three researchers, we graded about 70 papers each and sent the electronic copies along with the marks to the US site. About another 330 ungraded essays were then forwarded to the site for grading.

A number of interesting outcomes were noticed when we analyzed all the grades and as a result, a grader “A” had always considered himself a “hard” grader and considered grader “B” rather soft.

However, the purpose was not to check our own grading but to see how consistent the computer system handled the assessment. We were delighted to have our suspicions confirmed; the computer system had the same mean and standard deviation of marks as the three of us. We were satisfied that it worked.

There was an additional and quite unexpected result from the test. The system picked up several cases of plagiarism that we had failed to notice. In this case, the plagiarism was really that of one student copying the work of another student rather than from extracting text from another source.

The Weaknesses of the System

There are two important weaknesses and one minor weakness for our purposes in the system that we trialled. The first weakness is that for a successful implementation, one need to manually grade 200 essays and feed them into the system. The computer will then accurately and dependably grade as many more essays on that topic as is required. In small classes of less than a few hundred students, it becomes impractical.

The second weakness is the cost of using the system. As the system was American we had to pay in $US. With the exchange rate so poor, it cost about A$11,400 to grade a few hundred essays. This is simply not cost effective. If we were to use the same essay for several semesters then the per-unit grading costs would reduce substantially.

However, it is highly unlikely that we would want to use the same essay questions in consecutive semesters or even twice ever.

There is a third factor. The system is run at a site in the USA rather than on our own computer network at Curtin University. There is some lack of control and potential security risk in having the process run remotely.

Costing Considerations

Ideally, the system would be reasonably inexpensive, and certainly far cheaper than hiring grading staff. The grading system would be based on a single all-inclusive model answer supplied by the lecturer. Obviously, the system would need to assess with the same degree of accuracy as a manual grader. Finally, the system should be available to be run in-house on a PC or central server.

We currently pay exam graders at a rate of about A$25 per hour. It should be possible to reduce the cost of grading through an automated essay grading system by 90%. Our single experience with the American system as described above was that it cost about A$33 per essay of up to two pages in length.

Based on supplying 200 graded essays at a cost of A$3 per essay, the initial cost before paying for the grading service would be A$600. The grading service costs that we experienced were another A$10,800, bringing the total to A$11,400. In the ideal case, it would be beneficial for a University to own the grading system so the costs could be spread across many subjects and many departments. Even if the initial cost were in the thousands of dollars, the cost per essay or exam would become trivial.

There are economies of scale associated with the system, in that up to 2000 essays could have been graded for the A$11,400, but we did not have this number to grade, and so did not gain these benefits. If we had 2,000 essays to grade, the automated essay grading system would still have cost A$5.70 per essay, almost double the cost of grading manually.

Limitations to any Automated Grading System

To utilize any Automated Grading System the raw data, essays or examination answers, would need to be in a form that was computer readable. The most obvious form of this would be electronic documents in Word format. This is easily enough achieved where the student could write the essay on a computer.

However, when students sit for examinations this is normally done at desks with paper and pen. The resulting examination script is not easily transferred to a computer readable medium. On the other hand, we see that it is possible to have students sit an exam in a computer laboratory and submit their examination papers electronically. It would be difficult to have large numbers sit the exam simultaneously but it is not impractical to have two groups of students where as soon as the first group completes, the second group starts. In this way, with lab facilities of 200 PC’s the same examination could be sat by up to 400 students without compromising the examination paper.

Another possibility would be to give the students a take-home examination due within 24 hours. Any number of students would then be able to sit the exam at the same time and submit the exam papers electronically.

Another serious limitation to an essay grading system is that it grades a students’ knowledge of a given set of material. The model answer would contain only a set body of knowledge and would grade the student on the part of that knowledge the student was able to demonstrate. This may be acceptable in the early years of a course but probably not in more advanced studies.”

Local Literature and Studies

Automation of Grading System
Posted by Fr. Jerome Marquez, SVD
“The plan to automate the 2010 national election reverberates clearer to school faculty and staff as SJCS begins to automate its grading system starting this school year 2009-2010. This first quarter, faculty members of Preschool and Grade School Departments have tested the customized grading system in encoding the scholastic grades of their students. The Registrar’s Office, in coordination with the Information Technology Office, printed and released recently the first computerized report cards of the school. The shift to automate the grading system, hopefully, will increase efficiency of school staff and allow teachers to focus more on teaching than clerical recording. Other information and services may also be derived from this new system. High School grading system will later follow.”

Automation of Grading System in San Sebastian College
“Automation has had a notable impact in a wide range of industries beyond manufacturing (where it began). Largely automated telephone switchboards and answering machines have replaced once-ubiquitous telephone operators. Medical processes such as primary screening in electrocardiography or radiography and laboratory analysis of human genes, sera, cells, and tissues are carried out at much greater speed and accuracy by automated systems. Applying automation to Grading systems wherein it will also make a task easy and accurate.

Since San Sebastian College Recoletos began over 4 years ago, many things in this school have gone from manual to automate. In addition, because a typical problem in a starting school are Grading systems, I noticed that this system does not exist in this school. We all know that making an Automated Grading System means making the task for professors in computing and calculating grades will become easy and not only that the professors will have an advantage but the students will also get a gain, because it will also improve accuracy of calculations thus making the what we call “Hula of Grades” will become non – existent in the future of San Sebastian Recoletos. San Sebastian College Recoletos – Canlubang lacks a centralized grading system. It does not have a standalone program that will accept input, compute grades and print class cards all in a same program.”

Respondents of the Study
The respondents of the study will consist of a) Southdale International School students (130), b) faculty and staff (20).

Data Gathering Procedures
First, we discussed about the current grading system Southdale International School was using and we found out that they are still using the old manual way of computing and recording of grades which is really time consuming especially when large numbers of assignments are submitted at once.

Teachers of the said institution find themselves bogged down in their attempt to provide consistent evaluations and high quality feedback to students within as short a timeframe as is reasonable, usually a matter of days rather than weeks.

Educational administrators are also concerned with quality and timely feedback, but in addition must manage the cost of doing this work.

Clearly, an automated system would be a highly desirable addition to the educational tool-kit, particularly if it can provide less costly and more effective outcome.
After seeking the approval, we, the researchers, prepared the questionnaires, for our survey and we have them checked and approved by our researcher adviser Mr. Carlo C. Clerigo and have those surveys answered by the respondents, which are the students and the faculty and staffs of the said institution.

We tally the responses according to the questions in the statement of the problem. Then, the responses were put into tables and graphs, and interpret and analyze them.

Data Gathering Instruments
This study, which is “A Proposed Computerized Grading System” make use of the Questionnaire as its main instrument in gathering data. It also makes use of interview to support the data and information gathered from the questionnaires and it is supplemented by observations and documentary analysis.

Questionnaire. The researchers made use of this instrument to get a feedback to the respondents. Observation. The researchers made use of this to differentiate the manual and computerized way of grading system. Internet Research. Used for searching additional information.

The respondents of the survey we have conducted are the students enrolled in the institution, the faculty members and staff, and the “Others” consists of some people from outside the school. Questionnaires were given to the respondents to be answered accordingly, to be able to achieve the necessary information needed.

Figure 1.2: Tally of Survey Question for Automated Grading System

It shows the assessment of the respondents regarding the survey we made about the manual grading system. Looking at the table above, majority of the respondents are in favor of having an automated/computerized grading system in the institution.

Chapter IV
SUMMARY, CONCLUSIONS AND RECOMMENDATIONS

Summary
Based on the study that the researchers made, several problems can be encountered when using the manual way of recording and computing of grades. One example of that problem are unretrieve files or records of the students.

So, the researchers had determined first the viability of the software by conducting a survey in a form of questionnaires. After analyzing it, the researchers found that majority of the respondents, which are the enrolled students, the faculty and staffs and some people whom answered the said questionnaires is in favor in an automated grading system. Through descriptive method, which describes data and characteristics about the present condition of the system being studied, we have come up with the idea that applying automation to Grading systems will also make a task easy and accurate

Conclusion
Based on the findings, the researcher came up with a conclusion that a computerized/automated grading system is a must-have in a school. It is a best way of storing and computing grades of students that provides easy way in getting the grades and to make the procedure faster and efficient.

Recommendation
Based from the conclusion made, the researcher highly recommends that the school must have an automated/computerized grading system because computerization is focused on retrieving and storing information of students in a faster, more convenient way of storing files of the student enrollees in a computerized system that will lessen the effort of faculty staff in storing files of each the students. This will also serve as information for irregular students, transferees, and freshmen to get course and subject schedule. The information can be viewed in just a second without worrying that a single file is lost.