A new Moodle module supporting automatic verification of VHDL-based assignments.pdf

A new Moodle module supporting automatic verification
of VHDL-based assignments
Eladio Gutiérrez *, María A. Trenas, Julián Ramos, Francisco Corbera, Sergio Romero
Department of Computer Architecture, University of Málaga, 29071 Málaga, Spain
a r t i c l e i n f o
Article history:
Received 5 May 2009
Received in revised form 31 August 2009
Accepted 3 September 2009
Keywords:
Automatic assessment
Applications in subject areas
Distance education and telelearning
Learning management systems
Simulations
a b s t r a c t
This work describes a new Moodle module developed to give support to the practical content of a basic
computer organization course. This module goes beyond the mere hosting of resources and assignments.
It makes use of an automatic checking and verification engine that works on the VHDL designs submitted
by the students. The module automatically keeps up to date information about their state, and signifi-
cantly reduces the overload that a continuous assessment demands to the teacher. Additionally, this
new module is oriented to promote a collaborative teamwork allowing to define student teams in a more
operative way than built-in Moodle groups. The module has been designed according to the Moodle phi-
losophy and its application can be extended to other similar subjects.
Ó 2009 Elsevier Ltd. All rights reserved.
1. Introduction
At the present time, university education is strongly influenced by new technologies. The impact of learning management systems
(LMS) is particularly significant for science and engineering studies where they are but a particular case of the set of software tools applied
in these areas, which emerge from the field of Computer Science. But LMS systems are more than just a tool. Their use has a significant
influence on various aspects of the teaching/learning process. Their use encourages a more interactive learning and increases the possibil-
ities of virtual teaching. They make easier to implement pedagogical approaches of constructivist nature, that are characterized by the con-
struction of knowledge through the completion of different activities (Jonassen, 1999; Khalifa & Lam, 2002; Knuth & Cunnigham, 1993).
This encourages self-learning with an special emphasis in real world problem solving. And this is essential as refers to engineering studies,
as much of the work of an engineer will be to provide solutions in order to solve problems effectively. Collaborative learning (Tan, Lin, Yang,
& Zhao, 2008) is a perfect complement to constructivist teaching. It will encourage teamwork, another important skill in real world, but
also important as students will help each other, thus strengthening the learning process.
In general, LMSs provide web-based interfaces that support a wide range of activities. These include forums, content resources, ques-
tionnaires, chats, assignment and so on, which are, in general, sufficient for setting up standard courses. LMS may also integrate other tools
of great interest when teaching an engineering course, as it is the case of the automatic checking and verification of the student’s lab work.
This is a very time consuming task if manually done by the teacher (Rice, Beg, Waters, Bokreta, & Santiago-Avilés, 1999), becoming an
important overload. Its automation would greatly benefit the constructivist approach. But new technologies also bring negative effects,
such as the increase of the chances of plagiarism among students (Lathrop & Foss, 2000).
Focusing on Computer Science and Engineering studies, and related to the teaching of both high and low-level programming, a number
of approaches have been developed whose characteristics include automatic assessment (Bennedsen, Caspersen, & Kölling, 2008; Douce,
Livingstone, & Orwell, 2005; Jackson & Usher, 1997; Malmi, Korhonen, & Saikkonen, 2002) and the detection or avoidance of plagiarism
(Butakov & Scherbinin, 2009; Rosales et al., 2008). For example, a framework to teaching low-level input/output in mass courses by means
of a specific platform can be found (Rodríguez, Zamorano, Rosales, Dopico, & Pedraza, 2007). However this is not so extended in the area of
Computer Architecture, where the teaching tools mainly concentrate on the simulation, at the organizational level, of computer architec-
tures and their assembler language (Djordjevic, Nikolic, & Milenkovic, 2005; Moreno, González, Castilla, González, & Sigut, 2007), with few
developments on the simulation of the digital circuits at the low level (Kurmas, 2008; Poplawski & Kurmas, 2008). This is the case when
0360-1315/$ - see front matter Ó 2009 Elsevier Ltd. All rights reserved.
doi:10.1016/j.compedu.2009.09.006
* Corresponding author. Tel.: +34 952 13 2821; fax: +34 952 13 2790.
E-mail addresses: eladio@uma.es (E. Gutiérrez), matrenas@uma.es (M.A. Trenas), julian@uma.es (J. Ramos), corbera@uma.es (F. Corbera), sromero@uma.es (S. Romero).
Computers & Education 54 (2010) 562–577
Contents lists available at ScienceDirect
Computers & Education
journal homepage: www.elsevier.com/locate/compedu

teaching a basic course on Computer Architecture, an essential topic in the Computer Science & Engineering curricula, which generally in-
volves practical assignments related to the design and simulation of digital circuits and elementary processors. Such practical component is
usually performed by means of real CAD tools making use of schematic or VHDL (Very high speed integrated circuit Hardware Description
Language) design entries (Calazans & Moraes, 2001).
With the purpose of providing a better support for such a kind of practicals we have developed a specific Moodle module that provides
automatic assessment features, allowing an advanced management of assignments and submissions together with a enhanced feedback
information mechanisms for both teachers and students. All of this based on the automatic checking and verification of the student’s
lab work. Also it includes some organizational aspects oriented to promote student teamwork.
Motivations for choosing Moodle (Modular Object-Oriented Dynamic Learning Environment) include its modular organization based on
free software whose sources are freely available (Main Moodle site: http://moodle.org). This enables the developer to add new modules
using well-known interfaces. Additionally, Moodle has become a very popular platform with more than 50,000 registered sites at the begin-
ning of 2009, and at present it is widely used in higher and university education (Martı
´ n-Blas & Serrano-Fernández, 2009). Besides, it is the
platform currently used by the Virtual Campus of our university.
This paper is organized as follows. Next section introduces the context of the course the new module was developed for. Then, general
Moodle development guidelines are presented, prior to a general overview of our module. Sections 5 and 6 focus on the user interactions
with the system for both students and teachers, respectively. Due to its relevance, next section is devoted to the implementation of the
automatic verification support, followed by sections presenting site configuration issues and other features. The paper ends with a section
discussing our experience and some final conclusions.
2. Context of the course
Computer Technology is a basic course on computer architecture delivered in the Computer Science academic plan of the University of
Málaga (Spain). It includes 3.6 practical European credits from a total of 7.2. The objective of the course is to introduce how the CPU of a
computer system works at the register transfer level, covering the gap between the machine level and the assembler language. To this end,
the laboratory assignments (Corbera et al., 2006) consist of the design and implementation of various selected elements of the central pro-
cessing unit and memory system, as well as the whole control unit (both hardwired and microprogrammed) for a simplified MIPS based
architecture (Patterson & Hennessy, 2008). The whole work is developed as a hierarchical schematic using Xilinx Foundation as CAD tool
(Xilinx: http://www.xilinx.com).
In order to better guide the students, teachers should perform a functional evaluation using a simulation tool to verify whether the stu-
dents designs work or not according to the specifications. This is a very time consuming task, specially when there is a high number of
enrolled students (Rice et al., 1999). Therefore it commonly produces bottleneck when looking for a continuous assessment of students
and even provide them feedback about their evaluation. And this is an issue to solve, if we are to implement the recently proposed Euro-
pean Credit Transfer System (ECTS).
The European Credit Transfer System is affecting all the higher education system with a special focus on the teaching quality (Clausen,
2005; Reichert & Tauch, 2003). Its successful implementation involves changing the teaching practice carried out traditionally. The goal is
to exchange the exclusiveness of conventional master classes for a more partaking model, where the student’s work plays a significant role.
For this reason, teachers are required to evaluate the student effort (amount of effective work) and not only the amount of knowledge.
Currently the University of Málaga is enrolled in a ECTS experience in order to address successfully the imminent implementation of
the ECTS system.
In this context, a specific LMS was built for the Computer Technology Laboratory that has been in use from the academic year 2004 to
2007 which has been a great help in order to achieve most of the goals pursued by the ECTS experience (Gutiérrez, Ramos, Romero, & Tre-
nas, 2007). This prototype was conceived to receive lab works that students must upload after their verification. Students know that the
system will check their work against a battery of tests unknown for them. The prototype provided support for several aspects concerning
the laboratory work: material repository, planning and deadlines, assessment assistance, working at home, improved feedback, students
workload estimation, etc.
Nevertheless such a prototype suffered from various drawbacks. From the developer viewpoint, the system had a monolithic design
based on a great number of cgi-style programs making it difficult to modify or improve. From the teacher point of view the system was
initially configured for a fixed number of projects making difficult to propose new projects or make a deeper verification. Teachers could
not modify, without the assistance of the system managers, neither the project definition nor the testing bench. From the student view-
point, the system was isolated from the LMS platform used by our University, based on Moodle. This fact limited the interaction between
students in the system because activities like forums and messages were outside the prototype.
All these reasons led us to design and develop a specific Moodle module supporting computer technology practicals, that is being used
since the academic year 2007/2008. It inherits all features provided by the prototype system, but porting them to the Moodle philosophy.
On integrating the activity in Moodle, teachers and students can benefit from all the advantages provided by this platform. Hereafter we
will refer to this module as CTPracticals, after ‘‘Computer Technology Practicals”.
3. Moodle development
Moodle is a course management system able of handling a large number of courses and users, as it occurs with a university center. It is
freely provided as Open Source software under the GNU Public License, and it can be installed on computers running PHP, with SQL data-
base support, as MySQL. A typical Moodle installation is made up of three elements: a directory for the PHP files constituting the source
code of the application, another one with files containing data about courses and users, and a database which defines the different objects
that integrate the system.
Moodle basic organizational unit is the course, which is acceded through a web page such as the one shown in Fig. 1. A course is orga-
nized into sections that may correspond to topics or weeks, appearing in the middle column of the page. In each section, it is possible to
E. Gutiérrez et al. / Computers & Education 54 (2010) 562–577 563

include different resources and activities. The last ones will be assigned as home or class works to be developed by the students. At both
sides of the page may appear another elements, the blocks, containing different shortcuts or control elements. Users are another essential
Moodle object. They can enrol in the different courses as administrators, teachers or students. Each role is defined by its capabilities in a cer-
tain context, that is, the set of privileges when performing certain actions.
Each activity corresponds with the so called module. Module installation is easy, as it is performed by the copy of the directory contain-
ing its PHP sources in the general installation modules directory. The only requirement is that the directory should be named the same as
the activity. Moodle automatically recognizes new modules incorporated to the directory. Moodle kernel provides several basic modules
such as forums, surveys, quizzes, web pages, links, assignment, wikis, etc. Blocks installation is quite similar, a copy of the directory with
the PHP sources into the blocks directory. There are, as well, several basic blocks in Moodle kernel: calendar, course or activities admin-
istration, recent activity, etc. Together with the source files, each module or block may define its particular objects that will be mapped on
to new tables in the Moodle database. There is a great number of available modules and blocks, developed by the Moodle community (Main
Moodle site: http://moodle.org). Its modular structure with well defined interfaces facilitates the creation of new activities (modules) and
blocks without need of modifying the kernel.
4. Overview of the module CTPracticals
CTPracticals is a new Moodle activity available for both teachers and students. It is fully integrated in the Moodle system together with
the other ones available in a standard installation (for further details see Section 8). It involves two different elements: the activity module
itself, and a control block (see Fig. 1). The module allows to define different activity instances that will appear in the course sections. The
control block, on a side panel, provides access to control actions which will differ for teachers and students.
Fig. 1. Activities and control block provided by the module CTpracticals (teacher view).
Fig. 2. Overview of CTPracticals.
564 E. Gutiérrez et al. / Computers & Education 54 (2010) 562–577

Each instance of the activity involves the fulfilment of a certain lab project by the students. It will comprise one or several practicals,
each one associated to the different parts the project can be decomposed into. As an example, the activity may consist of the design of a
processor’s datapath, a project that could be organized as several practicals developing its ALU, register file, memory, etc. Thus, each prac-
tical is an assignment requiring the submission of a zip file by one of the students of the team. This zip file contains the project source files,
that is, the VHDL descriptions of the digital circuits corresponding to the assignment. Generally, these VHDL sources are created by export-
ing the circuit schematics in the CAD design tool used in the laboratory (Xilinx: http://www.xilinx.com).
A big picture of the module CTpracticals is shown in Fig. 2. It displays the user interactions with the system, as well as its internal data
flow. A more detailed description of how both students and teachers interact with the system will be provided in Sections 5 and 6, respec-
tively. Teachers are responsible of course configuration, including setting up the practicals and their testers. Students, grouped into teams,
will submit their assignments. The core of the system is based on the automatic verification engine, which allows a quicker and more effi-
cient feedback to the students. Its work is double: it first performs a formal checking of the submitted practical, and afterwards, if this one
is passed, a functional verification based on the simulation of the delivered digital circuit design. This is performed instantly and automat-
ically after the submission. This alleviates teachers evaluation effort, providing them with a useful tool in order to better guide their stu-
dents. Moreover, with an improved feedback, students will be able of refining their designs.
Each practical file submitted by each team will be stored in a specific directory for the module CTPracticals, located in the area of the
data directory of Moodle corresponding to their course. In fact, there will be a directory named after the team identifier and the prac-
tical, as they act as a unique key for its access. Its path will be stored in the object Work, together with the file’s name, in order to im-
prove its localization. As the number of submissions of the file is not limited if the deadline has not been reached, each delivery replaced
the previous one.
The new activity, CTPracticals, introduces five new objects into Moodle whose associated attributes are shown in Table 1:
Practical: Associated to each CTPracticals activity there can be one o more practicals. Each practical constitutes an assignment defined
by its own specification. A CTPracticals activity can be seen as a project containing various related practicals.
Team: A team is a set of students that collaboratively submit their practicals. Each team is defined by one unique identifier inside the
course. Each student from a team can submit, modify or download the file of a given practical for such a team. All commu-
nications from the system or teachers are made at team level, including those coming from the automatic project verification
engine.
Work: For a certain practical a specific zip file must be submitted by each team. This is designated as a work. Several restrictions are
applied, including the name of the file to submit.
Tester: A tester describes how to validate a work. It is used by the verification engine in order to check if a submitted practical fulfils
the specifications.
Work state: This includes not only which is the status of the work, according to the automatic verification engine, but also the different
reports generated during the verification process.
Each of these objects, together with the settings of the module and the definition of its instances result on new database tables linked to
the two Moodle native tables: users and courses (see Fig. 3)
5. Student interaction
Students interact with the system according to the use cases diagram shown in Fig. 4a. In order to encourage collaborative work, all
interactions with the CTPracticals are managed at the team level. At the beginning of the semester, students will access the control block
in order to configure their teams, and then, they will be able to access the different instances of the activity in order to submit their prac-
ticals and consult their results.
5.1. Team creation
A student needs to be part of a team in order to use the CTPracticals activities in the course. Moodle natively includes support to the
creation of student groups manually by the teacher, and there is an extension to it that allows students to select their groups. But besides
being a tedious work when performed by the teacher, the activities are associated to the students, and not to the groups. For example, the
student is the entity that submits its work to the activity module. Our teams work in a different way. For example, it is the team the entity
that submits its work to the module.
Students set up their teams at the beginning of the course, participating in a negotiation process. If the student does not belong already
to any team, he can chose between the creation of a new team, or to request his participation in a preexisting team. When a new team is
created it is assigned with a unique identifier that follows the team name template defined for the course. The identifier is automatically
generated by the system applying a hash function. Once the team has been created, its members may receive requests from another class-
mates asking for their inclusion, and they will be able of choosing between their acceptation or rejection. After the deadline expiration or if
the maximum number of members has been reached, only teachers will have the privilege of its modification.
5.2. Practical submission and feedback
After clicking one of the CTPracticals instances, the student accesses to a screen such as the one in Fig. 5. There he can consult deadlines,
access to work specifications, submit his practicals, download previous submissions and read detailed reports about them. The teacher can
configure the extent of these reports, that include both the current state of the submission and the results obtained after the application of
the verification engine.

6. Teacher interaction
Teacher interaction is captured in Fig. 4b. The side control block is accessed in order to administrate the module CTPracticals. Clicking
over the different instances of the activities it is possible to manage the practicals, its testers and the different kinds of reports produced
after the students submissions.
6.1. Control block
The side control block, displayed in Fig. 6a, allows the teacher to: visualize the general state of the different submissions at a glance
(Fig. 6b), to set up his preferences for the module (Fig. 6c), to manage the student teams, access to several statistics related to the submis-
sions results (Fig. 6d), and to update the Moodle gradebook. Note that the options course at a glance and statistics are quite useful to track
the progress of students throughout the course.
At the beginning of the course, teachers should configure the module CTPracticals accessing the preference interface (Fig. 6c). There they
can define aspects such as how the teams will be identified or about their composition. The main settings are shown in Table 2. Teams may
be configured to restrict their maximum number of students or to define the deadline for a student to register as a member of the team. The
Table 1
Major attributes of basic objects in CTPracticals. Each object has an associate unique id and also a creation/modification time.
CTPracticals activity instance
Name Name of the instance
Course IDa
Course where this instance is defined
Description Description of the instance
Grade Maximum total grade associated to the instance
Practical
Name Name of this practical
CTPractical IDa
Instance which this practical belongs to
Description Description of the practical
File nameb
Filename associated to this practical (to be submitted by each team)
Public in course/site True if this practical is visible in the course/site (to be imported in other instance of CTPracticals)
Show results True if students can see the results of the automatic verification of their submissions on submitting
Date to show results Results can be delayed to a given date
Deadline After this deadline students are not allowed to submit files for this practical
Grade Partial grade of this practical (sum of all partial grades must match the maximum grade of this activity)
Team
Name Name of the team (generated from the team name template)
Course IDa
Course where this team belongs to
Student IDa
List of students members of this team
Work (submission)
Practical IDa
Practical which this work belongs to
Team IDa
Team who submitted this work
Locationc
Path where submitted file is stored
ZIP filed
Reference (filename) to the submitted file
Tester
Name Name of the tester
Practical IDa
Practical which this tester is associated to
Description Description of the tester
Active True if this tester is active (applicable)
Is default True if this one is the tester to be applied when the practical is submitted; Results are referred precisely to the default tester
Locationc
Path where tester’s files are stored
Simulation script fileb,d
Reference (filename) to the simulation script file
VHDL testbenchb,d
Reference (filename) to the VHDL testbench file
Result fileb,d
Reference (filename) to the result file
Auxiliary filesb,d
Reference (filename) to others auxiliary files (e.g. memory contents, . . .)
Required filesb
Expression for files required in the submitted zip archive
Manual True if this is an manual (not automatic) tester
Public in course/site True if this practical is visible in the course/site
Work state
Work IDa
Work (submission) which this workstate is associated to
Tester IDa
Tested used in this verification
Error code Format error code (numerical value)
Result Is the simulation of submission OK?
Log for teacher Verification log for the teacher
Log for student Verification log for the student
Locationc
Path where workstate files are stored
Diff output filed
File with differences
List output filed
Logical output values (waveform) in textual form
Short feedback Short feedback message for students about this verification
a
Foreign key.
b
To be parsed replacing wildcard with team name.
c
Unique path in Moodle course data directory based on its id and foreign keys.
d
Has associated files.

team name template parameter is a pseudoregular alphanumerical expression used to generate an unique team identifier. This identifier can
be subsequently used to name both the files the students will submit and the testers for the practical. It always contains the [id] wildcard,
assigning a numerical identifier for each team. Certain modifiers allow to specify its length and if it will be or not filled with zeros. Another
supported wildcard is the Moodle group name ([group]) that may also be included in the team name. It is quite useful if a certain course is
shared by several classes, each one using its own Moodle group in order to differentiate its students. For example, template CT[id03] will
produce team identifiers such as CT001, CT002, CT003, . . . Template CT[id02]G[group], applied for a student belonging to the Moodle
group A would produce an identifier such as CT12GA.
Moodle
file name
public in course/site
show results
date to show results
deadline
grade
practicals
id
course id
name
description
CTPracticals
id
teams
course id
student id list
id
moodle students
...
id
...
moodle courses
works
id
team id
practical id
location
zip file
id
description
practical id
active
is default
location
script file
testbench file
result file
auxiliary files
public in course/site
testers
id
workstates
tester id
error code
result
teacher log
student log
location
diff output
list output
short feedback
work id
CTPracticals settings
id
course id
team template
max. members
deadline
main moodle group
id
CTPracticals id
description
Fig. 3. Tables added by CTPracticals.
Fig. 4. CTPracticals cases of use: (a) For students. (b) For teachers.

6.2. Activity management
Through an activity instance the teacher accesses to a screen such as the one in Fig. 7a. Three tabs allow the teacher to manage the prac-
ticals, testers and submissions, respectively. By default the user will be in the practical management tab.
6.2.1. Practicals management tab
This tab (Fig. 7) shows the different practicals in this instance of the activity, and allows to manage aspects like their order of appearance
and their scope, that is, if they are public in this course or in the whole Moodle site. This feature facilitates resource sharing between teach-
ers or between different courses. In fact, there is an specific sub-tab to import practicals to this instance of the activity from another one.
Both edition and creation of a certain practical is performed by clicking on the practical name. Table 1 highlights the main parameters
that the teacher may define in order to configure a practical. Some of them are of organizational nature, such as the submission deadline, or
the score assigned to each practical of the activity. For each practical it is possible to define an expression forcing a name to the file the
Fig. 5. Main student view of a CTPracticals activity.
Fig. 6. (a) Teacher view of the side control block. (b) Course at a glance. (c) Course preferences. (d) Statistics.

students will have to submit. It may contain the [team] wildcard which will be replaced by the team name. For example, for an expression
such as MYCPU[team], if the team’s identifier is CT33, the file should be named MYCPUCT33.zip. Also it may be controlled if automatic
verification results are shown to the student immediately after submission or if they will be delayed until a given date.
Note that the automatic verification is a feature that should be carefully used: the teacher must help students think by themselves. It is
necessary to prevent them from using the system as a debugger, working on a trial and error basis. They must learn how to design their test
batteries to check properly the results of the laboratory by themselves. For this reason, the teacher can restrict the visibility of the results by
controlling if the results of automatic verification are shown to the student immediately after submission, or if they will be delayed until a
given date or if they are totally hidden.
6.2.2. Testers management tab
Fig. 8a shows the appearance of this tab when a certain practical has been selected in order to edit its testers. Previously created testers
may be edited by clicking over their names, or new ones can be added. Testers may be activated or deactivated by the teacher. If deacti-
vated, they would not be applied by the verification engine. One of them is selected as the main one, meaning that it will be automatically
applied in each new submission and its results published to the students. Though is usual to have just one tester for each practical, teachers
may consider useful the existence of several of them, and to apply them under request. These additional testers may be useful when the
main tester is under development and it is necessary to check for fake positive or negative results.
As for the practicals, the scope of the testers may vary, and a sub-tab allows to import them. This enables the teachers to share testers or
reusing of the ones defined in preceding courses. Creation or edition of a tester is performed by use of the interface in Fig. 8, where its
parameters are defined after Table 1.
6.2.3. Submissions management tab
This tab, shown in Fig. 9a, allows to browse the diverse reports the system generates for each submission, as well as to operate over
them. This is performed via the sub-tabs Browse and Operate respectively. The Browse sub-tab provides access to a table displaying all
the teams and practicals for the activity. For each pair of team/practical the teacher is reported of its state, and he is able of accessing an-
other reports resulting from the automatic verification process: simulation log file, differences file, and a timing diagram containing its out-
put waveforms. As an example, Fig. 9a shows how it would appear if the difference files is the one selected.
As the system stores the work state for each submission of the practical, teachers can access to a record of their whole live cycle, and the
system provides them with different statistic reports. It is useful in order to evaluate the progression of the students along the course, the
time the students employ on a certain practical, or the life time of different errors before their amendment.
Fig. 7. (a) Teacher view of a CTPracticals activity (practical management tab). (b) Editing a practical. (c) Importing practicals.
Table 2
Basic course configuration for CTPracticals.
Course configuration for the module CTPracticals
Course ID Course identifier of this configuration
Team name template Expression for generating team names
Main student’s Moodle group Main Moodle group of a student belonging to several groups
Max. team members Maximum number of students in a team
Team definition deadline Deadline for a student to register as member of a team

How to operate over the submissions is shown in Fig. 9b. The main operation is the application of the testers in batch mode. This may be
performed by the teacher at anytime over all of the submissions or over just a subset of them. An interesting feature is that several testers
may be defined for the same practical. As batch verification process may require a considerable time, depending on the number of teams/
practicals, an estimated verification remaining timer is displayed while this process is in course. Such an estimation is based on previous
simulations. Also, in view of the results, it is possible to manually add a short feedback message to the student.
7. Automatic verification and assessment
The automatic verification engine is in the core of the module CTPracticals. It performs two checkups over the designs submitted by the
students. Firstly, there is a formal checking, looking for errors such as a bad file name, a missing file or erroneous signal names. If this is
passed, then a functional testing of the design comes, that is, if it fulfils the functional specifications. As students submit a VHDL description
of their digital circuit, this test requires the use of a VHDL simulator, external to the module.
Fig. 8. (a) Tester management tab. (b) Editing a tester.
Fig. 9. Submission tab: (a) In the browse subtab several pages can be accessed to see output files, logs, waveforms and download submitted works. (b) In the operation subtab
teachers can launch verifications for batches of practicals using different testers and also write short feedback messages to the students.

At this moment, the module makes use of the well known ModelSim VHDL simulator (Modelsim: http://www.model.com) that can be
run in command-line mode. It is composed basically of a compiler (vcom) and a functional simulator (vsim). Simulation of a VHDL project
implies to design a testbench file, which invokes the circuit to simulate and defines the stimulus for its input signals. As it is a VHDL source
file, it is compiled together with the other source files for the design project. It is also necessary to create a script for the vsim simulator,
defining the time window for the simulation and which signals are going to be analyzed, among others. TCL is the scripting language for
vsim.
Fig. 10 displays how does the verification engine work. The zip file submitted by the team is decompressed, and the VHDL sources are
extracted. Firstly, it checks if the required files are present or not. If so, the testbench is incorporated, and all the project sources compiled. If
there are no compilation errors, the functional tester is invoked, which will execute the simulation script. If simulation arrives to a success-
ful end, the simulation output will contain the evolution in time of the selected input signals. This is compared with a reference file, con-
taining the simulation output results for a correct practical. If both outputs are equal, the practical will be classified as a correct one, and the
practical will be graded accordingly. Note that this process involves that the teacher should previously have built and tested the design in
order to have available the right output.
As aforementioned, testers are defined for each practical through the interface shown in Fig. 8. Its attributes are summarized in Table 1.
The tester defines the three files enumerated in the previous paragraphs: the TCL simulation script, the VHDL testbench and the reference
result text file. Other auxiliary VHDL files may be required such as the content of certain ROM memories. A tester also includes the list of
required files to be checked in the submitted zip file. As an example, Fig. 11 displays the files involved in the functional verification of a
naive combinational circuit consisting of two input and one output buses. The script invokes the macro we are about to verify, and assigns
the stimulus to its inputs. Notice that team name templates may be used in these files, which will be replaced with the actual team name.
This feature prevents plagiarism to a certain extent, as macro names depends on the team identifier.
Verification process may start automatically or after a teacher’s request as a batch work. If automatically started, the main tester for the
practical (isdefault attribute in Table 1) will be the one applied on submission. The automatic verification process generates the object
workstate. It contains information of the results of both the formal checking and the functional simulation (see Table 1), the key of the
feedback mechanism for teachers, and as a consequence for students. The main one is the state of the practical, described by an error code
and the result of the simulation (see Table 3). Its value will inform about which format error has prevented the functional simulation of the
practical (see Fig. 10). If it was possible to perform the functional simulation, the correctness of the practical is reported on the result
attribute.
A log of the verification and simulation process is also part of the workstate. It provides details about the whole the process, of great help
when finding out the origin of an error. Teachers can access to a more verbose version, as some implementation details of the verification
tool (directories, database accesses, . . .) are hidden from students. Teachers may also consult the elapsed time compiling and simulating, of
interest to measure the server performance, and to predict the correction time for a high number of practicals and teams.
Two more files are generated, whose name and path are included in the workstate object. One of them is a textual listing with the values
of the output signals of the circuit on different time instants. A PHP viewer, developed inside our module, makes possible to visualize it as a
timing diagram (Fig. 12b). The other one is an HTML file comparing the output with the reference one. The differences are highlighted in a
similar way as the Unix vimdiff utility (Fig. 12a). This is a nice way to easily inspect differences and locate errors.
In addition, an escape operation mode allows the teacher to perform a manual assessment and feedback. If this mode is activated, there
is no an automatic verification of the practical and the teacher will include the result reports manually. This option is very useful for
descriptive stages of the design that are submitted as a document, not being a VHDL project. In manual mode, the tester result file is inter-
preted with a different meaning than this one of regular testers. Such a file must contain, in this case, a line per team indicating the result
(OK/Wrong) of manual verification and also an optional short feedback message.
Fig. 10. Verification engine flow for automatic assessment.

8. Site configuration, security issues and other features
Once the module has been installed, the site administrator must configure several settings of the module through the main adminis-
tration menu of Moodle (Fig. 13).
There are several settings related to the behaviour of simulations. This way, it is necessary to specify the path where the binaries of the
VHDL compiler (vcom) and simulator (vsim) are installed together with the license file. It is also possible to limit the number of simulations
(workstates) that are stored per practical and team to prevent spending an excessive amount of disk.
Another group of options are related to some security issues that may compromise the proper functioning of the server. Note that the
verification engine involves the execution of scripts that are external to the module. These include not only the simulation scripts and test-
bench designed by teachers, but also the VHDL code uploaded by the students themselves. This could represent an important risk which
has been mitigated using common mechanisms available on Unix platforms that act at several levels:
Safe user: Simulations can run under a different user to the one owning the web server. This prevents potential malicious actions over
the server daemons and files. This user must be firstly created in the platform by the administrator and specified in the settings.
Safe TCL interpreter: TCL, used for simulation scripts, provides a safe interpreter for execution of untrusted scripts. It restricts danger-
ous functionality of commands, like the deletion of files, and ensures that hosting applications are not harmed. Though, in general,
Fig. 11. Building a tester: (a) TCL script for the VHDL simulator (vsim). (b) Reference file with the right output result. (c) VHDL testbench invoking the macro under test. The
template string tg00 will be replaced by the team name.
Table 3
Status of a work after its verification.
Error code Result Meaning
ERROR0 OK/Wrong No format error, simulation ended with a right or wrong output.
ERROR1 n/a Format error due to missing required files
ERROR2 n/a Format error due to wrong compilation. Multiple reasons: syntax errors, missing macros, . . .
ERROR3 n/a Format error due to uncompleted simulation. Multiple reasons: wrong input/output interfaces, . . .
ERROR10 n/a In manual mode means that such a team lacks assessment.
n/a: Not applicable.
572 E. Gutiérrez et al. / Computers Education 54 (2010) 562–577

this safe mode could constrain the design of simulation scripts, it keeps enough functionality as to define the input and output signals
and run the simulation.
Chroot jail: This is the more sophisticated security mechanism implemented for simulations and it involves certain administrative
actions on the server. chroot is a Unix mechanism that changes the apparent disk root directory for a process and its children. It
allows to confine such processes in a sandbox which is restricted to a given folder where a mini-installation of the system resides.
Files outside the sandbox are not visible to the chrooted processes. Setting up a chroot jail requires to make a copy of the simulation
software in the sandbox. This can be done manually, but it is a laborious work because it is also necessary to include all libraries
which the simulator depends on. Fortunately some tools can ease this task by automating the sandbox creation, resolving library
dependencies and copying only those necessary files. An example of these tools is Jailkit (Jailkit: http://olivier.sessink.nl/jailkit/.),
the one used. Note that chrooting the simulations is optional and can be disabled in the module settings at the cost of some potential
risks.
Resource limitation: Simulation processes are executed in a shell with limits their resources such as the maximum amount of CPU
time or memory. This will prevent, for example, the possibility of an infinite loop during the simulation or the execution of scripts
eager for resources.
It is also necessary to set up certain privilege levels for Moodle users, as the module CTPracticals needs to behave differently for teachers
than for students. With this purpose some modifications have to be added to the standard Moodle installation:
Fig. 13. (a) Moodle module administration panel. (b) Site settings for CTPracticals.
Fig. 12. (a) After simulation, the difference file allows to search for possible errors. (b) Also the output can be viewed in a timing diagram viewer.
E. Gutiérrez et al. / Computers Education 54 (2010) 562–577 573

New user Moodle capabilities associated to CTPracticals: Each capability represents a permission that allows an user to perform a given
action. CTPracticals defines several capabilities which are added to the existing set of Moodle capabilities when the module is
installed. Capabilities associated to the teacher role include: team management, practical management, tester management, view
submissions and results, etc. The student role has not any of these management capabilities but instead, students can be registered
into teams and submit works. Observe that capabilities not only grant privileges but also they allow to distinguish roles, making the
resources (activities, control block) appear different for teachers than for students.
File protection: Moodle does not implement any native file protection system. Consequently, for the new module, a user might down-
load practicals belonging to other users if he knows the corresponding URL. This is solved patching the Moodle kernel archive
file.php, in charge of accessing data files through Moodle pages, adding a security verification when accessing to CTPracticals files.
This validates the login in the context of the course, and checks in the database that the requested file belongs to such a user.
During the module development, an important effort has been invested in order to fully integrate CTPracticals into the Moodle system. In
this sense, several complementary features have been implemented:
Backups: It is possible to include all CTPracticals activities in the Moodle backup. This allows not only to perform the backup of activ-
ities, but also to copy them from one course to another or between different sites. With this purpose a specific backup/restore library
has been created for CTPracticals including functions that are in charge of backing up module-dependent database tables and data
files. According to Moodle policies, the backup can be restricted to the definition of practicals and testers or it can also include user
information (teams, submissions, workstates, etc.).
Logs: The module CTPracticals records the user actions in the Moodle logging system. This information is used, for example, to get site
statistics or to measure the activity of the users. Additionally, a specific log has been defined that keeps track of students submissions
and their automatic verification results. Course statistics for CTPracticals are just obtained from such a log (Fig. 6d).
Cron: Moodle provides a cron service that allows modules to execute actions periodically. It has been used to clean up possible
orphan objects that do not belong to any existing course. It is the case of teams, objects that are attached to the course but not to
any activity. However the deletion of a course involves a recursive deletion of its different activities. Note that no orphan teams
result, when an activity CTPracticals or a team is deleted explicitly in a course, being deleted, in this case, all associated database
entries and files.
Gradebook: Scores generated by the verification engine are automatically included in the Moodle grade system as standard activities
do. It is possible to assign a score to each practical of the CTPracticals activity, up to the maximum grade assigned to it (Table 1). If the
verification process classifies a practical as OK, it will contribute with its score to the total grade of the activity as it appears in the
gradebook. As these scores are determined in a team basis, they need to be transferred to each student of the team. Having scores
available makes possible to combine those of different activities in order to obtain a global grade.
Multilanguage: Making use of the multilanguage features of Moodle, the contents of CTPracticals are available in two languages: Eng-
lish and Spanish.
9. Results
In this section, we analyze our experience with the new module in the teaching of the Computer Technology laboratory as described in
Section 2. The module CTPracticals was installed on a departmental server initially running Moodle 1.7, although it has been successfully
migrated to the last version 1.9. It has been extensively in use since the second semester of the academic year 2007/2008, after a trial per-
iod in the previous semester. During such an academic year a total of 410 students were enrolled in Computer Technology, comprising seven
different classes (classrooms), each one with its own Moodle course. Table 4 shows the distribution of students into groups, how many of
them were actually registered in the system, and the number of teams per group. Note that, teachers decided that students would work
individually in one of the groups, hence the number of registered students equals the number of teams. For the remaining groups, working
teams of up to three members were allowed, with teams of two being the typical case.
The laboratory work consists of designing a simple processor (CPU) using Xilinx Foundation series as CAD tool. The designs are devel-
oped as schematics, that are later exported to VHDL in order to be uploaded to the system. The work is broken down into several practicals
that students had to submit by a fixed deadline. The first two ones (1 and 2) deal with the design of the instruction format and instruction
cycle. These two practicals need to be verified manually by the teachers as they are documents describing these stages of the design. The
two subsequent ones correspond to the development of the datapath (3) and the memory system (4) of the processor. In the two last prac-
ticals the design of the control unit is proposed using two different methodologies, hardwired (5) and microprogrammed (6). These last
designs need to be completed with a loader module in charge of loading programs in the main memory (7). Table 5 summarizes the set
of such practicals. The execution time taken for simulation is shown for practicals with automatic verification. Simulation time represents
the average time that a student must wait for completing his submission or that the teacher must wait per work in batch verifications.
Figs. 14 and 15a depicts the web server usage and the Moodle platform activity (number of logins), respectively, for the period under
study. Both plots exhibit a typical behaviour with higher values for the mid/end months of the semester. These peaks also correspond to the
deadlines proposed for each practical, mainly for the two last ones which entail a higher complexity (both are designs of the processor
Table 4
Student groups involved in the experience.
Courses (classrooms) IA IB GA GB SA SB SC Total
Total number of students 52 51 63 54 70 59 61 410
Number of registered students 32 28 39 30 41 41 34 245
Number of registered teams 32 17 23 21 25 26 27 171

control unit). The slight decreasing activity in June is due to the fact that the teaching activities occupy only half of this month, and con-
sequently the average activity for such month decreases. Logically, the server usage is almost null during the summer holidays and it will
increase again for the next academic year. Minor activity of last months corresponds to the preparation work for the next year, developed
by teachers, and its use by other subjects involving less students.
Table 6 collects several figures related to the server workload. The number of different objects (activities, practicals, works and work-
states) created during the experience is shown, together with a breakdown of total submissions by result. In total, for 55 practicals (from all
the courses), almost 6000 submissions were done, resulting in an average of 820 per course and 35 per team. A weekly distribution of sub-
missions is detailed in Fig. 15b, where it may be observed again that the activity concentrates at the end of the semester. Students tend to
exhaust deadlines before final examinations. Note that practicals that are not working due to format errors or wrong simulation outputs
should be resubmitted, so the percentage of them may be higher than that of correct simulations, which need no further resubmission.
Table 5
Labworks proposed for designing a CPU.
Practical Type Simulation time (s)
1. Instruction format Manual –
2. Instruction cycle Manual –
3. Processor datapath Manual –
4. Memory system Automatic 4.58
5. Hardwired control unit Automatic 6.19
6. Microprogrammed control unit Automatic 6.67
7. Complete architecture with loader Automatic 7.31
Fig. 14. Server usage chart.
Fig. 15. (a) Moodle activity statistics. (b) Weekly submission history for all groups.
Table 6
Final figures for CTPracticals objects involved during the experience.
Total Mean per course
CTPracticals activities 12 1.71
Practicals 55 7.86
Testers 61 8.71
Works 778 111
Workstates 5723 818
Submissions with format errors 49.2% –
Submissions not OK 28.8% –
Submissions OK 15.9% –

The subjective perception about the experience was very positive, both for students and teachers. But in order to get a more useful feed-
back from students, an anonymous opinion poll was carried out. The aim was to evaluate two aspects of the experience: (i) how pleased
students were with the system and (ii) if students having also enrolled in previous years (because they did not pass the course) considered
the use of the system as an improvement compared to previous years. Table 7 shows the average marks for each item of the questionnaire
that were graded in the scale from 1 through 5 (1 = strongly disagree, 5 = strongly agree). Items labelled with 1.X correspond to aim (i) and
those labelled 2.X to aim (ii). In both groups of items the results were pretty high. It is remarkable that items related to the platform usabil-
ity is slightly higher than those related to the ability of providing useful information for debugging. After talks with students, looking for a
deeper feedback, we concluded that the reason for this is the difficulty to understand the output messages of compilers and simulators.
This point needs to be improved in future versions of the module, that should offer a clearer information from the automatic verification
subsystem. Finally, note that most of students of previous courses agrees that the use of the Moodle platform with CTPracticals has notably
ameliorated the laboratory dynamic with respect to prior years.
10. Conclusions
This paper introduces a new Moodle module, called CTPracticals, oriented to support subjects involving the development of digital cir-
cuits, as it is the case of a basic course on Computer Architecture in the context of Computer Science and Engineering studies. The module is
intended to operate over the VHDL designs submitted by the students, providing enriched features for such assignments.
In the first place, the module allows to register students into teams as they will collaboratively work on a design. The team is the basic
organizational unit of the module: all submissions, as well as the communication between teachers and students, are performed on a team
basis. Secondly, at the core of the module, an automatic verification engine processes the designs automatically as they are submitted. The
verification involves a format checking of submitted files prior to the simulation step of the VHDL design. The simulation output is con-
trasted with a valid behaviour description which allows to generate error reports or to assess the design as correct. Thirdly, an advanced
submission management allows students to know immediately the state of their work and also to access the result of the automatic assess-
ment or feedback messages from their teachers. Likewise, teachers have a big picture of the overall status of their class and they can launch
additional verifications in batch mode, send feedback messages to students, obtain statistics, etc. Thanks to automatic verification, teachers
are partially freed from the routine testing of submissions, mainly for courses with a large number of students, saving time that can be
dedicated to focus on the teaching goals.
The module is being extensively used since the academic year 2007/2008 in the subject Computer Technology where the design of a basic
CPU is proposed as labwork. Users, both teachers and students, are very satisfied with the system, as it has improved the laboratory dy-
namic, specially with respect to previous years. Besides, on being integrated into Moodle, users can benefit from all other facilities provided
by this platform.
Finally, due to the potentiality of the module, authors are planning to extend CTPracticals to other kinds of labworks. In this sense, other
subjects taught by the authors department are being considered, with practicals based on Matlab and assembly language. From the devel-
oper viewpoint, this will involve an adaptation of the verification engine to make it call the suitable applications.
Acknowledgements
This work has been partially supported by the educative innovation projects PIE06-072, PIE07-069 and PIE08-062 granted by the Uni-
versity of Málaga. The authors also thanks Pablo Carnero, graduated student who worked on early versions of the module. In closing, the
reader can visit the demo site http://guac.ac.uma.es/demo, logging in as user demoteacher with password 1234.
References
Bennedsen, J., Caspersen, M. E., Kölling, M. (Eds.). (2008). Reflections on the teaching of programming: Methods and implementations. Springer-Verlag.
Butakov, S., Scherbinin, V. (2009). The toolbox for local and global plagiarism detection. Computers and Education, 52(4), 781–788.
Calazans, N., Moraes, F. (2001). Integrating the teaching of computer organization and architecture with digital hardware design early in undergraduate courses. IEEE
Transactions on Education, 44(2), 109–119.
Clausen, T. (2005). Undergraduate engineering education challenged by the Bologna declaration. IEEE Transactions on Education, 48(2), 213–215.
Corbera, F., González, M., Gutiérrez, E., Ramos, J., Montiel, S., Trenas, M. A. (2006). Computer technology labworks (in Spanish). University of Málaga.
Djordjevic, J., Nikolic, B., Milenkovic, A. (2005). Flexible web-based educational system for teaching computer architecture and organization. IEEE Transactions on Education,
48(2), 264–273.
Douce, C., Livingstone, D., Orwell, J. (2005). Automatic test-based assessment of programming: A review. Journal on Educational Resources in Computing, 5(3), 1–13. 4.
Gutiérrez, E., Ramos, J., Romero, S., Trenas, M. (2007). A learning management system designed for a basic laboratory course on computer architecture. In e-learning’07
IADIS international conference.
Jackson, D., Usher, M. (1997). Grading student programs using ASSYST. SIGCSE Bulletin, 29(1), 335–339.
Table 7
Average marks of the student opinion poll (1 = strongly disagree, 5 = strongly agree).
The system
1.1 I find very useful to have available my lab work at any place and moment 4.8
1.2 I used the server to work o to submit the work out of the laboratory scheduling 4.5
1.3 The information provided by the system (format errors, teacher feedback, . . .) has been very useful 3.8
1.4 The system helped me to detect bugs and fix my practicals 3.4
Comparison with previous years (only for student enrolled in previous years)
2.1 The server facilitated the development of the lab work 3.8
2.2 The feedback from teachers has improved with the use of system 3.5
2.3 I had to attend a tutoring fewer times 3.7
2.4 In general, I prefer to use the system 4.5

Jailkit: A set of utilities to limit user accounts using chroot. http://olivier.sessink.nl/jailkit/.
Jonassen, D. (1999). Constructivist learning environment on the Web: engaging students in meaningful learning. In Education technology conference and exhibition.
Khalifa, M., Lam, R. (2002). Web-based learning: Effects on learning process and outcome. IEEE Transactions on Education, 45(4), 350–356.
Knuth, R., Cunnigham, D. (1993). Tools for constructivism. In T. M. Duffy, J. Lowyck, D. H. Jonassen, T. M. Welsh (Eds.), Designing environments for constructive learning (pp.
163–188).
Kurmas, Z. (2008). Improving student performance using automated testing of simulated digital logic circuits. In ITiCSE ’08: Proceedings of the 13th annual conference on
Innovation and technology in computer science education. ACM (pp. 265–270).
Lathrop, A., Foss, K. (2000). Student cheating and plagiarism in the Internet Era: A wake-up call. Libraries Unlimited.
Main Moodle site. http://moodle.org.
Malmi, L., Korhonen, A., Saikkonen, R. (2002). Experiences in automatic assessment on mass courses and issues for designing virtual courses. SIGCSE Bulletin, 34(3), 55–59.
Martı́n-Blas, T., Serrano-Fernández, A. (2009). The role of new technologies in the learning process: Moodle as a teaching tool in physics. Computers and Education, 52(1),
35–44.
Modelsim: A comprehensive simulation and debug environment for ASIC and FPGA designs. http://www.model.com.
Moreno, L., González, C., Castilla, I., González, E., Sigut, J. (2007). Applying a constructivist and collaborative methodological approach in engineering education. Computers
and Education, 49(3), 891–915.
Patterson, D. A., Hennessy, J. L. (2008). Computer organization and design: The hardware/software interface (4th ed.). Morgan Kaufmann Publishers Inc..
Poplawski, D. A., Kurmas, Z. (2008). JLS: A pedagogically targeted logic design and simulation tool. In ITiCSE ’08: Proceedings of the 13th annual conference on Innovation
and technology in computer science education. ACM (p. 314).
Reichert, S., Tauch, C. (2003). Trends in learning structures in european higher education. In EUA Graz convention.
Rice, W., Beg, Y., Waters, C., Bokreta, M., Santiago-Avilés, J. (1999). New pedagogical approaches to computer science education: A case study in peer learning. In Proceedings
of the 29th ASEE/IEEE frontiers in education conference.
Rodríguez, S., Zamorano, J., Rosales, F., Dopico, A. G., Pedraza, J. L. (2007). A framework for lab work management in mass courses. Application to low level input/output
without hardware. Computers and Education, 48(2), 153–170.
Rosales, F., García, A., Rodríguez, S., Pedraza, J. L., Méndez, R., Nieto, M. (2008). Detection of plagiarism in programming assignments. IEEE Transactions on Education, 51,
174–183.
Tan, W., Lin, S., Yang, Y., Zhao, X. (2008). Design on collaborative virtual learning community and learning process visualization. Lecture Notes in Computer Science, 5145,
424–433.
Xilinx: The programmable logic company. http://www.xilinx.com.

A new Moodle module supporting automatic verification of VHDL-based assignments.pdf

Recommended

Recommended

More Related Content

Similar to A new Moodle module supporting automatic verification of VHDL-based assignments.pdf

Similar to A new Moodle module supporting automatic verification of VHDL-based assignments.pdf (20)

More from Sabrina Ball

More from Sabrina Ball (20)

Recently uploaded

Recently uploaded (20)

A new Moodle module supporting automatic verification of VHDL-based assignments.pdf