You probably already know a little about what PEML is and why it was created. Some of the very rudimentary basics of PEML's syntax are shown here:
Your First PEML File
OK, enough with the syntax. Now start with this minimal version of an exercise and fill in your own content (you can download a slightly longer version here):
You can edit your own PEML files locally using any text editor, or you can edit PEML right in your web browser by opening our live editing/validation site in a separate tab:
The fields in this minimal document are:
Provide a unique identifier for your exercise. Globally unique. Any sequence of non-whitespace characters is ok, but you may wish to stick to existing naming conventions used in other domains. For example, you could use a unique URL (perhaps where this exercise's home definition lives), or something like a Java package name (a dotted name, perhaps including a university's domain name as a prefix), or your email combined with a unique exercise suffix of your own devising.
Provide a descriptive title that will be used as a human-readable label for the exercise. The intent is for this to be the "title" shown to students in various contexts, either when viewing a single exercise or when viewing lists of exercises. While there is no specific length limit, ideally titles should be no more than "one line" in size, because of the various contexts where they might be displayed.
While a license isn't strictly required, it is strongly recommended.
The id can be specified by a URL that identifies the license, or by a
name (or abbreviated name) that is in common use, such as any of the
keywords used by github (an excellent source for potential
license choices). You can even use "(C) 2021
author:, listing a license is better (everyone must
assume "all rights reserved" if you do not). See
license for more details).
Probably you. For an individual, either specify a unique, identifying
email address, or as shown here, an email address along with a name
using separate keys. For an organization, you can specify your information
author:, and then provide
specify the organization owning the copyright.
Provide your exercise's instructions here. This isn't required in all contexts (for example, if providing an auto-grading setup), but you probably want to.
Now you have a PEML description!
Identifying the Programming Language
If your exercise is a programming exercise, you probably will find it useful to identify the programming language it supports. PEML does this by allowing you to define the "programming system".
language: key is the required one--specify the language that
is supported using its common name (be careful of capitalization, since
some tools processing descriptions might not treat the name case-insensitively),
or its MIME type (to reduce ambiguity). Optionally, you can also specify
a version, if your supporting files are version-dependent. Feel free to
version dependency constraints, although check with your educational
tool to determine what is supported.
Note that the
version: keys here are
listed inside a
[systems] array. There's only one element in
the array in this example, but PEML does allow authors to express exercises
that support multiple programming systems. You can ignore that for now.
However, this is a good chance to recap PEML's array notation.
In short, arrays (lists) in PEML have keys that are surrounded by square brackets instead of using a colon. The end of the list is marked with a pair of empty square brackets (which can be omitted if they are at the end of the file). All the keys between these two markers are part of the list. Like ArchieML, if you look at the keys inside, as soon as a key is duplicated, that is taken as the start of a new entry in the list. So an array with multiple entries might look like this:
Associating Supporting Files
OK, so where's all the cool stuff? Like auto-grader inputs and all that?
PEML provides a very rich model for structuring this information for tool use. However, PEML relies heavily on convention over configuration to simplify the way those things are managed and to make it easier for authors to learn the minimal amount they need, and gradually add onto that core over time as more advanced situations arise.
Starting Files Provided to the Student
Suppose you want to provide some file(s) to the student as the starting
point for their solution. Just add them in the
folder next to the PEML description itself. We recommend placing each
exercise that uses additional resources in its own directory. This
approach works whether the PEML description is located in a folder
on the local machine, is packaged in a zip file or another form of
archive, or hosted in a repository. You could also use the
[src.starter.files] array key to provide this information
in the PEML description itself, implicitly providing them by co-location
is often simpler.
# Providing starter files for the user dir |-- exercise.peml +-- src +-- starter |-- file1.ext |-- file2.ext +-- file3.ext
Images Used in the Instructions
Suppose you want to provide images for use in your instructions.
Instructions are typically written in Markdown or vanilla HTML, but
can certainly refer to supplemental files, whether they be images,
separate pages describing APIs, examples students can download, etc.
While you could host these resources on your own website and use absolute
URLs, you may wish for them to be packaged with the exercise. You can
[public_html] key to specify these explicitly, or
simply place them in a
# Providing images files for the instructions dir |-- exercise.peml +-- public_html | |-- image1.png | |-- image2.png | +-- download_file.dat +-- src +-- starter |-- file1.ext |-- file2.ext +-- file3.ext
Test Case Files for Auto-grading
Suppose you want to provide some file(s) specifying the tests you want
to use to check the behavior of answers to the exercise. These could be
in the form of compilable program code, scripts, data files, or whatever
notation/format is used by the auto-grading tool reading your PEML
description. You can provide these as separate files under the
suites folder, which corresponds to the
# Providing test case files dir |-- exercise.peml +-- public_html | |-- image1.png | |-- image2.png | +-- download_file.dat +-- src | +-- starter | |-- file1.ext | |-- file2.ext | +-- file3.ext +-- suites |-- TestClass1.java +-- TestClass2.java
Other Supplemental Files for Auto-grading
Suppose your auto-grading tests use additional data files or other
resources that need to be available during execution of your tests.
Not all grading tools support such resources, but if they do, you
can provide them in the
which corresponds to the
Note: If you'd rather use a docker image to provide the environmental setup for environments (for building, running, testing, or even the student's starting environmet) and your tool supports this kind of usage, you probably want to just specify image information directly in the PEML file--see Environments in the data model).
# Providing data resources for use in testing dir |-- exercise.peml +-- public_html | |-- image1.png | |-- image2.png | +-- download_file.dat +-- src | +-- starter | |-- file1.ext | |-- file2.ext | +-- file3.ext +-- suites | |-- TestClass1.java | +-- TestClass2.java +-- environment +-- test |-- file4.ext |-- file5.ext +-- file6.ext
Top-level keys like
[suites] affect the whole exercise, which means they
apply to all programming systems that the exercise supports. When
the exercise only targets one programming language, that's probably
fine. If, however, you write your exercise so that it supports
multiple programming systems, you may wish to provide different
resources for each system. You can do that by using the same
directory structure for
suites/, but placing them underneath
systems/<language> (using the language name as
specified in the PEML file--if a MIME type is used, replace the '/'
in the MIME type with a '-').
Note: Most tools do not support exercises that support multiple programming systems. However, they should still support the first system listed, including system-specific settings provided in the way described here.
# Providing system-specific resources dir |-- exercise.peml +-- systems +-- Java | +-- src | | +-- starter | | |-- Class1.java | | +-- Class2.java | +-- suites | | |-- TestClass1.java | | +-- TestClass2.java | +-- environment | +-- test | |-- file3.ext | +-- file4.ext +-- python +-- src | +-- starter | |-- class1.py | +-- class2.py +-- suites | |-- test_class1.py | +-- test_class2.py +-- environment +-- test |-- file3.ext +-- file4.ext
If files are specified at both the global and system-specific levels, the files available are the union of both, where files with the same path names in both locations are overridden by the system-specific contents.
Using Data-driven Test Suites for Simple Cases
For exercises that have relatively simple scaffolding requirements for testing--that is, most or all of the tests follow the same basic format, but vary in some standardized ways such as different inputs or outputs--you may find writing test data directly into your PEML description to be more convenient than providing separate test cases.
For example, maybe you are working with a tool that only tests standard input/standard output behaviors, so every test you run consists only of a pair of input/output values. Then you might be able to describe your tests this way:
[suites] consist of a single test suite that contains two
test cases. Each test case here contains two values, named "stdin"
and "stdout". The tool translating this input into test cases would
need to recognize those names and know how to use them, so check with
your tool regarding support for this format and required naming conventions
for variables. However, this is a fairly simple way to mark up the values
when the situation permits.
In fact, the same content could be written in CSV, YAML, JSON, etc., depending on what your grading tool supports.
Here, the MIME types for the files were specified, but they can also be deduced from the file names, so the MIME type is redudant (not required). Also, because of the error-prone nature of manually quoting data in CSV format, PEML supports an alternative "text/x-unquoted-csv" where values are written in the same notation that would be used in the target programming system, allowing more natural use of expressions, native literal constructs, escapes, etc.
If you are lucky enough to have a tool that supports it, you may be able to provide a template used to translate a single test case record into executable code. You might even be able to designate tests as privately visible to instructors only, or publicly visible to students. For example:
If you haven't read through the whole PEML Introduction, that is your next step.
If you want to know more about how PEML came to be, why we're not using straight YAML or JSON, PEML's design goals, and its influences, read our About PEML page.
Finally, start digging into the PEML definition.