PEML

Tagging Exercises

 

The Programming Exercise Markup Language (PEML) is designed to provide an ultra-human-friendly authoring format for describing automatically graded programming assignments.

Describing Exercises through Tagging

This page describes the basics of tagging to label/categorize/describe exercises. While tagging isn't intended to be hard or to require specialized skill, some structure does provide added benefit to users who want to find or understand exercises. Authors who don't intend to share their PEML descriptions could omit tags entirely, since tags are optional. However, the information provided here is intended to allow authors who wish to provide descriptive information about their exercises to aid other users.

Basic Structure Common to All Tags

All tags are provided as part of a nested dictionary structure rooted at the tag key.

The general content of all the tagging fields is the same: a tag list is simply a sequence of strings. For convenience, lists of tags can be written as a single text value, with tags separated by delimiters. We recommend the use of semicolons as the most flexible delimiter choice for authors, since semicolons provide support for a wide range of tag styles and give maximum tool compatibility (for example, supporting multi-word tags, or even tag phrases that include commas): 

tag.topics: strings; loops; conditions

While we recommend the use of semicolons, tools should be more flexible in order to provide a more human-friendly authoring experience. If a tag string contains semicolons, then semicolons should be used as delimiters. Otherwise, if a tag string contains commas, then commas should be used as delimiters instead. In all cases, leading and trailing whitespace should be trimmed from individual tags in the list. 

# The preferred form:
tag.topics: strings; loops; conditions

# But this version means the same thing
tag.topics: strings, loops, conditions

For even greater control (or for tools generating descriptions), the ArchieML approach of defining an array of strings can even be used. Every tag list can be written as a single semicolon-delimited (or comma-delimited) string, or formulated as an array of strings: 

# More verbose array encoding, but equivalent
[tag.topics]
* strings
* loops
* conditions
[]

Describing the Topic of an Exercise

tag.topics optional: string or array

Schema:

"topics": { "oneOf": [
  { "type": "string", "minLength": 1 },
  {
    "type": "array",
    "items": { "type": "string", "minLength": 1 },
    "minItems": 1
  }
  ]}

The tag.topics tag list specifies the topic(s) that are the focus of the exercise--that is, what topic(s) or skill(s) is the learner supposed to be practicing in this exercise. This list is not intended to include topics the learner is supposed to already know or have mastered in order for them to understand the exercise in the first place.

For topic vocabulary, we encourage authors to use commonly used terms such as those that typically appear in textbooks associated with the kinds of courses the exercise is targeting, or terms taken from the ACM Computing Curricula guidelines (such as the topics listed in the body of knowledge in CS2013).

Describing Prerequisite Knowledge

tag.prerequisites optional: string or array or object

Schema:

"prerequisites": { "oneOf": [
  { "$ref": "tag_list" },
  {
    "type": "object",
    "properties": {
      "exposure": { "$ref": "tag_list" },
      "familiarity": { "$ref": "tag_list" },
      "mastery": { "$ref": "tag_list" }
    }
  }
]}

The tag.prerequisites tag list specifies the topic(s) that a learner is expected to already know in order to approach this exercise. In other words, this tag covers topics that enable the student to understand this exercise or its solution, but that are not the focus of this exercise--instead, simply prerequisite materials that enable the student to exercise the target knowledge/skills here.

If desired, an author can be more specific about the level of prior knowledge needed in prerequisite topics. To do this, rather than specifying tags using tag.prerequisites, the author can use one or more of three nested keys.

tag.prerequisites.exposure optional: string or array

Schema:

"exposure": { "oneOf": [
  { "type": "string", "minLength": 1 },
  {
    "type": "array",
    "items": { "type": "string", "minLength": 1 },
    "minItems": 1
  }
  ]}

The tag.prerequisites.exposure tag list is an optional way to be more specific about prerequisite knowledge or skills. Here, we're using a 3-category partitioning of how well a learner has mastered a topic by characterizing it in terms of exposure, familiarity, or mastery. A learner is said to have exposure to a topic if they have heard the term and/or seen the procedure, but may not be able to discuss or use it effectively without further instruction. This represents a weak level of background knowledge.

tag.prerequisites.familiarity optional: string or array

Schema:

"familiarity": { "oneOf": [
  { "type": "string", "minLength": 1 },
  {
    "type": "array",
    "items": { "type": "string", "minLength": 1 },
    "minItems": 1
  }
  ]}

The tag.prerequisites.familiarity tag list is an optional way to be more specific about prerequisite knowledge or skills. Here, we're using a 3-category partitioning of how well a learner has mastered a topic by characterizing it in terms of exposure, familiarity, or mastery. A learner is said to have familiarity with a topic if they can answer questions about or use the material/procedure, even in a new context, when instructed to do so. This represents a medium level of background knowledge.

tag.prerequisites.mastery optional: string or array

Schema:

"mastery": { "oneOf": [
  { "type": "string", "minLength": 1 },
  {
    "type": "array",
    "items": { "type": "string", "minLength": 1 },
    "minItems": 1
  }
  ]}

The tag.prerequisites.mastery tag list is an optional way to be more specific about prerequisite knowledge or skills. Here, we're using a 3-category partitioning of how well a learner has mastered a topic by characterizing it in terms of exposure, familiarity, or mastery. A learner is said to have mastery of a topic if they exhibit knowledge of or skill with the material/procedure, even in a new context, and even when not instructed to do so. This represents a high level of background knowledge.

Describing Exercise Metadata

In addition to tags about topic areas, tags can also be used in other ways to label exercises for various purposes, as described here. For labeling exercises by difficulty, see the difficulty key.

tag.style optional: string or array or object

Schema:

"style": { "oneOf": [
  { "type": "string", "minLength": 1 },
  {
    "type": "array",
    "items": { "type": "string", "minLength": 1 },
    "minItems": 1
  }
  ]}

The tag.style tag list specifies the "style" of exercise. Suggested values include:

  • code-writing
  • multiple-choice
  • fill-in-the-blank
  • bug-identification
  • bug-fixing
  • symbolic-execution (e.g., predicting the output)
  • project

Note: This list is likely to be incomplete, so ideas for a more complete list, or more standardized terms for these exercise styles, are definitely welcome.

tag.course optional: string or array or object

Schema:

"course": { "oneOf": [
  { "type": "string", "minLength": 1 },
  {
    "type": "array",
    "items": { "type": "string", "minLength": 1 },
    "minItems": 1
  }
  ]}

The tag.course tag list is used to indicate the course(s) associated with this exercise, if any. This could include anything from fairly general categories of courses ("cs1" or "programming languages") to specific institutional course numbers, or any combination.

tag.book optional: string or array or object

Schema:

"book": { "oneOf": [
  { "type": "string", "minLength": 1 },
  {
    "type": "array",
    "items": { "type": "string", "minLength": 1 },
    "minItems": 1
  }
  ]}

The tag.book tag list is used to indicate the textbook(s) associated with this exercise, if any. While exercises or programming projects that are taken directly from a textbook (or a textbook's associated question bank or lab manual) should already provide this information in the license.book key, the tag.book is more broadly applicable to independently-authored exercises that use materials/examples associated with a book or that expect the learner to have access to a book, even if they are not owned by the same copyright owner as the book itself.

tag.personal optional: string or array or object

Schema:

"personal": { "oneOf": [
  { "type": "string", "minLength": 1 },
  {
    "type": "array",
    "items": { "type": "string", "minLength": 1 },
    "minItems": 1
  }
  ]}

The tag.personal tag list is used to indicate personal tags that may only be meaningful to the original author/tagger, and are used for personal recall/categorization purposes. Values here may not be as relevant for other users.

Search