How to use Semgrep for Automatic Grading

Step 2: Define the Pattern

Each Match block has a template you can fill in. Here's how to define a pattern:

1. Select the Language (e.g., Python, JavaScript, etc.).

Fill in the Pattern: In Semgrep, you write patterns using a specific syntax. For example, if you want to ensure a student implements an add() function with two parameters in Python, the pattern would look like this:
yaml

rules:

- id: untitled_rule

  pattern: |

    def add($VAR1, $VAR2):

        ...

    return $SUM

  message: Semgrep found a match

  languages: [python]

  severity: WARNING

2. Explanation of the syntax:
   
   • The | symbol indicates that the pattern spans multiple lines.
   • $VAR1, $VAR2, and $SUM represent variables in the function. These variables can have any name.
   • The ellipsis (...) matches any code inside the function.
3. You can test your patterns using the Semgrep playground: Semgrep Playground.
   For more details on writing Semgrep rules, visit Semgrep Docs.

Step 3: Provide Hints (Optional)

You can add hints to help students if they fail to meet the required pattern:

1. Click the dialog box on the right side of the block settings.
2. Select "Uncollapsed when full marks are not achieved".
   This ensures that the hint will be displayed if the student doesn’t pass the test, guiding them towards the right solution.

Example Assignment: Calculator

Let’s look at a practical example. In this assignment, students are expected to code a basic calculator in Python. We want students to define 4 functions: add(), subtract(), multiply(), and divide(). The program should also catch divisions by zero. 

• Goal: Ensure that the add() function accepts two parameters.
• Pattern: As shown earlier, we’ll look for a function definition like def add($VAR1, $VAR2).

We added another pattern to check whether students implement divide() so it handles 0:

You don’t need to install Semgrep yourself in CodeGrade—CodeGrade will automatically handle installation if needed.

The Student View & How They Benefit

If a student doesn’t implement the required function (e.g., add()), they will see an error message like this:

This forces students to pay attention to the specific concepts and code structures discussed in class, improving their coding skills and ensuring that they don’t just game the system.

When to Use Semgrep

Semgrep is ideal for assignments where you want to teach students about fundamental code structures. Some common use cases include:

• Teaching function definitions and parameters.
• Enforcing code style guidelines.
• Preventing the use of advanced or incorrect techniques for beginner-level assignments.

For best results, combine Semgrep with other tests (like unit tests or I/O tests), as it’s easy for students to bypass the code structure test by simply including the pattern without fulfilling the functional requirements.