Testing and Verification

Relevant source files

• .github/workflows/rust.yml
• .github/workflows/website.yml
• .rumdl.toml
• CONTRIBUTING.md
• release-hook.sh
• src/cargo_toml.rs
• src/dev/check.rs
• src/init.rs
• tests/integration_tests.rs
• tests/test_exercises/exercises/compilation_failure.rs
• tests/test_exercises/exercises/compilation_success.rs
• tests/test_exercises/exercises/test_failure.rs

This document describes the testing and verification systems in Rustlings. It covers how exercises are verified for correctness during runtime and how the Rustlings application itself is tested to ensure it functions properly. For information about developer commands used to check exercise integrity, see Developer Commands.

Overview of Testing and Verification in Rustlings

Rustlings employs two primary types of testing mechanisms:

1. Exercise Verification: Used during normal operation to verify if a user's exercise solution is correct.
2. Integration Testing: Used by Rustlings developers to test the application's functionality.
3. Developer Check Pipeline: A comprehensive validation suite for the repository's integrity.

Verification Flow

The following diagram maps the high-level verification concepts to the specific code entities that implement them.

Sources: src/dev/check.rs30-202 tests/integration_tests.rs14-38 .github/workflows/rust.yml21-52 release-hook.sh14-16

Developer Check Pipeline

The dev check command is the primary tool for maintaining repository health. It performs several deep inspections of the codebase to ensure that the exercises, solutions, and configuration files are in a consistent state.

Validation Steps

The check module in src/dev/check.rs implements the following logic:

• check_cargo_toml: Ensures that the dev/Cargo.toml (or the initialized Cargo.toml) is synchronized with the exercises defined in info.toml. It uses append_bins to generate the expected [[bin]] list and compares it against the file content src/dev/check.rs30-57
• check_info_file_exercises: Validates exercise metadata. It checks for:
  • Name length (max 32) and forbidden characters (only alphanumeric and underscores allowed) src/dev/check.rs70-77
  • Presence of a main function to satisfy Language Server Protocols (LSP) and avoid compiler errors src/dev/check.rs107-112
  • Presence of at least one // TODO comment to guide the user src/dev/check.rs114-119
  • Consistency between the test boolean in info.toml and the presence of #[test] in the file src/dev/check.rs121-132
• check_unexpected_files: Scans the exercises directory for files not registered in info.toml. It allows only one level of nesting and specific files like README.md src/dev/check.rs145-197
• check_exercises_unsolved: Verifies that the exercises in their initial state (with // TODO comments) actually fail to compile or fail their tests, ensuring the user has something to solve src/dev/check.rs199-202

Sources: src/dev/check.rs21-27 src/dev/check.rs30-202 src/cargo_toml.rs29-66

Integration Testing Framework

The integration testing framework tests the Rustlings application itself by invoking the compiled binary and observing its behavior against a mock environment.

Integration Test Structure

The tests utilize a custom harness to abstract the process of spawning the Rustlings process and verifying its output.

Sources: tests/integration_tests.rs6-75

Test Command Runner

The integration tests use a Cmd harness that executes the binary found at the path provided by the environment variable CARGO_BIN_EXE_rustlings tests/integration_tests.rs38 It allows for asserting success or failure and checking if specific strings appear in stdout or stderr tests/integration_tests.rs47-64

Key Tests:

• run_compilation_success: Runs an exercise designed to pass tests/integration_tests.rs78-83
• run_compilation_failure: Runs an exercise with syntax errors to ensure Rustlings reports failure tests/integration_tests.rs86-91
• init: Tests the init command in a temporary directory, verifying it creates the rustlings/ directory and handles re-initialization attempts (idempotency) tests/integration_tests.rs133-159
• hint: Verifies that the hint command retrieves the correct text from info.toml for a specific exercise tests/integration_tests.rs124-130

Sources: tests/integration_tests.rs78-159 src/init.rs26-117

CI/CD and Verification Workflows

Rustlings uses GitHub Actions and local shell scripts to enforce quality and verify the Minimum Supported Rust Version (MSRV).

GitHub Actions (rust.yml)

The Check workflow runs on every push and pull request to the main branch .github/workflows/rust.yml3-15

• clippy: Runs cargo clippy -- --deny warnings to ensure no lint violations .github/workflows/rust.yml21-26
• fmt: Runs cargo fmt --all --check to enforce style consistency .github/workflows/rust.yml27-32
• test: Runs cargo test --workspace across Ubuntu, Windows, and macOS to ensure cross-platform compatibility .github/workflows/rust.yml33-44
• dev-check: Executes cargo dev check --require-solutions to ensure all exercises are valid and their corresponding solutions pass .github/workflows/rust.yml45-51

Release Hook and MSRV Verification

The release-hook.sh script is a local check used before publishing. It verifies the Minimum Supported Rust Version (MSRV), currently tested using cargo +1.88 to run the developer check suite release-hook.sh14-16

Sources: .github/workflows/rust.yml1-51 release-hook.sh1-16

Test Exercise Directory

To facilitate testing without relying on the actual curriculum (which may change), Rustlings includes a special set of test exercises in tests/test_exercises used exclusively for integration testing.

Exercise File	Purpose
compilation_success.rs	Simple empty main to test successful compilation tests/test_exercises/exercises/compilation_success.rs1
compilation_failure.rs	Broken syntax (unclosed let) to test compilation failure detection tests/test_exercises/exercises/compilation_failure.rs1-3
test_failure.rs	Contains assert!(false) to verify the test runner correctly detects failing unit tests tests/test_exercises/exercises/test_failure.rs1-9

Sources: tests/test_exercises/exercises/compilation_success.rs1 tests/test_exercises/exercises/compilation_failure.rs1-3 tests/test_exercises/exercises/test_failure.rs1-9