Skip to content

Conversation

@ellyxir
Copy link
Contributor

@ellyxir ellyxir commented Oct 27, 2025

Summary by cubic

Switch smoketests to use a bind-mounted labs directory and a Playwright MCP testing flow, enabling parallel runs with per-task outputs. Containers now exit when done; logs, patterns, and screenshots land in each task’s directory for easy monitoring and cleanup.

  • New Features

    • Bind-mount /app/labs in containers (no git clone).
    • Parallel runs controlled by RALPH_IDS (default 1–8).
    • Redirect output to tools/ralph/smoketest//ralph.log.
    • Playwright MCP configured at runtime after copying credentials.
    • Patterns written directly to per-task dirs; screenshots auto-moved by ID.
    • Container status polling and a success/partial/failure summary when finished.
  • Migration

    • Tail tools/ralph/smoketest//ralph.log to monitor instead of docker logs.
    • Edit RALPH_IDS in run_smoketest.sh to add tasks; run the script to start.
    • Deploy to http://localhost:8000 using space name ralph to avoid conflicts.
    • Use MCP Playwright tools directly (navigate/click/screenshot), not the Skill wrapper.
    • No artifact copying needed; all results live in tools/ralph/smoketest//.

ellyxir and others added 10 commits October 27, 2025 13:04
- Remove git clone from Dockerfile, use bind mount instead
- Remove MCP setup from Dockerfile (done in run script after credentials copied)
- Add 5 second sleep for mount to settle
- Update run_smoketest.sh to mount labs dir and copy credentials
- Configure Playwright MCP after copying credentials to avoid overwrite
- Update SMOKETEST_PROMPT.md with working Playwright MCP instructions
- Change recipe-dev to pattern-dev, fix paths, update port to 8000
- Add space name requirement (ralph${RALPH_ID}) to avoid parallel conflicts

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Redirect Ralph smoketest output to /app/smoketest/${RALPH_ID}/ralph.log
- Update README to document log location and bind mount behavior
- Users can now tail -f the logs instead of using docker logs
- All smoketest outputs (logs, results, patterns) now in one directory

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Changed SMOKETEST_PROMPT.md to create patterns in /app/smoketest/${RALPH_ID}/ instead of /app/labs/packages/patterns/
- Removed step 7 (copying pattern files) since they're now created in the final location
- Renumbered remaining steps (8→7, 9→8, 10→9)
- Updated README.md to document that patterns are created directly in smoketest directory for automatic cleanup

This approach simplifies the workflow and ensures patterns are automatically cleaned up when smoketests rerun. Tested successfully with ralph_1 container.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Container now exits when smoketest completes instead of staying running.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Script now monitors running containers and prints status every 10 seconds until all smoketests complete.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Show "Still running: 1 2 3" instead of "Still running: ralph_1 ralph_2 ralph_3" for cleaner output.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Show users where to find logs after smoketests start.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Updated SMOKETEST_PROMPT.md to instruct Ralph to name screenshots as ralph_${RALPH_ID}-<description>.png
- Updated run_smoketest.sh to automatically move screenshots to appropriate smoketest directories after completion
- Prevents screenshot conflicts between parallel Ralph instances

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Display where to find results and summarize how many SUCCESS/PARTIAL/FAILURE scores across all completed smoketests.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Added RALPH_IDS="1 2 3 4 5 6 7 8" to run_smoketest.sh for active tasks
- Added RALPH_IDS="1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20" to stop_smoketest.sh (generous range to catch any running containers)
- Replaced all hardcoded loops with $RALPH_IDS for easier configuration
- Updated README with instructions on adding new tasks

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@ellyxir ellyxir self-assigned this Oct 27, 2025
Line wrapping adjustment for screenshot naming instruction.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Copy link
Contributor

@cubic-dev-ai cubic-dev-ai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 issue found across 5 files

Prompt for AI agents (all 1 issues)

Understand the root cause of the following 1 issues and fix them.


<file name="tools/ralph/Dockerfile">

<violation number="1" location="tools/ralph/Dockerfile:78">
The new entrypoint tries to mkdir /app/smoketest/${RALPH_ID}, but that bind mount is root-owned when Docker creates it. Since the container runs as the unprivileged ralph user, mkdir fails with “Permission denied”, so the smoketest never starts. Please ensure the directory is writable (e.g., create/chown it before switching users).</violation>
</file>

React with 👍 or 👎 to teach cubic. Mention @cubic-dev-ai to give feedback, ask questions, or re-run the review.

- Run containers as host user with -u $(id -u):$(id -g) to avoid UID mismatch
- Pre-create smoketest directories on host with correct ownership
- Added prerequisites to README:
  - Docker installation link
  - User must be in docker group (explains sudo credential issue)
  - User must be logged into Claude CLI
- Removed mkdir from Dockerfile since host creates directories

This fixes permission issues on systems where container UID ≠ host UID.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Mount ~/.claude.json and ~/.claude as read-only volumes to /tmp/home
- Set HOME=/tmp/home so container can write config files
- Ensure smoketest parent directory exists before creating subdirs
- Removed docker cp commands (no longer needed)

This allows containers to run as host user without permission errors when
accessing /home/ralph (which is owned by the ralph user created in Dockerfile).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Removed shared ~/.claude mounts that caused conflicts between containers
- Copy credentials into each container's /tmp/home (writable for host user)
- Each container gets its own isolated copy of Claude credentials
- Fixes read-only filesystem error and prevents cross-container conflicts

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Moved installation from ralph user to root so they install to /usr/local/bin
instead of /home/ralph/.npm-global/bin. This fixes MODULE_NOT_FOUND errors
when container runs as host user with different UID.
- Document automatic image pull from Docker Hub
- Update manual run instructions to use host user and credential copying
- Replace outdated credential mount approach with current implementation
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants