Tests

Every test run on EDGEchat so far: how to run it again, what to expect, and what it showed. Each phase ends with a test on the dev PC (design, “Phases”); new tests are added here when they are first run.

Last updated 2026-10-23.

At a glance

A scratch config has no Claude login, so the CLI tests check how the plugin installs and starts its servers, not what Claude does with it. The P1 test on 2026-10-21 was the first run of a skill in a session.

A scratch config

CLAUDE_CONFIG_DIR points Claude Code at another settings folder, so a test never touches your own plugins, servers or login. Every CLI test below starts like this (Git Bash):

export CLAUDE_CONFIG_DIR=<an empty scratch folder>

CLI

Validate

claude plugin validate .
claude plugin validate .claude-plugin/plugin.json
claude plugin validate skills --strict

Last run 2026-10-25 with Claude Code 2.1.310, at 0.0.10: pass, the same one warning, plus two new “plugin directory” warnings that do not change the result: no icon (ICON_MISSING) and no privacyPolicyUrl (PRIVACY_URL_MISSING, since rednode is a remote server). Run 2026-10-23 with 2.1.302, at 0.0.5: pass, the one warning. First run 2026-10-16 with 2.1.299.

In P0, marketplace.json failed --strict until it had a top-level description.

Install from the checkout

claude plugin marketplace add "<path to this checkout>"
claude plugin install edge@edgemodders
claude plugin details edge@edgemodders
claude plugin configure edge@edgemodders
claude mcp list

Last run 2026-10-23 at 0.0.5 with Claude Code 2.1.302: pass: “6 userConfig options not yet set”; details lists the four servers, always-on cost about 185 tokens; configure lists the six options, gemini_api_key and openai_api_key sensitive; mcp list connects redcode, edgedeck and edgemat (rednode refused, game closed). Run 2026-10-21 at 0.0.2 with the results below. First run 2026-10-16 at 0.0.1, with the same results less edgedeck and its option.

With a local marketplace the plugin is read from the checkout (CLAUDE_PLUGIN_ROOT is the checkout itself), and the checkout’s path has spaces: redcode still connects.

Install from GitHub

What a user does. Needs the commit pushed.

claude plugin marketplace add Edgemodders/EDGEchat
claude plugin install edge@edgemodders
claude mcp list

One command does both, as the README gives it: claude plugin install edge --marketplace Edgemodders/EDGEchat, with --config <key>=<value> for settings. Run 2026-10-26 in a scratch config, with Claude Code 2.1.310 and --config game_path=...: pass. It added the marketplace in user settings, installed 0.0.14 at user scope, saved game_path, and said “4 userConfig options not yet set”.

Run 2026-10-17 at 8fd75a2, just after the repo was renamed to EDGEchat: pass. redcode connects; rednode is ECONNREFUSED with the game closed.

First run 2026-10-16 at 4f07db8, as Edgemodders/EDGEplugin: the same results as from the checkout; the plugin runs from <config>/plugins/cache/edgemodders/edge/0.0.1/, and redcode connects from there.

Update from GitHub

What an existing user sees when a release adds a server and an option. Install the old version from GitHub first, push the new one, then update.

claude plugin marketplace add Edgemodders/EDGEchat     # before the push: installs the old version
claude plugin install edge@edgemodders
# push the new version, then:
claude plugin marketplace update edgemodders
claude plugin update edge@edgemodders
claude plugin details edge@edgemodders
claude mcp list

First run 2026-10-21, 0.0.1 to 0.0.2 (5dbcc7c), Claude Code 2.1.302: pass, with one gap.

Plugin servers beside servers added by hand

Run 2026-10-16 in P0, on a throwaway copy of the plugin. Results, recorded in the design’s facts:

Unset options

Run 2026-10-16 in P1. A test server that writes its argv and environment to a file, with --game ${user_config.game_path} in its args and REDCODE_GAME_PATH, REDBRIDGE_EXE from options in its env.

Also seen: a server that uses an option userConfig does not declare is dropped, and validate says so. CLAUDE_PLUGIN_ROOT and CLAUDE_PLUGIN_DATA are in every server’s environment.

By hand

redcode server

Speaks MCP to the server directly, with the plugin’s options empty as when they are unset. Needs Node.js, the game and REDbridge, found from REDcode’s VS Code settings. From plugin 0.1.1 the plugin starts redcode-mcp 0.4.0 with npx alone; until then it went through scripts edcode-mcp.cmd, which cleared empty options.

node tools/test/mcp-probe.cjs --empty REDCODE_GAME_PATH --empty REDBRIDGE_EXE "npx -y @edgemodders/redcode-mcp@0.4.0" bridge_info

Last run 2026-10-27 with redcode-mcp 0.4.0, no wrapper: pass, REDbridge 0.11.0 (WolvenKit 10.0.0, from REDcode’s redcode.bridgePath), game 2.5 loaded, 57 archives, 592,071 files; exit 0. Earlier: 2026-10-23 through the wrapper with 0.3.0, pass; 2026-10-16 with 0.2.0, pass.

mesh_items (new in 0.4.0), checked 2026-10-27 through a small MCP client in the same way, game 2.5, REDbridge 0.11.0:

edgedeck server through the wrapper

Speaks MCP to the server directly and calls edgedeck_list, which only reads. Needs Node.js 20 or newer. With options set, it makes an empty library in the folder it is given, so give it a scratch folder.

node tools/test/mcp-probe.cjs --empty EDGEDECK_LIBRARY --empty EDGEDECK_GAME "scripts\edgedeck-mcp.cmd" edgedeck_list
EDGEDECK_LIBRARY='<scratch>\deck lib (x86)' EDGEDECK_GAME='<game folder>' node tools/test/mcp-probe.cjs "scripts\edgedeck-mcp.cmd" edgedeck_list

Last run 2026-11-02 with edgedeck-mcp 0.30.0, both options empty: pass, the default library (one mod now), deploy, undo and remove still flagged. Before that 2026-10-27 with 0.29.1: pass. Both options empty: the default library %USERPROFILE%EDGEdeck (empty on the dev PC now: no mods), game 2.5. A scratch library deck lib (x86) with the game folder: no mods, and the folder gets library. edgedeck_deploy, edgedeck_undo and edgedeck_remove still carry anthropic/requiresUserInteraction. Earlier: 2026-10-23 with 0.28.0, pass; 2026-10-21 with 0.27.3, pass.

What the first runs found, and why the wrapper is written as it is:

edgemat server

Speaks MCP to the server directly, with the plugin’s options empty. Needs Node.js 20 or newer. mat_library with no arguments only reads.

node tools/test/mcp-probe.cjs --empty EDGEMAT_GAME_PATH --empty REDBRIDGE_EXE --empty GEMINI_API_KEY --empty OPENAI_API_KEY "npx -y @edgemodders/edgemat-mcp@0.3.1" mat_library
node tools/test/mcp-probe.cjs --empty GEMINI_API_KEY --empty OPENAI_API_KEY "npx -y @edgemodders/edgemat-mcp@0.3.1" mat_generate '{"words":"red leather"}'

First run 2026-10-23 with edgemat-mcp 0.3.1: pass.

rednode with no token

Seen 2026-10-16 while the game ran, plugin installed in a scratch config with no rednode_token:

plugin:edge:rednode: http://127.0.0.1:41820/mcp (HTTP) - ✘ Failed to connect — Server rejected the configured Authorization header (HTTP 401). ... {"error":"A valid token is required"}

With the game closed it is ECONNREFUSED instead. /edge:setup tells the two apart this way.

OAuth sign-in for a plugin server

Whether a plugin’s HTTP server gets Claude Code’s OAuth sign-in, for connecting to REDnode by approving in the game instead of pasting a token (REDnode#19). Run 2026-10-26 with Claude Code 2.1.310 in a scratch config: a stand-in server on 127.0.0.1:41899 that answers /mcp with 401 and WWW-Authenticate: Bearer resource_metadata="...", serves the two .well-known documents and /register, and logs every request; a scratch plugin whose only server is {"type": "http", "url": "http://127.0.0.1:41899/mcp"}.

Then against REDnode itself, 2026-10-26, by the user on the dev PC: game 2.5, REDnode 0.40.0, then 0.40.1. REDnode’s test “MCP sign-in: test” in its docs/mcp.md, with a header-less rednode-signin added by hand beside the plugin’s (which still sends the token). Pass on 0.40.1, every step: authenticate, the same code in the browser and the game’s Connect window, Allow, game_state answered, Connected assistants listed it, still connected after a game restart, Revoke, Deny, a request lapsing after 2 minutes. On 0.40.0, signing in again after Revoke failed with “Unknown client”; 0.40.1 fixed it (REDnode#19, closed).

Plugin sign-in in VS Code

The plugin’s own rednode (0.0.13, the URL alone) signing in from the VS Code extension, then /edge:setup with all four servers. Run 2026-10-26 by the user on the dev PC: plugin 0.0.13 from GitHub, game 2.5, REDnode 0.40.1, the hand-added rednode removed first (claude mcp remove rednode -s user), game_path set back to the 2.5 folder. Pass.

Found on the way: the plugin’s settings form in VS Code has its Save configuration button below the fold with five options, and the user missed it, so game_path stayed on the 2.31 backup. It was set with claude plugin configure edge@edgemodders --values-stdin instead (a JSON object of strings on stdin; options left out keep their values). The README and /edge:setup now say to scroll down to the button.

P1 test in VS Code with the game

The test that ends P1 (design). Needs your own Claude Code login, so it runs in your real config.

1. Remove a rednode added by hand, which hides the plugin’s: claude mcp remove rednode -s user.
2. In the VS Code extension: /plugins, add the marketplace Edgemodders/EDGEchat, install edge.
3. Start the game and load a save. Open the REDnode launcher (F10 for REDnode’s overlay, then the REDNODE tab at the bottom left), and its Settings panel with the gear at the right of the Theme row (REDnode 0.32.2 or newer; before that the switches were in the launcher itself): MCP server on, Copy setup. Do not run the copied command. Put only the token (after Bearer ) in the plugin’s rednode_token: the gear on the edge row.
4. Start a new session and run /edge:setup.

First run 2026-10-21, plugin 0.0.2 installed from GitHub on a second PC (the user’s laptop, with the game, EDGEdeck and REDcode’s REDbridge), in the VS Code extension. Steered from the dev PC’s session, which could not run /plugins through Remote Control (“isn’t available in this environment”), so the user installed it locally. Pass, one check not seen.

Found on the way: the user did not know where “the REDNODE tab” was. 0.0.3 names the REDnode launcher everywhere. redcode did not fall back to REDcode’s VS Code setting on that PC (whether REDcode’s setting is set there, or the editor was EDGEkit, is not known), so game_path was needed.

Changing the game folder

Run 2026-10-23 on the dev PC, plugin 0.0.8, game closed. The game had updated to 2.5, which the EDGE tools do not support yet, so the user pointed the tools at a backup of 2.31. Pass once the plugin’s option was changed and Claude Code restarted.

To change folders: set the plugin’s game_path, not only REDcode’s setting, then restart Claude Code.

Asks before changes

Needs REDnode 0.26.16 or newer in the game, and plugin 0.0.5 or newer for the edgedeck rows (it runs edgedeck-mcp 0.28.0, the first with the flag). Run it in auto mode, the mode that skipped the prompt in P1. Game at a loaded save for the rednode rows, closed for the edgedeck rows. Deny every prompt, except the one allowed api_call.

First run 2026-10-23 on the user’s laptop, in the VS Code extension in auto mode, plugin 0.0.5, REDnode 0.32.1: pass. The user’s words: every prompt appeared as expected, and each was allowed or denied as the steps said.

Not run: “remove a mod” (edgedeck_remove carries the same flag in edgedeck-mcp 0.28.0; not tried, to keep the mods safe), the default mode, and /edge:setup with an old REDnode.

new-mod in the game

The P3 test: /edge:new-mod run by the user as a modder who does not code, with the game running: it asks what the mod should do, picks the kind of mod, makes the project, builds a first small version, and tests it in the game. Changes to the game and the game folder ask first. Run 2026-10-26 by the user on the dev PC: plugin 0.0.14, game 2.5, REDnode 0.40.1, REDbridge 0.11.0. Pass, by the user’s report; the details were not recorded.

Clean install

The P4 test: the README’s steps alone on a fresh PC or Windows account, with REDbridge from its releases zip and the .NET 10 runtime, then /edge:setup and the REDnode sign-in. Run 2026-10-26 by the user on the dev PC: plugin 0.0.14, game 2.5, REDnode 0.40.1, REDbridge 0.11.0. Pass, by the user’s report; the details were not recorded.

Swap and restore in the game

The wolvenkit-projects loop for 0.1.1: a WolvenKit project deployed by EDGEdeck, changed and swapped into the running game through REDswap, then EDGEdeck brought back in step. Run 2026-11-02 by the user on the dev PC, in claude --plugin-dir <checkout> (plugin 0.1.1 from the checkout, with game_path and edgedeck_library copied to edge@inline), game 2.5, REDnode 0.50.2 (REDswap 1.0.3), REDbridge 0.11.0, redcode-mcp 0.4.0, edgedeck-mcp 0.30.0. Pass.

The project: E:ModdingCyberpunk 2077RedmodsEDGEchatSwapTest, made with EDGEmat: the jacket of Items.Cop_01_Set_Jacket (cop_blue_black, four player meshes and a new .mlsetup), recoloured red. It packed as a REDmod (packed/mods/EDGEchatSwapTest/).

Found on the way:

EDGEkit one-click setup

The P0 decision (design): EDGEkit sets up the plugin in one click (EDGEkit#5). In an EDGEkit with Claude Code signed in, and the plugin not installed:

Run 2026-10-26 on the dev PC, EDGEkit 1.140.2 (signed in, Claude Code 2.1.310), plugin 0.0.15. Pass, by the user’s report and EDGEkit’s on #5, after one change in EDGEkit:

EDGEkit also pointed out that Claude Code leaves auto-update off for this marketplace; the README and setup now say how to turn it on.

Evals: archives-and-cr2w

The P2 test (design): each case runs with the plugin and without it, and the difference (Δ) is what the skill adds. Runs as you, on your Claude login; the plugin’s servers are not started (no mocks), so only the skill is tested. The report stays local with --no-publish; results go to evals/results/, which git ignores.

claude plugin eval . --tag archives-and-cr2w --runs 1 -j 3 --no-publish --trust-plugin

First run 2026-10-23, Claude Code 2.1.302, one run per arm: 3 cases, $0.49, 41 s.

The skill fired in every run with the plugin. Δ 0.00: general WolvenKit knowledge is already in the model, so this skill adds little. See the design’s P2 note.

Evals: rednode-mods

claude plugin eval . --tag rednode-mods --runs 1 -j 3 --no-publish --trust-plugin

Run 2026-10-25 at 0.0.10, Claude Code 2.1.310, guide copied from REDnode 00c1947 (0.37.0), with development folders and services in the skill, together with --tag new-mod and --judge-model sonnet: 5 cases, $1.16, 71 s, mean Δ +0.95. The three rednode-mods cases Δ +1.00 each as before (hud-widget still names red4ext/plugins/REDnode/apps); money-on-screen +1.00, cheaper-jackets +0.75. money-on-screen again loaded rednode-mods, not new-mod.

First run 2026-10-23, Claude Code 2.1.302, one run per arm, skill copied from REDnode 7e0d9ca (0.32.1): 3 cases, $0.63, 49 s. Δ +1.00: without the plugin, Claude failed every case; with it, it passed every one, and the skill fired each time.

Evals: the whole suite

Every case, three runs per arm, judged by Sonnet. The first runs used the default judge (Haiku), which failed correct answers that were long or covered more than the rubric asked (check-mod found all three planted problems and got FAIL three times; Sonnet passed the same case). Use --judge-model sonnet from now on.

claude plugin eval . --runs 3 -j 6 --judge-model sonnet --no-publish --trust-plugin

Run 2026-11-02 at 0.1.1, before the release, after the wiki was copied again (EDGEweb 842dae8) and the connect-claude and into-the-game rubrics said which correct answers are not failures: 15 cases, all pass with the plugin, mean Δ +0.73, $10.63, 6 min 38 s. connect-claude and into-the-game 1.00 each (Δ +1.00). As before, check-mod and new-mod did not fire on plain prompts (broken-package, money-on-screen pass through the other skills).

Run 2026-11-02, --tag wolvenkit-projects only, after the skill learned REDmods and swaps of undeployed archives: changed-after-swap 1.00 (Δ +0.33), pack-resources 1.00 (Δ 0.00), into-the-game 0.67: the failing answer had every PASS step again, as on 2026-10-27; the judge, a second time.

Run 2026-11-02, --tag rednode-mods only, after REDnode’s guide was copied again from 0.50.2: Δ +1.00 each (data-folder, hud-widget, mod-output), $2.39.

Run 2026-10-27 at 0.1.1 (local, before the push): 15 cases, mean Δ +0.66, $10.31, 5 min 15 s. Two cases short, both checked:

Every other case as at 0.0.15 (cheaper-jackets without the plugin: 0.42).

Run 2026-10-26 at 0.0.15, Claude Code 2.1.310, after the skills’ recheck on game 2.5: 15 cases (with edge-tools’s three), mean Δ +0.70, $9.92, 5 min 40 s. Two misses, both from graders out of date, not the plugin: money-on-screen failed one run because its rubric wanted the apps folder and the answer chose a REDnode development folder, which the skills teach from 0.0.10; cheaper-jackets’s checks-tweakdb regex wanted tweakdb_ where the answers said “check it with the TweakDB tools” (seen 2026-10-23 too). The rubric now takes either folder, and checks-tweakdb is an llm grader. Rerun of --tag new-mod with them: both 1.00 with the plugin (cheaper-jackets Δ +0.67, money-on-screen Δ +1.00), $1.62.

As before, new-mod and check-mod did not fire on plain-language prompts (rednode-mods and the others carried the answers); the commands are run as /edge:new-mod and /edge:check-mod, and passed that way in the game.

Run 2026-10-23 at 0.0.8, Claude Code 2.1.302: 12 cases, 10 pass, mean Δ +0.68, $7.91, 5 min 22 s.

Seen:

• With three runs, archives-and-cr2w shows some gain (Δ +0.67 and +0.22), where one run each showed none.
• new-mod and check-mod never fired on these plain-language prompts: Claude loaded rednode-mods instead, which carries the same knowledge, and the answers passed. The commands are for /edge:new-mod and /edge:check-mod; the in-game test runs them that way.
• cheaper-jackets lost points twice on checks-tweakdb, a regex for tweakdb_: those answers said they would check “with the TweakDB tools” without the tool’s name. The regex is stricter than the rubric means.

Evals: wolvenkit-projects

claude plugin eval . --tag wolvenkit-projects --runs 1 -j 3 --no-publish --trust-plugin

First run 2026-10-23, Claude Code 2.1.302, one run per arm: 3 cases, $0.47, 40 s. Δ +0.33.

The two Δ 0 cases test reasoning from a detailed prompt more than knowledge. They stay as they are: rewording prompts until plain Claude fails would only flatter the skill.

Evals: edge-tools

claude plugin eval . --tag edge-tools --runs 3 -j 6 --judge-model sonnet --no-publish --trust-plugin

First run 2026-10-26, Claude Code 2.1.310, plugin 0.0.12 (the wiki copied from EDGEweb 8b425af), three runs per arm: 3 cases, $1.78, 51 s. The skill fired in all 9 runs with the plugin. Mean Δ +0.93 as run; +1.00 without the regex grader removed after it (below).

rednode-mods in the game

What the evals cannot show: Claude building a package in the running game with the skill and the rednode server. Plugin 0.0.6 or newer, the game at a loaded save, Hot reload on in the REDnode launcher’s Settings panel (the gear at the right of the Theme row). In a new session:

Make me a small REDnode HUD widget at the top of the screen that shows V’s position. Build it in my game and check it works.

First run 2026-10-23 on the user’s laptop, in the VS Code extension, plugin 0.0.6, REDnode 0.32.1: pass, in the user’s words “it showed up properly and worked”. The package folder can be deleted afterwards.

init-project on a real project

Run 2026-10-26 by the user on the dev PC: plugin 0.0.14, game 2.5, REDnode 0.40.1, REDbridge 0.11.0. Pass, by the user’s report; the details were not recorded. Run on the mod made in new-mod in the game.

In a session with the plugin, in a WolvenKit project (for example one under E:\Modding\Cyberpunk 2077\Projects) and in a REDnode package folder:

/edge:init-project